(plan) system review and break down plans
This commit is contained in:
Story Crater Bot
+175
@@ -0,0 +1,175 @@
|
||||
# poimen-memory — task board
|
||||
|
||||
43 tasks — 36 build tasks plus **7 composition gates**, one per phase. One file
|
||||
per task, **self-contained**: inlined design facts, executable steps, acceptance
|
||||
criteria, a `Verify` section written for someone who did not build the thing, and
|
||||
the traps worth naming. Reading `DESIGN.md` is not required to do a task — it is
|
||||
linked as background only.
|
||||
|
||||
Each `Verify` section names the harness, the integration test with numbered
|
||||
assertions, the command to run, and the **false pass** — the shape of test that
|
||||
goes green while the feature is broken. Treat the false-pass list as part of the
|
||||
acceptance criteria, not commentary.
|
||||
|
||||
## Ordering — declared, never derived
|
||||
|
||||
**Phase order is the list below. Task ids are opaque and frozen.**
|
||||
|
||||
`M0.3` is `M0.3` forever, in whatever phase it currently sits, because its
|
||||
artifacts and cross-references key on that id. New tasks take new ids rather than
|
||||
renumbering neighbours. This is the `StepId` rule applied to the board itself —
|
||||
a board that renumbers to reorder has the bug it warns its own users about.
|
||||
|
||||
No phase starts until its predecessor's gate is green. The `gate` task at the end
|
||||
of each phase **is** that gate: it proves the phase's parts compose and that its
|
||||
swappable parts are genuinely swappable. Every build task is verified alone; the
|
||||
gate verifies the properties no single task owns.
|
||||
|
||||
## Two rules this board exists to protect
|
||||
|
||||
**1. The JSONL log is authoritative; the vault and the vector index are
|
||||
projections.** Anything that cannot be dropped and rebuilt byte-identically from
|
||||
the log has hidden inputs, and that is a bug. `M2.8` is the gate that enforces it.
|
||||
|
||||
**2. The update gate must discriminate, not summarize.** Agent transcripts are
|
||||
~43% tool results and mostly evidence-free. A gate that accepts most chunks is an
|
||||
expensive summarizer that will reproduce the memory-explosion failure the whole
|
||||
design exists to avoid. `M1.8` is the gate that enforces it, and **update-rate is
|
||||
the single number to watch.**
|
||||
|
||||
## Verification practice — script first, source second
|
||||
|
||||
A task is verified by running its command and reading the output, then opening
|
||||
the source. Reviewing the diff first is how an assertion that was quietly dropped
|
||||
still gets called done: the code looks right, and nothing proves the test ran.
|
||||
|
||||
Numbered assertion N in a `Verify` section is test fn `aN_<slug>`. The numbering
|
||||
is the contract — a test fn that does not exist should report missing rather than
|
||||
be silently absent from a green summary.
|
||||
|
||||
`cargo test` reporting `ok` with zero tests run is not a pass.
|
||||
|
||||
## Progress
|
||||
|
||||
**Source of truth is the `Status` field in each task file.** The tables below
|
||||
mirror it; a status changed here and not there is a lie.
|
||||
|
||||
Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
|
||||
| # | Phase | Ids | Tasks | ✅ | 🟡 | ⬜ | Gate |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | Read-only spine | M0.x | 8 | 0 | 0 | 8 | ⬜ M0.8 |
|
||||
| 2 | Gated loop at L1 | M1.x | 8 | 0 | 0 | 8 | ⬜ M1.8 |
|
||||
| 3 | Projections | M2.x | 8 | 0 | 0 | 8 | ⬜ M2.8 |
|
||||
| 4 | L2 synthesis + retrieval | M3.x | 4 | 0 | 0 | 4 | ⬜ M3.4 |
|
||||
| 5 | Skills | M4.x | 3 | 0 | 0 | 3 | ⬜ M4.3 |
|
||||
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
|
||||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||||
| | **Total** | | **43** | **0** | **0** | **43** | 0/7 green |
|
||||
|
||||
**Where the line is — 2026-08-18.** Nothing started. No crate exists yet: there
|
||||
is no `Cargo.toml` under `memory/`, so every task below is design only. M0.1 is
|
||||
the first thing that has to happen. `M2.2` (the CNPG manifest), `M5.4` (vLLM
|
||||
with LoRA), and all of `M6.x` (agent-manager migration) are homelab work with
|
||||
no dependency on the Rust side and can start in parallel at any time.
|
||||
|
||||
**M6 is a different repo, not a dependency of M0-M5.** It migrates
|
||||
`github.com/Riotpiaole/agent-manager`'s session store (a separate Go CLI tool,
|
||||
unrelated to this project's own memory system) from local sqlite to its own
|
||||
dedicated CNPG cluster. It rides in this board because it's homelab work
|
||||
happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
|
||||
landing in the same cluster around the same time need to look like siblings,
|
||||
not strangers — see M6.6's convention-consistency check.
|
||||
|
||||
## 1 — Read-only spine · M0.x
|
||||
|
||||
No model calls anywhere in this phase. The point is to prove the corpus parses
|
||||
and chunks sanely before spending inference on it.
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M0.1](M0.1-cargo-workspace.md) | Cargo workspace + crate skeletons | S | — | ⬜ |
|
||||
| [M0.2](M0.2-domain-types.md) | Domain types and sha256 identity | S | — | ⬜ |
|
||||
| [M0.3](M0.3-recordsource-and-chunkpolicy.md) | `RecordSource` trait + `ChunkPolicy` | M | — | ⬜ |
|
||||
| [M0.4](M0.4-tokenizer-sizing.md) | Tokenizer-backed chunk sizing | M | — | ⬜ |
|
||||
| [M0.5](M0.5-pi-session-adapter.md) | pi session adapter | M | — | ⬜ |
|
||||
| [M0.6](M0.6-claude-transcript-adapter.md) | Claude transcript adapter | S | — | ⬜ |
|
||||
| [M0.7](M0.7-ingest-dry-run.md) | `mem ingest --dry-run` | S | — | ⬜ |
|
||||
| [M0.8](M0.8-m0-gate.md) | **M0 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 2 — Gated loop at L1 · M1.x
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M1.1](M1.1-llm-chat-client.md) | `mem-llm` chat client | M | — | ⬜ |
|
||||
| [M1.2](M1.2-standing-query-loader.md) | Standing-query YAML loader | M | — | ⬜ |
|
||||
| [M1.3](M1.3-prompt-template.md) | GRU-Mem prompt template | M | — | ⬜ |
|
||||
| [M1.4](M1.4-gate-response-parser.md) | Gate-response parser | M | — | ⬜ |
|
||||
| [M1.5](M1.5-gated-loop.md) | The gated loop | L | — | ⬜ |
|
||||
| [M1.6](M1.6-jsonl-event-log.md) | JSONL event log writer | M | — | ⬜ |
|
||||
| [M1.7](M1.7-ingest-end-to-end.md) | `mem ingest` end to end | M | — | ⬜ |
|
||||
| [M1.8](M1.8-m1-gate.md) | **M1 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 3 — Projections · M2.x
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M2.1](M2.1-embeddings-client.md) | Embeddings client | S | — | ⬜ |
|
||||
| [M2.2](M2.2-memory-db-manifest.md) | CNPG `memory-db` + pgvector | M | homelab | ⬜ |
|
||||
| [M2.3](M2.3-schema-and-migrations.md) | Schema + sqlx migrations | M | — | ⬜ |
|
||||
| [M2.4](M2.4-pgvector-repo.md) | pgvector repository | M | — | ⬜ |
|
||||
| [M2.5](M2.5-obsidian-projector.md) | Obsidian projector | M | — | ⬜ |
|
||||
| [M2.6](M2.6-rebuild-from-log.md) | `mem rebuild --from-log` | M | — | ⬜ |
|
||||
| [M2.7](M2.7-verify-edges.md) | `mem verify` — edge closure | S | — | ⬜ |
|
||||
| [M2.8](M2.8-m2-gate.md) | **M2 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 4 — L2 synthesis and retrieval · M3.x
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M3.1](M3.1-l2-synthesis.md) | L2 synthesis pass | M | — | ⬜ |
|
||||
| [M3.2](M3.2-rerank-client.md) | Rerank client | S | — | ⬜ |
|
||||
| [M3.3](M3.3-mem-query.md) | `mem query` with provenance | M | — | ⬜ |
|
||||
| [M3.4](M3.4-m3-gate.md) | **M3 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 5 — Skills · M4.x
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M4.1](M4.1-skill-draft.md) | `mem skill draft` | M | — | ⬜ |
|
||||
| [M4.2](M4.2-derived-filter.md) | `derived: true` ingest filter | M | — | ⬜ |
|
||||
| [M4.3](M4.3-m4-gate.md) | **M4 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 6 — Post-training · M5.x
|
||||
|
||||
Python, separate from the Rust workspace. The boundary is the JSONL log.
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M5.1](M5.1-evidence-labeler.md) | `mem label` — evidence labeler | M | — | ⬜ |
|
||||
| [M5.2](M5.2-labeler-calibration.md) | Labeler calibration | M | — | ⬜ |
|
||||
| [M5.3](M5.3-training-corpus-export.md) | Training corpus export | M | — | ⬜ |
|
||||
| [M5.4](M5.4-vllm-lora-serving.md) | vLLM + `--enable-lora` | L | homelab | ⬜ |
|
||||
| [M5.5](M5.5-verl-training-loop.md) | verl training loop | L | — | ⬜ |
|
||||
| [M5.6](M5.6-m5-gate.md) | **M5 composition gate** | L | gate | ⬜ |
|
||||
|
||||
## 7 — agent-manager migration · M6.x
|
||||
|
||||
Separate repo (`github.com/Riotpiaole/agent-manager`, fork branch
|
||||
`add-headless-spawn`), separate cluster resource, no Rust/GRU-Mem
|
||||
dependency. Moves its session store off local sqlite onto a dedicated CNPG
|
||||
Postgres, reachable from the Mac client through a dedicated nginx route —
|
||||
durability-of-location, not a multi-host requirement.
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M6.1](M6.1-agent-manager-db-manifest.md) | CNPG `agent-manager-db` manifest | M | homelab | ⬜ |
|
||||
| [M6.2](M6.2-schema-port.md) | Postgres schema for agent-manager sessions | M | — | ⬜ |
|
||||
| [M6.3](M6.3-store-query-port.md) | store.go query port to Postgres | L | — | ⬜ |
|
||||
| [M6.4](M6.4-nginx-stream-routing.md) | nginx TCP routing to `agent-manager-db` | S | homelab | ⬜ |
|
||||
| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ |
|
||||
| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
|
||||
@@ -0,0 +1,98 @@
|
||||
# M0.1 — Cargo workspace + crate skeletons
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
|
||||
## Goal
|
||||
|
||||
The six-crate workspace, building clean, with the dependency direction fixed
|
||||
before any code exists to violate it.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
crates/
|
||||
mem-core/ domain types; Level; gate-response parser; the gated loop
|
||||
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
|
||||
mem-llm/ gateway client — chat, embeddings, rerank
|
||||
mem-ingest/ source adapters: pi sessions, claude transcripts
|
||||
mem-store/ JSONL log; pgvector repo; Obsidian projector
|
||||
mem-cli/ binary `mem`
|
||||
```
|
||||
|
||||
Dependency direction, enforced from the start:
|
||||
|
||||
```
|
||||
mem-cli -> mem-ingest, mem-store, mem-llm, mem-chunk, mem-core
|
||||
mem-store -> mem-core
|
||||
mem-ingest -> mem-chunk, mem-core
|
||||
mem-chunk -> mem-core
|
||||
mem-llm -> mem-core
|
||||
mem-core -> (nothing in this workspace)
|
||||
```
|
||||
|
||||
`mem-core` depends on no sibling. It holds the types every other crate speaks,
|
||||
so a dependency out of it is a cycle waiting to happen.
|
||||
|
||||
Workspace deps to pin now, so versions do not drift per crate: `tokio`,
|
||||
`futures`, `serde`, `serde_json`, `serde_yaml`, `anyhow`, `thiserror`,
|
||||
`sha2`, `clap`, `reqwest`, `tracing`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `Cargo.toml` at the repo root with `[workspace] members = [...]` and a
|
||||
`[workspace.dependencies]` block holding every shared crate version.
|
||||
2. Six member crates, each declaring deps as `foo.workspace = true`.
|
||||
3. `mem-cli` is the only `[[bin]]`; the rest are libraries.
|
||||
4. Add `rust-toolchain.toml` pinning a version, so CI and laptop agree.
|
||||
5. `.gitignore`: `target/`, `vault/` — but **not** `log/` and **not**
|
||||
`memory-tasks/`. The log is authoritative and the board carries acceptance
|
||||
criteria; both are tracked.
|
||||
6. Wire a CI job running `cargo build --workspace` and `cargo clippy --workspace
|
||||
-- -D warnings`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `cargo build --workspace` succeeds from a clean checkout.
|
||||
- `cargo clippy --workspace -- -D warnings` is clean.
|
||||
- `mem-core` has zero intra-workspace dependencies.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** cargo itself, plus a dependency assertion that does not trust the
|
||||
manifests to be read by a human.
|
||||
|
||||
**Integration test** — `tests/it_workspace.rs` in the root:
|
||||
1. `a1_all_members_build` — shell out to `cargo build --workspace`, assert exit 0.
|
||||
2. `a2_mem_core_has_no_sibling_deps` — parse `crates/mem-core/Cargo.toml`, assert
|
||||
no dependency name starts with `mem-`.
|
||||
3. `a3_dependency_direction` — parse every member manifest, build the edge set,
|
||||
assert it is a subset of the table above and that the graph is acyclic.
|
||||
4. `a4_log_and_tasks_are_tracked` — assert `.gitignore` matches neither
|
||||
`log/` nor `memory-tasks/`.
|
||||
|
||||
**Command:** `cargo test --workspace workspace`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the graph is acyclic **only**. Acyclic permits `mem-core ->
|
||||
mem-store`, which is backwards and still acyclic. Assertion 3 must check the
|
||||
edge set against the table, not just for cycles.
|
||||
- A CI job that runs `cargo build` in the root without `--workspace`. It builds
|
||||
the virtual manifest and can miss a member that does not compile.
|
||||
|
||||
## Traps
|
||||
|
||||
- Per-crate dependency versions instead of `workspace.dependencies`. They drift,
|
||||
and two versions of `serde` in one tree is a confusing type error much later.
|
||||
- Gitignoring `log/`. It is the authoritative record; ignoring it makes the whole
|
||||
authority model a fiction.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Repository layout
|
||||
@@ -0,0 +1,115 @@
|
||||
# M0.2 — Domain types and sha256 identity
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
The vocabulary every other crate speaks, and the content-hash identity the whole
|
||||
provenance graph hangs on.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```rust
|
||||
/// Closed. L0 evidence, L1 per-query memory, L2 project synthesis.
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum Level { L0, L1, L2 }
|
||||
|
||||
/// A normalised unit from any source. Adapters produce these; nothing
|
||||
/// downstream learns whether it came from pi, claude, or a socket.
|
||||
pub struct Record {
|
||||
pub role: Role, // User | Assistant | ToolResult | System
|
||||
pub text: String,
|
||||
pub timestamp: OffsetDateTime,
|
||||
pub provenance: Provenance, // source id + offset within it
|
||||
}
|
||||
|
||||
/// One or more Records, under the token budget, never split mid-Record.
|
||||
pub struct Chunk {
|
||||
pub t: u32, // 1-based turn index within a run
|
||||
pub records: Vec<Record>,
|
||||
pub tokens: usize,
|
||||
pub sha256: Sha256Hash,
|
||||
}
|
||||
|
||||
pub struct MemoryNode {
|
||||
pub level: Level,
|
||||
pub project: ProjectId,
|
||||
pub query_id: Option<QueryId>, // None at L2
|
||||
pub run_id: RunId,
|
||||
pub t: u32,
|
||||
pub text: String,
|
||||
pub sha256: Sha256Hash,
|
||||
pub parents: Vec<Sha256Hash>,
|
||||
}
|
||||
```
|
||||
|
||||
**Identity is the content hash, not a counter.** `sha256` is computed over the
|
||||
canonical serialization of the semantic content — for `Chunk`, the concatenated
|
||||
record texts and their provenance; for `MemoryNode`, `(level, project, query_id,
|
||||
text)`. It must **not** include the timestamp or the run id, or re-running the
|
||||
same input produces different hashes and `mem rebuild` stops being idempotent.
|
||||
|
||||
Newtypes with no `Default`: `ProjectId`, `QueryId`, `RunId`, `Sha256Hash`. A
|
||||
placeholder that type-checks is invisible — a hardcoded `"current"` compiles,
|
||||
passes tests, and makes every downstream result unattributable.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Declare `Level`, `Role`, `Record`, `Provenance`, `Chunk`, `MemoryNode`.
|
||||
2. Declare the newtypes. None derives `Default`. None has `From<String>` without
|
||||
validation.
|
||||
3. `fn content_hash(&self) -> Sha256Hash` on `Chunk` and `MemoryNode`, over a
|
||||
canonical byte encoding that excludes timestamps and run ids.
|
||||
4. `Level` serializes as the literal strings `"L0" | "L1" | "L2"` — the JSONL and
|
||||
the SQL `CHECK` constraint both depend on that spelling.
|
||||
5. Round-trip serde tests for every type.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Two `Chunk`s built from identical records in different runs hash identically.
|
||||
- Changing one character of any record text changes the hash.
|
||||
- `Level` round-trips through JSON as `"L0"`, not `0` and not `"l0"`.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** unit tests in `mem-core`, plus a hash-stability fixture committed
|
||||
as bytes.
|
||||
|
||||
**Integration test** — `tests/it_identity.rs`:
|
||||
1. `a1_same_content_same_hash` — build the same chunk twice with different
|
||||
`RunId` and timestamps, assert equal hashes.
|
||||
2. `a2_text_change_changes_hash` — flip one byte, assert the hash differs.
|
||||
3. `a3_level_wire_format` — `serde_json::to_string(&Level::L0) == "\"L0\""`.
|
||||
4. `a4_hash_stability_across_versions` — hash a committed fixture record set,
|
||||
assert it equals a hash literal written into the test. This catches a
|
||||
canonicalization change that would silently orphan every stored node.
|
||||
5. `a5_newtypes_have_no_default` — compile-fail test (`trybuild`) asserting
|
||||
`ProjectId::default()` does not compile.
|
||||
|
||||
**Command:** `cargo test -p mem-core identity`
|
||||
|
||||
**False pass:**
|
||||
- Asserting only that hashing is deterministic *within one process*. A hash that
|
||||
includes the timestamp is deterministic per run and still breaks rebuild.
|
||||
Assertion 1 must vary the run id and timestamp deliberately.
|
||||
- Omitting assertion 4. Without a committed expected hash, any future
|
||||
canonicalization change passes every other test and silently invalidates the
|
||||
database.
|
||||
|
||||
## Traps
|
||||
|
||||
- Including `run_id` or `created_at` in the hash. Rebuild then produces new nodes
|
||||
every time and `memory_edge` accumulates orphans.
|
||||
- Deriving `Default` on an id newtype "for tests". That default reaches
|
||||
production and every row attributes to the same fake project.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — The tier model, Storage schemas
|
||||
@@ -0,0 +1,106 @@
|
||||
# M0.3 — `RecordSource` trait + `ChunkPolicy`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
The seam where new input kinds arrive, shaped as a stream from the first commit
|
||||
so a future streaming source implements a trait instead of forcing a rewrite.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```rust
|
||||
pub trait RecordSource {
|
||||
/// Sources decide how to produce records; the chunker never learns
|
||||
/// whether they came from pi, claude, or a socket.
|
||||
fn records(self) -> impl Stream<Item = Result<Record>>;
|
||||
}
|
||||
|
||||
pub struct ChunkPolicy {
|
||||
pub max_tokens: usize, // 5000 — GRU-Mem paper default
|
||||
pub split_on: Boundary, // Boundary::Record — never mid-Record
|
||||
pub flush: FlushTrigger,
|
||||
}
|
||||
|
||||
pub enum FlushTrigger {
|
||||
Tokens(usize),
|
||||
// OrIdle(Duration) lands with the first streaming source. Carrying the
|
||||
// enum now means that change is one variant, not a signature change
|
||||
// threaded through the loop.
|
||||
}
|
||||
|
||||
pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
|
||||
```
|
||||
|
||||
Why a stream when both current sources are files: sources today have an EOF;
|
||||
telemetry, a live session tail, or a broker will not. Batch sources become
|
||||
streams for free via `futures::stream::iter`, so this costs nothing today and
|
||||
removes a rewrite later. The rest of the stack is already tokio.
|
||||
|
||||
**A single `Record` larger than `max_tokens` is not an error.** Tool results can
|
||||
be enormous. It becomes a chunk of one, over budget, and the chunker records that
|
||||
it did — silently truncating would destroy evidence, and silently dropping would
|
||||
lose it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
|
||||
2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
|
||||
record would exceed `max_tokens`, then yields.
|
||||
3. Never split a `Record`. An oversized single record yields alone, with
|
||||
`Chunk::over_budget = true`.
|
||||
4. `t` is 1-based and contiguous across the whole stream.
|
||||
5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
|
||||
chunking is testable with no I/O and no model.
|
||||
6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
|
||||
a `CharsOverFour` stub is enough here.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Chunk boundaries never fall inside a `Record`.
|
||||
- Chunk `t` values are 1-based, contiguous, no gaps.
|
||||
- An oversized single record yields one chunk flagged `over_budget`.
|
||||
- Concatenating all chunks' records reproduces the input sequence exactly.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `VecSource` + the stub counter. No files, no network.
|
||||
|
||||
**Integration test** — `tests/it_chunking.rs`:
|
||||
1. `a1_no_record_is_split` — for every chunk, every record equals some input
|
||||
record byte for byte.
|
||||
2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
|
||||
input sequence, same order, same length.
|
||||
3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
|
||||
4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
|
||||
one record and `over_budget = true`.
|
||||
5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
|
||||
appears whole in the output, not truncated and not dropped.
|
||||
6. `a6_empty_source` — zero records yields zero chunks, no panic.
|
||||
|
||||
**Command:** `cargo test -p mem-chunk chunking`
|
||||
|
||||
**False pass:**
|
||||
- Testing only with uniformly small records. The budget logic is never exercised
|
||||
and the oversized path never runs. Assertion 5 is the guard.
|
||||
- Asserting chunk count rather than losslessness. A chunker that drops the final
|
||||
partial chunk produces a plausible count and loses the tail — assertion 2 is
|
||||
what catches it.
|
||||
|
||||
## Traps
|
||||
|
||||
- Truncating an oversized record to fit the budget. That is silent evidence
|
||||
destruction, and the update gate will later be blamed for missing it.
|
||||
- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
|
||||
every test here, and defeats the entire reason this crate is separate.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one
|
||||
@@ -0,0 +1,92 @@
|
||||
# M0.4 — Tokenizer-backed chunk sizing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Count tokens with the tokenizer the serving model actually uses, so a chunk that
|
||||
fits locally also fits at the gateway.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
|
||||
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
|
||||
|
||||
Budget arithmetic, and why an approximation is not good enough:
|
||||
|
||||
```
|
||||
Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side)
|
||||
chunk 5000
|
||||
prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024)
|
||||
response 2048
|
||||
```
|
||||
|
||||
A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
|
||||
Undercount and the gateway rejects the request; overcount and chunks are smaller
|
||||
than they need to be, which multiplies the number of model calls.
|
||||
|
||||
`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
|
||||
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
|
||||
runtime: a build that needs the network is a build that fails offline, and a
|
||||
tokenizer that changes under you silently re-chunks the entire corpus.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
|
||||
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
|
||||
3. Assert at load that the vendored file's hash matches the recorded one.
|
||||
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
|
||||
`qwen2.5:3b-instruct`.
|
||||
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
|
||||
for cross-checking against the gateway's reported `prompt_tokens`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
|
||||
real records.
|
||||
- Loading with a modified tokenizer file fails loudly, not silently.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** the vendored tokenizer plus recorded gateway responses. The
|
||||
cross-check against the live gateway is a separate, network-gated test.
|
||||
|
||||
**Integration test** — `tests/it_tokens.rs`:
|
||||
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
|
||||
(ASCII, CJK, code, JSON, emoji), asserted exactly.
|
||||
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
|
||||
error naming the file.
|
||||
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
|
||||
real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
|
||||
`usage.prompt_tokens` to the local count minus the measured template overhead;
|
||||
assert within 2%.
|
||||
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
|
||||
locally-counted size exceeds the budget.
|
||||
|
||||
**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
|
||||
|
||||
**False pass:**
|
||||
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
|
||||
this corpus contains both — the caveman skill text alone is bilingual.
|
||||
- Comparing the local count to itself via a helper that calls the same function.
|
||||
Assertion 1's expected values must be recorded from the tokenizer once and
|
||||
written as literals.
|
||||
|
||||
## Traps
|
||||
|
||||
- Downloading the tokenizer at runtime. Offline builds break, and a silent
|
||||
upstream change re-chunks everything, which changes every chunk hash, which
|
||||
orphans every stored node.
|
||||
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
|
||||
ornith and is wrong; do not take a client-side config as the source of truth.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`
|
||||
@@ -0,0 +1,131 @@
|
||||
# M0.5 — pi session adapter
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
|
||||
from the path without guessing.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Layout, verified on this machine:
|
||||
|
||||
```
|
||||
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
|
||||
└─ cwd with / replaced by -, wrapped in leading and trailing --
|
||||
```
|
||||
|
||||
Record types observed in a real 2902-message session:
|
||||
|
||||
```
|
||||
message 2902
|
||||
model_change 112
|
||||
thinking_level_change 16
|
||||
compaction 8
|
||||
session 1 <- always first line
|
||||
```
|
||||
|
||||
Shapes:
|
||||
|
||||
```jsonc
|
||||
// first line
|
||||
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
|
||||
// message
|
||||
{"type":"message","id":"..","parentId":"..","timestamp":"..",
|
||||
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
|
||||
```
|
||||
|
||||
Role distribution in that same session — this is the whole reason the update gate
|
||||
exists:
|
||||
|
||||
```
|
||||
assistant 1445
|
||||
toolResult 1261 43%, mostly evidence-free
|
||||
user 196
|
||||
```
|
||||
|
||||
`cwd` in the `session` header is authoritative for the project key. The directory
|
||||
name is a lossy encoding (a real `-` in a path is indistinguishable from a
|
||||
separator) — **parse `cwd`, do not decode the directory name.**
|
||||
|
||||
`content` is not always a string. Assistant messages carry content blocks; tool
|
||||
results carry structured payloads. Normalise to text, and keep the block type in
|
||||
`Provenance` so a later filter can act on it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
|
||||
2. Read the first line, require `type == "session"`, take `cwd` as the project
|
||||
key. A file whose first line is not a session header is an error naming the
|
||||
file, not a skip.
|
||||
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
|
||||
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
|
||||
`toolResult -> Role::ToolResult`.
|
||||
5. Flatten `content` to text for all shapes; preserve the original block type in
|
||||
`Provenance`.
|
||||
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
|
||||
emit it as `Role::System` with the marker text, because a compaction boundary
|
||||
is where context was lost and that is worth seeing in the log.
|
||||
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
|
||||
8. A malformed line is a counted, reported skip — never a panic. These files are
|
||||
appended to by a live process and the last line may be a partial write.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Project key comes from `cwd`, matching for a path containing a literal `-`.
|
||||
- All three roles are emitted with correct counts on a real session.
|
||||
- A truncated final line is skipped with a warning, not a panic.
|
||||
- Compaction events appear in the record stream.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** two committed fixtures — one small hand-built session covering every
|
||||
record type and content shape, one real session copied verbatim (secrets
|
||||
scrubbed) for volume.
|
||||
|
||||
**Integration test** — `tests/it_pi_source.rs`:
|
||||
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
|
||||
key is `/tmp/my-project`, proving the directory name was not decoded.
|
||||
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
|
||||
3. `a3_content_shapes` — string content, block-array content, and structured tool
|
||||
result all flatten to non-empty text.
|
||||
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
|
||||
prior records and reports exactly one skip.
|
||||
5. `a5_missing_header` — file whose first line is a `message`; assert an error
|
||||
naming the path.
|
||||
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
|
||||
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
|
||||
record before reading the whole file (assert peak allocation, or instrument
|
||||
reads).
|
||||
|
||||
**Command:** `cargo test -p mem-ingest pi_source`
|
||||
|
||||
**False pass:**
|
||||
- Testing only against the hand-built fixture. It will contain the content shapes
|
||||
you thought of, which is the set you already handle. Assertion 2 against a real
|
||||
session is what finds the rest.
|
||||
- Asserting "no panic" on malformed input without asserting the *count* of
|
||||
skips. A source that silently drops every line panics never and ingests
|
||||
nothing.
|
||||
|
||||
## Traps
|
||||
|
||||
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
|
||||
is ambiguous the moment a path component contains `-`, which is common.
|
||||
- Treating `content` as `String`. It parses for user messages and fails on
|
||||
assistant blocks, so the bug appears to be about assistants specifically and
|
||||
wastes an afternoon.
|
||||
- Materializing the file. These reach tens of MB and the trait is a stream for a
|
||||
reason.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
|
||||
@@ -0,0 +1,92 @@
|
||||
# M0.6 — Claude transcript adapter
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
The second `RecordSource`, which is the one that proves the trait is real.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
~/.claude/projects/-Users-rockliang-workplace-Poimen/<uuid>.jsonl
|
||||
└─ cwd, / replaced by -, no wrapping dashes (differs from pi)
|
||||
```
|
||||
|
||||
15 MB across 7 transcripts on this machine. Record types observed:
|
||||
|
||||
```
|
||||
attachment 150 queue-operation 138 assistant 134 user 81
|
||||
file-history-snapshot 69 system 69 ai-title 21 last-prompt 21 mode 21
|
||||
```
|
||||
|
||||
Fields, read from any line: `sessionId`, `cwd`, `gitBranch`.
|
||||
Typed records dispatch on `type`; the ones that matter here:
|
||||
|
||||
- `user` / `assistant` — content under `message.content`
|
||||
- `system` with `subtype == "api_error"` — a failed turn, worth keeping
|
||||
- `summary` — carries `summary`
|
||||
- everything else — ignore
|
||||
|
||||
The filename stem is a UUID. The encoding differs from pi (no wrapping `--`),
|
||||
which is exactly why the project key must come from the `cwd` **field**, as in
|
||||
M0.5, and not from the directory name.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `ClaudeTranscriptSource` in `mem-ingest`, implementing `RecordSource`.
|
||||
2. Take `cwd` from the first line carrying it; error if no line does.
|
||||
3. Emit records for `user`, `assistant`, and `system`+`api_error`.
|
||||
4. Flatten `message.content` across its shapes, as in M0.5.
|
||||
5. `Provenance` = `claude:<uuid>` plus line offset.
|
||||
6. Reuse the content-flattening and malformed-line handling from M0.5 — extract
|
||||
them into a shared helper rather than copying, since divergence between two
|
||||
flatteners is a bug that only shows up on one source.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Both sources satisfy `RecordSource` with no changes to `mem-chunk`.
|
||||
- Project keys from pi and claude for the same directory resolve to the same
|
||||
`ProjectId`.
|
||||
- `api_error` system records survive into the stream.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a committed transcript fixture plus a real one, secrets scrubbed.
|
||||
|
||||
**Integration test** — `tests/it_claude_source.rs`:
|
||||
1. `a1_project_from_cwd_field` — assert the key comes from `cwd`, not the
|
||||
directory name, using a fixture where they would differ.
|
||||
2. `a2_same_project_across_sources` — a pi session and a claude transcript for
|
||||
the same directory yield an equal `ProjectId`. This is the assertion that
|
||||
makes cross-source memory possible at all.
|
||||
3. `a3_role_mapping` — user/assistant/api_error appear; `attachment`,
|
||||
`queue-operation`, `file-history-snapshot` do not.
|
||||
4. `a4_shared_flattener` — the same content-block fixture flattens identically
|
||||
through both sources (call both, compare strings).
|
||||
|
||||
**Command:** `cargo test -p mem-ingest claude_source`
|
||||
|
||||
**False pass:**
|
||||
- Testing each source in isolation. The point of this task is that they agree;
|
||||
assertions 2 and 4 are the only ones that check it, and both are cross-source.
|
||||
|
||||
## Traps
|
||||
|
||||
- Copying the flattener instead of sharing it. The two will drift, and the
|
||||
resulting bug looks like "claude transcripts lose tool output" rather than
|
||||
"there are two flatteners".
|
||||
- Assuming the pi encoding. Claude has no wrapping `--`, so a decoder written
|
||||
against pi silently produces a different project key for the same directory —
|
||||
and cross-source memory quietly splits in two.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Context
|
||||
@@ -0,0 +1,94 @@
|
||||
# M0.7 — `mem ingest --dry-run`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.4, M0.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
See the chunk plan for a real project before spending a single model call on it.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
`--dry-run` makes **zero network calls**. That is the property, not a side
|
||||
effect: this is the last checkpoint before inference, and its value is telling
|
||||
you the corpus is sane while feedback is still free.
|
||||
|
||||
Output shape:
|
||||
|
||||
```
|
||||
project /Users/rockliang/workplace/Poimen/agent-rust
|
||||
sources pi:4 files claude:2 files
|
||||
records 3118 (assistant 1445 toolResult 1261 user 196 system 216)
|
||||
chunks 412 over-budget 3
|
||||
tokens min 84 p50 4870 p95 5000 max 11204
|
||||
```
|
||||
|
||||
`over-budget` counts single records exceeding the chunk budget (M0.3). A nonzero
|
||||
count is expected — large tool results — and is worth surfacing because it
|
||||
predicts requests the gateway may reject.
|
||||
|
||||
Numbers to sanity-check against: a real pi session in this project has 2902
|
||||
messages with the role split above, and toolResult being ~43% is the signal that
|
||||
the update gate has something to discriminate.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem ingest --project <path-or-key> --dry-run`.
|
||||
2. Resolve sources: scan both source roots for directories whose `cwd` matches
|
||||
the project. Report which files matched.
|
||||
3. Stream records through `mem-chunk` with the real tokenizer; accumulate stats
|
||||
without retaining chunk bodies.
|
||||
4. Print the table above. Machine-readable variant behind `--format json`.
|
||||
5. `--limit <n>` to stop after n chunks, for iterating on a large project.
|
||||
6. Exit non-zero if zero sources matched — a silent empty plan reads like success.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- No network syscall occurs during `--dry-run`.
|
||||
- Stats are computed streaming; memory does not scale with corpus size.
|
||||
- Zero matched sources exits non-zero with a message naming the project key.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** the fixtures from M0.5/M0.6, plus a network guard.
|
||||
|
||||
**Integration test** — `tests/it_dry_run.rs`:
|
||||
1. `a1_no_network` — run the command with outbound TCP blocked (inject a
|
||||
`reqwest` client that panics on use, or set an unroutable proxy); assert exit 0.
|
||||
2. `a2_counts_match_sources` — record and role counts equal the sum of what the
|
||||
two adapters yield independently.
|
||||
3. `a3_chunk_count_matches_chunker` — the reported chunk count equals
|
||||
`chunks(...).count()` computed separately.
|
||||
4. `a4_over_budget_reported` — fixture with one oversized record; assert
|
||||
`over-budget 1`.
|
||||
5. `a5_empty_project_fails` — unknown project key exits non-zero, message names
|
||||
the key.
|
||||
6. `a6_constant_memory` — run against a 50 MB fixture; assert peak RSS stays
|
||||
under a bound well below file size.
|
||||
|
||||
**Command:** `cargo test -p mem-cli dry_run`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the command exits 0 and printing looks right. A dry run that matched
|
||||
no sources also exits 0 and prints a tidy table of zeros — assertion 5 is what
|
||||
separates them.
|
||||
- Computing stats by collecting chunks into a `Vec` first. Every assertion here
|
||||
passes and assertion 6 is the only one that fails, which is why it is present.
|
||||
|
||||
## Traps
|
||||
|
||||
- Making `--dry-run` construct the LLM client "but not call it". Construction
|
||||
reads credentials and can fail; the guarantee is no network, and the cheapest
|
||||
way to keep it is to not build the client at all.
|
||||
- Reporting p50 chunk size only. The max is the interesting number — it predicts
|
||||
which requests will be rejected downstream.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verification, P1
|
||||
@@ -0,0 +1,87 @@
|
||||
# M0.8 — M0 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M0 — Read-only spine |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M0 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the parts compose and that the swappable part is genuinely swappable —
|
||||
the properties no single M0 task owns.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The claim this phase makes: **a new input kind is added by implementing
|
||||
`RecordSource`, and nothing downstream changes.** If that is false, the crate
|
||||
split bought nothing and the streaming work later will be a rewrite.
|
||||
|
||||
The gate proves it by adding a third source that resembles nothing already
|
||||
supported, and asserting the rest of the pipeline is untouched.
|
||||
|
||||
Second claim: the whole phase is free. No model calls, so the gate is a CI job
|
||||
that runs on every push without a gateway or credentials.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `SyntheticSource` in the test tree only — generates records from a
|
||||
seed, has no file format, and deliberately produces a record larger than the
|
||||
chunk budget and a run of empty-text records.
|
||||
2. Assert `chunks()` handles it with **no change** to `mem-chunk`.
|
||||
3. Assert dependency direction still holds (M0.1 assertion 3) after three sources
|
||||
exist — this is when someone is tempted to reach backwards.
|
||||
4. Run `mem ingest --dry-run` over all three sources for one project and diff the
|
||||
summary against a committed expected file.
|
||||
5. Wire the whole thing as a required CI job.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Three sources, one `RecordSource`, zero source-specific branches in `mem-chunk`
|
||||
or `mem-cli`.
|
||||
- Dry-run summary diffs empty against the committed expectation.
|
||||
- CI job passes with no network and no credentials.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** committed expected-output file, diffed. The script's output is the
|
||||
review artifact.
|
||||
|
||||
**Integration test** — `tests/it_m0_gate.rs`:
|
||||
1. `a1_third_source_needs_no_downstream_change` — `SyntheticSource` flows through
|
||||
`chunks()`; assert `git diff --stat crates/mem-chunk crates/mem-cli` is empty
|
||||
for the commit that added it (enforced by a CI step, not by the test binary).
|
||||
2. `a2_no_source_specific_branches` — grep `crates/mem-chunk` and `crates/mem-cli`
|
||||
for the strings `pi:`, `claude:`, `sessionId`, `toolResult`; assert none
|
||||
appear outside `mem-ingest`.
|
||||
3. `a3_dry_run_golden` — run the dry run over all three sources, diff against
|
||||
`expected/m0-gate.txt`; empty diff is the only pass.
|
||||
4. `a4_offline` — the entire gate runs with networking disabled.
|
||||
5. `a5_lossless_end_to_end` — records in equals records out, across all three
|
||||
sources composed.
|
||||
|
||||
**Command:** `cargo test --workspace m0_gate`
|
||||
|
||||
**False pass:**
|
||||
- Adding `SyntheticSource` in a way that mirrors the pi format. It then exercises
|
||||
the same code path and proves nothing about generality. It must have no file,
|
||||
no header line, and an oversized record.
|
||||
- Assertion 3 passing because the expected file was regenerated in the same
|
||||
commit. A changed `expected/` file in a diff is a claim that the contract
|
||||
changed, and must be reviewed as one.
|
||||
|
||||
## Traps
|
||||
|
||||
- Skipping assertion 2 because "obviously there are no source-specific branches".
|
||||
There will be — the first `if provenance.starts_with("pi:")` gets added to fix
|
||||
a real bug and is entirely reasonable in isolation.
|
||||
- Letting the gate need credentials. The value of an offline gate is that it runs
|
||||
on every push; the moment it needs a gateway key it becomes a nightly job that
|
||||
nobody watches.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, Verification
|
||||
@@ -0,0 +1,107 @@
|
||||
# M1.1 — `mem-llm` chat client
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Talk to the homelab gateway, with the two non-obvious details that cost a day to
|
||||
find already baked in.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
base https://api.riotpiao.com/v1
|
||||
route POST /v1/qwen/chat/completions qwen2.5:3b-instruct
|
||||
POST /v1/ornith/chat/completions ornith:35b
|
||||
POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools)
|
||||
```
|
||||
|
||||
**Auth is `apikey:`, not `Authorization: Bearer`.** Kong's `key-auth` compares
|
||||
the whole header value against the stored key, so the OpenAI SDK convention
|
||||
returns 401. Verified:
|
||||
|
||||
```
|
||||
-H "apikey: $KEY" -> 200
|
||||
-H "Authorization: $KEY" -> 200
|
||||
-H "Authorization: Bearer $KEY" -> 401
|
||||
```
|
||||
|
||||
**One provider per route.** `baseUrl` is per-route and the model id is `ornith:35b`
|
||||
with the tag, not `ornith`. `/v1/models` advertises `/v1/score` which does not
|
||||
work — do not trust that list as a capability probe.
|
||||
|
||||
Request bodies above ~10.6 KB used to fail with
|
||||
`{"error":{"message":"[] is too short - 'messages'"}}`; the Kong body buffer was
|
||||
raised to 16m and it is fixed. If that error ever reappears, it is the buffer,
|
||||
not the client.
|
||||
|
||||
Send **no `tools` array**. The controller needs none, and the reasoning route
|
||||
rejects any request carrying one.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `ChatClient::new(base_url, api_key, model)` in `mem-llm`.
|
||||
2. Send the `apikey` header. Read the key from `MEM_API_KEY`, never from a
|
||||
committed file.
|
||||
3. `complete(system, user, max_tokens) -> Completion { text, usage, latency }`.
|
||||
4. Timeout default 300s — local models are slow to first token and a cold load
|
||||
can take minutes.
|
||||
5. Retry on 5xx and timeout with exponential backoff, max 3. **Do not retry 4xx**
|
||||
— a 400 is a malformed request and retrying it just costs three times as much.
|
||||
6. On any error, include the response body in the error. The useful information
|
||||
is always in the body, never the status.
|
||||
7. `MEM_LLM_RECORD=<dir>` writes every request/response pair to disk, for
|
||||
building fixtures without hand-writing them.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A real completion round-trips against the gateway.
|
||||
- A 401 is reported as an auth error naming the header convention.
|
||||
- A 4xx is not retried; a 5xx is.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `wiremock` for the offline tests; one `#[ignore]` test against the
|
||||
live gateway.
|
||||
|
||||
**Integration test** — `tests/it_chat_client.rs`:
|
||||
1. `a1_sends_apikey_header` — assert the mock received `apikey` and **no**
|
||||
`authorization` header.
|
||||
2. `a2_no_tools_field` — assert the serialized body has no `tools` key at all,
|
||||
not merely an empty array.
|
||||
3. `a3_retries_5xx` — mock 503 twice then 200; assert 3 requests and success.
|
||||
4. `a4_does_not_retry_4xx` — mock 400; assert exactly 1 request and an error
|
||||
carrying the body text.
|
||||
5. `a5_timeout_is_configurable` — mock a 2s delay with a 1s timeout; assert a
|
||||
timeout error.
|
||||
6. `a6_live_smoke` — `#[ignore]`; real gateway, `qwen2.5:3b-instruct`, prompt
|
||||
"reply with exactly: pong", assert the text contains `pong`.
|
||||
|
||||
**Command:** `cargo test -p mem-llm chat_client` (add `-- --ignored` for a6)
|
||||
|
||||
**False pass:**
|
||||
- Testing only against the mock. The mock accepts whatever header you send it;
|
||||
assertion 6 against the live gateway is the only thing that proves the auth
|
||||
convention is right.
|
||||
- Asserting `tools: []` is absent by checking `body.tools.is_empty()`. An empty
|
||||
array serialized into the request is still a `tools` key, and that is what the
|
||||
reasoning route rejects. Assert on the raw JSON.
|
||||
|
||||
## Traps
|
||||
|
||||
- Using `Authorization: Bearer`. It is what every SDK does and it 401s here.
|
||||
- Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it
|
||||
tripled the load and produced identical failures more slowly.
|
||||
- A 60s timeout. That is the value that made `ornith` look broken when it was
|
||||
merely cold.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verified facts
|
||||
@@ -0,0 +1,103 @@
|
||||
# M1.2 — Standing-query YAML loader
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M0.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Load the standing questions that give the update gate its referent, and fail at
|
||||
load rather than mid-run when one is wrong.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```yaml
|
||||
# queries/poimen.yaml
|
||||
project: poimen
|
||||
roots:
|
||||
- /Users/rockliang/workplace/Poimen/agent-rust # matched against session cwd
|
||||
sources: [pi, claude]
|
||||
queries:
|
||||
- id: architecture-decisions
|
||||
question: What architectural decisions were made, with reasoning and rejected alternatives?
|
||||
- id: infra-root-causes
|
||||
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
|
||||
synthesis:
|
||||
question: What is the current state of this project, and what should someone know before working on it?
|
||||
exit_gate: true
|
||||
defaults:
|
||||
memory_budget: 1024
|
||||
chunk_tokens: 5000
|
||||
exit_gate: false # L1 default — see below
|
||||
```
|
||||
|
||||
**Why a question is mandatory.** The GRU-Mem memory agent is `φθ(Q, C_t, M_{t-1})`
|
||||
and its update gate is defined as "does this chunk contain useful information
|
||||
*about the problem*". With no `Q` the gate has no referent, and `r_update` is
|
||||
undefinable — which forecloses post-training (M5) entirely. A query with an empty
|
||||
question is a load error, not a warning.
|
||||
|
||||
**`exit_gate: false` at L1 is deliberate.** Paper §3.3: for "what are *all* the
|
||||
X" questions you cannot know evidence is sufficient without reading everything,
|
||||
so the paper itself provides a without-exit-gate inference mode. L1 extraction is
|
||||
that shape. The gate is still *recorded* — its signal is needed for M5.
|
||||
|
||||
`query.id` is stable and frozen. It names the L1 memory, the Obsidian note, and
|
||||
the log directory; renaming it orphans all three.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `QuerySet::load(path)` in `mem-core`, `serde_yaml`.
|
||||
2. Validate at load: non-empty `project`; at least one query; every `id` unique,
|
||||
non-empty, and `[a-z0-9-]+`; every `question` non-empty; `memory_budget > 0`.
|
||||
3. Every failure names the file, the query id, and the field.
|
||||
4. `mem query validate <file>` prints the resolved set and exits non-zero on any
|
||||
error.
|
||||
5. Defaults apply per query and are overridable per query.
|
||||
6. `id` collision across two files for the same project is an error.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- An empty or missing `question` fails at load with a message naming the id.
|
||||
- An id outside `[a-z0-9-]+` fails at load — it becomes a filename.
|
||||
- Defaults resolve; per-query overrides win.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** table-driven over fixture YAML files, one per failure mode.
|
||||
|
||||
**Integration test** — `tests/it_query_loader.rs`:
|
||||
1. `a1_valid_loads` — the reference file above resolves to the expected struct.
|
||||
2. `a2_empty_question_rejected` — error text contains the query id.
|
||||
3. `a3_duplicate_id_rejected` — error names both occurrences.
|
||||
4. `a4_bad_id_charset_rejected` — `infra/root causes` is rejected, message
|
||||
mentions the filename constraint.
|
||||
5. `a5_defaults_and_overrides` — a query without `exit_gate` gets `false`; one
|
||||
with `true` keeps it.
|
||||
6. `a6_l1_exit_gate_defaults_false` — assert explicitly, because a silent flip to
|
||||
`true` truncates every extraction and looks like a model quality problem.
|
||||
7. `a7_missing_question_field` — absent key behaves as empty, same error.
|
||||
|
||||
**Command:** `cargo test -p mem-core query_loader`
|
||||
|
||||
**False pass:**
|
||||
- Testing only the happy path. Every assertion except 1 and 5 is a rejection
|
||||
test, and rejection is the entire point of load-time validation.
|
||||
- Asserting an error occurred without asserting the message names the offending
|
||||
id. "invalid config" sends someone to read the whole file by hand.
|
||||
|
||||
## Traps
|
||||
|
||||
- Allowing an empty question "for now". It loads, the gate has no referent, the
|
||||
model updates on nearly everything, and it reads as a bad model rather than a
|
||||
bad config.
|
||||
- Letting `id` contain `/` or spaces. It is a path segment in three places.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Standing queries
|
||||
@@ -0,0 +1,120 @@
|
||||
# M1.3 — GRU-Mem prompt template
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Assemble the memory-agent prompt exactly as the paper specifies, because the
|
||||
model's ability to emit parseable gates depends on the format it was aligned to.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Paper Figure 10a, reproduced verbatim — this is the contract, not a starting
|
||||
point to improvise on:
|
||||
|
||||
```
|
||||
You are presented with a problem, a section of an article that may contain the
|
||||
answer to the problem, and a previous memory. Please read the provided section
|
||||
carefully. You should reason about whether the new section contains useful
|
||||
information about the problem, and then update the memory with the new
|
||||
information that helps to answer the problem.
|
||||
Be sure to retain all relevant details from the previous memory while adding any
|
||||
new, useful information. You should also carefully judge whether you have
|
||||
collected enough information to answer the problem.
|
||||
You should reason about whether the new section contains useful information, what
|
||||
to update, and what to do next first between <think> and </think>.
|
||||
If the new section contains useful information about the problem, you should
|
||||
first generate <check>yes</check>. After that, update the new memory between
|
||||
<update> and </update>.
|
||||
If the new section does not contain useful information about the problem, you
|
||||
should first generate <check>no</check>. After that, you should keep the previous
|
||||
memory unchanged between <update> and </update>.
|
||||
In the end, if you haven't collected enough information for the problem, return
|
||||
<next>continue</next>. ONLY when enough information is collected, return
|
||||
<next>end</next>.
|
||||
<problem> {prompt} </problem>
|
||||
<memory> {memory} </memory>
|
||||
<section> {chunk} </section>
|
||||
```
|
||||
|
||||
Substitutions for this system: `{prompt}` = the standing question, `{memory}` =
|
||||
`M_{t-1}` or the literal `No previous memory` at `t=1` (the paper's own case
|
||||
studies show that exact string), `{chunk}` = the rendered chunk.
|
||||
|
||||
Budget, against the 32768 cap:
|
||||
|
||||
```
|
||||
system + template ~400
|
||||
question ~100
|
||||
memory <=1024
|
||||
chunk <=5000
|
||||
response 2048
|
||||
------
|
||||
~8600 headroom is comfortable
|
||||
```
|
||||
|
||||
Chunk rendering: each record as `[role] text`, records separated by a blank line.
|
||||
Role labels matter — the model uses them to tell a tool result from a decision.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `PromptBuilder` in `mem-core` producing `(system, user)`.
|
||||
2. Template verbatim as above. Any deviation gets a comment saying why.
|
||||
3. `t=1` renders `No previous memory` — not empty, not `null`.
|
||||
4. Render chunk records as `[role] text`, blank-line separated.
|
||||
5. Assert the assembled prompt fits the budget before sending; over budget is an
|
||||
error naming the component that overflowed, not a truncation.
|
||||
6. `mem prompt --project P --query Q --chunk N` prints the exact prompt, for
|
||||
eyeballing what the model actually sees.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Assembled prompt matches a committed golden file byte for byte.
|
||||
- `t=1` contains `No previous memory`.
|
||||
- Over-budget assembly errors and names the offending component.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** golden-file comparison. The prompt is a contract; a diff in it is a
|
||||
change to the contract.
|
||||
|
||||
**Integration test** — `tests/it_prompt.rs`:
|
||||
1. `a1_golden_t1` — first turn against `expected/prompt-t1.txt`, exact match.
|
||||
2. `a2_golden_tn` — turn with a prior memory against `expected/prompt-tn.txt`.
|
||||
3. `a3_no_previous_memory_literal` — assert the exact string at `t=1`.
|
||||
4. `a4_all_tags_present` — `<problem>`, `<memory>`, `<section>` each appear
|
||||
exactly once.
|
||||
5. `a5_role_labels_rendered` — a chunk with all four roles renders all four
|
||||
labels.
|
||||
6. `a6_over_budget_errors` — a 20000-token chunk errors, message contains
|
||||
`section`.
|
||||
7. `a7_budget_headroom` — for the real fixture corpus, assert every assembled
|
||||
prompt is under 32768 minus 2048.
|
||||
|
||||
**Command:** `cargo test -p mem-core prompt`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the prompt "contains" the question. A template that dropped the
|
||||
`<check>` instructions still contains it, and the model then emits prose the
|
||||
parser cannot read. Golden-file equality is the assertion that holds.
|
||||
- Skipping assertion 7 by testing only small fixtures. Budget overflow appears
|
||||
at p95 chunk size, not at the median.
|
||||
|
||||
## Traps
|
||||
|
||||
- Improving the wording. The 3B model's gate reliability comes from this exact
|
||||
format; a cleaner rewrite is an unmeasured change to the one thing M1.8 gates on.
|
||||
- Rendering an empty `<memory></memory>` at `t=1`. The paper's traces show
|
||||
`No previous memory`, and an empty tag reads to the model as "memory exists and
|
||||
is empty", which is a different claim.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Standing queries · paper Fig 10a
|
||||
@@ -0,0 +1,113 @@
|
||||
# M1.4 — Gate-response parser
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn the model's tagged output into `(U_t, M̂_t, E_t)`, strictly — because a
|
||||
lenient parser silently fabricates gate decisions.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Expected response shape:
|
||||
|
||||
```
|
||||
<think>...</think>
|
||||
<check>yes|no</check>
|
||||
<update>candidate memory, or the previous memory verbatim</update>
|
||||
<next>continue|end</next>
|
||||
```
|
||||
|
||||
Semantics, from the paper:
|
||||
|
||||
| tag | value | meaning |
|
||||
|---|---|---|
|
||||
| `<check>` | `yes` | `U_t = true` — memory becomes `M̂_t` |
|
||||
| `<check>` | `no` | `U_t = false` — memory stays `M_{t-1}`, chunk discarded |
|
||||
| `<next>` | `continue` | `E_t = false` |
|
||||
| `<next>` | `end` | `E_t = true` |
|
||||
|
||||
**Strict parsing is the design, matching the paper's `r_format`:** it awards 1
|
||||
only when *every* turn in the trajectory parses, 0 otherwise, "because we can not
|
||||
infer whether the incorrect format is caused by the previous erroneous parsing".
|
||||
|
||||
So: exactly one of each tag, properly closed, `<check>` content exactly `yes` or
|
||||
`no` after trimming, `<next>` exactly `continue` or `end`. Anything else is a
|
||||
`ParseError` naming which tag failed and carrying the raw text.
|
||||
|
||||
**A parse failure must not default.** Defaulting `U_t` to `false` silently drops
|
||||
evidence; defaulting to `true` pollutes memory. The loop (M1.5) decides the
|
||||
retry policy; the parser only reports.
|
||||
|
||||
Reasoning models emit `<think>` natively, which can nest or repeat. Extract by
|
||||
locating the *last* `</think>` before the first `<check>`, not by regex over the
|
||||
whole body.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `parse_gate_response(&str) -> Result<GateResponse, ParseError>` in `mem-core`.
|
||||
2. `GateResponse { think: String, update_gate: bool, candidate: String, exit_gate: bool }`.
|
||||
3. Reject duplicates of any tag, a missing tag, an unclosed tag, and any
|
||||
`<check>`/`<next>` value outside the allowed set.
|
||||
4. `ParseError` variants name the tag and include the raw response, truncated.
|
||||
5. Trim surrounding whitespace inside tags; do not otherwise normalise — memory
|
||||
text is preserved verbatim.
|
||||
6. When `U_t = false`, still capture `candidate` so the log can show what the
|
||||
model *would* have written. The loop ignores it; the record is diagnostic.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All four tags parse from a well-formed response.
|
||||
- Every malformed shape errors, naming the failing tag.
|
||||
- No input produces a default `U_t` or `E_t`.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** table-driven over recorded real responses plus hand-built malformed
|
||||
cases. Capture real ones with `MEM_LLM_RECORD` from M1.1.
|
||||
|
||||
**Integration test** — `tests/it_gate_parser.rs`:
|
||||
1. `a1_wellformed_yes_continue` — `U=true`, `E=false`, candidate matches.
|
||||
2. `a2_wellformed_no_end` — `U=false`, `E=true`.
|
||||
3. `a3_missing_check_errors` — error names `check`.
|
||||
4. `a4_duplicate_update_errors` — two `<update>` blocks error.
|
||||
5. `a5_bad_check_value_errors` — `<check>maybe</check>` errors, message shows the
|
||||
value.
|
||||
6. `a6_unclosed_tag_errors` — `<update>` never closed.
|
||||
7. `a7_nested_think` — a response with `<think>` inside `<think>` still finds the
|
||||
right boundary.
|
||||
8. `a8_no_defaults` — property test over 1000 random mutations of a valid
|
||||
response: every result is either an exact parse or an error, never a
|
||||
silently-defaulted `GateResponse`.
|
||||
9. `a9_real_responses` — every recorded real response parses.
|
||||
|
||||
**Command:** `cargo test -p mem-core gate_parser`
|
||||
|
||||
**False pass:**
|
||||
- A regex that finds the first `<check>` and stops. It passes 1–2 and silently
|
||||
accepts duplicates, which is assertion 4's job.
|
||||
- Testing only hand-written responses. Real 3B output has whitespace,
|
||||
markdown fences and stray prose the author would not think to write —
|
||||
assertion 9 is the only one that sees it.
|
||||
- Omitting assertion 8. A parser with `unwrap_or(false)` anywhere passes every
|
||||
positive test and quietly halves the update rate.
|
||||
|
||||
## Traps
|
||||
|
||||
- Defaulting on parse failure. The one thing that must not happen: a fabricated
|
||||
gate decision is indistinguishable from a real one in the log, and it poisons
|
||||
the M5 training data at the source.
|
||||
- Normalising the candidate memory (collapsing whitespace, stripping markdown).
|
||||
Memory text is content; the hash and every downstream projection depend on it
|
||||
being verbatim.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Architecture · paper §3.2.1 `r_format`
|
||||
@@ -0,0 +1,125 @@
|
||||
# M1.5 — The gated loop
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | L — 3+ days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.4 |
|
||||
|
||||
## Goal
|
||||
|
||||
The recurrence itself: `U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})`, with the level as a
|
||||
parameter so L2 reuses it unchanged.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Paper Algorithm 1, transcribed:
|
||||
|
||||
```
|
||||
t <- 1; M_0 <- None
|
||||
while t <= T:
|
||||
U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})
|
||||
if U_t == True: M_t <- M̂_t # update
|
||||
else: M_t <- M_{t-1} # retain, discard chunk
|
||||
if use_exit_gate and E_t == True: break
|
||||
t <- t + 1
|
||||
answer = ψθ(Q, M_t)
|
||||
```
|
||||
|
||||
Two decisions this task must not get wrong:
|
||||
|
||||
**`use_exit_gate` is a parameter, false at L1.** `E_t` is always *recorded*
|
||||
regardless — its signal is the M5 training target, and on a future unbounded
|
||||
stream it becomes the only termination condition. Recording a gate you do not act
|
||||
on is deliberate, not dead code.
|
||||
|
||||
**The memory budget is enforced by the loop, not hoped for from the model.**
|
||||
`memory_budget` is 1024 tokens. If `M̂_t` exceeds it the loop does **not** silently
|
||||
truncate — truncation mid-sentence corrupts the memory for every later turn. It
|
||||
records a `budget_exceeded` event and retains `M_{t-1}`, treating the turn as
|
||||
`U_t = false`. Memory that stops growing is recoverable; memory that is
|
||||
truncated garbage is not.
|
||||
|
||||
The loop is generic over the input stream, so L2 (M3.1) passes L1 memories in
|
||||
place of chunks with no other change.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `run_loop(level, query, source: impl Stream<Item = Chunk>, cfg) -> RunOutcome`
|
||||
in `mem-core`.
|
||||
2. Per turn: build prompt (M1.3), call model (M1.1), parse (M1.4), apply the
|
||||
update rule, emit events.
|
||||
3. On parse error: retry the same chunk up to 2 times. Still failing, record
|
||||
`parse_failed`, treat as `U_t = false`, continue. **Never** default the gate.
|
||||
4. On `U_t = true`: emit an L0 `evidence` event for the chunk, then an L1
|
||||
`memory` event whose `parents` include that evidence sha plus the previous
|
||||
memory's sha.
|
||||
5. Enforce `memory_budget` as above.
|
||||
6. Honour `E_t` only when `use_exit_gate`; always record it.
|
||||
7. Emit `run_end` with `chunks_seen`, `chunks_used`, `final_memory_sha`.
|
||||
8. Cancellation: a dropped future must not leave a half-written log. Emit events
|
||||
only after a turn fully resolves.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `U_t = false` leaves memory byte-identical to the previous turn.
|
||||
- `U_t = true` replaces memory and links parents.
|
||||
- `exit_gate = false` processes every chunk even when `E_t = true` throughout.
|
||||
- Over-budget candidate retains prior memory and records the event.
|
||||
- Two parse failures then success consumes 3 calls for one chunk.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a scripted fake `ChatClient` returning canned responses per turn, so
|
||||
the whole loop runs with no network and fully determined gate sequences.
|
||||
|
||||
**Integration test** — `tests/it_gated_loop.rs`:
|
||||
1. `a1_retain_on_no` — scripted `no` for 5 turns; assert final memory equals
|
||||
initial and `chunks_used == 0`.
|
||||
2. `a2_update_on_yes` — `yes` at turn 3 only; assert memory equals turn 3's
|
||||
candidate and `chunks_used == 1`.
|
||||
3. `a3_exit_gate_off_reads_all` — `end` at every turn with `use_exit_gate=false`;
|
||||
assert all 10 chunks processed.
|
||||
4. `a4_exit_gate_on_stops` — same script, `use_exit_gate=true`; assert it stops
|
||||
at turn 1.
|
||||
5. `a5_exit_always_recorded` — in a3, assert 10 `gate` events carry `exit=true`
|
||||
despite not acting on them.
|
||||
6. `a6_budget_exceeded_retains` — candidate of 4000 tokens; assert memory
|
||||
unchanged, one `budget_exceeded` event, turn counted as not-used.
|
||||
7. `a7_parse_retry` — fail twice then succeed; assert 3 calls, one memory event.
|
||||
8. `a8_parse_failure_is_not_an_update` — fail 3 times; assert `U_t` false,
|
||||
`parse_failed` recorded, loop continues.
|
||||
9. `a9_parents_linked` — every memory event's `parents` contains the evidence sha
|
||||
from the same turn.
|
||||
10. `a10_level_is_a_parameter` — run the identical script at L1 and L2; assert the
|
||||
only difference in emitted events is the `level` field.
|
||||
|
||||
**Command:** `cargo test -p mem-core gated_loop`
|
||||
|
||||
**False pass:**
|
||||
- Testing with a fake that always returns `yes`. Every assertion about the retain
|
||||
path is skipped, and retain is the path that matters — it is what makes this a
|
||||
gate rather than a summarizer.
|
||||
- Asserting `chunks_used` without asserting memory bytes. A loop that updates
|
||||
memory on `no` but counts correctly passes a count-only test; assertion 1
|
||||
compares bytes.
|
||||
- Omitting assertion 10. Without it, L2 in M3.1 becomes a copy of this loop, and
|
||||
the two drift.
|
||||
|
||||
## Traps
|
||||
|
||||
- Truncating over-budget memory to fit. It corrupts every subsequent turn's
|
||||
input, and the damage is attributed to the model.
|
||||
- Acting on `E_t` at L1 because "the model said it had enough". Paper §3.3 is
|
||||
explicit that this is wrong for exhaustive questions, and it silently truncates
|
||||
extraction in a way that looks like poor recall.
|
||||
- Writing log events before the turn resolves. A cancelled run then leaves a
|
||||
memory event with no matching gate event and `mem verify` fails on a file that
|
||||
was merely interrupted.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Architecture, tier model · paper Alg 1
|
||||
@@ -0,0 +1,104 @@
|
||||
# M1.6 — JSONL event log writer
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Write the authoritative record — the one artifact everything else is derived
|
||||
from, and the one that must survive a crash mid-run.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Path: `log/<project>/<query-id>/<run-id>.jsonl`. Append-only, one object per line.
|
||||
**Every record carries `level`.**
|
||||
|
||||
```jsonl
|
||||
{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
|
||||
{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."}
|
||||
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
|
||||
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
|
||||
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<sha>"],"sha256":"..."}
|
||||
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}
|
||||
```
|
||||
|
||||
`evidence` appears **only** when the update gate opened. That is what makes
|
||||
update-rate directly measurable from the log — `gate` records give the
|
||||
denominator, `evidence` records the numerator.
|
||||
|
||||
This log is authoritative: the vault and pgvector are projections rebuilt from
|
||||
it. Two consequences — it is tracked in git, and it is never rewritten in place.
|
||||
|
||||
A run that crashes leaves a file with no `run_end`. That is a valid, readable
|
||||
state meaning "incomplete", not corruption. Readers must handle it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `LogWriter::open(project, query_id, run_id)` in `mem-store`, creating parents.
|
||||
2. `append(event)` serializes one line and **flushes**. An unflushed buffer loses
|
||||
the last turns of exactly the run you want to debug.
|
||||
3. `run_id` is a ULID — lexicographically sortable by creation time, so listing
|
||||
runs in order is a directory sort.
|
||||
4. `LogReader` streams events back, tolerating a truncated final line.
|
||||
5. `replay_memory_at(t)` reconstructs `M_t` from the events alone, proving the log
|
||||
is sufficient.
|
||||
6. `stats()` computes chunks seen/used and update-rate from a log file.
|
||||
7. Never open in truncate mode. Append only.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Every emitted record has a `level`.
|
||||
- `evidence` count equals the count of `gate` records with `update: true`.
|
||||
- A file with no `run_end` reads cleanly and reports `incomplete`.
|
||||
- `replay_memory_at(t)` matches the memory the loop held at `t`.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** the scripted loop from M1.5 writing to a temp dir, plus a corrupted
|
||||
fixture.
|
||||
|
||||
**Integration test** — `tests/it_event_log.rs`:
|
||||
1. `a1_every_record_has_level` — parse every line, assert `level` present and in
|
||||
`{L0,L1,L2}`.
|
||||
2. `a2_evidence_matches_update_gates` — count `gate.update==true`, assert equal to
|
||||
the `evidence` count.
|
||||
3. `a3_replay_equals_live` — for every `t`, `replay_memory_at(t)` equals the
|
||||
memory the loop held. This is the assertion that proves the authority model.
|
||||
4. `a4_truncated_tail_reads` — chop the last line mid-object; assert all prior
|
||||
events parse and the run reports `incomplete`.
|
||||
5. `a5_no_run_end_is_incomplete` — a log ending after a `memory` event reports
|
||||
incomplete, not an error.
|
||||
6. `a6_append_only` — write, reopen, write again; assert the first events survive.
|
||||
7. `a7_flush_per_event` — kill the process (or drop without close) after 3
|
||||
appends; assert 3 lines on disk.
|
||||
8. `a8_run_id_sorts_by_time` — three runs, assert lexicographic order equals
|
||||
chronological order.
|
||||
|
||||
**Command:** `cargo test -p mem-store event_log`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the file parses. A writer that omits `evidence` events entirely
|
||||
produces a perfectly parseable log with an update-rate of zero — assertion 2 is
|
||||
what catches it.
|
||||
- Testing replay only at the final `t`. A writer that records only the final
|
||||
memory passes that and fails assertion 3 at every intermediate turn.
|
||||
- Omitting assertion 7. Buffered writes pass every test that closes the file
|
||||
properly, and lose data in precisely the crash case the log exists for.
|
||||
|
||||
## Traps
|
||||
|
||||
- Opening with truncate. One accidental re-run erases the authoritative record,
|
||||
and the projections are the only surviving copy — inverted authority.
|
||||
- Gitignoring `log/`. Makes the whole "JSONL is authoritative" claim a fiction.
|
||||
`agent-rust/.gitignore` has a bare `tasks` entry that untracks its whole board;
|
||||
do not repeat it here.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Storage schemas, authority model
|
||||
@@ -0,0 +1,102 @@
|
||||
# M1.7 — `mem ingest` end to end
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
One command that reads a real project and produces a real log — and reports the
|
||||
number that says whether the gate works.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem ingest --project poimen --query infra-root-causes
|
||||
mem ingest --project poimen # all queries in the set
|
||||
mem ingest --project poimen --limit 50 # first 50 chunks, for iterating
|
||||
mem ingest --project poimen --resume # skip chunks already in the log
|
||||
```
|
||||
|
||||
Progress output, because a run that prints nothing cannot be distinguished from
|
||||
one that has hung:
|
||||
|
||||
```
|
||||
[ 17/412] t=17 update=yes mem=142tok 1.9s
|
||||
[ 18/412] t=18 update=no mem=142tok 0.8s
|
||||
...
|
||||
run 01HXYZ chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
|
||||
```
|
||||
|
||||
**Update-rate is the headline number.** Tool results are ~43% of records and
|
||||
mostly evidence-free; a correct gate rejects the large majority of chunks. A rate
|
||||
above ~30% means the gate is not discriminating and the run is an expensive
|
||||
summarizer — that is the paper's memory-explosion failure and it is what M1.8
|
||||
gates on.
|
||||
|
||||
Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every
|
||||
chunk costs a model call, so `--resume` is not a nicety.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
|
||||
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
|
||||
3. Report update-rate in the summary and as `--format json`.
|
||||
4. `--resume` reads the existing log, finds the highest `t` with a `gate` record,
|
||||
and restarts from `t+1` with that turn's memory.
|
||||
5. `--limit` caps chunks processed.
|
||||
6. Exit non-zero if the run did not reach `run_end`.
|
||||
7. Ctrl-C finishes the in-flight turn, writes `run_end`, exits — no half-turn.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A real project produces a complete log with `run_end`.
|
||||
- Reported update-rate equals the value computed independently from the log.
|
||||
- `--resume` on a complete log is a no-op; on a partial one it continues.
|
||||
- Interrupt produces a valid log.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** scripted client for determinism, plus one live `#[ignore]` run.
|
||||
|
||||
**Integration test** — `tests/it_ingest.rs`:
|
||||
1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event
|
||||
counts match the script.
|
||||
2. `a2_update_rate_matches_log` — compare the reported rate to
|
||||
`LogWriter::stats()` recomputed from the file.
|
||||
3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model
|
||||
calls.
|
||||
4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the
|
||||
next call is t=11 and memory at t=11 equals the replayed memory at t=10.
|
||||
5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has
|
||||
`run_end`, and the last `gate` has a matching `memory`-or-not decision.
|
||||
6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records.
|
||||
7. `a7_live_smoke` — `#[ignore]`; real gateway, `--limit 20` on a real project;
|
||||
assert `run_end` and **print** the update-rate for a human to read.
|
||||
|
||||
**Command:** `cargo test -p mem-cli ingest` (add `-- --ignored` for a7)
|
||||
|
||||
**False pass:**
|
||||
- Asserting only that the command exits 0. A run whose gate always answers `no`
|
||||
exits 0, writes a valid log, and has learned nothing — the update-rate is the
|
||||
only thing that distinguishes it, which is why a7 prints it rather than
|
||||
merely asserting a run happened.
|
||||
- Resuming by counting lines rather than reading the highest `t` with a `gate`
|
||||
record. Line counts break the moment an `evidence` record is present, i.e. as
|
||||
soon as the gate ever opened.
|
||||
|
||||
## Traps
|
||||
|
||||
- No progress output. A 14-minute run that prints nothing is indistinguishable
|
||||
from a hang, and the first instinct will be to kill it.
|
||||
- Resume that replays from `t=1` with the old memory. It costs a full run and
|
||||
produces a log with duplicate turns that `mem verify` will reject.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verification, P2
|
||||
@@ -0,0 +1,99 @@
|
||||
# M1.8 — M1 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M1 — Gated loop at L1 |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Answer the only question that matters at this stage: **did we build a gate, or an
|
||||
expensive summarizer?**
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
This gate runs against the **live gateway on a real project** and asserts
|
||||
properties of the resulting log. It is the first phase that costs money in wall
|
||||
clock, and the first that can fail for reasons no unit test can see.
|
||||
|
||||
Two thresholds, both from the paper:
|
||||
|
||||
**Update-rate < 30%.** Agent transcripts are ~43% tool results, mostly
|
||||
evidence-free. Paper Figure 6 shows the failure mode directly: the ungated
|
||||
MemAgent's memory climbs to its 1024-token ceiling and saturates, after which
|
||||
"the accumulated noise can further impede subsequent updates". A high update-rate
|
||||
is that curve starting.
|
||||
|
||||
**Memory size flat, not climbing.** Plot `memory.tokens` against `t`. GRU-Mem's
|
||||
curve is low and roughly flat; the ungated curve rises to the cap and stays
|
||||
pinned. A monotonic climb means the gate is open too often even if the rate
|
||||
looks acceptable.
|
||||
|
||||
Third property, cheap and load-bearing: **the same run twice produces the same
|
||||
chunk hashes**. Temperature affects the model's text, not the chunking; if chunk
|
||||
shas differ between runs, identity includes something it should not (M0.2) and
|
||||
every projection will churn.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Run `mem ingest --project poimen` for all standing queries against the live
|
||||
gateway.
|
||||
2. Compute per query: update-rate, memory-token series, parse-failure count,
|
||||
elapsed.
|
||||
3. Assert the thresholds below.
|
||||
4. Emit `expected/m1-gate.txt` with the summary table; commit it. Subsequent runs
|
||||
diff against it, and a changed expectation is a reviewable claim.
|
||||
5. Sample 20 gate decisions and have the 32B `reasoning` model audit them; report
|
||||
agreement. Advisory at this gate, and the seed of M5.2's calibration.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Update-rate < 30% on every standing query.
|
||||
- Memory tokens ≤ 1024 and not monotonically increasing.
|
||||
- Parse-failure rate < 5%.
|
||||
- Chunk shas stable across two runs.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** live gateway. Long-running; a nightly or on-demand job, not
|
||||
per-push.
|
||||
|
||||
**Integration test** — `tests/it_m1_gate.rs`, all `#[ignore]` by default:
|
||||
1. `a1_update_rate_under_threshold` — per query, assert < 0.30, print actual.
|
||||
2. `a2_memory_bounded` — every `memory.tokens` ≤ 1024.
|
||||
3. `a3_memory_not_climbing` — fit a line to tokens vs `t`; assert the slope is
|
||||
below a small positive bound. Do not assert non-increasing — a legitimately
|
||||
growing memory rises early then plateaus.
|
||||
4. `a4_parse_failure_rate` — `parse_failed` / turns < 0.05.
|
||||
5. `a5_chunk_sha_stable` — two runs, same chunk shas in the same order.
|
||||
6. `a6_evidence_traceable` — every `evidence` sha appears as a parent of some
|
||||
`memory` record.
|
||||
7. `a7_judge_audit` — sample 20 decisions, ask the 32B model, print agreement.
|
||||
Advisory; does not fail the gate.
|
||||
|
||||
**Command:** `cargo test --workspace m1_gate -- --ignored --nocapture`
|
||||
|
||||
**False pass:**
|
||||
- Running the gate on a tiny `--limit`. Update-rate on the first 20 chunks is
|
||||
noise; the failure mode is cumulative and needs a full project.
|
||||
- Asserting the rate without printing it. The number is the artifact — a run at
|
||||
29% passes and is telling you something a boolean hides.
|
||||
- Asserting memory is non-increasing rather than bounded-slope. Real memory does
|
||||
grow early, and a strict assertion here fails on correct behaviour, which
|
||||
trains everyone to skip the gate.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating a high update-rate as a model-quality problem first. Check the
|
||||
prompt (M1.3 golden file) and the parser (M1.4 defaults) before blaming the
|
||||
3B model — a parser with `unwrap_or(true)` produces exactly this symptom.
|
||||
- Tuning the threshold to whatever the first run produced. 30% comes from the
|
||||
corpus composition; moving it to accommodate a bad result deletes the gate.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verification · paper Fig 6, §4.2
|
||||
@@ -0,0 +1,86 @@
|
||||
# M2.1 — Embeddings client
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn text into 768-dim vectors, batched, against the gateway's TEI endpoint.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
POST /v1/embeddings
|
||||
{"model":"nomic-ai/nomic-embed-text-v2-moe","input":["..."]}
|
||||
-> {"object":"list","data":[{"embedding":[...768 floats...]}],"usage":{...}}
|
||||
```
|
||||
|
||||
**768 dimensions**, probed and confirmed. It is the `vector(768)` in the schema
|
||||
and in the HNSW index; a model swap is a schema migration, not a config change.
|
||||
|
||||
**Batch limit is 32.** Verified: 1200 inputs returned
|
||||
`{"message":"batch size 1200 > maximum allowed batch size 32","code":413}`.
|
||||
Chunk the input list accordingly.
|
||||
|
||||
This route currently has **no auth** — no `konghq.com/plugins` annotation, so
|
||||
`model-key-auth` never attaches. Send the `apikey` header anyway: the route
|
||||
should be fixed, and a client that only works while auth is broken breaks when
|
||||
it is fixed.
|
||||
|
||||
Bodies here are large (many texts × long strings) and the Kong buffer is 16m, so
|
||||
batching also keeps requests well inside it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `EmbeddingsClient::embed(texts: &[String]) -> Result<Vec<Vec<f32>>>` in `mem-llm`.
|
||||
2. Split into batches of ≤32, preserving input order in the output.
|
||||
3. Assert every returned vector is exactly 768 long; a mismatch is an error
|
||||
naming the model, not a silent pad or truncate.
|
||||
4. Reuse M1.1's client config: `apikey` header, retry on 5xx only, generous
|
||||
timeout.
|
||||
5. `embed_one` convenience wrapper.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- 100 texts return 100 vectors in input order.
|
||||
- Every vector is 768-dim.
|
||||
- A dimension mismatch errors loudly.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `wiremock` offline, one `#[ignore]` live test.
|
||||
|
||||
**Integration test** — `tests/it_embeddings.rs`:
|
||||
1. `a1_batches_at_32` — 100 inputs produce exactly 4 requests.
|
||||
2. `a2_order_preserved` — mock returns identifiable vectors; assert output order
|
||||
matches input order across batch boundaries.
|
||||
3. `a3_dimension_asserted` — mock returns a 512-dim vector; assert an error
|
||||
naming the model.
|
||||
4. `a4_apikey_sent` — assert the header is present even though the route does not
|
||||
require it.
|
||||
5. `a5_live_dims` — `#[ignore]`; real gateway, assert 768.
|
||||
|
||||
**Command:** `cargo test -p mem-llm embeddings` (add `-- --ignored` for a5)
|
||||
|
||||
**False pass:**
|
||||
- Testing with ≤32 inputs. The batching path never runs and order-across-batches
|
||||
— the thing most likely to be wrong — is never exercised.
|
||||
- Trusting the response order within a batch without asserting it. Assertion 2
|
||||
must use distinguishable vectors, not a length check.
|
||||
|
||||
## Traps
|
||||
|
||||
- Hardcoding 768 in three places. Put it in one constant that the schema
|
||||
migration also references, so a model change is one edit and one migration.
|
||||
- Assuming no auth is needed because it currently works without a key. That route
|
||||
is missing its plugin annotation, which is a bug scheduled to be fixed.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verified facts, pgvector
|
||||
@@ -0,0 +1,102 @@
|
||||
# M2.2 — CNPG `memory-db` + pgvector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | homelab |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
|
||||
## Goal
|
||||
|
||||
A Postgres with pgvector, provisioned the way everything else in the cluster is:
|
||||
through git, with no manual `psql`.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**pgvector needs no custom image.** Verified on the running cluster:
|
||||
|
||||
```
|
||||
$ psql -tAc "select name,default_version,installed_version
|
||||
from pg_available_extensions where name='vector'"
|
||||
vector|0.7.0|
|
||||
```
|
||||
|
||||
on the stock `ghcr.io/cloudnative-pg/postgresql:16.2`. Available, not yet
|
||||
installed — `CREATE EXTENSION` is all that is missing.
|
||||
|
||||
**The operator is CNPG 1.30.0**, which supports declarative extensions on the
|
||||
`Database` CRD (`kubectl explain database.spec.extensions` resolves). So the
|
||||
extension is git-managed too — no manual step, consistent with the GitOps rule
|
||||
that infrastructure changes flow through version control.
|
||||
|
||||
Follow `k8s/infra/databases/temporal-db.yaml` exactly: 3 instances, `imageName`
|
||||
pinned, `enableSuperuserAccess: false`, `storageClass: longhorn-cnpg`,
|
||||
`enablePodMonitor: true`, control-plane tolerations, `podAntiAffinityType:
|
||||
preferred`.
|
||||
|
||||
Storage: 10Gi matches the existing clusters. At 768 dims × 4 bytes, a vector is
|
||||
~3 KB; tens of thousands of nodes is well under a gigabyte, so 10Gi is generous
|
||||
and consistent rather than tight.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `k8s/infra/databases/memory-db.yaml` — `Cluster` + `Database` with
|
||||
`extensions: [{name: vector, ensure: present}]`.
|
||||
2. Namespace `memory`, created by the ArgoCD app that owns it.
|
||||
3. Add to the owning kustomization's explicit resource list — an unlisted file is
|
||||
silently dropped with no error and no drift shown.
|
||||
4. Commit, push, let ArgoCD sync. **No `kubectl apply`.**
|
||||
5. Verify the extension installed and the app user can create tables.
|
||||
6. Record the connection string convention in the repo README; the password comes
|
||||
from the CNPG-generated secret, never committed.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `Cluster` reaches `Cluster in healthy state` with 3 instances.
|
||||
- `select extversion from pg_extension where extname='vector'` returns a version.
|
||||
- ArgoCD shows the app `Synced/Healthy`.
|
||||
- No manual `psql` was run to get there.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `kubectl` and `psql` read-only checks after sync.
|
||||
|
||||
**Integration test** — `verify/m2.2.sh`, output diffed against `expected/m2.2.txt`:
|
||||
1. `a1_cluster_healthy` — `kubectl get cluster -n memory memory-db` reports 3/3
|
||||
ready.
|
||||
2. `a2_extension_installed` — `select extname, extversion from pg_extension where
|
||||
extname='vector'` returns one row.
|
||||
3. `a3_declarative_not_manual` — `kubectl get database -n memory memory-db-vector
|
||||
-o jsonpath='{.spec.extensions}'` shows the declaration, proving it came from
|
||||
git.
|
||||
4. `a4_argocd_synced` — the owning app is `Synced/Healthy`.
|
||||
5. `a5_app_user_can_ddl` — as `app`, `CREATE TABLE t(v vector(768)); DROP TABLE t;`
|
||||
succeeds.
|
||||
6. `a6_hnsw_available` — `CREATE INDEX ... USING hnsw` on that temp table
|
||||
succeeds, proving 0.7.0 has the index type the schema needs.
|
||||
|
||||
**Command:** `bash verify/m2.2.sh | diff - expected/m2.2.txt`
|
||||
|
||||
**False pass:**
|
||||
- Checking `pg_available_extensions` instead of `pg_extension`. Available means
|
||||
the files are on disk; installed means `CREATE EXTENSION` ran. The whole task
|
||||
is the second one.
|
||||
- Verifying after a manual `CREATE EXTENSION`. It passes and proves nothing about
|
||||
the declarative path, which is the actual deliverable. Assertion 3 is the guard.
|
||||
|
||||
## Traps
|
||||
|
||||
- Forgetting the kustomization resource list. The file sits in git, ArgoCD reports
|
||||
Synced, and the objects never exist — silent, and the failure surfaces later as
|
||||
a connection error.
|
||||
- Adding `prune: true` semantics without thinking about operator-created children.
|
||||
CNPG creates Services, Secrets and PVCs owned by the Cluster; if ArgoCD's
|
||||
tracking label propagates to them, prune fights the operator. The
|
||||
`llm-serving` app already had to set `prune: false` for exactly this reason.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — pgvector · `k8s/infra/databases/temporal-db.yaml`
|
||||
@@ -0,0 +1,110 @@
|
||||
# M2.3 — Schema + sqlx migrations
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
The tables the provenance graph lives in, with the constraints that make a
|
||||
malformed graph impossible rather than merely unlikely.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_node (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')),
|
||||
project TEXT NOT NULL,
|
||||
query_id TEXT, -- NULL at L2
|
||||
run_id TEXT NOT NULL,
|
||||
t INT NOT NULL,
|
||||
source TEXT, -- set at L0
|
||||
text TEXT NOT NULL,
|
||||
sha256 TEXT NOT NULL UNIQUE, -- content identity, from M0.2
|
||||
embedding vector(768) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE TABLE memory_edge (
|
||||
child_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
|
||||
parent_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
|
||||
PRIMARY KEY (child_sha, parent_sha)
|
||||
);
|
||||
CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops);
|
||||
CREATE INDEX ON memory_node (project, level);
|
||||
```
|
||||
|
||||
`sha256 UNIQUE` is what makes rebuild idempotent — re-inserting identical content
|
||||
is a conflict to ignore, not a duplicate row. It is also why the hash must exclude
|
||||
run ids and timestamps (M0.2).
|
||||
|
||||
`ON DELETE CASCADE` on both edge columns: dropping a node should not leave
|
||||
dangling edges. Rebuild drops everything anyway, but a partial cleanup should not
|
||||
be able to corrupt the graph.
|
||||
|
||||
`query_id` is NULL at L2 by design — L2 spans queries. Enforce it:
|
||||
`CHECK ((level = 'L2') = (query_id IS NULL))`.
|
||||
|
||||
Cosine distance, not L2: these are normalised text embeddings and cosine is what
|
||||
the model was trained for. `vector_cosine_ops` must match the operator the query
|
||||
uses (`<=>`), or the index is silently ignored and every query is a seq scan.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `migrations/0001_init.sql` with the above, plus the `query_id`/level CHECK.
|
||||
2. `CREATE EXTENSION IF NOT EXISTS vector;` first — the declarative extension
|
||||
(M2.2) should have run, and this makes local dev work too.
|
||||
3. Wire `sqlx::migrate!()` and run at startup.
|
||||
4. `sqlx prepare` for offline compile-time query checking in CI.
|
||||
5. Add a `schema_version` sanity query the repo layer asserts on connect.
|
||||
6. Document that changing the embedding model is a migration, because the column
|
||||
width is part of the schema.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Migrations apply to a clean database and are idempotent.
|
||||
- Inserting a duplicate `sha256` conflicts rather than duplicating.
|
||||
- An `L2` row with a non-null `query_id` is rejected by the CHECK.
|
||||
- The HNSW index is used by a cosine-distance query.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a disposable database — `sqlx::test` or testcontainers with the same
|
||||
image tag as production, `ghcr.io/cloudnative-pg/postgresql:16.2`.
|
||||
|
||||
**Integration test** — `tests/it_schema.rs`:
|
||||
1. `a1_migrate_clean` — apply to an empty database, assert both tables exist.
|
||||
2. `a2_migrate_idempotent` — apply twice, assert no error.
|
||||
3. `a3_sha_unique` — insert the same sha twice, assert a unique violation.
|
||||
4. `a4_level_check` — `level='L3'` rejected; `level='L2'` with a `query_id`
|
||||
rejected; `level='L1'` without one rejected.
|
||||
5. `a5_edge_fk` — an edge referencing a missing sha is rejected.
|
||||
6. `a6_cascade` — delete a node, assert its edges are gone.
|
||||
7. `a7_hnsw_is_used` — `EXPLAIN` a `ORDER BY embedding <=> $1 LIMIT 10` query and
|
||||
assert the plan contains `Index Scan` on the HNSW index, not `Seq Scan`.
|
||||
|
||||
**Command:** `cargo test -p mem-store schema`
|
||||
|
||||
**False pass:**
|
||||
- Testing the schema against SQLite or plain Postgres without pgvector. It will
|
||||
accept `vector(768)` as an unknown type in some configurations and every
|
||||
vector assertion becomes meaningless. Use the production image.
|
||||
- Omitting assertion 7. An index created with the wrong opclass exists, reports
|
||||
healthy, and is never used — queries just get slower as the table grows, which
|
||||
reads as a scaling problem rather than a wrong index.
|
||||
|
||||
## Traps
|
||||
|
||||
- `vector_l2_ops` with a `<=>` query, or the reverse. The index is silently
|
||||
ignored. This is the single most common pgvector mistake.
|
||||
- Making `query_id` NOT NULL because L1 always has one. L2 then cannot be stored,
|
||||
and the workaround is a sentinel string that pollutes every group-by.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — pgvector schema
|
||||
@@ -0,0 +1,99 @@
|
||||
# M2.4 — pgvector repository
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.3, M2.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Write the projection into Postgres idempotently, so rebuild is safe to run at any
|
||||
time and produces the same rows.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```rust
|
||||
async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>;
|
||||
async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
|
||||
async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize)
|
||||
-> Result<Vec<ScoredNode>>;
|
||||
async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
|
||||
async fn clear_project(&self, project: &ProjectId) -> Result<()>;
|
||||
```
|
||||
|
||||
`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
|
||||
identical node is the same node; re-running rebuild must not duplicate or churn
|
||||
rows. Same for edges on the composite key.
|
||||
|
||||
`search` orders by `embedding <=> $1` — cosine distance, matching the
|
||||
`vector_cosine_ops` index. Any other operator silently drops to a seq scan.
|
||||
|
||||
Edges are inserted **after** both endpoints exist, or the foreign key rejects
|
||||
them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
|
||||
optimisation; a single-pass insert fails on the first forward reference.
|
||||
|
||||
Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
|
||||
rebuild — batch across nodes, not per node.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `PgRepo::connect(url)` with a pool; run migrations on connect.
|
||||
2. Implement the five methods above.
|
||||
3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
|
||||
multi-row statement.
|
||||
4. Two-pass write: nodes, then edges.
|
||||
5. `clear_project` deletes nodes for one project; edges cascade.
|
||||
6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert
|
||||
to a similarity score here. The reranker (M3.2) wants the ordering, and a
|
||||
lossy conversion hides ties.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Upserting the same node twice leaves one row.
|
||||
- Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
|
||||
- `search` returns nearest-first and respects the level filter.
|
||||
- `clear_project` removes only that project.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** disposable Postgres with the production image; a deterministic fake
|
||||
embedder (hash → fixed vector) so vector assertions are exact.
|
||||
|
||||
**Integration test** — `tests/it_pg_repo.rs`:
|
||||
1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
|
||||
2. `a2_two_pass_required` — single-pass insert with a forward edge reference
|
||||
fails; two-pass succeeds. Proves the ordering constraint is real.
|
||||
3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
|
||||
order matches hand-computed cosine distance.
|
||||
4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
|
||||
L1.
|
||||
5. `a5_project_isolation` — two projects with identical text; search one, assert
|
||||
no cross-project results.
|
||||
6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
|
||||
orphan edges remain.
|
||||
7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
|
||||
8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.
|
||||
|
||||
**Command:** `cargo test -p mem-store pg_repo`
|
||||
|
||||
**False pass:**
|
||||
- Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
|
||||
which removes the only check that the distance operator matches the index.
|
||||
- Testing `search` with one project in the database. Assertion 5 is the only one
|
||||
that catches a missing `WHERE project = $1`, and that bug leaks another
|
||||
project's memory into every answer.
|
||||
|
||||
## Traps
|
||||
|
||||
- Converting distance to similarity in the repo. It loses precision, and the
|
||||
reranker downstream wants candidates in order rather than scores.
|
||||
- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
|
||||
would do, and rebuild goes from seconds to minutes.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
|
||||
@@ -0,0 +1,113 @@
|
||||
# M2.5 — Obsidian projector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Render the log as a vault a human reads, with the tier graph as the link graph.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
vault/<project>/
|
||||
index.md L2 synthesis, links every L1 note
|
||||
<query-id>.md L1, one per standing query
|
||||
evidence/<source>-<t>.md L0, only with --emit-evidence-notes
|
||||
```
|
||||
|
||||
```markdown
|
||||
---
|
||||
project: poimen
|
||||
level: L1
|
||||
query_id: infra-root-causes
|
||||
updated: 2026-08-17
|
||||
chunks_seen: 412
|
||||
chunks_used: 17
|
||||
run_id: 01HXYZ...
|
||||
---
|
||||
# Infra root causes — poimen
|
||||
|
||||
<final memory text, verbatim>
|
||||
|
||||
## Provenance
|
||||
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer
|
||||
```
|
||||
|
||||
**Deterministic output is the requirement, not a nicety.** M2.8 asserts that
|
||||
rebuilding produces a byte-identical vault. That means: stable key order in
|
||||
frontmatter, no timestamp of *generation* (only `updated` derived from the log),
|
||||
sorted provenance lists, and `\n` line endings.
|
||||
|
||||
`updated` comes from the run's timestamp in the log — not `now()`. A generation
|
||||
timestamp makes every rebuild a diff and destroys the gate.
|
||||
|
||||
L0 notes default off: 17 per query is fine, but it grows unbounded across
|
||||
projects and queries. Citations inline give the same provenance without the file
|
||||
count.
|
||||
|
||||
Wikilinks are `[[<source-note-name>]]`. The link target may not exist as a file
|
||||
when evidence notes are off — that is fine and normal in Obsidian, and it still
|
||||
shows in the graph view as an unresolved node.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `ObsidianProjector::project(log_dir, vault_dir, opts)` in `mem-store`.
|
||||
2. Read the log; take the final `memory` record per query for L1, and the L2
|
||||
record for `index.md`.
|
||||
3. Frontmatter with a fixed key order; `updated` from the log.
|
||||
4. Provenance section from `parents`, sorted by source then `t`.
|
||||
5. `--emit-evidence-notes` writes L0 notes; default off.
|
||||
6. Write with `\n`, no trailing whitespace, exactly one trailing newline.
|
||||
7. A note whose L1 memory is empty is still written, with a body saying no
|
||||
evidence was found — an absent file is indistinguishable from a failed run.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Two projections of the same log produce byte-identical files.
|
||||
- Frontmatter key order is stable.
|
||||
- `updated` reflects the run, not the projection.
|
||||
- Every L1 note links to its L2 index and vice versa.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a committed log fixture and a committed expected vault tree.
|
||||
|
||||
**Integration test** — `tests/it_projector.rs`:
|
||||
1. `a1_byte_identical_twice` — project into two temp dirs, assert every file's
|
||||
bytes are equal. This is M2.8's core property, tested early.
|
||||
2. `a2_no_generation_timestamp` — project, sleep 1s, project again, assert equal.
|
||||
Catches `now()` leaking into output.
|
||||
3. `a3_frontmatter_key_order` — assert the exact key sequence.
|
||||
4. `a4_golden_tree` — diff the whole output against `expected/vault/`, empty diff.
|
||||
5. `a5_empty_memory_still_writes` — a log with zero updates produces a note
|
||||
saying so.
|
||||
6. `a6_links_bidirectional` — every L1 note appears in `index.md` and links back.
|
||||
7. `a7_evidence_notes_flag` — off by default; on, produces one note per L0 node.
|
||||
8. `a8_line_endings` — no `\r`, exactly one trailing `\n`.
|
||||
|
||||
**Command:** `cargo test -p mem-store projector`
|
||||
|
||||
**False pass:**
|
||||
- Comparing files by parsed content rather than bytes. Key reordering and
|
||||
whitespace churn both pass, and both fail M2.8 later, where the cause is much
|
||||
harder to find.
|
||||
- Testing with a single query. Assertion 6 needs at least two L1 notes to catch a
|
||||
link built from the wrong id.
|
||||
|
||||
## Traps
|
||||
|
||||
- `updated: {now}`. The most natural thing to write, and it makes every rebuild
|
||||
dirty, which trains everyone to ignore the diff that M2.8 depends on.
|
||||
- Serializing frontmatter from a `HashMap`. Iteration order is unspecified and
|
||||
the output churns between runs on the same input.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Obsidian vault
|
||||
@@ -0,0 +1,98 @@
|
||||
# M2.6 — `mem rebuild --from-log`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.4, M2.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Drop both projections and rebuild them from the log alone — the command that
|
||||
makes "the log is authoritative" a testable claim instead of a slogan.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem rebuild --from-log --project poimen # both projections
|
||||
mem rebuild --from-log --project poimen --vault-only
|
||||
mem rebuild --from-log --project poimen --db-only
|
||||
```
|
||||
|
||||
The claim: **anything not reconstructible from the log has a hidden input, and
|
||||
that is a bug.** Rebuild is the executable form of that claim. If it needs the
|
||||
existing vault or database to produce correct output, something is being carried
|
||||
across that is not in the record.
|
||||
|
||||
Rebuild does **no model calls except embeddings**. Gate decisions, memory text and
|
||||
provenance are all in the log already; re-running the controller would produce
|
||||
different text and defeat the purpose.
|
||||
|
||||
Order matters: clear → insert all nodes → insert all edges → project vault. Edges
|
||||
before nodes violates the foreign key (M2.4).
|
||||
|
||||
Embeddings are the expensive part. Cache by `sha256` so a rebuild after a vault
|
||||
template change does not re-embed unchanged nodes.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem rebuild --from-log --project P`.
|
||||
2. Read every log file for the project, in run-id order.
|
||||
3. `clear_project`, then two-pass node/edge insert, batching embeddings.
|
||||
4. Project the vault (M2.5), overwriting.
|
||||
5. Embedding cache keyed by sha, on disk under `.cache/`, so it survives runs.
|
||||
6. Report counts: nodes by level, edges, embeddings computed vs cached.
|
||||
7. Refuse to run if any log file is incomplete (no `run_end`) unless `--allow-partial`
|
||||
— rebuilding from a half-run silently produces a half-memory.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Rebuild from an empty database and empty vault produces the full state.
|
||||
- Rebuild twice produces identical database rows and identical vault bytes.
|
||||
- No controller model calls occur.
|
||||
- An incomplete log is refused by default.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** log fixture, disposable Postgres, temp vault. A controller client
|
||||
that panics if called.
|
||||
|
||||
**Integration test** — `tests/it_rebuild.rs`:
|
||||
1. `a1_from_empty` — drop everything, rebuild, assert node counts per level match
|
||||
the log's records.
|
||||
2. `a2_idempotent_db` — rebuild twice, assert row count unchanged and no
|
||||
`created_at` churn on existing rows.
|
||||
3. `a3_idempotent_vault` — rebuild twice, assert vault bytes identical.
|
||||
4. `a4_no_controller_calls` — inject a panicking chat client; assert rebuild
|
||||
succeeds.
|
||||
5. `a5_embedding_cache` — second rebuild computes zero embeddings.
|
||||
6. `a6_edge_order` — a log whose first memory references a later-inserted parent
|
||||
still rebuilds, proving two-pass.
|
||||
7. `a7_incomplete_refused` — a log with no `run_end` exits non-zero; with
|
||||
`--allow-partial` it succeeds.
|
||||
8. `a8_log_is_sufficient` — delete the vault and the database entirely, rebuild,
|
||||
and assert the result equals a committed golden. This is the authority claim.
|
||||
|
||||
**Command:** `cargo test -p mem-cli rebuild`
|
||||
|
||||
**False pass:**
|
||||
- Rebuilding on top of existing state. It masks every hidden input, because the
|
||||
missing piece is already there from the previous run. Assertions 1 and 8 must
|
||||
start from nothing.
|
||||
- Asserting row counts only. A rebuild that inserts the right number of rows with
|
||||
wrong `parents` passes; assertion 6 and M2.7's edge closure are what check the
|
||||
graph.
|
||||
|
||||
## Traps
|
||||
|
||||
- Re-running the controller during rebuild. It produces different memory text
|
||||
every time, the vault never stabilises, and M2.8 can never pass.
|
||||
- Caching embeddings by node id rather than content hash. Ids change between
|
||||
rebuilds; hashes do not, which is the whole point of content identity.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Authority model
|
||||
@@ -0,0 +1,90 @@
|
||||
# M2.7 — `mem verify` — edge closure
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Assert the provenance graph is well-formed, so a memory with no traceable
|
||||
evidence is caught rather than believed.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Invariants, checked against the log and the database independently:
|
||||
|
||||
1. Every L1 `memory` record has at least one L0 parent. A memory with no evidence
|
||||
came from somewhere the record does not explain.
|
||||
2. Every `parents` sha resolves to a node that exists.
|
||||
3. Every `evidence` sha appears as a parent of at least one memory. Evidence that
|
||||
nothing cites was written for no reason.
|
||||
4. `evidence` count equals `gate.update == true` count (also M1.6 a2, re-checked
|
||||
here across the whole project rather than one run).
|
||||
5. No edge is self-referential; no cycles.
|
||||
6. Levels are consistent: an L1 node's parents are L0; an L2 node's are L1.
|
||||
|
||||
Invariant 6 is the one that catches a tier confusion, and it is the one most
|
||||
likely to break when M3.1 adds the L2 pass — an L2 node accidentally parented to
|
||||
L0 evidence would still look plausible in the vault.
|
||||
|
||||
`mem verify` is a read-only diagnostic. It never repairs; repair is `mem rebuild`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem verify --project P [--db] [--log]`, defaulting to both.
|
||||
2. Check invariants 1–6, collecting **all** violations rather than failing on the
|
||||
first — one run should tell you everything wrong.
|
||||
3. Report per violation: invariant, level, sha, run id, and the log line number.
|
||||
4. Exit non-zero on any violation.
|
||||
5. `--format json` for machine consumption.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A clean project reports zero violations, exit 0.
|
||||
- Each invariant has a fixture that violates it and is detected.
|
||||
- All violations are reported in one run, not just the first.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** hand-built log fixtures, one per invariant, plus a clean one.
|
||||
|
||||
**Integration test** — `tests/it_verify.rs`:
|
||||
1. `a1_clean_passes` — the good fixture, zero violations, exit 0.
|
||||
2. `a2_orphan_memory` — L1 with empty `parents`; detected as invariant 1.
|
||||
3. `a3_dangling_parent` — parent sha not present; invariant 2.
|
||||
4. `a4_uncited_evidence` — evidence nothing references; invariant 3.
|
||||
5. `a5_evidence_gate_mismatch` — 3 update-gates but 2 evidence records;
|
||||
invariant 4.
|
||||
6. `a6_cycle` — A parents B, B parents A; invariant 5.
|
||||
7. `a7_level_mismatch` — L2 node parented directly to an L0 node; invariant 6.
|
||||
8. `a8_reports_all` — a fixture violating three invariants at once; assert all
|
||||
three appear in one run's output.
|
||||
9. `a9_db_and_log_agree` — introduce a violation in the database only; assert
|
||||
`--db` catches it and `--log` does not, proving the two checks are independent.
|
||||
|
||||
**Command:** `cargo test -p mem-cli verify`
|
||||
|
||||
**False pass:**
|
||||
- Checking the database only. The log is authoritative; a log-level violation
|
||||
that rebuild happens to smooth over is still a bug in the writer, and
|
||||
assertion 9 is what keeps the two checks honest.
|
||||
- Failing fast on the first violation. It passes every single-violation fixture
|
||||
and makes assertion 8 impossible, which in practice means three rebuild cycles
|
||||
to find three problems.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating invariant 3 as fatal. Uncited evidence is a real smell, but a
|
||||
legitimate case exists: the final turn updates memory and the run is cut short
|
||||
before the memory record flushes. Report it, and let the gate decide severity.
|
||||
- Skipping invariant 6 because L2 does not exist yet. It is cheap now and it is
|
||||
precisely what M3.1 will break.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — tier model, Verification
|
||||
@@ -0,0 +1,100 @@
|
||||
# M2.8 — M2 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the authority model: the log is sufficient, and both projections are
|
||||
genuinely derived.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The claim under test — poimen's own principle, applied here:
|
||||
|
||||
> Nothing derived is authoritative. If it cannot be dropped and rebuilt
|
||||
> byte-identically, it has hidden inputs and that is a bug.
|
||||
|
||||
The gate is destructive by design: it **deletes** the vault and truncates the
|
||||
database, rebuilds from the log alone, and diffs. Anything that survives only
|
||||
because it was already there is a hidden input, and this is the only test that
|
||||
finds it.
|
||||
|
||||
```sh
|
||||
rm -rf vault/poimen
|
||||
psql -c "delete from memory_node where project = 'poimen'"
|
||||
mem rebuild --from-log --project poimen
|
||||
git -C vault diff --exit-code # empty diff is the only pass
|
||||
```
|
||||
|
||||
`git diff --exit-code` on a tracked vault is the assertion. It compares against
|
||||
what was committed, so it also catches a projector change that was not intended.
|
||||
|
||||
Run it twice: once from empty (sufficiency) and once on top of itself
|
||||
(idempotence). Both must produce the same bytes.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `verify/m2.8.sh` performing the destructive rebuild above.
|
||||
2. Assert the vault diff is empty and the database node counts match the log.
|
||||
3. Run `mem verify` and assert zero violations.
|
||||
4. Second rebuild without clearing; assert still empty diff and unchanged row
|
||||
count.
|
||||
5. Assert no controller model calls (embeddings are allowed and expected).
|
||||
6. Commit `expected/m2.8.txt` with the count summary; diff against it.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Vault rebuilt from nothing is byte-identical to the committed vault.
|
||||
- Database node/edge counts match the log's records exactly.
|
||||
- `mem verify` reports zero violations.
|
||||
- Second rebuild changes nothing.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** disposable database, git-tracked vault, real log. Long-running; a
|
||||
nightly or on-demand job.
|
||||
|
||||
**Integration test** — `verify/m2.8.sh`, output diffed against `expected/m2.8.txt`:
|
||||
1. `a1_vault_from_empty` — delete vault, rebuild, `git diff --exit-code` empty.
|
||||
2. `a2_db_from_empty` — truncate, rebuild, counts per level equal the log's.
|
||||
3. `a3_verify_clean` — `mem verify` exits 0.
|
||||
4. `a4_rebuild_idempotent` — rebuild again, diff still empty, row count unchanged.
|
||||
5. `a5_no_controller_calls` — assert zero calls to the chat route during rebuild
|
||||
(count via the record dir from M1.1, or a proxy).
|
||||
6. `a6_projection_independence` — `--vault-only` then `--db-only` produces the
|
||||
same end state as a combined rebuild.
|
||||
7. `a7_log_alone_suffices` — move the log to a fresh checkout with no vault and no
|
||||
database, rebuild, diff against the committed vault. The strongest form of
|
||||
the claim.
|
||||
|
||||
**Command:** `bash verify/m2.8.sh | diff - expected/m2.8.txt`
|
||||
|
||||
**False pass:**
|
||||
- Running the gate without deleting the vault first. A projector that only writes
|
||||
changed files produces an empty diff trivially, and the hidden input survives.
|
||||
- Diffing an untracked vault. `git diff` on untracked files reports nothing, so
|
||||
the assertion passes vacuously. The vault must be committed, or the script must
|
||||
compare against a committed golden tree explicitly.
|
||||
- Allowing controller calls "because it is easier". Rebuild then produces new
|
||||
memory text each run and the gate can never pass — at which point the usual fix
|
||||
is to weaken the gate.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating a non-empty diff as a projector bug by default. It is equally likely to
|
||||
be a *hash* bug: if `sha256` includes a timestamp (M0.2), every rebuild produces
|
||||
new nodes and the vault churns. Check identity before blaming rendering.
|
||||
- Running against production data with the destructive script and no backup. The
|
||||
log is the record; if that is intact, everything is recoverable — which is
|
||||
exactly why the log must be tracked in git before this gate is first run.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Authority model, Verification
|
||||
@@ -0,0 +1,102 @@
|
||||
# M3.1 — L2 synthesis pass
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3 — L2 synthesis and retrieval |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Project-level memory across the per-query memories — using the same loop, with
|
||||
the exit gate switched on.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem synthesize --project poimen
|
||||
```
|
||||
|
||||
L2 is **not new machinery**. It is `run_loop` (M1.5) with:
|
||||
|
||||
| | L1 | L2 |
|
||||
|---|---|---|
|
||||
| input stream | `Chunk` from sources | L1 `MemoryNode`s |
|
||||
| question | per-query question | `synthesis.question` |
|
||||
| `use_exit_gate` | false | **true** |
|
||||
| `query_id` | set | NULL |
|
||||
| parents | L0 evidence shas | L1 memory shas |
|
||||
|
||||
**Why the exit gate flips on.** At L1 the input is hundreds of chunks and the
|
||||
question is exhaustive ("what are *all* the X"), which is exactly the case paper
|
||||
§3.3 says to run without the gate. At L2 the input is a handful of memories and
|
||||
"enough evidence" is genuinely decidable, which is the case the gate was designed
|
||||
for and where the paper measures its 4× speedup.
|
||||
|
||||
If M1.5 was written correctly this task is mostly wiring. If it needs changes to
|
||||
`run_loop`, the level was not really a parameter — and assertion a10 in M1.5
|
||||
existed to prevent exactly that.
|
||||
|
||||
Ordering: L1 memories enter the stream in a stable order (query id, ascending),
|
||||
so synthesis is reproducible.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem synthesize --project P` reads the final L1 memory per standing query.
|
||||
2. Wrap them as the loop's input stream, in sorted query-id order.
|
||||
3. Run `run_loop` with `level = L2`, `use_exit_gate = true`, the synthesis
|
||||
question, `query_id = None`.
|
||||
4. Write to `log/<project>/_synthesis/<run-id>.jsonl`.
|
||||
5. `parents` on the L2 memory are the L1 memory shas consumed up to that turn.
|
||||
6. Refuse to run if any standing query has no completed L1 run — synthesizing
|
||||
over a partial set silently produces a partial picture.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- No change to `run_loop` is required.
|
||||
- The exit gate fires and stops early on a real project.
|
||||
- L2 parents are L1 shas, never L0.
|
||||
- Sorted input order makes two runs consume memories in the same sequence.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** scripted client, plus one live run.
|
||||
|
||||
**Integration test** — `tests/it_l2.rs`:
|
||||
1. `a1_reuses_run_loop` — assert `mem synthesize` calls the same `run_loop`
|
||||
symbol; a duplicated loop is a review failure, and a `#[deny]`-style test here
|
||||
is a grep asserting `fn run_loop` appears exactly once in the workspace.
|
||||
2. `a2_exit_gate_on` — scripted `end` at turn 2 of 5; assert it stops at 2.
|
||||
3. `a3_parents_are_l1` — every L2 parent sha resolves to an L1 node.
|
||||
4. `a4_query_id_null` — the L2 record has no `query_id`.
|
||||
5. `a5_stable_input_order` — two runs consume L1 memories in identical order.
|
||||
6. `a6_refuses_partial` — one query with no completed run; assert non-zero exit
|
||||
naming the query.
|
||||
7. `a7_level_check_holds` — run `mem verify`; invariant 6 (level consistency)
|
||||
passes.
|
||||
8. `a8_live` — `#[ignore]`; real project, assert an L2 memory is produced and
|
||||
print whether the exit gate fired and at which turn.
|
||||
|
||||
**Command:** `cargo test -p mem-cli l2` (add `-- --ignored` for a8)
|
||||
|
||||
**False pass:**
|
||||
- Copying `run_loop` into an L2-specific function. Everything passes, the two
|
||||
drift within a month, and the tier model quietly becomes two implementations.
|
||||
Assertion 1 is the guard.
|
||||
- Testing the exit gate with a script that never says `end`. The gate's effect is
|
||||
invisible and a `use_exit_gate` that is ignored passes.
|
||||
|
||||
## Traps
|
||||
|
||||
- Feeding L0 evidence into L2 "for more detail". It blows the context budget and
|
||||
breaks the level invariant; L2 reads memories, and if they are inadequate the
|
||||
fix is at L1.
|
||||
- Synthesizing over whatever L1 runs happen to exist. A missing query produces a
|
||||
confident summary of an incomplete project, which is worse than no summary.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — tier model · paper §3.3
|
||||
@@ -0,0 +1,88 @@
|
||||
# M3.2 — Rerank client
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3 — L2 synthesis and retrieval |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Reorder vector-recall candidates by actual relevance, because embedding distance
|
||||
is a coarse filter.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
POST /v1/rerank
|
||||
{"query":"what is rust","texts":["rust is a language","bananas"]}
|
||||
-> [{"index":0,"score":0.98176736},{"index":1,"score":0.00008833366}]
|
||||
```
|
||||
|
||||
`BAAI/bge-reranker-base` via TEI. Note the **response shape is a bare array**, not
|
||||
an OpenAI-style `{"data": [...]}` envelope — this route does not follow the chat
|
||||
convention.
|
||||
|
||||
The discrimination is real: 0.98 vs 0.00009 on that probe, four orders of
|
||||
magnitude. Embedding cosine on the same pair would be far closer, which is why
|
||||
recall-then-rerank beats recall alone.
|
||||
|
||||
`index` refers to the position in the submitted `texts` array; results come back
|
||||
**sorted by score**, so the index is the only way to map back. Do not assume
|
||||
order.
|
||||
|
||||
Batch limits apply as with embeddings — keep candidate lists modest (top-50 from
|
||||
recall is plenty).
|
||||
|
||||
## Steps
|
||||
|
||||
1. `RerankClient::rerank(query, texts) -> Result<Vec<Scored>>` in `mem-llm`.
|
||||
2. Parse the bare array; map `index` back to the caller's items.
|
||||
3. Preserve the caller's item type: take `&[T]`, return `Vec<(T, f32)>` so the
|
||||
caller does not re-associate by position.
|
||||
4. Same `apikey` header, retry and timeout policy as M1.1.
|
||||
5. Empty input returns empty without a request.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Results map correctly back to input items via `index`.
|
||||
- A more relevant text scores above a less relevant one on a live call.
|
||||
- Empty input makes no request.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `wiremock` offline, one `#[ignore]` live test.
|
||||
|
||||
**Integration test** — `tests/it_rerank.rs`:
|
||||
1. `a1_bare_array_parsed` — mock returns `[{"index":1,...},{"index":0,...}]`;
|
||||
assert parsing succeeds.
|
||||
2. `a2_index_mapping` — with the out-of-order mock above, assert the returned
|
||||
items correspond to inputs 1 and 0 respectively, not 0 and 1.
|
||||
3. `a3_empty_no_request` — empty texts; assert zero requests.
|
||||
4. `a4_apikey_sent` — header present.
|
||||
5. `a5_live_discriminates` — `#[ignore]`; real gateway, query "what is rust"
|
||||
against `["rust is a language","bananas"]`; assert the first scores at least
|
||||
10× the second.
|
||||
|
||||
**Command:** `cargo test -p mem-llm rerank` (add `-- --ignored` for a5)
|
||||
|
||||
**False pass:**
|
||||
- Assuming the response preserves input order. A mock that returns results in
|
||||
input order passes a naive test and the live endpoint returns them sorted,
|
||||
which silently mislabels every result. Assertion 2 must use an out-of-order
|
||||
mock.
|
||||
- Expecting an OpenAI envelope. It will fail immediately against the live route,
|
||||
but a mock written to match the wrong shape hides that until integration.
|
||||
|
||||
## Traps
|
||||
|
||||
- Re-associating results by position instead of by `index`. The scores are right
|
||||
and attached to the wrong documents — a bug that looks like poor retrieval
|
||||
quality rather than a mapping error.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verified facts, retrieval
|
||||
@@ -0,0 +1,97 @@
|
||||
# M3.3 — `mem query` with provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3 — L2 synthesis and retrieval |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.2, M2.4 |
|
||||
|
||||
## Goal
|
||||
|
||||
Ask the memory a question and get an answer that can be traced back to the
|
||||
session it came from.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem query "why did requests over 10KB fail?"
|
||||
mem query --project poimen --levels L1,L2 --k 10 "..."
|
||||
```
|
||||
|
||||
Pipeline: embed the question (M2.1) → HNSW recall top-k over `memory_node`
|
||||
filtered by project and level (M2.4) → rerank the candidates (M3.2) → return with
|
||||
provenance walked down through `memory_edge`.
|
||||
|
||||
Default levels are **L1 and L2**, not L0. L1/L2 are synthesized answers; L0 is raw
|
||||
evidence and returning it by default buries the answer in transcript. `--levels
|
||||
L0` exists for "show me the actual source".
|
||||
|
||||
Provenance walk: for each returned node, follow `memory_edge` to its parents and
|
||||
report source and turn. One hop for L1 (to evidence), two for L2 (through L1).
|
||||
|
||||
Recall wide, rerank narrow: take 50 from HNSW, rerank, return 5. Cosine distance
|
||||
alone puts "bananas" close enough to matter; the reranker separated the same pair
|
||||
by four orders of magnitude.
|
||||
|
||||
Output is human-readable by default, `--format json` for programmatic use.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem query [--project P] [--levels L] [--k N] <question>`.
|
||||
2. Embed, recall 10×k, rerank, truncate to k.
|
||||
3. Walk edges to build a provenance list per hit; deduplicate sources.
|
||||
4. Render: score, level, query id, the memory text, then provenance lines.
|
||||
5. Project defaults to the one inferred from `$PWD`; `--project` overrides. No
|
||||
project and no match is an error, not an empty result over everything.
|
||||
6. `--explain` prints the recall candidates and their pre-rerank distances, for
|
||||
debugging retrieval quality.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A question with a known answer returns the right L1 note first.
|
||||
- Every hit carries at least one resolvable provenance entry.
|
||||
- `--levels L0` returns evidence; the default does not.
|
||||
- Reranking changes the order versus raw recall on at least one real query.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a seeded database from a real log, deterministic fake embedder for
|
||||
the offline assertions, live for quality.
|
||||
|
||||
**Integration test** — `tests/it_query.rs`:
|
||||
1. `a1_known_answer` — seeded with the poimen log, query "why did requests over
|
||||
10KB fail?"; assert the top hit is the `infra-root-causes` L1 node.
|
||||
2. `a2_provenance_resolves` — every hit's provenance shas exist in
|
||||
`memory_node`.
|
||||
3. `a3_default_excludes_l0` — assert no L0 nodes in default output.
|
||||
4. `a4_levels_flag` — `--levels L0` returns evidence nodes.
|
||||
5. `a5_rerank_reorders` — capture pre- and post-rerank order; assert they differ
|
||||
on at least one fixture query, proving the reranker is wired and not a no-op.
|
||||
6. `a6_project_isolation` — two projects seeded; assert no cross-project hits.
|
||||
7. `a7_no_project_errors` — unresolvable project exits non-zero.
|
||||
8. `a8_l2_two_hop_provenance` — an L2 hit's provenance resolves through L1 to L0
|
||||
sources.
|
||||
|
||||
**Command:** `cargo test -p mem-cli query`
|
||||
|
||||
**False pass:**
|
||||
- Asserting only that results are returned. A pipeline where the reranker returns
|
||||
the input unchanged still returns results, and assertion 5 is the only check
|
||||
that it is doing anything.
|
||||
- Testing provenance existence without resolving it. A hit carrying parent shas
|
||||
that do not exist in the table looks fine in the output and is useless.
|
||||
|
||||
## Traps
|
||||
|
||||
- Returning L0 by default. The answer is there but buried in raw transcript, and
|
||||
the tool reads as low quality when it is a display default.
|
||||
- Recalling exactly k then reranking. Reranking cannot recover a relevant
|
||||
document that recall never returned; the width of the recall is what determines
|
||||
the ceiling.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
|
||||
@@ -0,0 +1,93 @@
|
||||
# M3.4 — M3 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3 — L2 synthesis and retrieval |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the whole stack answers a real question with real provenance — the first
|
||||
point at which the system is useful rather than merely correct.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The gate is a small set of **known-answer questions** with hand-written expected
|
||||
sources, committed to the repo. Each names a fact that genuinely appears in the
|
||||
ingested sessions and the session it appears in.
|
||||
|
||||
Seed set, all drawn from real work in this corpus:
|
||||
|
||||
| question | expected to cite |
|
||||
|---|---|
|
||||
| why did requests over 10KB fail? | the Kong body-buffer / `client_body_buffer_size` finding |
|
||||
| why did `Authorization: Bearer` return 401? | the Kong key-auth header finding |
|
||||
| what causes the 504 on a cold ornith start? | the timeouts-on-Ingress-vs-Service finding |
|
||||
|
||||
This is a retrieval quality gate, so it is graded, not boolean: report
|
||||
**hit rate at k=5** and **provenance precision** (fraction of cited sources that
|
||||
actually contain the fact). A single failing question is information, not
|
||||
necessarily a stop.
|
||||
|
||||
The gate also re-runs `mem verify` including the level invariant, because M3.1 is
|
||||
the change most likely to break it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `verify/known-answers.yaml` — question, expected node, expected source.
|
||||
2. `verify/m3.4.sh` runs each through `mem query --format json`.
|
||||
3. Compute hit rate at 5 and provenance precision; print both.
|
||||
4. Assert the thresholds below.
|
||||
5. Run `mem verify`; assert zero violations.
|
||||
6. Assert L2 exists and its provenance resolves two hops.
|
||||
7. Commit `expected/m3.4.txt`; diff.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Hit rate at k=5 ≥ 0.8 on the known-answer set.
|
||||
- Provenance precision ≥ 0.9 — a citation that does not contain the fact is worse
|
||||
than no citation.
|
||||
- `mem verify` clean, including level consistency.
|
||||
- Every L2 hit resolves to L0 sources.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** live gateway, seeded database from real logs. On-demand, not
|
||||
per-push.
|
||||
|
||||
**Integration test** — `verify/m3.4.sh` diffed against `expected/m3.4.txt`:
|
||||
1. `a1_hit_rate` — ≥ 0.8, print actual.
|
||||
2. `a2_provenance_precision` — for each cited source, fetch the L0 text and
|
||||
assert it contains the expected fact substring; ≥ 0.9.
|
||||
3. `a3_verify_clean` — zero violations.
|
||||
4. `a4_l2_two_hop` — every L2 hit resolves through L1 to a real source.
|
||||
5. `a5_rerank_contributes` — hit rate with reranking is ≥ hit rate without. If
|
||||
reranking makes it worse, the wiring is wrong (probably index mapping, M3.2).
|
||||
6. `a6_no_cross_project` — a question about another project returns nothing from
|
||||
this one.
|
||||
|
||||
**Command:** `bash verify/m3.4.sh | diff - expected/m3.4.txt`
|
||||
|
||||
**False pass:**
|
||||
- Writing the known-answer set after seeing what the system returns. It then
|
||||
measures nothing. Write the questions and expected sources from the sessions
|
||||
first, independently of any query output.
|
||||
- Measuring hit rate without provenance precision. A system that returns the
|
||||
right note with fabricated citations scores 1.0 on hits and is untrustworthy —
|
||||
assertion 2 is the one that matters for whether anyone can act on an answer.
|
||||
|
||||
## Traps
|
||||
|
||||
- Tuning `k` until the hit rate passes. k=50 will hit almost everything and the
|
||||
metric stops meaning anything; the gate specifies k=5 for that reason.
|
||||
- Accepting a5 failing as "reranker is just not helping". It far more often means
|
||||
the `index` mapping in M3.2 is wrong and scores are attached to the wrong
|
||||
documents.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verification
|
||||
@@ -0,0 +1,114 @@
|
||||
# M4.1 — `mem skill draft`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M4 — Skills |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn a memory note into a draft skill — the step that makes the memory *do*
|
||||
something rather than only be read.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem skill draft --from poimen/infra-root-causes
|
||||
-> vault/skills/_drafts/poimen-infra-root-causes/SKILL.md
|
||||
```
|
||||
|
||||
**A skill is a projection, not a level.** L0/L1/L2 are descriptive — what
|
||||
happened. A skill is procedural — what to do next time. The gated loop does not
|
||||
produce it: "does this chunk contain evidence for Q" has no meaning when the
|
||||
output is an instruction.
|
||||
|
||||
Format is free, because `SKILL.md` is YAML frontmatter plus markdown, which is
|
||||
exactly an Obsidian note. Verified against a real installed skill:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: keyword-research
|
||||
description: 'Use when the user asks to "find keywords"... Not for X — use Y.'
|
||||
when_to_use: "Use when starting keyword research for a new page..."
|
||||
argument-hint: "<topic or seed keyword> [market/language]"
|
||||
---
|
||||
```
|
||||
|
||||
So the same file is a vault note and a loadable skill, with no conversion.
|
||||
|
||||
**`description` is the whole game.** It is the trigger — a skill whose
|
||||
description does not match how the user actually phrases the request never fires,
|
||||
no matter how good the body is. Note the real example above spends half its
|
||||
description on *negative* routing ("Not for X — use Y").
|
||||
|
||||
Follow the existing rubric rather than inventing one: the installed
|
||||
`grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and
|
||||
a four-dimension rubric — conciseness, actionability, workflow clarity,
|
||||
progressive disclosure.
|
||||
|
||||
**Drafts land in `_drafts/` and are never auto-loaded.** A directory, not a
|
||||
frontmatter flag, because a directory cannot be accidentally globbed into
|
||||
`--skill`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem skill draft --from <project>/<query-id>` reads the L1 or L2 note.
|
||||
2. Prompt the model to convert descriptive memory into procedural instruction,
|
||||
with the rubric's four dimensions in the prompt.
|
||||
3. Emit frontmatter: `name`, `description`, `when_to_use`, plus
|
||||
`generated_from: <sha>` and `generated_at`.
|
||||
4. Write to `vault/skills/_drafts/<project>-<query-id>/SKILL.md`.
|
||||
5. Refuse to write outside `_drafts/`. Promotion is a human `git mv`.
|
||||
6. `--dry-run` prints without writing.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Output parses as valid frontmatter + markdown.
|
||||
- `generated_from` resolves to a real node sha.
|
||||
- The file lands in `_drafts/` and nowhere else.
|
||||
- Promotion is not automated anywhere in the codebase.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** seeded vault and log; scripted model client for determinism.
|
||||
|
||||
**Integration test** — `tests/it_skill_draft.rs`:
|
||||
1. `a1_valid_frontmatter` — parse the output; assert `name`, `description`,
|
||||
`when_to_use`, `generated_from` present and non-empty.
|
||||
2. `a2_generated_from_resolves` — the sha exists in `memory_node`.
|
||||
3. `a3_writes_only_to_drafts` — assert the path contains `_drafts/`; attempt to
|
||||
pass a path outside it and assert refusal.
|
||||
4. `a4_no_promotion_path` — grep the workspace for any code writing to
|
||||
`vault/skills/` that is not under `_drafts/`; assert none. Promotion must be
|
||||
manual.
|
||||
5. `a5_description_is_trigger_shaped` — assert `description` contains at least one
|
||||
phrasing cue (a quoted user phrase or "Use when"), matching the installed
|
||||
examples.
|
||||
6. `a6_dry_run_writes_nothing` — assert no file created.
|
||||
7. `a7_idempotent` — same input twice produces identical bytes apart from
|
||||
`generated_at`.
|
||||
|
||||
**Command:** `cargo test -p mem-cli skill_draft`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the file was written without asserting *where*. The entire safety
|
||||
property of this task is the location, and a draft written to `vault/skills/`
|
||||
is immediately loadable.
|
||||
- Accepting any non-empty `description`. A one-line restatement of the title
|
||||
never triggers, so the feature appears to work and the skill never fires —
|
||||
assertion 5 is a weak but real guard.
|
||||
|
||||
## Traps
|
||||
|
||||
- Auto-promoting "when the draft looks good". That closes the loop this design
|
||||
deliberately leaves open, and there is no external verifier inside it.
|
||||
- Generating the body from L0 evidence. Skills are procedure distilled from
|
||||
synthesis; raw transcript produces a narrative, not an instruction.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Skills, the procedural projection
|
||||
@@ -0,0 +1,109 @@
|
||||
# M4.2 — `derived: true` ingest filter
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M4 — Skills |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M4.1, M0.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Stop the system learning from its own output.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The cycle, and it is the only one in the design:
|
||||
|
||||
```
|
||||
emitted skill is loaded into a session
|
||||
│
|
||||
▼
|
||||
appears verbatim in that session's transcript
|
||||
│
|
||||
▼
|
||||
transcript is ingested as evidence
|
||||
│
|
||||
▼
|
||||
reinforces the memory that produced the skill
|
||||
```
|
||||
|
||||
No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
|
||||
what actually stops it.
|
||||
|
||||
Mechanism: every emitted artifact records its content hash in a manifest. During
|
||||
ingest, a record whose normalised text matches a known artifact is tagged
|
||||
`derived: true` and **excluded from evidence** — it is still recorded in the log
|
||||
so the exclusion is visible and auditable, but the gate never sees it.
|
||||
|
||||
Matching must survive the model reformatting the text slightly. Exact hash on the
|
||||
whole record is too brittle: a skill quoted with different indentation would slip
|
||||
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
|
||||
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
|
||||
threshold.
|
||||
|
||||
Threshold is a tradeoff and should be logged, not hidden: too low excludes
|
||||
genuine discussion *about* a skill, too high lets the cycle run.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
|
||||
`{name, sha256, shingles, emitted_at}`.
|
||||
2. `mem skill draft` appends to it.
|
||||
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
|
||||
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
|
||||
5. Log a `derived_excluded` event with the record's provenance and the artifact
|
||||
it matched, so exclusions are auditable and a false positive is findable.
|
||||
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
|
||||
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A record quoting an emitted skill verbatim is excluded.
|
||||
- A record quoting it with different whitespace and fences is also excluded.
|
||||
- A record merely *mentioning* the skill by name is not excluded.
|
||||
- Every exclusion is logged with what it matched.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a fixture manifest with one artifact, plus records in several
|
||||
paraphrase grades.
|
||||
|
||||
**Integration test** — `tests/it_derived_filter.rs`:
|
||||
1. `a1_verbatim_excluded` — exact copy of an artifact is excluded.
|
||||
2. `a2_reformatted_excluded` — same content, different indentation and code
|
||||
fences; excluded.
|
||||
3. `a3_mention_not_excluded` — "I used the infra-root-causes skill" is kept. This
|
||||
is the false-positive guard, and over-filtering silently starves the memory.
|
||||
4. `a4_unrelated_not_excluded` — random session text is kept.
|
||||
5. `a5_exclusion_logged` — assert a `derived_excluded` event naming the matched
|
||||
artifact.
|
||||
6. `a6_verify_catches_leak` — insert an L0 node matching an artifact directly into
|
||||
the database; assert `mem verify --derived-filter` fails.
|
||||
7. `a7_threshold_configurable` — assert the threshold is read from config and
|
||||
appears in the run record.
|
||||
8. `a8_no_manifest_is_safe` — with no manifest file, ingest proceeds and filters
|
||||
nothing, rather than failing.
|
||||
|
||||
**Command:** `cargo test -p mem-ingest derived_filter`
|
||||
|
||||
**False pass:**
|
||||
- Testing only the verbatim case. Exact-match filtering passes and the realistic
|
||||
case — a model that reformats what it quotes — walks straight through.
|
||||
Assertion 2 is the one that matters.
|
||||
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
|
||||
of a topic once a skill about it exists, which quietly makes the memory worse
|
||||
the more skills you write.
|
||||
|
||||
## Traps
|
||||
|
||||
- Filtering on record *hash*. One character of whitespace defeats it, and the
|
||||
cycle runs while the filter reports itself working.
|
||||
- Silent exclusion. Without assertion 5's log event, a false positive is
|
||||
invisible — memory just gets thinner and nobody knows why.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
|
||||
@@ -0,0 +1,87 @@
|
||||
# M4.3 — M4 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M4 — Skills |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M4 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the loop stays open — that a generated skill cannot silently become
|
||||
training data for the memory that generated it.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Two properties, and both must hold. Either one alone is insufficient:
|
||||
|
||||
1. **A draft is not loadable.** It lives in `_drafts/`, and pointing a real agent
|
||||
at `vault/skills/` must not surface it.
|
||||
2. **A promoted skill's text never enters evidence.** Even after a human promotes
|
||||
it and it appears in a session, the derived filter keeps it out of L0.
|
||||
|
||||
The gate exercises the full cycle deliberately: draft a skill, promote it, run a
|
||||
session that loads it, ingest that session, and assert the skill's content did
|
||||
not become evidence.
|
||||
|
||||
Property 1 is tested with the real loader, not by inspecting paths — the
|
||||
question is what an agent actually sees.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Draft a skill from a real note.
|
||||
2. `pi --skill vault/skills/ --list-skills` (or the equivalent enumeration);
|
||||
assert the draft is absent.
|
||||
3. Promote it with `git mv` out of `_drafts/`; assert it now appears.
|
||||
4. Synthesize a session transcript that quotes the promoted skill, in three
|
||||
grades: verbatim, reformatted, and merely referenced.
|
||||
5. Ingest it; assert the first two are excluded and the third is kept.
|
||||
6. Run `mem verify --derived-filter`; assert clean.
|
||||
7. Commit `expected/m4.3.txt`; diff.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Draft absent from a real loader's skill enumeration.
|
||||
- Promoted skill present.
|
||||
- Verbatim and reformatted quotes excluded from evidence; a bare mention kept.
|
||||
- `mem verify --derived-filter` reports zero leaks.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** the real `pi` binary for enumeration, seeded vault and log.
|
||||
|
||||
**Integration test** — `verify/m4.3.sh` diffed against `expected/m4.3.txt`:
|
||||
1. `a1_draft_not_loadable` — enumerate skills; assert the draft name is absent.
|
||||
2. `a2_promoted_is_loadable` — after `git mv`, assert present. Proves assertion 1
|
||||
is not passing because the loader is broken.
|
||||
3. `a3_verbatim_excluded` — ingest, assert no L0 node matches.
|
||||
4. `a4_reformatted_excluded` — same.
|
||||
5. `a5_mention_kept` — the referencing record survives as evidence.
|
||||
6. `a6_verify_clean` — `mem verify --derived-filter` exits 0.
|
||||
7. `a7_exclusions_auditable` — assert `derived_excluded` events name the artifact.
|
||||
|
||||
**Command:** `bash verify/m4.3.sh | diff - expected/m4.3.txt`
|
||||
|
||||
**False pass:**
|
||||
- Asserting the draft is absent without assertion 2. If the loader silently fails
|
||||
to enumerate anything, assertion 1 passes trivially and the guarantee is
|
||||
untested.
|
||||
- Testing exclusion only on the draft. The draft is not the risk — a *promoted*
|
||||
skill is the one that actually reaches sessions, and it is the one the filter
|
||||
must catch.
|
||||
|
||||
## Traps
|
||||
|
||||
- Automating promotion inside the gate script and leaving it there. The gate
|
||||
needs to promote something to test the promoted path; make it obvious that the
|
||||
script is the only place it happens, and that production has no such path.
|
||||
- Reading a passing gate as "the cycle is impossible". It means the two guards
|
||||
hold today. Adding a new emitted artifact type without adding it to the
|
||||
manifest reopens the loop, and only assertion 6 will notice.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
|
||||
@@ -0,0 +1,108 @@
|
||||
# M5.1 — `mem label` — evidence labeler
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Produce the per-chunk ground truth `U_t` that `r_update` needs, since this corpus
|
||||
does not come with evidence labels.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Paper `r_update`:
|
||||
|
||||
```
|
||||
r_update_t = +1 if U_t is correct
|
||||
-1 if U_t is incorrect
|
||||
```
|
||||
|
||||
"Correct" means: for chunks containing evidence for `Q`, the agent should emit
|
||||
`<check>yes</check>`; for chunks without, `<check>no</check>`. That requires
|
||||
knowing which chunks contain evidence.
|
||||
|
||||
The paper had it for free — synthetic NIAH tasks place the needle deliberately,
|
||||
and HotpotQA ships supporting facts. **We have neither.** Agent transcripts have
|
||||
no annotation of which turn contained the answer.
|
||||
|
||||
Cheapest honest substitute: **distant supervision from the 32B model.** Ask
|
||||
`reasoning` (DeepSeek-R1-Distill-Qwen-32B, vLLM) per `(question, chunk)` whether
|
||||
the chunk contains evidence. It is ~10× the controller's size and sees each chunk
|
||||
independently, without the memory state that might bias the 3B's decision.
|
||||
|
||||
This inherits the labeler's bias, which is why M5.2 exists and must run before
|
||||
anyone trains on these labels.
|
||||
|
||||
Constraint: `reasoning` has a **16384 total context** and vLLM rejects
|
||||
`input + max_tokens > 16384`. A 5000-token chunk plus question plus instructions
|
||||
fits with room; keep `max_tokens` small (labels are one token of signal) and do
|
||||
not batch chunks into one prompt.
|
||||
|
||||
The labeler emits a binary label plus a short justification. Keep the
|
||||
justification — it is what makes M5.2's disagreement analysis possible.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem label --project P --query Q` reads chunks from the log.
|
||||
2. Per chunk, prompt `reasoning`: question, chunk, "does this contain evidence for
|
||||
the question? Answer yes or no, then one sentence why."
|
||||
3. Send **no tools** — the reasoning route rejects any request carrying them.
|
||||
4. Write `label/<project>/<query-id>.jsonl`:
|
||||
`{"chunk_sha":"...","t":7,"label":true,"why":"...","model":"reasoning","ts":"..."}`.
|
||||
5. Resumable: skip chunks already labelled.
|
||||
6. Report the label rate — the fraction of chunks the labeler calls evidence.
|
||||
Compare it to the controller's update-rate from M1.7; a large gap is the
|
||||
finding, not a bug.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Every chunk in the log gets exactly one label.
|
||||
- Labels key on `chunk_sha`, so they survive re-chunking only if content is
|
||||
unchanged.
|
||||
- Resume skips completed work.
|
||||
- Label rate is reported alongside the controller's update-rate.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** scripted client offline; one `#[ignore]` live run.
|
||||
|
||||
**Integration test** — `tests/it_label.rs`:
|
||||
1. `a1_one_label_per_chunk` — no duplicates, no gaps against the log's chunks.
|
||||
2. `a2_keyed_by_sha` — labels reference `chunk_sha`, not `t`, so reordering the
|
||||
log does not corrupt them.
|
||||
3. `a3_no_tools_sent` — assert the request body has no `tools` key.
|
||||
4. `a4_context_budget` — assert every labeling prompt is under
|
||||
16384 − max_tokens.
|
||||
5. `a5_resume` — label, rerun, assert zero new calls.
|
||||
6. `a6_justification_kept` — every label has non-empty `why`.
|
||||
7. `a7_rate_reported` — the summary prints both label rate and the controller's
|
||||
update-rate.
|
||||
8. `a8_live` — `#[ignore]`; 20 real chunks through `reasoning`; print the labels
|
||||
and justifications for a human to sanity-check.
|
||||
|
||||
**Command:** `cargo test -p mem-cli label` (add `-- --ignored` for a8)
|
||||
|
||||
**False pass:**
|
||||
- Keying labels by `t`. A re-chunk shifts every `t`, the labels silently
|
||||
misalign, and the training set is quietly wrong in a way nothing downstream can
|
||||
detect.
|
||||
- Dropping the justification to save space. M5.2 then has nothing to analyse and
|
||||
the calibration step degenerates into a single agreement number with no way to
|
||||
understand it.
|
||||
|
||||
## Traps
|
||||
|
||||
- Batching several chunks into one labeling prompt to save calls. The labels
|
||||
become order-dependent and the 16K context is exceeded on the third chunk.
|
||||
- Treating the 32B's labels as ground truth. They are a *proxy*, and M5.2 is the
|
||||
task that measures how good a proxy.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.1
|
||||
@@ -0,0 +1,100 @@
|
||||
# M5.2 — Labeler calibration
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Measure how good the proxy is, before training a policy to imitate it.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
M5.1's labels are distant supervision. Training on them without measuring
|
||||
agreement means the policy learns the 32B model's bias and reports it as
|
||||
improvement — and under a single-model deployment there is no competing variant
|
||||
whose divergence would make that visible.
|
||||
|
||||
Method: hand-label a stratified holdout, compare, and report more than accuracy.
|
||||
|
||||
- **Sample size**: 100 chunks is enough to distinguish 0.7 from 0.9 agreement.
|
||||
- **Stratify** by the labeler's own answer — 50 it called evidence, 50 it did
|
||||
not. Random sampling from a corpus where ~5% is evidence gives ~5 positives,
|
||||
and the positive class is the one that matters.
|
||||
- **Report Cohen's κ, not raw agreement.** With a 95/5 class balance, a labeler
|
||||
that always says "no" scores 95% agreement and is useless. κ corrects for
|
||||
chance.
|
||||
- Also report precision and recall on the positive class separately. They fail
|
||||
differently: low recall silently starves memory, low precision pollutes it.
|
||||
|
||||
Disagreements are the artifact. Read them; they are usually either a genuinely
|
||||
ambiguous chunk or a question that is too vague — and the second is fixable at
|
||||
M1.2 and worth much more than a better labeler.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem label sample --project P --n 100 --stratified` writes a blind
|
||||
worksheet — chunk text and question, **no** labeler answer visible.
|
||||
2. Hand-label it. Record the human labels separately.
|
||||
3. `mem label calibrate` joins them; reports agreement, κ, precision, recall,
|
||||
and the confusion matrix.
|
||||
4. Dump all disagreements with both justifications side by side.
|
||||
5. Commit the holdout and the human labels; they are reusable for every future
|
||||
labeler change.
|
||||
6. Gate: κ ≥ 0.6 before the labels are used for training.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Worksheet hides the labeler's answer.
|
||||
- κ, precision, recall and the confusion matrix are all reported.
|
||||
- Disagreements are dumped with justifications.
|
||||
- The holdout is committed and reusable.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** a synthetic labeled set with known agreement, so the statistics
|
||||
themselves are testable.
|
||||
|
||||
**Integration test** — `tests/it_calibration.rs`:
|
||||
1. `a1_worksheet_is_blind` — assert the labeler's answer appears nowhere in the
|
||||
output file.
|
||||
2. `a2_stratified` — assert the sample is ~50/50 by labeler answer, not corpus
|
||||
proportional.
|
||||
3. `a3_kappa_correct` — feed a set with hand-computed κ; assert the reported
|
||||
value matches to 3 decimals.
|
||||
4. `a4_kappa_vs_accuracy` — a synthetic all-negative labeler on a 95/5 set:
|
||||
assert accuracy > 0.9 **and** κ ≈ 0. This is the assertion that justifies
|
||||
reporting κ at all.
|
||||
5. `a5_confusion_matrix` — all four cells match hand counts.
|
||||
6. `a6_disagreements_dumped` — count equals off-diagonal total; each carries both
|
||||
justifications.
|
||||
7. `a7_holdout_stable` — rerunning the sampler with the same seed reproduces the
|
||||
same chunks.
|
||||
|
||||
**Command:** `cargo test -p mem-cli calibration`
|
||||
|
||||
**False pass:**
|
||||
- Reporting accuracy only. On this class balance it is nearly meaningless, and it
|
||||
will look excellent right up until the trained policy learns to always answer
|
||||
"no".
|
||||
- Hand-labeling with the labeler's answer visible. Anchoring makes agreement look
|
||||
high and the whole exercise decorative — assertion 1 is a real safeguard, not
|
||||
hygiene.
|
||||
|
||||
## Traps
|
||||
|
||||
- Sampling proportionally. A 5%-positive corpus yields five positives in a
|
||||
hundred, and precision on the positive class — the number that decides whether
|
||||
memory gets polluted — is estimated from five examples.
|
||||
- Treating low κ as "the labeler needs a better prompt". Check the *questions*
|
||||
first: an ambiguous standing question makes evidence genuinely undecidable, and
|
||||
no labeler can fix that.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Risks
|
||||
@@ -0,0 +1,114 @@
|
||||
# M5.3 — Training corpus export
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn the log plus labels into trajectories verl can train on — the boundary
|
||||
between the Rust side and the Python side.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**The JSONL log is the boundary.** Rust produces it; Python consumes it. Nothing
|
||||
else crosses, which is what keeps the two halves independent.
|
||||
|
||||
A training example is a **trajectory**, not a turn: the paper's advantage mixes a
|
||||
trajectory-level term with a turn-level one (`Â = α·Â_traj + (1−α)·Â_turn`,
|
||||
α=0.9), so turns must stay grouped by run.
|
||||
|
||||
Per turn, verl needs: the exact prompt sent, the exact response, and the rewards.
|
||||
|
||||
```jsonl
|
||||
{"trajectory_id":"01HXYZ","turns":[
|
||||
{"t":1,"prompt":"<full assembled prompt>","response":"<think>...</next>",
|
||||
"r_update":-1,"parsed":true},
|
||||
...],
|
||||
"r_exit":-0.5,"r_format":1,"r_outcome":null}
|
||||
```
|
||||
|
||||
Reward assembly, from the paper:
|
||||
|
||||
- `r_update_t` = +1 if the recorded `U_t` matches M5.1's label, −1 otherwise.
|
||||
- `r_exit` — one value per trajectory: `0` if `t_exit == t_last_evidence`,
|
||||
`−0.75` if earlier, `−0.5` if later. `t_last_evidence` is the largest `t` whose
|
||||
label is true. **Note L1 runs never exit** (exit gate off), so every L1
|
||||
trajectory is a "late" exit at −0.5 unless the exit signal is taken from the
|
||||
recorded `E_t` rather than the loop's behaviour — take it from the record.
|
||||
- `r_format` = 1 only if **every** turn in the trajectory parsed, 0 otherwise.
|
||||
Strict, because a malformed turn may be caused by the previous one.
|
||||
- `r_outcome` is null. We have no answer-correctness signal; the paper's
|
||||
`is_equiv(A, Â)` has no analogue in extraction. Say so explicitly rather than
|
||||
fabricating one.
|
||||
|
||||
The prompt must be the **exact bytes sent**, not re-assembled. Re-assembly drifts
|
||||
from what the model actually saw the moment M1.3's template changes.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem export --project P --format verl --out corpus/`.
|
||||
2. Join log turns to labels by `chunk_sha`.
|
||||
3. Reconstruct each turn's prompt from the recorded request if `MEM_LLM_RECORD`
|
||||
captured it; otherwise fail loudly rather than re-assembling.
|
||||
4. Compute rewards as above; carry `r_outcome: null` through.
|
||||
5. Group by run into trajectories, ordered by `t`.
|
||||
6. Emit a summary: trajectories, turns, positive/negative `r_update` split,
|
||||
`r_format` pass rate, `t_last_evidence` distribution.
|
||||
7. Refuse to export if M5.2's κ is below the threshold or absent.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Turns are grouped into trajectories, ordered.
|
||||
- Prompts are byte-exact recordings, never re-assembled.
|
||||
- `r_format` is 0 for a trajectory with any unparsed turn.
|
||||
- Export is refused without calibration.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** log fixture with known labels and a deliberately unparsed turn.
|
||||
|
||||
**Integration test** — `tests/it_export.rs`:
|
||||
1. `a1_trajectory_grouping` — turns grouped by run, `t` ascending, none lost.
|
||||
2. `a2_r_update_signs` — matching label → +1, mismatching → −1, checked per turn.
|
||||
3. `a3_r_format_strict` — a trajectory with one unparsed turn scores 0 overall,
|
||||
not per turn.
|
||||
4. `a4_r_exit_from_record` — assert `r_exit` derives from the recorded `E_t`, not
|
||||
from whether the loop stopped. A fixture where the gate said `end` at t=5 but
|
||||
the loop continued must score as an exit at 5.
|
||||
5. `a5_prompt_is_recorded_bytes` — assert the exported prompt equals the recorded
|
||||
request body; corrupt the recording and assert export fails rather than
|
||||
silently re-assembling.
|
||||
6. `a6_r_outcome_null` — assert the field is present and null, not omitted and not
|
||||
zero.
|
||||
7. `a7_refuses_without_calibration` — no κ file → non-zero exit naming M5.2.
|
||||
8. `a8_summary_counts` — reported splits match a hand count on the fixture.
|
||||
|
||||
**Command:** `cargo test -p mem-cli export`
|
||||
|
||||
**False pass:**
|
||||
- Re-assembling prompts at export time. Every assertion except 5 passes, and the
|
||||
policy is trained on prompts the model never saw — which shows up as a training
|
||||
run that will not converge, with no obvious cause.
|
||||
- Applying `r_format` per turn. It looks more granular and it is wrong: the paper
|
||||
is explicit that the strictness exists because a bad turn may be caused by the
|
||||
previous one.
|
||||
- Emitting `r_outcome: 0` instead of null. Zero is a real reward value and the
|
||||
trainer will use it as signal.
|
||||
|
||||
## Traps
|
||||
|
||||
- Deriving `r_exit` from loop behaviour at L1. The gate is switched off there, so
|
||||
every trajectory scores as a late exit and the exit signal becomes constant
|
||||
noise the policy cannot learn from.
|
||||
- Joining labels by `t`. Same failure as M5.1's trap — a re-chunk misaligns
|
||||
everything silently.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2
|
||||
@@ -0,0 +1,102 @@
|
||||
# M5.4 — vLLM + `--enable-lora`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | L — 3+ days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | homelab |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
|
||||
## Goal
|
||||
|
||||
A serving path that can load the memory adapter — because the current one cannot.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Ollama cannot hot-swap LoRA adapters.** The controller runs on
|
||||
`qwen2.5:3b-instruct` served by Ollama today, which is fine for prompted-only use
|
||||
and a dead end for post-training. vLLM supports `--enable-lora` with
|
||||
`--lora-modules name=path`, serving base plus adapters from one resident model.
|
||||
|
||||
The pattern already exists in this cluster: the `reasoning` predictor is
|
||||
`vllm/vllm-openai:v0.11.0` under KServe, adopted into ArgoCD as
|
||||
`k8s/apps/llm-serving/reasoning.yaml`. Copy its shape.
|
||||
|
||||
**VRAM is the constraint that makes an adapter the right answer.** One GPU,
|
||||
`OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A
|
||||
separate full memory model evicts something, and eviction is a weights reload
|
||||
measured in tens of seconds — `ornith`'s cold start already blew a 60s gateway
|
||||
timeout once. A LoRA rides on a resident base for near-zero extra VRAM.
|
||||
|
||||
Two hard-won operational facts to carry over:
|
||||
|
||||
1. **Kong reads timeouts from the Service, not the Ingress.** `konghq.com/read-timeout`
|
||||
on an Ingress is ignored; it must reach the predictor Service, and KServe
|
||||
propagates InferenceService annotations there. Getting this wrong produces a
|
||||
504 at exactly 60s on the first cold request.
|
||||
2. **Readiness must mean "can serve", not "process is up."** vLLM's startup probe
|
||||
needs a long `failureThreshold` — model load plus torch compile measured 108s
|
||||
+ 55s on the 32B. Report ready too early and the first request 504s.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `k8s/apps/llm-serving/memory.yaml` — InferenceService, vLLM, Qwen2.5-3B-Instruct.
|
||||
2. Args: `--enable-lora`, `--max-lora-rank 32`, `--max-model-len 32768`,
|
||||
`--served-model-name memory`.
|
||||
3. Adapter storage: a PVC or an initContainer fetching from object storage;
|
||||
`--lora-modules memory-v1=/mnt/adapters/memory-v1`.
|
||||
4. Kong timeout annotations on the **InferenceService** metadata so KServe
|
||||
propagates them to the Service.
|
||||
5. Startup probe with a generous `failureThreshold`, gated on the OpenAI
|
||||
`/health` endpoint.
|
||||
6. New Kong route `/v1/memory/chat/completions` in `llm-routes.yaml`, with the
|
||||
`model-key-auth` plugin — in namespace `llm-serving`, since a KongPlugin
|
||||
reference resolves in the annotated object's own namespace and a dangling one
|
||||
fails open.
|
||||
7. Commit, push, let ArgoCD sync. No `kubectl apply`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Base model answers through `/v1/memory/chat/completions`.
|
||||
- A named adapter is selectable via the `model` field.
|
||||
- Unauthenticated requests to the new route return 401.
|
||||
- Cold start does not 504.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `kubectl` and `curl` against the live gateway after sync.
|
||||
|
||||
**Integration test** — `verify/m5.4.sh` diffed against `expected/m5.4.txt`:
|
||||
1. `a1_isvc_ready` — InferenceService reports Ready.
|
||||
2. `a2_base_completion` — a completion naming the base model returns 200.
|
||||
3. `a3_adapter_selectable` — with a dummy adapter mounted, request `model:
|
||||
memory-v1`; assert 200 and that it differs from the base response.
|
||||
4. `a4_auth_enforced` — no key → 401; `apikey` header → 200.
|
||||
5. `a5_cold_start_no_504` — delete the pod, wait for Ready, immediately send a
|
||||
request; assert 200, not 504. This is the regression test for the timeout bug.
|
||||
6. `a6_timeouts_on_service` — assert `konghq.com/read-timeout` is present on the
|
||||
predictor **Service**, not only the Ingress.
|
||||
7. `a7_vram_headroom` — with the memory model resident alongside the others,
|
||||
assert `nvidia-smi` free memory stays above a threshold.
|
||||
|
||||
**Command:** `bash verify/m5.4.sh | diff - expected/m5.4.txt`
|
||||
|
||||
**False pass:**
|
||||
- Testing after the model is warm. The 504 bug only appears on the first request
|
||||
after a restart, which is exactly the case assertion 5 forces.
|
||||
- Checking the timeout annotation on the Ingress. It is ignored there, and its
|
||||
presence is what makes the bug hard to see.
|
||||
|
||||
## Traps
|
||||
|
||||
- Adding a third resident model without checking VRAM. Something gets evicted,
|
||||
and the symptom is a slow unrelated model rather than an obvious failure.
|
||||
- Putting the KongPlugin in the wrong namespace. It fails **open** — the route
|
||||
serves unauthenticated and looks healthy, which is how the model routes ran
|
||||
with no auth for a while.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Separate weights · `k8s/apps/llm-serving/`
|
||||
@@ -0,0 +1,120 @@
|
||||
# M5.5 — verl training loop
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | L — 3+ days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M5.3, M5.4 |
|
||||
|
||||
## Goal
|
||||
|
||||
Train the LoRA that makes the update and exit gates better than prompting alone.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Python, using verl (`github.com/volcengine/verl`), which is what the paper used.
|
||||
Lives outside the Rust workspace; the JSONL corpus is the only interface.
|
||||
|
||||
Paper hyperparameters, Table 3 — start here rather than guessing:
|
||||
|
||||
```
|
||||
chunk size 5000
|
||||
max prompt length 8192
|
||||
max response length 2048
|
||||
clip ratio 0.20
|
||||
learning rate 1e-6
|
||||
temperature (train) 1.0 top_p 1.0
|
||||
temperature (val) 1.0 top_p 0.7
|
||||
train batch size 128
|
||||
rollout N 16
|
||||
mini batch size 128
|
||||
LR warmup steps 20
|
||||
```
|
||||
|
||||
Advantage, paper §3.2.2 — the part most likely to be implemented wrong:
|
||||
|
||||
```
|
||||
Â_traj = r_traj_g − mean over the GROUP of trajectories
|
||||
Â_turn = r_update_{g,t} − mean over turns AT STEP t across groups
|
||||
 = α·Â_traj + (1−α)·Â_turn α = 0.9
|
||||
```
|
||||
|
||||
Two distinct baselines. `Â_turn` is normalised across groups **at the same `t`**,
|
||||
and the group size at step `t` can differ from the trajectory group size, because
|
||||
trajectories that exited early have fewer turns.
|
||||
|
||||
α=0.9 is the paper's default and the ablation (Figure 8) explains why: at α=1
|
||||
there is no update-gate reward and accuracy on evidence-free chunks collapses —
|
||||
the model updates indiscriminately, which is exactly the failure this whole
|
||||
system exists to avoid.
|
||||
|
||||
Expect instability. The paper's own limitations section says the extra rewards
|
||||
"reduce training stability, requiring a smaller off-policy degree and longer
|
||||
convergence time."
|
||||
|
||||
## Steps
|
||||
|
||||
1. `training/` directory, Python, `uv`-managed. System Python is 3.9.6; this
|
||||
needs 3.11+.
|
||||
2. Corpus loader for M5.3's format.
|
||||
3. Configure verl for LoRA on Qwen2.5-3B-Instruct, rank 16–32.
|
||||
4. Implement the three rewards and the two-baseline advantage, α configurable.
|
||||
5. Log per step: update accuracy split by evidence-present and evidence-free,
|
||||
exact-exit ratio, format correctness, mean response length, validation reward.
|
||||
6. Hold out a validation split by **project**, not by trajectory — same-project
|
||||
trajectories share vocabulary and leak.
|
||||
7. Export the adapter, version it, publish where M5.4 can mount it.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Training runs to convergence on the validation reward.
|
||||
- Both advantage terms are computed with their own baselines.
|
||||
- Update accuracy on evidence-free chunks does not collapse.
|
||||
- The adapter loads in M5.4's server.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** pytest for reward and advantage maths; a short training run for the
|
||||
loop itself.
|
||||
|
||||
**Integration test** — `training/tests/test_rewards.py`:
|
||||
1. `a1_r_update_signs` — matching label +1, mismatching −1.
|
||||
2. `a2_r_exit_bands` — exact 0, early −0.75, late −0.5.
|
||||
3. `a3_r_format_strict` — any unparsed turn zeroes the whole trajectory.
|
||||
4. `a4_traj_baseline` — `Â_traj` uses the group mean; hand-computed fixture.
|
||||
5. `a5_turn_baseline_at_step_t` — `Â_turn` normalises across groups at the same
|
||||
`t`; a fixture with unequal trajectory lengths must not misalign. This is the
|
||||
assertion that catches the most likely implementation error.
|
||||
6. `a6_alpha_mix` — α=1 yields pure trajectory advantage; α=0 pure turn.
|
||||
7. `a7_alpha_1_degenerates` — train 50 steps at α=1 on a fixture; assert
|
||||
evidence-free accuracy drops relative to α=0.9, reproducing the paper's
|
||||
Figure 8b.
|
||||
8. `a8_validation_split_by_project` — assert no project appears in both splits.
|
||||
9. `a9_adapter_loads` — export, mount in M5.4, assert a completion returns 200.
|
||||
|
||||
**Command:** `uv run pytest training/tests -v`
|
||||
|
||||
**False pass:**
|
||||
- Normalising `Â_turn` over the whole batch rather than per step `t`. It trains,
|
||||
loss goes down, and the turn-level signal is diluted into noise — assertion 5
|
||||
is the only thing that catches it.
|
||||
- Splitting validation by trajectory. Same-project trajectories share phrasing
|
||||
and file paths, so validation reward looks excellent and generalisation is
|
||||
untested.
|
||||
- Skipping assertion 7 as "too slow". It is the only end-to-end evidence that the
|
||||
update reward is wired to anything.
|
||||
|
||||
## Traps
|
||||
|
||||
- Tuning α before the rewards are verified. Every α is wrong if `r_update`'s sign
|
||||
is flipped, and the symptom looks identical.
|
||||
- Training on labels whose κ was never measured. The policy learns the labeler,
|
||||
and there is no held-out signal that would reveal it — M5.2 exists for this and
|
||||
M5.3 refuses to export without it.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.2, Table 3, Fig 8
|
||||
@@ -0,0 +1,106 @@
|
||||
# M5.6 — M5 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M5 — Post-training |
|
||||
| Size | L — 3+ days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Establish that the trained adapter is actually better than prompting — and that
|
||||
"better" was measured on data it never saw.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The comparison is **adapter vs. the prompted baseline**, on a held-out project,
|
||||
using the same corpus and the same prompt.
|
||||
|
||||
The paper's own result sets the expectation: Figure 9 shows RL helps but the
|
||||
prompted workflow already works, and the gains concentrate on harder tasks. So
|
||||
a modest improvement is the realistic success case; a dramatic one is a reason to
|
||||
check for leakage first.
|
||||
|
||||
Metrics, and why each is present:
|
||||
|
||||
| metric | why |
|
||||
|---|---|
|
||||
| update accuracy, evidence-present | recall — does it catch evidence |
|
||||
| update accuracy, evidence-free | precision — does it resist noise. **This is the one that collapses at α=1** |
|
||||
| exact-exit ratio | did the exit gate learn anything |
|
||||
| format correctness | the parser's job got easier or harder |
|
||||
| memory token curve | the Figure 6 shape — flat, or climbing to the cap |
|
||||
| wall clock per run | the paper claims up to 400% speedup with the exit gate |
|
||||
|
||||
**Held-out project is non-negotiable.** Same-project trajectories share file
|
||||
paths, error strings and phrasing; measuring on them is measuring memorisation.
|
||||
|
||||
Report both, always: an adapter that improves evidence-present accuracy while
|
||||
degrading evidence-free is worse for this system, because polluted memory
|
||||
degrades every subsequent turn.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Pick a project excluded from training. Ingest it with the prompted baseline;
|
||||
record all metrics.
|
||||
2. Ingest the same project with the adapter, identical prompt and chunking.
|
||||
3. Compare on the M5.2 holdout labels.
|
||||
4. Assert the thresholds below.
|
||||
5. Plot the memory-token curve for both; commit it as the Figure 6 analogue.
|
||||
6. Commit `expected/m5.6.txt`; diff.
|
||||
7. If the adapter loses, keep it versioned and record why — a negative result with
|
||||
the reason is worth more than a rerun with different hyperparameters.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Evidence-present accuracy ≥ baseline.
|
||||
- Evidence-free accuracy ≥ baseline. Never traded away.
|
||||
- Format correctness ≥ baseline.
|
||||
- Memory token curve flat, not climbing to the cap.
|
||||
- Measured on a project absent from training.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** live gateway with both models; the held-out project; M5.2's labels.
|
||||
|
||||
**Integration test** — `verify/m5.6.sh` diffed against `expected/m5.6.txt`:
|
||||
1. `a1_holdout_is_unseen` — assert the eval project appears in no training
|
||||
trajectory. Check the corpus, not the config.
|
||||
2. `a2_evidence_present_accuracy` — adapter ≥ baseline; print both.
|
||||
3. `a3_evidence_free_accuracy` — adapter ≥ baseline; print both.
|
||||
4. `a4_format_correctness` — adapter ≥ baseline.
|
||||
5. `a5_memory_curve_flat` — slope below the M1.8 bound for both; assert the
|
||||
adapter is no worse.
|
||||
6. `a6_exit_ratio_reported` — print exact/early/late exit ratios. Advisory: the
|
||||
exit gate is off at L1, so this measures the signal, not behaviour.
|
||||
7. `a7_same_prompt` — assert both runs used a byte-identical prompt template.
|
||||
8. `a8_wall_clock` — report both; no threshold, since the exit gate is off at L1.
|
||||
|
||||
**Command:** `bash verify/m5.6.sh | diff - expected/m5.6.txt`
|
||||
|
||||
**False pass:**
|
||||
- Comparing on a project that was in training. Everything improves and none of it
|
||||
generalises — assertion 1 checks the corpus rather than trusting the split
|
||||
config.
|
||||
- Reporting a single combined accuracy. It hides the precision/recall trade that
|
||||
matters most: a model that says "yes" more often scores better on
|
||||
evidence-present and pollutes memory, and the combined number can improve while
|
||||
the system gets worse.
|
||||
- Different prompts between runs. Any measured difference then attributes to the
|
||||
adapter and is partly the prompt — assertion 7 is cheap and removes the doubt.
|
||||
|
||||
## Traps
|
||||
|
||||
- Retraining until the gate passes without changing anything principled. That is
|
||||
fitting the gate, and the held-out project stops being held out the third time
|
||||
you look at it.
|
||||
- Reading a small improvement as failure. The paper's own baseline works; the
|
||||
adapter's value here is as much about stability on evidence-free chunks as
|
||||
headline accuracy.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Verification, Risks · paper Fig 9
|
||||
@@ -0,0 +1,122 @@
|
||||
# M6.1 — CNPG `agent-manager-db` manifest
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | homelab |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
|
||||
## Goal
|
||||
|
||||
A dedicated Postgres for agent-manager's session store, provisioned the way
|
||||
everything else in the cluster is: through git, with no manual `psql`. Same
|
||||
pattern this project already used for `memory-db` (M2.2), applied to a
|
||||
different, unrelated app.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**agent-manager is a separate repo**, not part of this Rust workspace:
|
||||
`github.com/Riotpiaole/agent-manager` (fork of `github.com/YoanWai/
|
||||
agent-manager`), local checkout `~/workplace/agent-manager`, branch
|
||||
`add-headless-spawn`. Its session store is `internal/store/store.go` — see
|
||||
M6.2/M6.3 for the schema and query port.
|
||||
|
||||
**Existing CNPG pattern, verified on the live cluster: 3 Clusters today, one
|
||||
per app, each ClusterIP-only (not LAN-reachable):**
|
||||
|
||||
| namespace/name | app |
|
||||
|---|---|
|
||||
| `cicd/forgejo-db` | Forgejo |
|
||||
| `iam/authentik-db` | Authentik |
|
||||
| `temporal/temporal-db` | Temporal |
|
||||
|
||||
**No shared/multi-tenant DB cluster** — every app gets its own dedicated
|
||||
CNPG `Cluster`. `agent-manager-db` follows the same rule; it does not join
|
||||
`memory-db` (M2.2's cluster) even though both are new Postgres instances
|
||||
touched by the same person around the same time. Different app, different
|
||||
cluster.
|
||||
|
||||
Follow `k8s/infra/databases/temporal-db.yaml` exactly, same as M2.2 did:
|
||||
`imageName` pinned, `enableSuperuserAccess: false`, `storageClass:
|
||||
longhorn-cnpg`, `enablePodMonitor: true`, control-plane tolerations,
|
||||
`podAntiAffinityType: preferred`.
|
||||
|
||||
**Instance count — open question, default to convention.** Motivation for
|
||||
this whole migration is durability-of-location, not HA (single-machine
|
||||
usage, not a multi-host shared-session requirement). The 3 existing
|
||||
clusters are all 3-instance. Default to 3 instances for consistency with
|
||||
every other app in the cluster rather than special-casing this one to 1;
|
||||
revisit only if resource pressure on the homelab nodes makes it a real
|
||||
tradeoff.
|
||||
|
||||
Storage: session rows are tiny (`sessions`, `groups`, `settings`,
|
||||
`review_*` — no blobs beyond a `snapshot TEXT` pane capture per session).
|
||||
1Gi is generous; no need for `memory-db`'s 10Gi (that one holds
|
||||
768-dim vectors).
|
||||
|
||||
## Steps
|
||||
|
||||
1. `k8s/infra/databases/agent-manager-db.yaml` — `Cluster` + `Database`,
|
||||
namespace `agent-manager`, no extensions (plain relational, no
|
||||
pgvector).
|
||||
2. Namespace `agent-manager`, created by the ArgoCD app that owns it.
|
||||
3. Add to the owning kustomization's explicit resource list — an unlisted
|
||||
file is silently dropped with no error and no drift shown (the M2.2
|
||||
task file names this exact trap).
|
||||
4. Commit, push to **both** Forgejo origin and the GitHub mirror — verify
|
||||
which `repoURL` the eventual ArgoCD `Application` for this app actually
|
||||
watches before assuming either push is the one that matters (`kong`
|
||||
app, for example, tracks the GitHub mirror specifically, not Forgejo).
|
||||
5. Let ArgoCD sync. **No `kubectl apply`.**
|
||||
6. Verify the app user can create tables (schema arrives in M6.2, but a
|
||||
throwaway `CREATE TABLE t(id text); DROP TABLE t;` proves connectivity
|
||||
here).
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `Cluster` reaches `Cluster in healthy state`.
|
||||
- ArgoCD shows the app `Synced/Healthy`.
|
||||
- No manual `psql` was run to get there.
|
||||
- Service is ClusterIP-only — not reachable from the LAN directly (M6.4's
|
||||
nginx route is the only path in).
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `kubectl` and `psql` read-only checks after sync.
|
||||
|
||||
**Integration test** — `verify/m6.1.sh`, output diffed against
|
||||
`expected/m6.1.txt`:
|
||||
1. `a1_cluster_healthy` — `kubectl get cluster -n agent-manager
|
||||
agent-manager-db` reports all instances ready.
|
||||
2. `a2_clusterip_only` — `kubectl get svc -n agent-manager -o
|
||||
jsonpath='{.items[*].spec.type}'` contains no `LoadBalancer` or
|
||||
`NodePort`.
|
||||
3. `a3_argocd_synced` — the owning app is `Synced/Healthy`.
|
||||
4. `a4_app_user_can_ddl` — as `app`, `CREATE TABLE t(id text); DROP TABLE
|
||||
t;` succeeds.
|
||||
5. `a5_no_lan_route_yet` — connection attempt from outside the cluster
|
||||
network fails at this point in the plan (M6.4 hasn't landed).
|
||||
|
||||
**Command:** `bash verify/m6.1.sh | diff - expected/m6.1.txt`
|
||||
|
||||
**False pass:**
|
||||
- Confirming sync without checking service type. A `Cluster` can be
|
||||
healthy and `Synced` while someone fat-fingered a `LoadBalancer` type
|
||||
into the manifest, silently violating the "dedicated ingress, not raw
|
||||
LAN IP" network-path decision this whole migration made. Assertion 2 is
|
||||
the guard.
|
||||
|
||||
## Traps
|
||||
|
||||
- Forgetting the kustomization resource list (same trap M2.2 already
|
||||
named) — file sits in git, ArgoCD reports Synced, objects never exist.
|
||||
- Adding `prune: true` without accounting for CNPG-operator-created
|
||||
children (Services, Secrets, PVCs). M2.2's Traps section already hit
|
||||
this on `llm-serving`; same fix applies here (`prune: false`).
|
||||
|
||||
---
|
||||
|
||||
Background: `k8s/infra/databases/temporal-db.yaml` · [M2.2](M2.2-memory-db-manifest.md) (same pattern, different app)
|
||||
@@ -0,0 +1,243 @@
|
||||
# M6.2 — Postgres schema for agent-manager sessions
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
The sqlite schema agent-manager's `internal/store/store.go` builds up
|
||||
through 14 `ALTER TABLE` migrations, collapsed into one Postgres-native
|
||||
schema — with the sqlite workarounds (integer booleans, dual-encoded
|
||||
timestamps) removed rather than carried over.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Current schema, read directly from `internal/store/store.go` (6 tables, no
|
||||
FK constraints anywhere — every relationship is enforced in Go, not SQL):
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
GROUPS ||--o{ SESSIONS : "group_name (app-level, no FK)"
|
||||
SESSIONS ||--o{ SESSIONS : "parent_id (self-ref, app-level, no FK)"
|
||||
SESSIONS ||--o| REVIEW_TARGETS : "session_id (app-level, no FK)"
|
||||
SESSIONS ||--o{ REVIEW_BASES : "session_id (app-level, no FK)"
|
||||
SESSIONS ||--o| REVIEW_SCOPES : "session_id (app-level, no FK)"
|
||||
SETTINGS {
|
||||
text key PK
|
||||
text value
|
||||
}
|
||||
SESSIONS {
|
||||
text id PK
|
||||
text name
|
||||
text tool
|
||||
text cwd
|
||||
text group_name "app-level FK -> GROUPS.name"
|
||||
text status
|
||||
int archived "bool 0/1 in sqlite"
|
||||
int created_at "unix nanos, dual-encoded in sqlite"
|
||||
int last_status_at "unix nanos"
|
||||
text agent_session_id
|
||||
text pending_inputs "JSON array blob"
|
||||
int pending_claimed "bool 0/1"
|
||||
text launch_prompt
|
||||
int sort_order
|
||||
int acked "bool 0/1"
|
||||
text snapshot
|
||||
text worktree_repo
|
||||
text worktree_branch
|
||||
int agent_launched_at "unix nanos"
|
||||
text retired_agent_session_id
|
||||
text parent_id "app-level FK -> SESSIONS.id, self"
|
||||
}
|
||||
GROUPS {
|
||||
text name PK
|
||||
int sort_order
|
||||
text path
|
||||
int archived "bool 0/1"
|
||||
text worktree
|
||||
}
|
||||
REVIEW_TARGETS {
|
||||
text session_id PK
|
||||
text repo_root
|
||||
}
|
||||
REVIEW_BASES {
|
||||
text session_id PK
|
||||
text repo_root PK
|
||||
text base_ref
|
||||
}
|
||||
REVIEW_SCOPES {
|
||||
text session_id PK
|
||||
text scope
|
||||
}
|
||||
```
|
||||
|
||||
Target Postgres DDL:
|
||||
|
||||
```sql
|
||||
CREATE TABLE groups (
|
||||
name TEXT PRIMARY KEY,
|
||||
sort_order INTEGER NOT NULL DEFAULT 0,
|
||||
path TEXT NOT NULL DEFAULT '',
|
||||
archived BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
worktree TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
CREATE TABLE sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
tool TEXT NOT NULL,
|
||||
cwd TEXT NOT NULL,
|
||||
group_name TEXT NOT NULL REFERENCES groups(name),
|
||||
status TEXT NOT NULL,
|
||||
archived BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
last_status_at TIMESTAMPTZ NOT NULL,
|
||||
agent_session_id TEXT NOT NULL DEFAULT '',
|
||||
pending_inputs JSONB NOT NULL DEFAULT '[]',
|
||||
pending_claimed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
launch_prompt TEXT NOT NULL DEFAULT '',
|
||||
sort_order INTEGER NOT NULL DEFAULT 0,
|
||||
acked BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
snapshot TEXT NOT NULL DEFAULT '',
|
||||
worktree_repo TEXT NOT NULL DEFAULT '',
|
||||
worktree_branch TEXT NOT NULL DEFAULT '',
|
||||
agent_launched_at TIMESTAMPTZ,
|
||||
retired_agent_session_id TEXT NOT NULL DEFAULT '',
|
||||
parent_id TEXT REFERENCES sessions(id)
|
||||
);
|
||||
|
||||
CREATE TABLE settings (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE review_targets (
|
||||
session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
repo_root TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE review_bases (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
repo_root TEXT NOT NULL,
|
||||
base_ref TEXT NOT NULL,
|
||||
PRIMARY KEY (session_id, repo_root)
|
||||
);
|
||||
|
||||
CREATE TABLE review_scopes (
|
||||
session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
scope TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
**Decision: add real FKs, not app-level-only.** Current Go code manually
|
||||
cascades `review_targets`/`review_bases`/`review_scopes` deletes before
|
||||
deleting a `sessions` row (`Delete`, `DeleteChild` in store.go) — `ON
|
||||
DELETE CASCADE` on those three removes that manual bookkeeping entirely.
|
||||
`group_name -> groups(name)` and `parent_id -> sessions(id)` get FKs too,
|
||||
since Postgres makes them free and they catch the exact class of bug
|
||||
`validParent`/`ensureGroup` exist in Go to prevent by hand. This is a
|
||||
behavior change from sqlite (constraint violation now possible on write
|
||||
paths that previously just wrote garbage) — M6.3 must add error handling
|
||||
for FK violations at the ~4 call sites that write `group_name`/
|
||||
`parent_id` without having already validated the reference through
|
||||
`validParent`/`ensureGroup`.
|
||||
|
||||
**Booleans** — `BOOLEAN`, not `INTEGER`. Removes `boolToInt()`/`!= 0` at
|
||||
every read/write site in store.go.
|
||||
|
||||
**Timestamps** — `TIMESTAMPTZ`, not the `encodeTime`/`decodeTime`
|
||||
nanosecond-or-legacy-seconds hack (store.go:1369-1396). That hack exists
|
||||
only because sqlite has no native timestamp type and old rows needed a
|
||||
seconds/nanos disambiguation heuristic (`secondsCeiling`). A fresh
|
||||
Postgres database has no legacy rows — the whole function pair is deleted,
|
||||
not ported. `agent_launched_at` becomes nullable (`NULL` = "never
|
||||
restarted") instead of the sqlite sentinel `0`.
|
||||
|
||||
**`pending_inputs`** — `JSONB`, not `TEXT` holding a JSON string. Same
|
||||
`json.Marshal`/`Unmarshal` round-trip in Go either way, but `JSONB` lets
|
||||
Postgres validate the shape on write instead of accepting malformed JSON
|
||||
that only fails on the next read.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Write `agent-manager` migration files (this repo's `sqlx`-migration
|
||||
convention doesn't apply — agent-manager is a separate Go repo; use
|
||||
whatever migration tool its `add-headless-spawn` branch already has, or
|
||||
a plain `.sql` file run once at `Open()` if it has none — check before
|
||||
introducing a new dependency).
|
||||
2. `CREATE TABLE` in FK-dependency order: `groups`, then `sessions`
|
||||
(references `groups` and itself), then `settings`,
|
||||
`review_targets`/`review_bases`/`review_scopes`.
|
||||
3. No indexes beyond the primary keys are needed yet — `ListSessions`
|
||||
filters on `archived` and orders by `group_name, sort_order,
|
||||
created_at`; add a composite index only if M6.6's gate shows it's slow,
|
||||
not preemptively.
|
||||
4. Confirm `CREATE EXTENSION` is not needed anywhere (plain relational
|
||||
schema, no pgvector) — unlike this project's own `memory-db`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Schema applies to a clean `agent-manager-db` database.
|
||||
- `archived`/`acked`/`pending_claimed` are `BOOLEAN`, not `INTEGER`.
|
||||
- `created_at`/`last_status_at`/`agent_launched_at` are `TIMESTAMPTZ`.
|
||||
- Deleting a `sessions` row cascades `review_targets`/`review_bases`/
|
||||
`review_scopes` without the Go code doing it manually.
|
||||
- Inserting a session with an unknown `group_name` is rejected by the FK,
|
||||
not silently written.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** disposable Postgres, same image as M6.1's `Cluster`
|
||||
(`ghcr.io/cloudnative-pg/postgresql:16.2`) — or a local `postgres:16`
|
||||
container for fast iteration, since agent-manager's own test suite doesn't
|
||||
need the real cluster.
|
||||
|
||||
**Integration test** — extend agent-manager's existing store tests
|
||||
(`internal/store/*_test.go` already has `timeenc_test.go`; that file is
|
||||
deleted in M6.3, its coverage folded into this schema's tests) with:
|
||||
1. `a1_migrate_clean` — apply to an empty database, assert all 6 tables
|
||||
exist.
|
||||
2. `a2_bool_columns_are_boolean` — `information_schema.columns` reports
|
||||
`boolean` for `archived`, `acked`, `pending_claimed`.
|
||||
3. `a3_timestamp_columns_are_timestamptz` — same, for `created_at`,
|
||||
`last_status_at`, `agent_launched_at`.
|
||||
4. `a4_review_cascade` — insert a session + a `review_targets` row for it,
|
||||
delete the session, assert `review_targets` is empty without a separate
|
||||
`DELETE FROM review_targets` call.
|
||||
5. `a5_group_fk_rejects_unknown` — insert a session with a `group_name`
|
||||
that has no matching `groups` row, assert it's rejected.
|
||||
6. `a6_parent_fk_self_ref` — insert two sessions where the second's
|
||||
`parent_id` points at the first, assert it succeeds; point it at a
|
||||
nonexistent id, assert it's rejected.
|
||||
|
||||
**Command:** `go test ./internal/store/... -run TestSchema`
|
||||
|
||||
**False pass:**
|
||||
- Testing the schema against sqlite (leftover `modernc.org/sqlite` import)
|
||||
instead of real Postgres. `BOOLEAN`/`TIMESTAMPTZ`/`JSONB`/FK-cascade
|
||||
behavior all differ or are silently accepted-but-ignored by sqlite —
|
||||
every assertion above becomes meaningless against the wrong engine.
|
||||
|
||||
## Traps
|
||||
|
||||
- Keeping `ON DELETE CASCADE` off `group_name`/`parent_id` FKs by
|
||||
accident (only adding it to the `review_*` ones). A group delete or
|
||||
session delete then hits an FK violation instead of the graceful
|
||||
"session has terminals of its own; move them out first" error
|
||||
`PlaceSession` already gives — check M6.3 preserves that message instead
|
||||
of leaking a raw constraint-violation error to the CLI.
|
||||
- Making `parent_id` `NOT NULL DEFAULT ''` (matching the sqlite default)
|
||||
instead of nullable. `'' REFERENCES sessions(id)` is never satisfiable
|
||||
except by a literal empty-string-id row, which doesn't exist — every
|
||||
top-level session's insert then fails the FK. Must be `NULL` for "no
|
||||
parent."
|
||||
|
||||
---
|
||||
|
||||
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
|
||||
@@ -0,0 +1,160 @@
|
||||
# M6.3 — store.go query port to Postgres
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | L — 3+ days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M6.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Every query in `internal/store/store.go` rewritten against the M6.2 schema,
|
||||
sqlite dropped entirely (decided: full port, not a dual sqlite/Postgres
|
||||
backend — this is single-machine usage, no standalone-without-cluster
|
||||
requirement to preserve).
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Read directly from `internal/store/store.go` (1397 lines) on the
|
||||
`add-headless-spawn` branch:
|
||||
|
||||
- **65 call sites** (`db.Exec`, `db.QueryRow`, `db.Query`, `tx.Exec`,
|
||||
`tx.QueryRow`) use sqlite `?` positional placeholders. Postgres
|
||||
(`pgx` or `lib/pq`) needs `$1, $2, ...` — mechanical but must be done
|
||||
per-statement since arg count varies 1–16 across sites.
|
||||
- **Upserts are already Postgres-compatible.** All 9 upsert sites
|
||||
(`SetReviewRepo`, `SetReviewBase`, `SetReviewScope`, `SetSetting`,
|
||||
`ensureGroup`, `CreateGroup`, `AddGroup`, `createSession`'s group
|
||||
insert, `PlaceSession`'s group insert) already use `ON
|
||||
CONFLICT(...) DO UPDATE SET ... = excluded....` / `ON CONFLICT(...) DO
|
||||
NOTHING`, valid as-is in Postgres. No rewrite beyond the placeholder
|
||||
swap — this corrects an earlier assumption that these were sqlite
|
||||
`INSERT OR REPLACE` and needed real rework.
|
||||
- `boolToInt()`/`archived != 0`-style conversions at ~8 sites go away —
|
||||
scan `int archived`/`int acked`/`int pendingClaimed`/`int` locals in
|
||||
`ListSessions`, `Get` and replace with `bool` directly.
|
||||
- `encodeTime()`/`decodeTime()` calls at every session-timestamp
|
||||
read/write go away — pass `time.Time` straight through; `pgx` handles
|
||||
`TIMESTAMPTZ` natively. Delete `timeenc_test.go` along with the
|
||||
functions it tests (M6.2 already noted its coverage folds into the
|
||||
schema tests instead).
|
||||
- `db.SetMaxOpenConns(1)` + `PRAGMA journal_mode=WAL`
|
||||
(store.go:76-79) gave sqlite serialized writes for free. Some call
|
||||
sites lean on that implicit serialization — most notably
|
||||
`createSession`'s `sort_order = (SELECT COALESCE(MAX(sort_order)+1, 0)
|
||||
FROM sessions WHERE group_name = ? AND parent_id = ?)` subquery, which
|
||||
races under concurrent Postgres writers with a real connection pool.
|
||||
Wrap it in `SELECT ... FOR UPDATE` inside the existing transaction (the
|
||||
function already opens one) rather than trusting single-writer
|
||||
semantics that no longer exist. Same pattern applies to
|
||||
`ReorderSession`/`SwapSessionOrder`/`ReorderGroup`/`SwapGroupOrder`'s
|
||||
read-then-renumber-then-write sequences.
|
||||
- FK violations are new failure modes M6.2 introduced (group/parent FKs).
|
||||
`createSession` and `PlaceSession` already validate `group_name`/
|
||||
`parent_id` through `validParent`/`ensureGroup` before writing, so those
|
||||
paths shouldn't hit a live constraint in practice — but wrap the actual
|
||||
`INSERT`/`UPDATE` error and translate a `23503` (foreign_key_violation)
|
||||
SQLSTATE into the same descriptive errors the pre-validation already
|
||||
produces, so a race between the check and the write degrades to a clear
|
||||
error instead of a raw driver error reaching the CLI.
|
||||
- `driver: modernc.org/sqlite` import and the blank `_ "modernc.org/
|
||||
sqlite"` in `Open()` are deleted; replaced with `pgx` (`github.com/
|
||||
jackc/pgx/v5/stdlib` for `database/sql` compatibility, keeping the rest
|
||||
of the file's `*sql.DB`-based code unchanged) or a native `pgx.Pool` —
|
||||
pick `pgx/v5/stdlib` unless a later task needs pgx-native features
|
||||
(e.g. `COPY`), since it's the smaller diff against the existing
|
||||
`database/sql` code.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Swap the driver import and `Open()`'s connection string handling
|
||||
(sqlite file path -> Postgres DSN, likely from an env var or flag the
|
||||
CLI already has a slot for — check `cmd/` for how `Open()` is called
|
||||
today).
|
||||
2. Delete `db.SetMaxOpenConns(1)` / `PRAGMA journal_mode=WAL`; size the
|
||||
pool deliberately instead (`SetMaxOpenConns` to something sane for a
|
||||
single-machine client, e.g. 5–10).
|
||||
3. Delete `init()`'s `CREATE TABLE IF NOT EXISTS` + 14-migration list —
|
||||
M6.2's migration owns schema creation now; `Open()` just connects and
|
||||
optionally runs a `schema_version` sanity check.
|
||||
4. Mechanically convert every `?` to `$N` across the 65 call sites, in
|
||||
file order, verifying arg count against placeholder count each time
|
||||
(this is where an off-by-one is easiest to introduce silently).
|
||||
5. Remove `boolToInt`/`archived != 0` conversions; scan struct fields
|
||||
directly as `bool`.
|
||||
6. Remove `encodeTime`/`decodeTime`/`secondsCeiling`; pass `time.Time`
|
||||
directly. Delete `timeenc_test.go`.
|
||||
7. Add `SELECT ... FOR UPDATE` (or equivalent explicit locking) to the
|
||||
sort-order read-then-write sequences named above.
|
||||
8. Add FK-violation (`23503`) error translation at the write sites that
|
||||
can theoretically race past their own pre-validation.
|
||||
9. Run agent-manager's full existing test suite against a real Postgres
|
||||
(M6.1's cluster, or local `postgres:16` for iteration) — every current
|
||||
test should still pass unmodified in intent, only in backing store.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Zero references to `modernc.org/sqlite`, `?` placeholders,
|
||||
`boolToInt`, `encodeTime`/`decodeTime` remain in `internal/store/`.
|
||||
- Full existing store test suite passes against Postgres.
|
||||
- Concurrent `CreateSession` calls (simulated) never produce duplicate
|
||||
`sort_order` values within the same `group_name`/`parent_id`.
|
||||
- A `spawn` CLI round trip (create session, update status, delete) works
|
||||
end-to-end against the M6.1 cluster.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `internal/store/*_test.go` running against the disposable
|
||||
Postgres from M6.2's harness.
|
||||
|
||||
**Integration test** — extend/rename the existing store test files:
|
||||
1. `a1_no_sqlite_references` — `grep -r "modernc.org/sqlite" internal/`
|
||||
returns nothing.
|
||||
2. `a2_no_bare_placeholders` — `grep -rE '\?[,)]' internal/store/store.go`
|
||||
inside SQL string literals returns nothing (manual review of any
|
||||
`?` that's part of a non-SQL string, e.g. a Go format verb, to avoid a
|
||||
false positive).
|
||||
3. `a3_existing_suite_passes` — `go test ./internal/store/...` green
|
||||
against Postgres.
|
||||
4. `a4_concurrent_sort_order` — spawn N goroutines each calling
|
||||
`CreateSession` into the same group concurrently; assert the resulting
|
||||
`sort_order` values are a dense 0..N-1 permutation with no duplicates.
|
||||
5. `a5_fk_violation_translated` — attempt `PlaceSession` with a
|
||||
`parentID` deleted between the read and the write (simulate via a
|
||||
second connection); assert the returned error is the existing
|
||||
descriptive one, not a raw pgx driver error.
|
||||
6. `a6_spawn_roundtrip` — `spawn` CLI subcommand creates a session,
|
||||
`UpdateStatus`, then `Delete`; assert no error and the row is gone.
|
||||
|
||||
**Command:** `go test ./internal/... -run TestStore -v`
|
||||
|
||||
**False pass:**
|
||||
- Running the suite against sqlite still (leftover build tag or import)
|
||||
while believing it validated Postgres. Assertion 1 is the guard —
|
||||
without it, the whole task could report green while the driver swap
|
||||
never actually happened.
|
||||
- Skipping assertion 4. A `sort_order` race is invisible in every
|
||||
single-threaded test and only shows up as duplicate/out-of-order
|
||||
sessions under real concurrent use, which is exactly the class of bug
|
||||
removing `SetMaxOpenConns(1)` introduces.
|
||||
|
||||
## Traps
|
||||
|
||||
- Converting `?` to `$N` by simple find-and-replace in file order without
|
||||
re-checking each statement's actual arg list — a multi-arg statement
|
||||
reordered during earlier edits (e.g. `createSession`'s 15-arg INSERT)
|
||||
will silently bind the wrong value to the wrong column, and Go's
|
||||
`database/sql` won't catch a type-compatible mismatch (e.g. two `TEXT`
|
||||
columns swapped).
|
||||
- Forgetting `pending_inputs` is now `JSONB` (M6.2), not `TEXT` — the
|
||||
existing `json.Marshal`/`Unmarshal` round-trip in
|
||||
`encodePendingInputs`/`pendingInputState` still works unchanged (pgx
|
||||
scans `JSONB` into `[]byte` the same as `TEXT`), but don't add an extra
|
||||
marshal layer thinking the column type changed the Go-side contract.
|
||||
|
||||
---
|
||||
|
||||
Background: [M6.2](M6.2-schema-port.md) · `internal/store/store.go`, `internal/spawn/spawn.go` (agent-manager, `add-headless-spawn` branch)
|
||||
@@ -0,0 +1,122 @@
|
||||
# M6.4 — nginx TCP routing to `agent-manager-db`
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | S — <1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | homelab |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
A path from the Mac client running agent-manager to the cluster-internal,
|
||||
ClusterIP-only `agent-manager-db` — through the shared ingress controller,
|
||||
matching this homelab's existing pattern, not a raw LAN IP or a tunnel.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Decided, not open:** dedicated ingress routing through nginx, not
|
||||
`kubectl port-forward`/SSH tunnel and not a MetalLB `LoadBalancer` IP.
|
||||
Matches the pattern already used elsewhere in this homelab of routing
|
||||
through the shared ingress controller rather than exposing raw
|
||||
per-service LAN IPs.
|
||||
|
||||
**Postgres is not HTTP.** The standard nginx-ingress `Ingress` resource is
|
||||
HTTP(S)-oriented (host/path routing, TLS termination via SNI on 443).
|
||||
Postgres speaks its own binary wire protocol on 5432. The ingress
|
||||
controller needs `stream {}` block config (TCP/UDP passthrough) or a
|
||||
dedicated `TCP` mode `Service`/`ConfigMap` entry — whichever the specific
|
||||
nginx-ingress deployment in this cluster supports (check
|
||||
`k8s/infra/ingress/` for how it's deployed and whether `tcp-services`
|
||||
ConfigMap wiring already exists for anything else, since this may be the
|
||||
first TCP passthrough case in the cluster).
|
||||
|
||||
**No existing precedent in this homelab** — `forgejo-db`, `authentik-db`,
|
||||
`temporal-db` are all consumed only by pods inside the same cluster over
|
||||
their ClusterIP Service, never from outside. `agent-manager-db` is the
|
||||
first case of an external (Mac) client needing to reach a CNPG cluster,
|
||||
which is why this task exists as dedicated work rather than "just add a
|
||||
Service."
|
||||
|
||||
## Steps
|
||||
|
||||
1. Confirm how nginx-ingress is deployed in this cluster (`k8s/infra/
|
||||
ingress/`) and whether it already exposes a `tcp-services` ConfigMap
|
||||
or `stream {}` snippet mechanism — ingress-nginx (the community
|
||||
controller) supports TCP passthrough via a `tcp-services` ConfigMap
|
||||
mapping `<external-port>: <namespace>/<service>:<port>`; confirm this
|
||||
is the controller in use before assuming that config shape.
|
||||
2. Pick an external port for Postgres traffic (5432 is already the
|
||||
in-cluster default; an external port distinct from any other exposed
|
||||
service avoids collision — check what's already claimed).
|
||||
3. Add the `tcp-services` (or equivalent) entry routing that external
|
||||
port to `agent-manager-db-rw.agent-manager.svc.cluster.local:5432`
|
||||
(CNPG's read-write Service name convention — confirm against the
|
||||
actual Service name M6.1's `Cluster` generates).
|
||||
4. Expose that port on the ingress controller's `Service`/`LoadBalancer`
|
||||
(this is the one LAN-facing port for this whole feature — the DB
|
||||
itself stays ClusterIP-only, only the ingress controller's existing
|
||||
external IP gains a new port).
|
||||
5. Commit, push to both remotes, verify which `repoURL` the owning
|
||||
ArgoCD `Application` watches (same caveat as M6.1) before assuming a
|
||||
push landed, let ArgoCD sync.
|
||||
6. Test connectivity from the Mac client: `psql
|
||||
postgresql://<user>@<homelab-ingress-host>:<external-port>/
|
||||
agent_manager` (credentials from M6.5).
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `psql` (or `pgx`) from outside the cluster reaches `agent-manager-db`
|
||||
through the ingress controller's external IP/port.
|
||||
- `agent-manager-db`'s own Service remains ClusterIP-only — no
|
||||
`LoadBalancer`/`NodePort` added to it directly (that would defeat the
|
||||
point of routing through nginx).
|
||||
- TLS/auth on the connection is Postgres's own (`sslmode`, password auth)
|
||||
— nginx `stream {}` passthrough does not terminate or inspect the
|
||||
Postgres protocol, so it adds no auth of its own. Confirm this is
|
||||
acceptable given the homelab's network boundary (LAN-only ingress
|
||||
exposure, not public internet) before treating it as done.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `psql` from the Mac client (outside the cluster network),
|
||||
plus `kubectl` checks on the ingress controller's config.
|
||||
|
||||
**Integration test** — `verify/m6.4.sh` diffed against
|
||||
`expected/m6.4.txt`:
|
||||
1. `a1_external_connects` — `psql
|
||||
postgresql://app@<ingress-host>:<port>/agent_manager -c 'select 1'`
|
||||
from the Mac client succeeds.
|
||||
2. `a2_db_service_still_clusterip` — `kubectl get svc -n agent-manager
|
||||
agent-manager-db-rw -o jsonpath='{.spec.type}'` is `ClusterIP`.
|
||||
3. `a3_argocd_synced` — the ingress-owning app is `Synced/Healthy` after
|
||||
the config change.
|
||||
4. `a4_wrong_port_refused` — connecting to a random unmapped port on the
|
||||
same ingress host fails (proves the mapping is port-specific, not an
|
||||
accidental catch-all passthrough).
|
||||
|
||||
**Command:** `bash verify/m6.4.sh | diff - expected/m6.4.txt`
|
||||
|
||||
**False pass:**
|
||||
- Testing connectivity from inside the cluster (e.g. `kubectl exec` into
|
||||
a pod and `psql` the ClusterIP directly). That was already true before
|
||||
this task and proves nothing about the ingress path — assertion 1 must
|
||||
run from the actual Mac client, outside the cluster network.
|
||||
|
||||
## Traps
|
||||
|
||||
- Reusing port 5432 externally on the ingress controller's existing
|
||||
external IP if anything else is already listening there (unlikely for
|
||||
Postgres specifically, but worth a `kubectl get svc -n <ingress-ns>
|
||||
<ingress-svc> -o yaml` check before assuming the port is free).
|
||||
- `stream {}` / `tcp-services` config living outside the GitOps-tracked
|
||||
kustomization because it's a ConfigMap edit that "felt like a quick
|
||||
manual fix." Same hard rule as everything else: commit + push +
|
||||
ArgoCD sync, no manual `kubectl apply` to the ingress controller's
|
||||
config.
|
||||
|
||||
---
|
||||
|
||||
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `k8s/infra/ingress/` (nginx-ingress deployment, controller type to confirm)
|
||||
@@ -0,0 +1,110 @@
|
||||
# M6.5 — Postgres credentials for the Mac client
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | S — <1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | homelab |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
The `agent-manager` process running on the Mac gets a Postgres connection
|
||||
string/credentials, managed the same ksops way the rest of this homelab
|
||||
handles secrets — not a password pasted into a local config file or env
|
||||
var by hand.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Existing pattern to follow: ksops-managed secret**, same as
|
||||
`model-invoke-apikey` elsewhere in this homelab. CNPG itself already
|
||||
generates an in-cluster Secret for the `app` user
|
||||
(`agent-manager-db-app` by its usual naming convention) — the work here is
|
||||
getting that credential (or a dedicated read-write user, if reusing the
|
||||
CNPG-generated superuser-adjacent `app` credential isn't desired) to a
|
||||
process running outside the cluster, on the Mac.
|
||||
|
||||
**Two credentials touch two different trust boundaries:**
|
||||
- Cluster-internal: CNPG's own generated Secret, already ksops-free
|
||||
(CNPG manages it, not this repo).
|
||||
- Mac client: needs that same username/password (or a separate,
|
||||
narrower-scoped user) delivered to `~/workplace/agent-manager`'s
|
||||
runtime config, via a ksops-encrypted file in git rather than a
|
||||
manually-copied value.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Decide: reuse CNPG's auto-generated `app` user, or create a dedicated
|
||||
`agent_manager_client` role scoped to only the `sessions`/`groups`/
|
||||
`settings`/`review_*` tables (narrower blast radius if the credential
|
||||
ever leaks from a Mac laptop, which is a meaningfully different threat
|
||||
model than a credential that only ever lives inside the cluster).
|
||||
2. If a dedicated role: add it via CNPG's declarative `Database` /
|
||||
`postInitSQL` (or a one-time migration in M6.2's schema setup) rather
|
||||
than a manual `psql` grant.
|
||||
3. Encrypt the resulting connection string (or user/password pair) with
|
||||
ksops, following the exact file layout `model-invoke-apikey` uses.
|
||||
4. Commit the encrypted secret to git (safe — that's the point of ksops),
|
||||
push to both remotes.
|
||||
5. Wire agent-manager's config loading (check `cmd/` / existing config
|
||||
file handling on the `add-headless-spawn` branch) to read the
|
||||
decrypted value at runtime — decide whether decryption happens via a
|
||||
`sops exec-env`-style wrapper the Mac invokes, or a decrypted file
|
||||
materialized once locally and gitignored, matching whatever
|
||||
`model-invoke-apikey`'s consumers already do.
|
||||
6. Verify agent-manager connects using only the ksops-sourced credential
|
||||
— no plaintext password anywhere in the repo or in shell history.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- No Postgres password appears in plaintext in git, in
|
||||
`~/workplace/agent-manager`'s tracked config, or was typed directly
|
||||
into a `kubectl`/`psql` command during setup.
|
||||
- The credential is encrypted with ksops in the same repo location/
|
||||
pattern as `model-invoke-apikey`.
|
||||
- agent-manager on the Mac successfully authenticates through M6.4's
|
||||
ingress route using this credential.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** manual — this is a secret-handling task, not one to automate
|
||||
a fake credential through.
|
||||
|
||||
**Checklist** (no `verify/*.sh`, since scripting a real credential check
|
||||
means committing something that either leaks a real secret or asserts
|
||||
nothing):
|
||||
1. `git grep -i "password"` in both the homelab repo and
|
||||
`~/workplace/agent-manager` shows only ksops-encrypted blobs or
|
||||
references to environment/config loading, never a literal value.
|
||||
2. The ksops secret file's structure matches `model-invoke-apikey`'s
|
||||
(same encryption provider, same key layout) — a side-by-side diff of
|
||||
the YAML structure (not values) confirms this.
|
||||
3. `psql` from the Mac using the decrypted credential (via M6.4's route)
|
||||
succeeds.
|
||||
4. Revoking/rotating the credential (delete the CNPG Secret or drop the
|
||||
dedicated role, if one was created) and re-running step 3 fails
|
||||
cleanly — confirms the client isn't caching or falling back to
|
||||
something else.
|
||||
|
||||
**False pass:**
|
||||
- Confirming connectivity once with a credential typed in manually during
|
||||
setup, then wiring the ksops path afterward without re-testing that the
|
||||
*ksops-sourced* value is what actually authenticates. Checklist item 4
|
||||
(rotate and confirm the old path is really gone) is the guard against
|
||||
"it worked, but only because of leftover state."
|
||||
|
||||
## Traps
|
||||
|
||||
- Granting the dedicated role (if created) superuser or
|
||||
database-owner-equivalent privileges "to avoid permission errors during
|
||||
setup" and never narrowing it afterward — defeats the reason to create
|
||||
a dedicated role at all.
|
||||
- Storing the decrypted credential in a file the Mac client reads that
|
||||
isn't gitignored, recreating the plaintext-secret problem one directory
|
||||
away from the ksops-encrypted source of truth.
|
||||
|
||||
---
|
||||
|
||||
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `model-invoke-apikey` (ksops pattern reference, this homelab)
|
||||
@@ -0,0 +1,150 @@
|
||||
# M6.6 — M6 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M6 — agent-manager migration |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the schema, the query port, the network path, and the credentials
|
||||
bind together into one working system — not four pieces that each passed
|
||||
their own task in isolation. This is the property no single M6 task owns:
|
||||
M6.1 proves the cluster is healthy, M6.2 proves the schema applies, M6.3
|
||||
proves the queries work against a local test database, M6.4 proves a raw
|
||||
TCP connection reaches the cluster, M6.5 proves the credential decrypts —
|
||||
none of them alone proves agent-manager, running for real on the Mac,
|
||||
through the real ingress route, with the real ksops credential, against
|
||||
the real cluster, does its actual job: track a session end-to-end without
|
||||
losing or corrupting data.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**"Bind together smoothly" means two concrete things here, not a vibe:**
|
||||
|
||||
1. **The full network path is exercised, not simulated.** M6.3's tests run
|
||||
against a disposable local Postgres — that validates the SQL, not the
|
||||
route. This gate is the first (and only) task that runs agent-manager
|
||||
unmodified, on the actual Mac, through M6.4's nginx `stream {}` route,
|
||||
authenticating with M6.5's ksops-sourced credential, against M6.1's
|
||||
real cluster.
|
||||
2. **Schema conventions match the rest of this homelab, not just
|
||||
"compiles."** This project's own `memory-db` (M2.x) and
|
||||
`agent-manager-db` (M6.x) are two unrelated Postgres schemas landing in
|
||||
the same cluster around the same time. They don't share data or a
|
||||
cluster (M6.1 already ruled that out), but a reviewer scanning
|
||||
`k8s/infra/databases/` should find the same shape twice: same
|
||||
`Cluster`/`Database` CRD structure, same `storageClass`, same
|
||||
`enableSuperuserAccess: false`, same GitOps-only provisioning
|
||||
discipline. "Binds together smoothly" includes that consistency check,
|
||||
not just agent-manager working in isolation.
|
||||
|
||||
**What a session round trip actually touches**, so the test isn't
|
||||
shallow: `CreateSession` (writes `sessions` + touches `groups` via the
|
||||
`ON CONFLICT DO NOTHING` insert) -> `UpdateStatus` -> `SetAgentSessionID`
|
||||
-> `SetReviewRepo` (writes `review_targets`) -> `Delete` (must cascade
|
||||
`review_targets` via the FK, per M6.2's decision, with no leftover row).
|
||||
That single flow crosses all 4 non-`settings` tables and exercises both
|
||||
the FK-cascade decision and the placeholder-conversion correctness from
|
||||
M6.3 in one pass.
|
||||
|
||||
## Steps
|
||||
|
||||
1. On the Mac, with agent-manager built from the fully-ported
|
||||
`add-headless-spawn` branch (M6.3 complete) and configured to use
|
||||
M6.4/M6.5's route and credential: run the `spawn` CLI subcommand to
|
||||
create a real session.
|
||||
2. Drive it through the full lifecycle above (status update, agent
|
||||
session id capture, review target set, delete) using agent-manager's
|
||||
own CLI/TUI, not a hand-rolled SQL script — the point is proving the
|
||||
actual client works, not that Postgres accepts hand-written SQL.
|
||||
3. Kill and restart agent-manager mid-lifecycle (after step 2's status
|
||||
update, before delete); confirm it reconnects and reads back the same
|
||||
state — proves the connection isn't accidentally caching state
|
||||
client-side that masks a write that never actually landed.
|
||||
4. Diff `k8s/infra/databases/agent-manager-db.yaml` against
|
||||
`k8s/infra/databases/memory-db.yaml` field-by-field for the
|
||||
convention-consistency check.
|
||||
5. Confirm both ArgoCD Applications (agent-manager's and this project's
|
||||
`memory-db`, once it exists) are tracked from the `repoURL` each
|
||||
actually watches — re-verify per M6.1's caveat, since this is the
|
||||
final point where a "pushed but ArgoCD never saw it" mistake would
|
||||
otherwise go unnoticed until much later.
|
||||
6. Commit `expected/m6.6.txt`; diff.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A session created, updated, and deleted through agent-manager's real
|
||||
CLI on the Mac round-trips correctly through the full network path.
|
||||
- A mid-lifecycle restart does not lose or duplicate state.
|
||||
- Deleting the session leaves zero orphan rows in `review_targets`/
|
||||
`review_bases`/`review_scopes` (FK cascade, not app-level cleanup).
|
||||
- `agent-manager-db.yaml` and `memory-db.yaml` match on every field that
|
||||
isn't inherently app-specific (name, storage size).
|
||||
- No manual `kubectl apply`/`psql` anywhere in the setup this gate
|
||||
exercises.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** the real Mac client, the real cluster, agent-manager's own
|
||||
CLI — this gate deliberately does not use a disposable/local database,
|
||||
since proving the disposable path works is exactly what M6.1-M6.5 already
|
||||
did.
|
||||
|
||||
**Integration test** — `verify/m6.6.sh` diffed against
|
||||
`expected/m6.6.txt`:
|
||||
1. `a1_full_roundtrip` — create/update/set-review/delete via the real CLI;
|
||||
assert no error at any step.
|
||||
2. `a2_no_orphan_review_rows` — after delete, query `review_targets`/
|
||||
`review_bases`/`review_scopes` directly (from inside the cluster, as a
|
||||
final-state check) for the deleted session's id; assert zero rows,
|
||||
with no explicit `DELETE FROM review_*` having been issued by the CLI
|
||||
(proves the FK cascade did the work, not leftover manual-cleanup code
|
||||
nobody removed).
|
||||
3. `a3_survives_restart` — kill agent-manager between status-update and
|
||||
delete; restart; assert the status update is still visible before
|
||||
proceeding to delete.
|
||||
4. `a4_schema_convention_match` — diff the two `Cluster` manifests' non
|
||||
app-specific fields; assert empty diff.
|
||||
5. `a5_repourl_confirmed` — for each of the two ArgoCD Applications
|
||||
involved, print which `repoURL` it watches and confirm it matches
|
||||
which remote was actually pushed.
|
||||
6. `a6_no_manual_apply_in_history` — review the shell history / session
|
||||
log from M6.1 through M6.5 for a `kubectl apply` or `psql` write
|
||||
command that wasn't inside an explicitly-flagged debugging exception;
|
||||
assert none exist outside that exception.
|
||||
|
||||
**Command:** `bash verify/m6.6.sh | diff - expected/m6.6.txt`
|
||||
|
||||
**False pass:**
|
||||
- Running assertion 1 against M6.3's disposable local Postgres instead of
|
||||
the real cluster because it's faster/already running. That's exactly
|
||||
the "four pieces that each passed in isolation" failure mode this gate
|
||||
exists to catch — it must hit M6.1's actual cluster through M6.4's
|
||||
actual route.
|
||||
- Treating a schema diff (assertion 4) as advisory and skipping it when
|
||||
short on time. A convention mismatch here is invisible today and
|
||||
becomes the thing a future reviewer trips over when comparing the two
|
||||
`k8s/infra/databases/*.yaml` files months later with no memory of why
|
||||
they differ.
|
||||
|
||||
## Traps
|
||||
|
||||
- Discovering during this gate that M6.4's nginx route works from inside
|
||||
the homelab LAN but not from wherever the Mac actually sits (VPN,
|
||||
different subnet, etc.) — a gap none of M6.1-M6.5's narrower tests
|
||||
would have caught, since this is the first task that tests from the
|
||||
Mac's actual network position rather than "outside the cluster" in the
|
||||
abstract.
|
||||
- Fixing a gate failure by loosening the gate (e.g. deleting assertion 2
|
||||
because the cascade "mostly works") instead of fixing the underlying
|
||||
FK/migration issue. Same discipline this project's other gates
|
||||
(M0.8, M1.8, M2.8...) already hold to.
|
||||
|
||||
---
|
||||
|
||||
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · [M6.2](M6.2-schema-port.md) · [M6.3](M6.3-store-query-port.md) · [M6.4](M6.4-nginx-stream-routing.md) · [M6.5](M6.5-credentials-secret.md)
|
||||
Reference in New Issue
Block a user