commit 9d28b63ff2efda1376d9e4685c80a683e991440e Author: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed Aug 19 09:52:07 2026 -0700 (plan) system review and break down plans diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..eaa2d4f --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,486 @@ +# Poimen Memory — gated recurrent knowledge extraction from agent context + +Target repo: `/Users/rockliang/workplace/Poimen/memory` (empty git repo, no remote, no commits). + +## Context + +Agent sessions accumulate faster than they can be read, and almost all of the volume is noise. One pi session in this project measured: + +``` +assistant 1445 +toolResult 1261 <- 43%, mostly evidence-free (ls output, file reads) +user 196 ++ 8 compaction events +``` + +Compaction already fires 8 times per session, which means context is being *discarded* rather than *retained* — the knowledge produced (root causes, decisions, gotchas) evaporates when the window rolls. Meanwhile the corpus is already project-partitioned on disk: + +- `~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen--/*.jsonl` — `cwd` in the session header gives the project key +- `~/.claude/projects//.jsonl` — 15 MB, 7 transcripts +- `Poimen/agent-rust/tasks/artifacts//` — coder/reviewer JSONL per attempt + +Intended outcome: a durable, queryable project memory built by reading that history chunk-by-chunk and keeping only what answers standing questions — surfaced as an Obsidian vault a human reads and a pgvector index an agent queries. + +The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the problem: the **update gate** refuses to write evidence-free chunks into memory (the 43% of toolResults), and the **exit gate** stops the scan once evidence is sufficient. The paper reports up to 400% speedup and *better* accuracy than ungated recurrent memory, because unchecked memory growth actively degrades later updates. + +## Verified facts this plan depends on + +| Fact | Value | How known | +|---|---|---| +| Embedding dims | **768** | probed `/v1/embeddings` with `nomic-ai/nomic-embed-text-v2-moe` | +| pgvector in CNPG | **available, v0.7.0**, not yet installed | `pg_available_extensions` on `forgejo-db-2`, stock image `ghcr.io/cloudnative-pg/postgresql:16.2` | +| CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` | +| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` | +| Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** | +| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value | + +The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response. + +**Side fix:** `~/.pi/agent/models.json` declares `contextWindow: 131072` for ornith. Wrong — Ollama caps at 32768, so long prompts truncate silently. Correct to 32768. + +## The tier model + +Memory is **levelled**, and every event in the log carries its level. The paper has one flat memory; a project knowledge base needs three. + +| Level | What it is | Produced by | Bounded | +|---|---|---|---| +| **L0** | evidence chunk — the verbatim source span the update gate accepted | update gate opening at L1 | no, but sparse (~17 of 412 chunks) | +| **L1** | per-query memory — GRU-Mem's `M_t` for one standing question | gated loop over L0 chunk stream | 1024 tokens | +| **L2** | project synthesis — memory across the L1 memories of one project | gated loop over L1 memories | 1024 tokens | + +**The tiering is not new machinery.** L2 is the same controller, same prompt, same two gates — run with the L1 memories as its chunk stream and a project-level question. The recurrence is the algorithm applied to its own output, so `mem-core` implements one loop and the level is a parameter. Two consequences worth having on purpose: + +- **Exit gate flips by level.** Off at L1 (see below), reasonably *on* at L2, where the input is a handful of memories rather than hundreds of chunks and "enough evidence" is actually decidable. +- **Levels form a provenance graph, not a pile.** Each L1 node records the L0 nodes that produced it; each L2 node records its L1 parents. That graph *is* the Obsidian link structure and the `parent_id` edges in Postgres — one relationship expressed in both projections. + +## Architecture + +``` +pi sessions / claude transcripts / loop.sh artifacts + │ + ▼ project resolver (cwd -> project id) + [ Chunk Parser ] 5000-token chunks, split on message boundaries + │ + ▼ +┌──────────────────────────────────────────────────────┐ +│ GRU-Mem Controller qwen2.5:3b-instruct │ +│ reason about chunk vs question │ +│ yes|no -> update gate U_t │ +│ candidate memory M̂_t │ +│ continue|end -> exit gate E_t │ +└───────────────┬──────────────────────────────────────┘ + U_t=yes │ U_t=no + ┌───────┴────────┐ + ▼ ▼ + emit L0 evidence discard chunk + M_t <- M̂_t M_t <- M_{t-1} + │ + ▼ L1 memory per standing query + └──────► same loop, input = L1 memories ──► L2 project synthesis + │ + ▼ +┌──────────────────────────────────────────────────────┐ +│ JSONL event log AUTHORITATIVE │ +│ Obsidian vault derived projection │ +│ pgvector index derived projection │ +└──────────────────────────────────────────────────────┘ +``` + +**Authority model — the load-bearing decision.** The JSONL log is the only source of truth. The vault and the vector index are projections that must be droppable and rebuildable byte-identically from the log. This is poimen's own §1 principle ("nothing derived is authoritative; if it cannot be dropped and rebuilt, it has hidden inputs and that is a bug") applied here, and it buys three things: re-embedding after a model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, and the post-training corpus is the log itself. + +## Standing queries + +The paper's memory agent is `φθ(Q, C_t, M_{t-1})` — **it requires a question**. The update gate is defined as "does this chunk contain useful information *about the problem*". Without `Q` the gate has no referent, and `r_update` becomes undefinable, which forecloses post-training. + +So each project declares durable questions in YAML. One query = one L1 memory = one Obsidian note. + +```yaml +# queries/poimen.yaml +project: poimen +sources: + - pi:--Users-rockliang-workplace-Poimen-agent-rust-- + - claude:-Users-rockliang-workplace-Poimen +queries: + - id: architecture-decisions + question: What architectural decisions were made, with reasoning and rejected alternatives? + - id: infra-root-causes + question: What infrastructure bugs were found, what was the root cause, how was it isolated? + - id: open-questions + question: What questions were raised and left unresolved? +synthesis: # the L2 pass + question: What is the current state of this project, and what should someone know before working on it? + exit_gate: true +``` + +**Exit gate defaults off at L1.** Paper §3.3 makes this call itself: for "what are *all* the X" questions you cannot know evidence is sufficient without reading everything, so they provide a w/o-EG inference mode. L1 extraction is exactly that shape. Keep the gate *recorded* (its signal is needed for RL) but do not act on it. Enable at L2 and for interactive retrieval, where the paper measures 4× speedup. + +## Separate weights — yes, specifically a LoRA adapter + +Confirming the instinct, with the reason that actually matters here: + +- **Base:** Qwen2.5-3B-Instruct, already resident as `qwen2.5:3b-instruct` +- **Memory policy:** LoRA adapter, rank 16–32, ~30–60 MB + +Why an adapter rather than a fine-tuned model: + +1. **VRAM.** One GPU, `OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A separate full memory model evicts something, and eviction here is a weights reload measured in tens of seconds — we already watched `ornith` cold-start blow a 60s gateway timeout. A LoRA rides on the resident base for near-zero extra VRAM. +2. **Iteration.** Post-training emits a ~50 MB adapter, not a 6 GB model. Swap without redeploying. +3. **Reversibility.** A gate-behaviour regression reverts by pointing at the previous adapter. + +**Infra consequence to accept up front:** Ollama cannot hot-swap LoRA adapters. Serving one means either moving the memory model to a **vLLM** InferenceService with `--enable-lora` (the `reasoning` predictor is already vLLM v0.11.0, so the pattern exists), or merging into a GGUF for Ollama and losing swappability. P1–P4 run prompted-only on the stock model, so this is deferred, not dodged. + +## Repository layout + +Rust workspace at `Poimen/memory`: + +``` +Cargo.toml workspace +crates/ + mem-core/ domain types; Level enum; gate-response parser; the gated loop + mem-chunk/ RecordSource trait; chunking policy; flush triggers (see below) + mem-llm/ gateway client — chat completions, embeddings, rerank + mem-ingest/ source adapters: pi sessions, claude transcripts, loop.sh artifacts + mem-store/ JSONL log writer/reader; pgvector repo; Obsidian projector + mem-cli/ binary `mem`: ingest | synthesize | rebuild | query | verify | skill | label +memory-tasks/ the task board — INDEX.md + one file per task, tracked +queries/ standing query YAML, one file per project +vault/ Obsidian output (own git repo or gitignored) +log/ JSONL event log — authoritative, tracked +``` + +Crates: `sqlx` (postgres, runtime-tokio-rustls) + `pgvector` (sqlx feature), `tokenizers` for chunk sizing against the real Qwen2 tokenizer, `futures`, `serde`/`serde_json`, `clap`, `reqwest`. + +Reuse rather than reinvent: `mem-llm`'s request shape and the `apikey` header convention are proven in `agent-rust/loop.sh`; response parsing mirrors its `extract_text()`. + +### `mem-chunk` — its own crate, stream-shaped from day one + +Chunking is split out because it is the seam where new input kinds arrive. Sources today are files with an EOF; telemetry, a live session tail, or a broker will not have one. Designing the boundary as a stream now means a future source implements a trait rather than forcing the loop to be rewritten — and it costs nothing, since the rest of the stack is already tokio (`sqlx` runtime-tokio, `reqwest`). + +```rust +// mem-chunk +pub trait RecordSource { + /// Normalised records: role, text, timestamp, provenance. Sources decide + /// how to produce them; the chunker never learns about pi vs claude vs a socket. + fn records(self) -> impl Stream>; +} + +pub struct ChunkPolicy { + pub max_tokens: usize, // 5000, paper default + pub split_on: Boundary, // never mid-message + pub flush: FlushTrigger, // see below +} + +pub fn chunks(src: S, p: ChunkPolicy) -> impl Stream; +``` + +Batch sources become streams for free via `futures::stream::iter`, so `mem-ingest` gets no more complex today. + +**The one thing that genuinely differs for streams is the flush trigger.** A file chunker emits a partial chunk at EOF; a stream has no EOF, so a partially-filled chunk would sit forever. `FlushTrigger` is therefore `Tokens(n)` today and gains `OrIdle(Duration)` when a stream source lands — carrying it in the policy now means the later change is one enum variant, not a signature change through the loop. + +Worth noting for whenever that happens: the **exit gate changes meaning on an unbounded source**. At L1 over a finite transcript it is switched off because "read everything" is well defined (paper §3.3). Over a live stream there is no everything, so the gate stops being an optimisation and becomes the only termination condition — which is an argument for keeping it trained even while it is switched off. + +## Storage schemas + +### JSONL event log — authoritative + +`log///.jsonl`. **Every record carries `level`.** + +```jsonl +{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."} +{"type":"chunk","level":"L0","t":1,"source":"pi:2026-07-21T16-23-59_019f857d","span":[0,42],"sha256":"..."} +{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812} +{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."} +{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":[""],"sha256":"..."} +{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."} +``` + +The L2 pass writes the same record types with `"level":"L2"`, `"input_level":"L1"`, and `parents` holding L1 shas. `evidence` records appear only when the update gate opened, so update-rate is directly measurable and memory at any `t` is replayable. + +### pgvector — projection + +One table across all levels, because retrieval wants to search them together and filter: + +```sql +CREATE TABLE memory_node ( + id BIGSERIAL PRIMARY KEY, + level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')), + project TEXT NOT NULL, + query_id TEXT, -- null at L2 + run_id TEXT NOT NULL, + t INT NOT NULL, + source TEXT, -- set at L0 + text TEXT NOT NULL, + sha256 TEXT NOT NULL UNIQUE, + embedding vector(768) NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE TABLE memory_edge ( -- provenance: child <- parent + child_sha TEXT NOT NULL REFERENCES memory_node(sha256), + parent_sha TEXT NOT NULL REFERENCES memory_node(sha256), + PRIMARY KEY (child_sha, parent_sha) +); +CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops); +CREATE INDEX ON memory_node (project, level); +``` + +```mermaid +erDiagram + MEMORY_NODE ||--o{ MEMORY_EDGE : "child_sha -> sha256" + MEMORY_NODE ||--o{ MEMORY_EDGE : "parent_sha -> sha256" + + MEMORY_NODE { + bigserial id PK + text level "L0 | L1 | L2" + text project + text query_id "NULL at L2" + text run_id + int t + text source "set at L0" + text text + text sha256 UK "content identity" + vector_768 embedding + timestamptz created_at + } + + MEMORY_EDGE { + text child_sha PK,FK + text parent_sha PK,FK + } +``` + +One table across L0/L1/L2 (not three) — retrieval searches all levels together and filters by `level`. `memory_edge` is the provenance graph: L1 rows point back at the L0 chunks that produced them, L2 rows point back at L1 parents. `sha256 UNIQUE` is content identity (dedup key, hash excludes run id/timestamp — M0.2) and is what `memory_edge` actually references, not the surrogate `id`. + +Retrieval: HNSW recall filtered by level, then `bge-reranker-base` via `/v1/rerank` for precision — that endpoint scored 0.98 vs 0.00009 on a discrimination probe, so it earns its place. Default query searches L1+L2 and walks `memory_edge` down to L0 for citations. + +Infra — new CNPG cluster with the extension managed declaratively (CNPG 1.30 supports this, so **no manual `psql`**, consistent with the GitOps hard rule). Follows `k8s/infra/databases/temporal-db.yaml` exactly: + +```yaml +# k8s/infra/databases/memory-db.yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: { name: memory-db, namespace: memory } +spec: + instances: 3 + imageName: ghcr.io/cloudnative-pg/postgresql:16.2 + bootstrap: { initdb: { database: memory, owner: app, encoding: UTF8, localeCollate: C, localeCType: C } } + enableSuperuserAccess: false + storage: { size: 10Gi, storageClass: longhorn-cnpg } + monitoring: { enablePodMonitor: true } + affinity: + podAntiAffinityType: preferred + tolerations: [{ key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }] +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Database +metadata: { name: memory-db-vector, namespace: memory } +spec: + name: memory + owner: app + cluster: { name: memory-db } + extensions: [{ name: vector, ensure: present }] +``` + +### Obsidian vault — projection + +The tier graph becomes the note graph: + +``` +vault/poimen/ + index.md <- L2 synthesis, links to every L1 note + infra-root-causes.md <- L1 + architecture-decisions.md <- L1 + evidence/ <- L0, optional (--emit-evidence-notes, default off) +vault/skills/ + _drafts//SKILL.md <- machine-written, never auto-loaded + /SKILL.md <- human-promoted, loadable +``` + +```markdown +--- +project: poimen +level: L1 +query_id: infra-root-causes +updated: 2026-08-17 +chunks_seen: 412 +chunks_used: 17 +--- +# Infra root causes — poimen + + + +## Provenance +- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer +``` + +L0 defaults to inline citations rather than notes — 17 per query is manageable but grows unbounded across projects. The flag exists for when the graph view is worth the file count. + +## Skills — the procedural projection + +**A skill is a projection, not a level.** L0/L1/L2 are all *descriptive* — what happened. A skill is *procedural* — what to do next time. That is a change of modality, not a further compression, so the gated loop does not produce it: the update gate's question ("does this chunk contain evidence for Q") has no meaning when the output is an instruction. + +The format is free. `SKILL.md` is YAML frontmatter plus markdown, which is exactly an Obsidian note — verified against `~/.claude/skills/seo-geo-claude-skills/research/keyword-research/SKILL.md`, whose frontmatter carries `name`, `description`, `when_to_use`, `argument-hint`. So `vault/skills//SKILL.md` is simultaneously a vault note and a loadable skill, with no conversion step: + +```sh +pi --skill vault/skills/ # or set skillsPath in ~/.pi/config.json +ln -s .../vault/skills/ ~/.claude/skills/ +``` + +`mem skill draft --from poimen/infra-root-causes` reads an L1 or L2 note and writes a draft. Emission should follow the existing authoring rubric rather than inventing one — the installed `grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and a four-dimension rubric (conciseness, actionability, workflow clarity, progressive disclosure). Description quality is the whole game: a skill whose `description` does not match how the user actually phrases the request never fires. + +**Drafts are never auto-loaded, and promotion is manual.** This is the one place the system can close a loop on itself, and the failure is subtle: + +> a memory-derived skill is auto-loaded → it appears in future session transcripts → those transcripts are ingested as evidence → the memory that produced the skill is reinforced by its own output + +No external verifier breaks that cycle. It is the same hazard poimen §16 names when it gates "automatic workflow mutation without human approval" by default, and it is why `_drafts/` is a separate directory rather than a frontmatter flag — a directory cannot be accidentally globbed into `--skill`. + +Two mechanical guards: + +1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff. +2. **Provenance marks derived text.** Every emitted skill carries `generated_from: ` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data. + +## Phases + +**P1 — Read-only spine.** `mem-ingest` implements `RecordSource` for pi sessions and Claude transcripts; `mem-chunk` chunks to 5000 tokens on message boundaries; `mem ingest --dry-run` prints the chunk plan with no model calls. Both sources are batch, but they go through the stream interface so the seam is exercised from the first commit rather than retrofitted. + +**P2 — Gated loop at L1, prompted only.** `mem-llm` + gate-response parser (paper Figure 10a prompt, adapted for standing queries). Writes the JSONL log with L0 evidence and L1 memory records. Exit gate recorded, not acted on. This is the paper's "w/o RL" baseline, which Figure 9 shows already works. + +**P3 — Projections.** Obsidian projector and pgvector repo, both rebuildable via `mem rebuild --from-log`. Infra commit for `memory-db`. + +**P4 — L2 synthesis and retrieval.** `mem synthesize` runs the same loop over L1 memories with the exit gate on. `mem query` — embed, HNSW recall by level, rerank, return with provenance walked through `memory_edge`. + +**P5 — Skill drafting.** `mem skill draft --from ` emits `vault/skills/_drafts//SKILL.md` with `generated_from` provenance; `mem-ingest` grows the `derived: true` exclusion filter. Promotion stays manual. Cheap to build and it is the phase that makes the memory *do* something rather than only be read. + +**P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above. + +## Task breakdown + +Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M.` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id. + +``` +memory-tasks/ + INDEX.md board + phase order + progress mirror + M0.1-cargo-workspace.md + M0.2-domain-types.md + ... + M5.6-m5-gate.md +``` + +Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole board there. Naming this `memory-tasks/` sidesteps that pattern — and it should be **tracked**, since the task files carry the acceptance criteria. + +**M0 — Read-only spine** (no model calls anywhere in this phase) + +| id | task | size | deps | +|---|---|---|---| +| M0.1 | Cargo workspace + six crate skeletons, CI builds clean | S | — | +| M0.2 | `mem-core` domain types: `Level`, `Record`, `Chunk`, `MemoryNode`, sha256 identity | S | M0.1 | +| M0.3 | `mem-chunk`: `RecordSource` trait, `ChunkPolicy`, `FlushTrigger::Tokens` | M | M0.2 | +| M0.4 | Tokenizer-backed sizing against the real Qwen2 tokenizer | M | M0.3 | +| M0.5 | `mem-ingest`: pi session adapter (`~/.pi/agent/sessions//*.jsonl`) | M | M0.3 | +| M0.6 | `mem-ingest`: claude transcript adapter (`~/.claude/projects/**/.jsonl`) | S | M0.5 | +| M0.7 | `mem ingest --dry-run` — chunk plan, token histogram, source breakdown | S | M0.4, M0.6 | +| M0.8 | **M0 gate** — both adapters through one `RecordSource`, no source-specific code past the trait | M | gate | + +**M1 — Gated loop at L1** + +| id | task | size | deps | +|---|---|---|---| +| M1.1 | `mem-llm` chat client — `apikey` header, retry, timeout | M | M0.1 | +| M1.2 | Standing-query YAML loader + schema validation, unresolved id fails at load | M | M0.2 | +| M1.3 | GRU-Mem prompt template (paper Fig 10a), memory + chunk + question assembly | M | M1.2 | +| M1.4 | Gate-response parser: `///`, strict, malformed = hard error | M | M1.3 | +| M1.5 | The gated loop — `U_t` mutate-or-retain, `E_t` recorded not acted on, 1024-token budget | L | M1.4 | +| M1.6 | JSONL event log writer, `level` on every record, `parents` on memory | M | M1.5 | +| M1.7 | `mem ingest` end to end + update-rate reported on stdout | M | M1.6 | +| M1.8 | **M1 gate** — full run on `poimen`, update-rate < 30%, memory tokens flat not climbing | M | gate | + +**M2 — Projections** + +| id | task | size | deps | +|---|---|---|---| +| M2.1 | `mem-llm` embeddings client, 768-dim, batched | S | M1.1 | +| M2.2 | `k8s/infra/databases/memory-db.yaml` — CNPG Cluster + Database with `extensions: [vector]` | M | — | +| M2.3 | `memory_node` / `memory_edge` schema + sqlx migrations, HNSW indexes | M | M2.2 | +| M2.4 | `mem-store` pgvector repo — upsert by sha, edge insert | M | M2.3, M2.1 | +| M2.5 | Obsidian projector — frontmatter, wikilinks, L0 citations, `--emit-evidence-notes` | M | M1.6 | +| M2.6 | `mem rebuild --from-log` — drop and rebuild both projections | M | M2.4, M2.5 | +| M2.7 | `mem verify` — every L1 has ≥1 L0 parent, every parent sha resolves | S | M2.6 | +| M2.8 | **M2 gate** — rebuild is byte-identical (`git -C vault diff --exit-code` empty) | M | gate | + +**M3 — L2 synthesis and retrieval** + +| id | task | size | deps | +|---|---|---|---| +| M3.1 | L2 pass — same loop, input `Stream`, exit gate **on** | M | M1.5 | +| M3.2 | `mem-llm` rerank client (`bge-reranker-base`) | S | M1.1 | +| M3.3 | `mem query` — embed, HNSW recall filtered by level, rerank, walk edges to L0 | M | M3.2, M2.4 | +| M3.4 | **M3 gate** — known-answer query returns the right L1 node with a real L0 citation | M | gate | + +**M4 — Skills** + +| id | task | size | deps | +|---|---|---|---| +| M4.1 | `mem skill draft --from ` → `_drafts/`, frontmatter incl. `generated_from` | M | M3.1 | +| M4.2 | `derived: true` ingest filter — emitted artifacts excluded from evidence | M | M4.1, M0.5 | +| M4.3 | **M4 gate** — draft absent from `--list-skills`; no L0 node matches an emitted artifact | M | gate | + +**M5 — Post-training** (Python, separate from the Rust workspace; boundary is the JSONL) + +| id | task | size | deps | +|---|---|---|---| +| M5.1 | `mem label` — 32B `reasoning` as offline evidence labeler, writes `U_t` ground truth | M | M1.6 | +| M5.2 | Labeler calibration — hand-label a holdout, measure agreement before trusting it | M | M5.1 | +| M5.3 | Training corpus export from the log to verl's expected format | M | M5.1 | +| M5.4 | vLLM InferenceService for Qwen2.5-3B with `--enable-lora` (GitOps, homelab) | L | — | +| M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 | +| M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate | + +Total 38 tasks, 6 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. + +## Verification + +```bash +# P1 — corpus parses, chunk plan sane, zero model calls +cargo run -p mem-cli -- ingest --project poimen --dry-run + +# P2 — one query end to end +cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes +jq -r 'select(.type=="gate" and .level=="L1") | .update' log/poimen/infra-root-causes/*.jsonl | sort | uniq -c +# expect: far more false than true. Update-rate > ~30% means the gate is not +# discriminating — that is the paper's memory-explosion failure, Figure 6 is +# the reference shape. +jq -r 'select(.type=="memory") | .tokens' log/.../*.jsonl | tail -1 +# expect: <= 1024 and roughly flat over t, not monotonically climbing + +# levels are well-formed and edges close +jq -r '.level' log/poimen/**/*.jsonl | sort | uniq -c # L0/L1 present +cargo run -p mem-cli -- verify --project poimen +# asserts: every L1 memory has >=1 L0 parent; every parent sha exists + +# P3 — projections truly derived +cargo run -p mem-cli -- rebuild --from-log --project poimen +git -C vault diff --exit-code # empty: rebuild is byte-identical +psql -c "select level, count(*) from memory_node group by level;" + +# P4 — synthesis and retrieval +cargo run -p mem-cli -- synthesize --project poimen # expect exit gate to fire +cargo run -p mem-cli -- query "why did requests over 10KB fail?" +# expect: infra-root-causes L1 node, Kong body-buffer passage, L0 citation + +# P5 — skill drafts land unloadable, and the cycle stays open +cargo run -p mem-cli -- skill draft --from poimen/infra-root-causes +ls vault/skills/_drafts/ # draft here, NOT in vault/skills/ +pi --skill vault/skills/ --list-skills # draft must not appear +cargo run -p mem-cli -- verify --derived-filter --project poimen +# asserts: no L0 evidence node text matches an emitted skill artifact +``` + +The decisive P2 metric is **update-rate**, the one number distinguishing a working gate from an expensive summarizer. toolResults are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks. + +## Risks + +- **3B gate quality unmeasured on this corpus.** The paper evaluates on QA benchmarks with clean evidence labels; agent transcripts are messier. Mitigation: P2's update-rate is a cheap early read, and the 32B `reasoning` model can spot-audit a sample before committing to P5. +- **L2 inherits L1's errors with no path back to source.** Synthesis over memories cannot recover evidence the L1 gate wrongly discarded. `memory_edge` makes the omission *visible* (an L1 note with suspiciously few parents) but not recoverable without a re-run. +- **Self-reinforcement through skills.** The only cycle in the system: emitted skill → future session context → ingested as evidence → reinforces the memory that emitted it. Guarded by manual promotion plus the `derived: true` ingest filter, and both must hold. Audit it by checking that no L0 evidence node's text matches an emitted artifact. +- **No ground-truth evidence labels.** `r_update` needs them. Distant supervision from the 32B labeler inherits its bias; hold out a hand-labelled set to measure agreement before trusting it. +- **Vault/log divergence.** Hand edits are overwritten on rebuild. Either make the vault read-only or add an `## Notes` region the projector preserves. Decide before anyone starts editing. +- **Ollama has no LoRA path.** P5 forces the vLLM decision. Do not discover this at P5. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4e7e773 --- /dev/null +++ b/README.md @@ -0,0 +1,226 @@ +# poimen-memory + +Gated recurrent memory over agent context. Reads session history chunk-by-chunk, +keeps only what answers standing questions, projects result into an Obsidian +vault and a pgvector index. + +**Status: design complete, no code yet.** 37 tasks in [memory-tasks/](memory-tasks/INDEX.md), +0 done. Start at [M0.1](memory-tasks/M0.1-cargo-workspace.md). + +## Problem + +Agent sessions grow faster than anyone reads them, and most of the volume is +noise. One real pi session in this project: + +``` +assistant 1445 +toolResult 1261 43% — ls output, file reads, mostly evidence-free +user 196 ++ 8 compaction events +``` + +Compaction fires 8 times per session. Context gets *discarded*, not retained — +root causes, decisions and gotchas evaporate when window rolls. + +## Mechanism + +GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled +gates on a recurrent memory loop: + +- **update gate** — memory only mutates when chunk contains evidence. Blocks the + memory explosion that ungated recurrent memory hits. +- **exit gate** — stop scanning once evidence sufficient. + +Paper reports up to 400% speedup and *better* accuracy than ungated, because +unbounded memory growth degrades later updates. + +``` +sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections +``` + +Controller emits structured output; loop acts on it: + +``` + reason about chunk vs question + yes|no -> U_t, update or discard + candidate memory M̂_t + continue|end -> E_t, exit or continue +``` + +## Memory tiers + +| Level | What | From | Bounded | +|---|---|---|---| +| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) | +| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok | +| **L2** | project synthesis | gated loop over L1 memories | 1024 tok | + +L2 is not new machinery — same loop, same prompt, L1 memories as input stream. +Level is a parameter. + +Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1 +parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks. + +## Standing queries + +Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is +defined as "does this chunk contain useful information *about the problem*". No +`Q`, no gate, and `r_update` becomes undefinable, which kills post-training. + +So each project declares durable questions. One query = one L1 memory = one note. + +```yaml +# queries/poimen.yaml +project: poimen +roots: [/Users/rockliang/workplace/Poimen/agent-rust] +queries: + - id: infra-root-causes + question: What infrastructure bugs were found, what was the root cause, how was it isolated? + - id: architecture-decisions + question: What architectural decisions were made, with reasoning and rejected alternatives? +synthesis: + question: What is the current state of this project, and what should someone know before working on it? + exit_gate: true +``` + +**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all* +the X" questions you cannot know evidence is sufficient without reading +everything. L1 extraction is that shape. At L2 input is a handful of memories and +sufficiency is decidable. Gate still *recorded* at L1 — signal needed for +post-training. + +## Authority model + +**JSONL log authoritative. Vault and vector index are projections.** + +Anything not rebuildable byte-identically from the log has hidden inputs, and +that is a bug. Gate M2.8 enforces it destructively: + +```sh +rm -rf vault/poimen +psql -c "delete from memory_node where project='poimen'" +mem rebuild --from-log --project poimen +git -C vault diff --exit-code # empty diff is the only pass +``` + +Buys three things: re-embedding after model change is a rebuild not a migration, +Obsidian edits cannot corrupt the record, post-training corpus is the log itself. + +## Skills + +A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what +happened. A skill is procedural — what to do next time. Gated loop does not +produce it. + +Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian +note. So `vault/skills//SKILL.md` is both, no conversion: + +```sh +pi --skill vault/skills/ +ln -s .../vault/skills/ ~/.claude/skills/ +``` + +**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one +cycle in the design: + +``` +emitted skill auto-loads -> appears in future transcripts +-> ingested as evidence -> reinforces the memory that emitted it +``` + +No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be +globbed into `--skill`), and every artifact carries `generated_from` so ingest +tags matching chunks `derived: true` and refuses them as evidence. + +## Separate weights + +Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned +model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding +`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction +is a weights reload measured in tens of seconds. Adapter rides the resident base. + +Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression +reverts by pointing at previous adapter. + +**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora` +(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4 +run prompted-only, so decision is deferred, not dodged. + +## Layout + +``` +DESIGN.md full design, 460 lines +memory-tasks/ 37 task files + INDEX.md — tracked +crates/ + mem-core/ domain types; Level; gate parser; the gated loop + mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger + mem-llm/ gateway client — chat, embeddings, rerank + mem-ingest/ source adapters: pi sessions, claude transcripts + mem-store/ JSONL log; pgvector repo; Obsidian projector + mem-cli/ binary `mem` +queries/ standing query YAML per project +log/ JSONL event log — authoritative, tracked +vault/ Obsidian output +``` + +`mem-chunk` is separate and stream-shaped from day one. Sources today are files +with an EOF; telemetry or a live tail will not have one. `RecordSource` returns +`impl Stream`; batch sources become streams via +`futures::stream::iter`, so it costs nothing now and removes a rewrite later. + +## Commands + +```sh +mem ingest --project poimen --dry-run # chunk plan, zero model calls +mem ingest --project poimen --query infra-root-causes +mem synthesize --project poimen # L2 pass, exit gate on +mem rebuild --from-log --project poimen # drop and rebuild projections +mem verify --project poimen # provenance graph closure +mem query "why did requests over 10KB fail?" +mem skill draft --from poimen/infra-root-causes +mem label --project poimen # evidence labels for training +``` + +## Verified environment facts + +Checked against the running cluster, not assumed: + +| Fact | Value | +|---|---| +| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` | +| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) | +| pgvector | **0.7.0 available in stock CNPG image**, no custom build | +| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` | +| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config | +| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone | +| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** | +| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` | + +Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response. + +## Phases + +Each ends in a composition gate. No phase starts until predecessor gate is green. + +| | Phase | Tasks | Gate asserts | +|---|---|---|---| +| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline | +| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing | +| M2 | Projections | 8 | rebuild byte-identical from log alone | +| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 | +| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence | +| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project | + +**Update-rate is the number to watch.** It is what distinguishes a gate from an +expensive summarizer. Tool results are 43% of records and mostly evidence-free, +so a correct gate rejects the large majority of chunks. + +M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA) +is homelab work independent of the rest of M5. + +## Reading order + +1. This file +2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice +3. [DESIGN.md](DESIGN.md) — full design, schemas, risks +4. Individual task files — self-contained, no DESIGN.md read required diff --git a/tasks/INDEX.md b/tasks/INDEX.md new file mode 100644 index 0000000..aaa6bb5 --- /dev/null +++ b/tasks/INDEX.md @@ -0,0 +1,175 @@ +# poimen-memory — task board + +43 tasks — 36 build tasks plus **7 composition gates**, one per phase. One file +per task, **self-contained**: inlined design facts, executable steps, acceptance +criteria, a `Verify` section written for someone who did not build the thing, and +the traps worth naming. Reading `DESIGN.md` is not required to do a task — it is +linked as background only. + +Each `Verify` section names the harness, the integration test with numbered +assertions, the command to run, and the **false pass** — the shape of test that +goes green while the feature is broken. Treat the false-pass list as part of the +acceptance criteria, not commentary. + +## Ordering — declared, never derived + +**Phase order is the list below. Task ids are opaque and frozen.** + +`M0.3` is `M0.3` forever, in whatever phase it currently sits, because its +artifacts and cross-references key on that id. New tasks take new ids rather than +renumbering neighbours. This is the `StepId` rule applied to the board itself — +a board that renumbers to reorder has the bug it warns its own users about. + +No phase starts until its predecessor's gate is green. The `gate` task at the end +of each phase **is** that gate: it proves the phase's parts compose and that its +swappable parts are genuinely swappable. Every build task is verified alone; the +gate verifies the properties no single task owns. + +## Two rules this board exists to protect + +**1. The JSONL log is authoritative; the vault and the vector index are +projections.** Anything that cannot be dropped and rebuilt byte-identically from +the log has hidden inputs, and that is a bug. `M2.8` is the gate that enforces it. + +**2. The update gate must discriminate, not summarize.** Agent transcripts are +~43% tool results and mostly evidence-free. A gate that accepts most chunks is an +expensive summarizer that will reproduce the memory-explosion failure the whole +design exists to avoid. `M1.8` is the gate that enforces it, and **update-rate is +the single number to watch.** + +## Verification practice — script first, source second + +A task is verified by running its command and reading the output, then opening +the source. Reviewing the diff first is how an assertion that was quietly dropped +still gets called done: the code looks right, and nothing proves the test ran. + +Numbered assertion N in a `Verify` section is test fn `aN_`. The numbering +is the contract — a test fn that does not exist should report missing rather than +be silently absent from a green summary. + +`cargo test` reporting `ok` with zero tests run is not a pass. + +## Progress + +**Source of truth is the `Status` field in each task file.** The tables below +mirror it; a status changed here and not there is a lie. + +Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked + +| # | Phase | Ids | Tasks | ✅ | 🟡 | ⬜ | Gate | +|---|---|---|---|---|---|---|---| +| 1 | Read-only spine | M0.x | 8 | 0 | 0 | 8 | ⬜ M0.8 | +| 2 | Gated loop at L1 | M1.x | 8 | 0 | 0 | 8 | ⬜ M1.8 | +| 3 | Projections | M2.x | 8 | 0 | 0 | 8 | ⬜ M2.8 | +| 4 | L2 synthesis + retrieval | M3.x | 4 | 0 | 0 | 4 | ⬜ M3.4 | +| 5 | Skills | M4.x | 3 | 0 | 0 | 3 | ⬜ M4.3 | +| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 | +| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 | +| | **Total** | | **43** | **0** | **0** | **43** | 0/7 green | + +**Where the line is — 2026-08-18.** Nothing started. No crate exists yet: there +is no `Cargo.toml` under `memory/`, so every task below is design only. M0.1 is +the first thing that has to happen. `M2.2` (the CNPG manifest), `M5.4` (vLLM +with LoRA), and all of `M6.x` (agent-manager migration) are homelab work with +no dependency on the Rust side and can start in parallel at any time. + +**M6 is a different repo, not a dependency of M0-M5.** It migrates +`github.com/Riotpiaole/agent-manager`'s session store (a separate Go CLI tool, +unrelated to this project's own memory system) from local sqlite to its own +dedicated CNPG cluster. It rides in this board because it's homelab work +happening alongside M2.2/M5.4, and because the two projects' Postgres schemas +landing in the same cluster around the same time need to look like siblings, +not strangers — see M6.6's convention-consistency check. + +## 1 — Read-only spine · M0.x + +No model calls anywhere in this phase. The point is to prove the corpus parses +and chunks sanely before spending inference on it. + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M0.1](M0.1-cargo-workspace.md) | Cargo workspace + crate skeletons | S | — | ⬜ | +| [M0.2](M0.2-domain-types.md) | Domain types and sha256 identity | S | — | ⬜ | +| [M0.3](M0.3-recordsource-and-chunkpolicy.md) | `RecordSource` trait + `ChunkPolicy` | M | — | ⬜ | +| [M0.4](M0.4-tokenizer-sizing.md) | Tokenizer-backed chunk sizing | M | — | ⬜ | +| [M0.5](M0.5-pi-session-adapter.md) | pi session adapter | M | — | ⬜ | +| [M0.6](M0.6-claude-transcript-adapter.md) | Claude transcript adapter | S | — | ⬜ | +| [M0.7](M0.7-ingest-dry-run.md) | `mem ingest --dry-run` | S | — | ⬜ | +| [M0.8](M0.8-m0-gate.md) | **M0 composition gate** | M | gate | ⬜ | + +## 2 — Gated loop at L1 · M1.x + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M1.1](M1.1-llm-chat-client.md) | `mem-llm` chat client | M | — | ⬜ | +| [M1.2](M1.2-standing-query-loader.md) | Standing-query YAML loader | M | — | ⬜ | +| [M1.3](M1.3-prompt-template.md) | GRU-Mem prompt template | M | — | ⬜ | +| [M1.4](M1.4-gate-response-parser.md) | Gate-response parser | M | — | ⬜ | +| [M1.5](M1.5-gated-loop.md) | The gated loop | L | — | ⬜ | +| [M1.6](M1.6-jsonl-event-log.md) | JSONL event log writer | M | — | ⬜ | +| [M1.7](M1.7-ingest-end-to-end.md) | `mem ingest` end to end | M | — | ⬜ | +| [M1.8](M1.8-m1-gate.md) | **M1 composition gate** | M | gate | ⬜ | + +## 3 — Projections · M2.x + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M2.1](M2.1-embeddings-client.md) | Embeddings client | S | — | ⬜ | +| [M2.2](M2.2-memory-db-manifest.md) | CNPG `memory-db` + pgvector | M | homelab | ⬜ | +| [M2.3](M2.3-schema-and-migrations.md) | Schema + sqlx migrations | M | — | ⬜ | +| [M2.4](M2.4-pgvector-repo.md) | pgvector repository | M | — | ⬜ | +| [M2.5](M2.5-obsidian-projector.md) | Obsidian projector | M | — | ⬜ | +| [M2.6](M2.6-rebuild-from-log.md) | `mem rebuild --from-log` | M | — | ⬜ | +| [M2.7](M2.7-verify-edges.md) | `mem verify` — edge closure | S | — | ⬜ | +| [M2.8](M2.8-m2-gate.md) | **M2 composition gate** | M | gate | ⬜ | + +## 4 — L2 synthesis and retrieval · M3.x + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M3.1](M3.1-l2-synthesis.md) | L2 synthesis pass | M | — | ⬜ | +| [M3.2](M3.2-rerank-client.md) | Rerank client | S | — | ⬜ | +| [M3.3](M3.3-mem-query.md) | `mem query` with provenance | M | — | ⬜ | +| [M3.4](M3.4-m3-gate.md) | **M3 composition gate** | M | gate | ⬜ | + +## 5 — Skills · M4.x + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M4.1](M4.1-skill-draft.md) | `mem skill draft` | M | — | ⬜ | +| [M4.2](M4.2-derived-filter.md) | `derived: true` ingest filter | M | — | ⬜ | +| [M4.3](M4.3-m4-gate.md) | **M4 composition gate** | M | gate | ⬜ | + +## 6 — Post-training · M5.x + +Python, separate from the Rust workspace. The boundary is the JSONL log. + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M5.1](M5.1-evidence-labeler.md) | `mem label` — evidence labeler | M | — | ⬜ | +| [M5.2](M5.2-labeler-calibration.md) | Labeler calibration | M | — | ⬜ | +| [M5.3](M5.3-training-corpus-export.md) | Training corpus export | M | — | ⬜ | +| [M5.4](M5.4-vllm-lora-serving.md) | vLLM + `--enable-lora` | L | homelab | ⬜ | +| [M5.5](M5.5-verl-training-loop.md) | verl training loop | L | — | ⬜ | +| [M5.6](M5.6-m5-gate.md) | **M5 composition gate** | L | gate | ⬜ | + +## 7 — agent-manager migration · M6.x + +Separate repo (`github.com/Riotpiaole/agent-manager`, fork branch +`add-headless-spawn`), separate cluster resource, no Rust/GRU-Mem +dependency. Moves its session store off local sqlite onto a dedicated CNPG +Postgres, reachable from the Mac client through a dedicated nginx route — +durability-of-location, not a multi-host requirement. + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M6.1](M6.1-agent-manager-db-manifest.md) | CNPG `agent-manager-db` manifest | M | homelab | ⬜ | +| [M6.2](M6.2-schema-port.md) | Postgres schema for agent-manager sessions | M | — | ⬜ | +| [M6.3](M6.3-store-query-port.md) | store.go query port to Postgres | L | — | ⬜ | +| [M6.4](M6.4-nginx-stream-routing.md) | nginx TCP routing to `agent-manager-db` | S | homelab | ⬜ | +| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ | +| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ | + +--- + +Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch) diff --git a/tasks/M0.1-cargo-workspace.md b/tasks/M0.1-cargo-workspace.md new file mode 100644 index 0000000..1b401ee --- /dev/null +++ b/tasks/M0.1-cargo-workspace.md @@ -0,0 +1,98 @@ +# M0.1 — Cargo workspace + crate skeletons + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | — | + +## Goal + +The six-crate workspace, building clean, with the dependency direction fixed +before any code exists to violate it. + +## Facts (inlined — no spec read needed) + +``` +crates/ + mem-core/ domain types; Level; gate-response parser; the gated loop + mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger + mem-llm/ gateway client — chat, embeddings, rerank + mem-ingest/ source adapters: pi sessions, claude transcripts + mem-store/ JSONL log; pgvector repo; Obsidian projector + mem-cli/ binary `mem` +``` + +Dependency direction, enforced from the start: + +``` +mem-cli -> mem-ingest, mem-store, mem-llm, mem-chunk, mem-core +mem-store -> mem-core +mem-ingest -> mem-chunk, mem-core +mem-chunk -> mem-core +mem-llm -> mem-core +mem-core -> (nothing in this workspace) +``` + +`mem-core` depends on no sibling. It holds the types every other crate speaks, +so a dependency out of it is a cycle waiting to happen. + +Workspace deps to pin now, so versions do not drift per crate: `tokio`, +`futures`, `serde`, `serde_json`, `serde_yaml`, `anyhow`, `thiserror`, +`sha2`, `clap`, `reqwest`, `tracing`. + +## Steps + +1. `Cargo.toml` at the repo root with `[workspace] members = [...]` and a + `[workspace.dependencies]` block holding every shared crate version. +2. Six member crates, each declaring deps as `foo.workspace = true`. +3. `mem-cli` is the only `[[bin]]`; the rest are libraries. +4. Add `rust-toolchain.toml` pinning a version, so CI and laptop agree. +5. `.gitignore`: `target/`, `vault/` — but **not** `log/` and **not** + `memory-tasks/`. The log is authoritative and the board carries acceptance + criteria; both are tracked. +6. Wire a CI job running `cargo build --workspace` and `cargo clippy --workspace + -- -D warnings`. + +## Acceptance + +- `cargo build --workspace` succeeds from a clean checkout. +- `cargo clippy --workspace -- -D warnings` is clean. +- `mem-core` has zero intra-workspace dependencies. + +## Verify + +**Harness:** cargo itself, plus a dependency assertion that does not trust the +manifests to be read by a human. + +**Integration test** — `tests/it_workspace.rs` in the root: +1. `a1_all_members_build` — shell out to `cargo build --workspace`, assert exit 0. +2. `a2_mem_core_has_no_sibling_deps` — parse `crates/mem-core/Cargo.toml`, assert + no dependency name starts with `mem-`. +3. `a3_dependency_direction` — parse every member manifest, build the edge set, + assert it is a subset of the table above and that the graph is acyclic. +4. `a4_log_and_tasks_are_tracked` — assert `.gitignore` matches neither + `log/` nor `memory-tasks/`. + +**Command:** `cargo test --workspace workspace` + +**False pass:** +- Asserting the graph is acyclic **only**. Acyclic permits `mem-core -> + mem-store`, which is backwards and still acyclic. Assertion 3 must check the + edge set against the table, not just for cycles. +- A CI job that runs `cargo build` in the root without `--workspace`. It builds + the virtual manifest and can miss a member that does not compile. + +## Traps + +- Per-crate dependency versions instead of `workspace.dependencies`. They drift, + and two versions of `serde` in one tree is a confusing type error much later. +- Gitignoring `log/`. It is the authoritative record; ignoring it makes the whole + authority model a fiction. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Repository layout diff --git a/tasks/M0.2-domain-types.md b/tasks/M0.2-domain-types.md new file mode 100644 index 0000000..ee9c3df --- /dev/null +++ b/tasks/M0.2-domain-types.md @@ -0,0 +1,115 @@ +# M0.2 — Domain types and sha256 identity + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.1 | + +## Goal + +The vocabulary every other crate speaks, and the content-hash identity the whole +provenance graph hangs on. + +## Facts (inlined — no spec read needed) + +```rust +/// Closed. L0 evidence, L1 per-query memory, L2 project synthesis. +#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum Level { L0, L1, L2 } + +/// A normalised unit from any source. Adapters produce these; nothing +/// downstream learns whether it came from pi, claude, or a socket. +pub struct Record { + pub role: Role, // User | Assistant | ToolResult | System + pub text: String, + pub timestamp: OffsetDateTime, + pub provenance: Provenance, // source id + offset within it +} + +/// One or more Records, under the token budget, never split mid-Record. +pub struct Chunk { + pub t: u32, // 1-based turn index within a run + pub records: Vec, + pub tokens: usize, + pub sha256: Sha256Hash, +} + +pub struct MemoryNode { + pub level: Level, + pub project: ProjectId, + pub query_id: Option, // None at L2 + pub run_id: RunId, + pub t: u32, + pub text: String, + pub sha256: Sha256Hash, + pub parents: Vec, +} +``` + +**Identity is the content hash, not a counter.** `sha256` is computed over the +canonical serialization of the semantic content — for `Chunk`, the concatenated +record texts and their provenance; for `MemoryNode`, `(level, project, query_id, +text)`. It must **not** include the timestamp or the run id, or re-running the +same input produces different hashes and `mem rebuild` stops being idempotent. + +Newtypes with no `Default`: `ProjectId`, `QueryId`, `RunId`, `Sha256Hash`. A +placeholder that type-checks is invisible — a hardcoded `"current"` compiles, +passes tests, and makes every downstream result unattributable. + +## Steps + +1. Declare `Level`, `Role`, `Record`, `Provenance`, `Chunk`, `MemoryNode`. +2. Declare the newtypes. None derives `Default`. None has `From` without + validation. +3. `fn content_hash(&self) -> Sha256Hash` on `Chunk` and `MemoryNode`, over a + canonical byte encoding that excludes timestamps and run ids. +4. `Level` serializes as the literal strings `"L0" | "L1" | "L2"` — the JSONL and + the SQL `CHECK` constraint both depend on that spelling. +5. Round-trip serde tests for every type. + +## Acceptance + +- Two `Chunk`s built from identical records in different runs hash identically. +- Changing one character of any record text changes the hash. +- `Level` round-trips through JSON as `"L0"`, not `0` and not `"l0"`. + +## Verify + +**Harness:** unit tests in `mem-core`, plus a hash-stability fixture committed +as bytes. + +**Integration test** — `tests/it_identity.rs`: +1. `a1_same_content_same_hash` — build the same chunk twice with different + `RunId` and timestamps, assert equal hashes. +2. `a2_text_change_changes_hash` — flip one byte, assert the hash differs. +3. `a3_level_wire_format` — `serde_json::to_string(&Level::L0) == "\"L0\""`. +4. `a4_hash_stability_across_versions` — hash a committed fixture record set, + assert it equals a hash literal written into the test. This catches a + canonicalization change that would silently orphan every stored node. +5. `a5_newtypes_have_no_default` — compile-fail test (`trybuild`) asserting + `ProjectId::default()` does not compile. + +**Command:** `cargo test -p mem-core identity` + +**False pass:** +- Asserting only that hashing is deterministic *within one process*. A hash that + includes the timestamp is deterministic per run and still breaks rebuild. + Assertion 1 must vary the run id and timestamp deliberately. +- Omitting assertion 4. Without a committed expected hash, any future + canonicalization change passes every other test and silently invalidates the + database. + +## Traps + +- Including `run_id` or `created_at` in the hash. Rebuild then produces new nodes + every time and `memory_edge` accumulates orphans. +- Deriving `Default` on an id newtype "for tests". That default reaches + production and every row attributes to the same fake project. + +--- + +Background: [DESIGN.md](../DESIGN.md) — The tier model, Storage schemas diff --git a/tasks/M0.3-recordsource-and-chunkpolicy.md b/tasks/M0.3-recordsource-and-chunkpolicy.md new file mode 100644 index 0000000..4768f1b --- /dev/null +++ b/tasks/M0.3-recordsource-and-chunkpolicy.md @@ -0,0 +1,106 @@ +# M0.3 — `RecordSource` trait + `ChunkPolicy` + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.2 | + +## Goal + +The seam where new input kinds arrive, shaped as a stream from the first commit +so a future streaming source implements a trait instead of forcing a rewrite. + +## Facts (inlined — no spec read needed) + +```rust +pub trait RecordSource { + /// Sources decide how to produce records; the chunker never learns + /// whether they came from pi, claude, or a socket. + fn records(self) -> impl Stream>; +} + +pub struct ChunkPolicy { + pub max_tokens: usize, // 5000 — GRU-Mem paper default + pub split_on: Boundary, // Boundary::Record — never mid-Record + pub flush: FlushTrigger, +} + +pub enum FlushTrigger { + Tokens(usize), + // OrIdle(Duration) lands with the first streaming source. Carrying the + // enum now means that change is one variant, not a signature change + // threaded through the loop. +} + +pub fn chunks(src: S, p: ChunkPolicy) -> impl Stream; +``` + +Why a stream when both current sources are files: sources today have an EOF; +telemetry, a live session tail, or a broker will not. Batch sources become +streams for free via `futures::stream::iter`, so this costs nothing today and +removes a rewrite later. The rest of the stack is already tokio. + +**A single `Record` larger than `max_tokens` is not an error.** Tool results can +be enormous. It becomes a chunk of one, over budget, and the chunker records that +it did — silently truncating would destroy evidence, and silently dropping would +lose it. + +## Steps + +1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`. +2. Implement `chunks()` as a `Stream` adapter that accumulates until the next + record would exceed `max_tokens`, then yields. +3. Never split a `Record`. An oversized single record yields alone, with + `Chunk::over_budget = true`. +4. `t` is 1-based and contiguous across the whole stream. +5. Provide `VecSource(Vec)` implementing `RecordSource` for tests, so + chunking is testable with no I/O and no model. +6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one; + a `CharsOverFour` stub is enough here. + +## Acceptance + +- Chunk boundaries never fall inside a `Record`. +- Chunk `t` values are 1-based, contiguous, no gaps. +- An oversized single record yields one chunk flagged `over_budget`. +- Concatenating all chunks' records reproduces the input sequence exactly. + +## Verify + +**Harness:** `VecSource` + the stub counter. No files, no network. + +**Integration test** — `tests/it_chunking.rs`: +1. `a1_no_record_is_split` — for every chunk, every record equals some input + record byte for byte. +2. `a2_lossless` — flatten all chunk records; assert the sequence equals the + input sequence, same order, same length. +3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`. +4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly + one record and `over_budget = true`. +5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it + appears whole in the output, not truncated and not dropped. +6. `a6_empty_source` — zero records yields zero chunks, no panic. + +**Command:** `cargo test -p mem-chunk chunking` + +**False pass:** +- Testing only with uniformly small records. The budget logic is never exercised + and the oversized path never runs. Assertion 5 is the guard. +- Asserting chunk count rather than losslessness. A chunker that drops the final + partial chunk produces a plausible count and loses the tail — assertion 2 is + what catches it. + +## Traps + +- Truncating an oversized record to fit the budget. That is silent evidence + destruction, and the update gate will later be blamed for missing it. +- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes + every test here, and defeats the entire reason this crate is separate. + +--- + +Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one diff --git a/tasks/M0.4-tokenizer-sizing.md b/tasks/M0.4-tokenizer-sizing.md new file mode 100644 index 0000000..a9af470 --- /dev/null +++ b/tasks/M0.4-tokenizer-sizing.md @@ -0,0 +1,92 @@ +# M0.4 — Tokenizer-backed chunk sizing + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.3 | + +## Goal + +Count tokens with the tokenizer the serving model actually uses, so a chunk that +fits locally also fits at the gateway. + +## Facts (inlined — no spec read needed) + +The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem +paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE. + +Budget arithmetic, and why an approximation is not good enough: + +``` +Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side) +chunk 5000 +prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024) +response 2048 +``` + +A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus. +Undercount and the gateway rejects the request; overcount and chunks are smaller +than they need to be, which multiplies the number of model calls. + +`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored +`tokenizer.json`. **Vendor the file into the repo** rather than downloading at +runtime: a build that needs the network is a build that fails offline, and a +tokenizer that changes under you silently re-chunks the entire corpus. + +## Steps + +1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo. +2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing. +3. Assert at load that the vendored file's hash matches the recorded one. +4. Make `max_tokens` and the model id configurable, defaulting to 5000 and + `qwen2.5:3b-instruct`. +5. Add a `mem tokens ` debug subcommand printing the token count of a file, + for cross-checking against the gateway's reported `prompt_tokens`. + +## Acceptance + +- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of + real records. +- Loading with a modified tokenizer file fails loudly, not silently. + +## Verify + +**Harness:** the vendored tokenizer plus recorded gateway responses. The +cross-check against the live gateway is a separate, network-gated test. + +**Integration test** — `tests/it_tokens.rs`: +1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts + (ASCII, CJK, code, JSON, emoji), asserted exactly. +2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an + error naming the file. +3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10 + real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare + `usage.prompt_tokens` to the local count minus the measured template overhead; + assert within 2%. +4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's + locally-counted size exceeds the budget. + +**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3) + +**False pass:** +- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and + this corpus contains both — the caveman skill text alone is bilingual. +- Comparing the local count to itself via a helper that calls the same function. + Assertion 1's expected values must be recorded from the tokenizer once and + written as literals. + +## Traps + +- Downloading the tokenizer at runtime. Offline builds break, and a silent + upstream change re-chunks everything, which changes every chunk hash, which + orphans every stored node. +- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for + ornith and is wrong; do not take a client-side config as the source of truth. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk` diff --git a/tasks/M0.5-pi-session-adapter.md b/tasks/M0.5-pi-session-adapter.md new file mode 100644 index 0000000..b6610ad --- /dev/null +++ b/tasks/M0.5-pi-session-adapter.md @@ -0,0 +1,131 @@ +# M0.5 — pi session adapter + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.3 | + +## Goal + +Turn pi's session JSONL into normalised `Record`s, and resolve the project key +from the path without guessing. + +## Facts (inlined — no spec read needed) + +Layout, verified on this machine: + +``` +~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/_.jsonl + └─ cwd with / replaced by -, wrapped in leading and trailing -- +``` + +Record types observed in a real 2902-message session: + +``` +message 2902 +model_change 112 +thinking_level_change 16 +compaction 8 +session 1 <- always first line +``` + +Shapes: + +```jsonc +// first line +{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"} +// message +{"type":"message","id":"..","parentId":"..","timestamp":"..", + "message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}} +``` + +Role distribution in that same session — this is the whole reason the update gate +exists: + +``` +assistant 1445 +toolResult 1261 43%, mostly evidence-free +user 196 +``` + +`cwd` in the `session` header is authoritative for the project key. The directory +name is a lossy encoding (a real `-` in a path is indistinguishable from a +separator) — **parse `cwd`, do not decode the directory name.** + +`content` is not always a string. Assistant messages carry content blocks; tool +results carry structured payloads. Normalise to text, and keep the block type in +`Provenance` so a later filter can act on it. + +## Steps + +1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`. +2. Read the first line, require `type == "session"`, take `cwd` as the project + key. A file whose first line is not a session header is an error naming the + file, not a skip. +3. Stream subsequent lines; emit a `Record` per `type == "message"`. +4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`, + `toolResult -> Role::ToolResult`. +5. Flatten `content` to text for all shapes; preserve the original block type in + `Provenance`. +6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** — + emit it as `Role::System` with the marker text, because a compaction boundary + is where context was lost and that is worth seeing in the log. +7. `Provenance` = `pi:` plus the record's `id` and line offset. +8. A malformed line is a counted, reported skip — never a panic. These files are + appended to by a live process and the last line may be a partial write. + +## Acceptance + +- Project key comes from `cwd`, matching for a path containing a literal `-`. +- All three roles are emitted with correct counts on a real session. +- A truncated final line is skipped with a warning, not a panic. +- Compaction events appear in the record stream. + +## Verify + +**Harness:** two committed fixtures — one small hand-built session covering every +record type and content shape, one real session copied verbatim (secrets +scrubbed) for volume. + +**Integration test** — `tests/it_pi_source.rs`: +1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the + key is `/tmp/my-project`, proving the directory name was not decoded. +2. `a2_role_counts` — on the real fixture, assert exact counts per role. +3. `a3_content_shapes` — string content, block-array content, and structured tool + result all flatten to non-empty text. +4. `a4_truncated_tail` — append half a JSON object; assert the source yields all + prior records and reports exactly one skip. +5. `a5_missing_header` — file whose first line is a `message`; assert an error + naming the path. +6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`. +7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first + record before reading the whole file (assert peak allocation, or instrument + reads). + +**Command:** `cargo test -p mem-ingest pi_source` + +**False pass:** +- Testing only against the hand-built fixture. It will contain the content shapes + you thought of, which is the set you already handle. Assertion 2 against a real + session is what finds the rest. +- Asserting "no panic" on malformed input without asserting the *count* of + skips. A source that silently drops every line panics never and ingests + nothing. + +## Traps + +- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--` + is ambiguous the moment a path component contains `-`, which is common. +- Treating `content` as `String`. It parses for user messages and fails on + assistant blocks, so the bug appears to be about assistants specifically and + wastes an afternoon. +- Materializing the file. These reach tens of MB and the trait is a stream for a + reason. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk` diff --git a/tasks/M0.6-claude-transcript-adapter.md b/tasks/M0.6-claude-transcript-adapter.md new file mode 100644 index 0000000..74c2b63 --- /dev/null +++ b/tasks/M0.6-claude-transcript-adapter.md @@ -0,0 +1,92 @@ +# M0.6 — Claude transcript adapter + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.5 | + +## Goal + +The second `RecordSource`, which is the one that proves the trait is real. + +## Facts (inlined — no spec read needed) + +``` +~/.claude/projects/-Users-rockliang-workplace-Poimen/.jsonl + └─ cwd, / replaced by -, no wrapping dashes (differs from pi) +``` + +15 MB across 7 transcripts on this machine. Record types observed: + +``` +attachment 150 queue-operation 138 assistant 134 user 81 +file-history-snapshot 69 system 69 ai-title 21 last-prompt 21 mode 21 +``` + +Fields, read from any line: `sessionId`, `cwd`, `gitBranch`. +Typed records dispatch on `type`; the ones that matter here: + +- `user` / `assistant` — content under `message.content` +- `system` with `subtype == "api_error"` — a failed turn, worth keeping +- `summary` — carries `summary` +- everything else — ignore + +The filename stem is a UUID. The encoding differs from pi (no wrapping `--`), +which is exactly why the project key must come from the `cwd` **field**, as in +M0.5, and not from the directory name. + +## Steps + +1. Implement `ClaudeTranscriptSource` in `mem-ingest`, implementing `RecordSource`. +2. Take `cwd` from the first line carrying it; error if no line does. +3. Emit records for `user`, `assistant`, and `system`+`api_error`. +4. Flatten `message.content` across its shapes, as in M0.5. +5. `Provenance` = `claude:` plus line offset. +6. Reuse the content-flattening and malformed-line handling from M0.5 — extract + them into a shared helper rather than copying, since divergence between two + flatteners is a bug that only shows up on one source. + +## Acceptance + +- Both sources satisfy `RecordSource` with no changes to `mem-chunk`. +- Project keys from pi and claude for the same directory resolve to the same + `ProjectId`. +- `api_error` system records survive into the stream. + +## Verify + +**Harness:** a committed transcript fixture plus a real one, secrets scrubbed. + +**Integration test** — `tests/it_claude_source.rs`: +1. `a1_project_from_cwd_field` — assert the key comes from `cwd`, not the + directory name, using a fixture where they would differ. +2. `a2_same_project_across_sources` — a pi session and a claude transcript for + the same directory yield an equal `ProjectId`. This is the assertion that + makes cross-source memory possible at all. +3. `a3_role_mapping` — user/assistant/api_error appear; `attachment`, + `queue-operation`, `file-history-snapshot` do not. +4. `a4_shared_flattener` — the same content-block fixture flattens identically + through both sources (call both, compare strings). + +**Command:** `cargo test -p mem-ingest claude_source` + +**False pass:** +- Testing each source in isolation. The point of this task is that they agree; + assertions 2 and 4 are the only ones that check it, and both are cross-source. + +## Traps + +- Copying the flattener instead of sharing it. The two will drift, and the + resulting bug looks like "claude transcripts lose tool output" rather than + "there are two flatteners". +- Assuming the pi encoding. Claude has no wrapping `--`, so a decoder written + against pi silently produces a different project key for the same directory — + and cross-source memory quietly splits in two. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Context diff --git a/tasks/M0.7-ingest-dry-run.md b/tasks/M0.7-ingest-dry-run.md new file mode 100644 index 0000000..7cdf59a --- /dev/null +++ b/tasks/M0.7-ingest-dry-run.md @@ -0,0 +1,94 @@ +# M0.7 — `mem ingest --dry-run` + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.4, M0.6 | + +## Goal + +See the chunk plan for a real project before spending a single model call on it. + +## Facts (inlined — no spec read needed) + +`--dry-run` makes **zero network calls**. That is the property, not a side +effect: this is the last checkpoint before inference, and its value is telling +you the corpus is sane while feedback is still free. + +Output shape: + +``` +project /Users/rockliang/workplace/Poimen/agent-rust +sources pi:4 files claude:2 files +records 3118 (assistant 1445 toolResult 1261 user 196 system 216) +chunks 412 over-budget 3 +tokens min 84 p50 4870 p95 5000 max 11204 +``` + +`over-budget` counts single records exceeding the chunk budget (M0.3). A nonzero +count is expected — large tool results — and is worth surfacing because it +predicts requests the gateway may reject. + +Numbers to sanity-check against: a real pi session in this project has 2902 +messages with the role split above, and toolResult being ~43% is the signal that +the update gate has something to discriminate. + +## Steps + +1. `mem ingest --project --dry-run`. +2. Resolve sources: scan both source roots for directories whose `cwd` matches + the project. Report which files matched. +3. Stream records through `mem-chunk` with the real tokenizer; accumulate stats + without retaining chunk bodies. +4. Print the table above. Machine-readable variant behind `--format json`. +5. `--limit ` to stop after n chunks, for iterating on a large project. +6. Exit non-zero if zero sources matched — a silent empty plan reads like success. + +## Acceptance + +- No network syscall occurs during `--dry-run`. +- Stats are computed streaming; memory does not scale with corpus size. +- Zero matched sources exits non-zero with a message naming the project key. + +## Verify + +**Harness:** the fixtures from M0.5/M0.6, plus a network guard. + +**Integration test** — `tests/it_dry_run.rs`: +1. `a1_no_network` — run the command with outbound TCP blocked (inject a + `reqwest` client that panics on use, or set an unroutable proxy); assert exit 0. +2. `a2_counts_match_sources` — record and role counts equal the sum of what the + two adapters yield independently. +3. `a3_chunk_count_matches_chunker` — the reported chunk count equals + `chunks(...).count()` computed separately. +4. `a4_over_budget_reported` — fixture with one oversized record; assert + `over-budget 1`. +5. `a5_empty_project_fails` — unknown project key exits non-zero, message names + the key. +6. `a6_constant_memory` — run against a 50 MB fixture; assert peak RSS stays + under a bound well below file size. + +**Command:** `cargo test -p mem-cli dry_run` + +**False pass:** +- Asserting the command exits 0 and printing looks right. A dry run that matched + no sources also exits 0 and prints a tidy table of zeros — assertion 5 is what + separates them. +- Computing stats by collecting chunks into a `Vec` first. Every assertion here + passes and assertion 6 is the only one that fails, which is why it is present. + +## Traps + +- Making `--dry-run` construct the LLM client "but not call it". Construction + reads credentials and can fail; the guarantee is no network, and the cheapest + way to keep it is to not build the client at all. +- Reporting p50 chunk size only. The max is the interesting number — it predicts + which requests will be rejected downstream. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verification, P1 diff --git a/tasks/M0.8-m0-gate.md b/tasks/M0.8-m0-gate.md new file mode 100644 index 0000000..feb1198 --- /dev/null +++ b/tasks/M0.8-m0-gate.md @@ -0,0 +1,87 @@ +# M0.8 — M0 composition gate + +| Field | Value | +|---|---| +| Phase | M0 — Read-only spine | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M0 | + +## Goal + +Prove the parts compose and that the swappable part is genuinely swappable — +the properties no single M0 task owns. + +## Facts (inlined — no spec read needed) + +The claim this phase makes: **a new input kind is added by implementing +`RecordSource`, and nothing downstream changes.** If that is false, the crate +split bought nothing and the streaming work later will be a rewrite. + +The gate proves it by adding a third source that resembles nothing already +supported, and asserting the rest of the pipeline is untouched. + +Second claim: the whole phase is free. No model calls, so the gate is a CI job +that runs on every push without a gateway or credentials. + +## Steps + +1. Implement `SyntheticSource` in the test tree only — generates records from a + seed, has no file format, and deliberately produces a record larger than the + chunk budget and a run of empty-text records. +2. Assert `chunks()` handles it with **no change** to `mem-chunk`. +3. Assert dependency direction still holds (M0.1 assertion 3) after three sources + exist — this is when someone is tempted to reach backwards. +4. Run `mem ingest --dry-run` over all three sources for one project and diff the + summary against a committed expected file. +5. Wire the whole thing as a required CI job. + +## Acceptance + +- Three sources, one `RecordSource`, zero source-specific branches in `mem-chunk` + or `mem-cli`. +- Dry-run summary diffs empty against the committed expectation. +- CI job passes with no network and no credentials. + +## Verify + +**Harness:** committed expected-output file, diffed. The script's output is the +review artifact. + +**Integration test** — `tests/it_m0_gate.rs`: +1. `a1_third_source_needs_no_downstream_change` — `SyntheticSource` flows through + `chunks()`; assert `git diff --stat crates/mem-chunk crates/mem-cli` is empty + for the commit that added it (enforced by a CI step, not by the test binary). +2. `a2_no_source_specific_branches` — grep `crates/mem-chunk` and `crates/mem-cli` + for the strings `pi:`, `claude:`, `sessionId`, `toolResult`; assert none + appear outside `mem-ingest`. +3. `a3_dry_run_golden` — run the dry run over all three sources, diff against + `expected/m0-gate.txt`; empty diff is the only pass. +4. `a4_offline` — the entire gate runs with networking disabled. +5. `a5_lossless_end_to_end` — records in equals records out, across all three + sources composed. + +**Command:** `cargo test --workspace m0_gate` + +**False pass:** +- Adding `SyntheticSource` in a way that mirrors the pi format. It then exercises + the same code path and proves nothing about generality. It must have no file, + no header line, and an oversized record. +- Assertion 3 passing because the expected file was regenerated in the same + commit. A changed `expected/` file in a diff is a claim that the contract + changed, and must be reviewed as one. + +## Traps + +- Skipping assertion 2 because "obviously there are no source-specific branches". + There will be — the first `if provenance.starts_with("pi:")` gets added to fix + a real bug and is entirely reasonable in isolation. +- Letting the gate need credentials. The value of an offline gate is that it runs + on every push; the moment it needs a gateway key it becomes a nightly job that + nobody watches. + +--- + +Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, Verification diff --git a/tasks/M1.1-llm-chat-client.md b/tasks/M1.1-llm-chat-client.md new file mode 100644 index 0000000..cf2f8b3 --- /dev/null +++ b/tasks/M1.1-llm-chat-client.md @@ -0,0 +1,107 @@ +# M1.1 — `mem-llm` chat client + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.1 | + +## Goal + +Talk to the homelab gateway, with the two non-obvious details that cost a day to +find already baked in. + +## Facts (inlined — no spec read needed) + +``` +base https://api.riotpiao.com/v1 +route POST /v1/qwen/chat/completions qwen2.5:3b-instruct + POST /v1/ornith/chat/completions ornith:35b + POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools) +``` + +**Auth is `apikey:`, not `Authorization: Bearer`.** Kong's `key-auth` compares +the whole header value against the stored key, so the OpenAI SDK convention +returns 401. Verified: + +``` +-H "apikey: $KEY" -> 200 +-H "Authorization: $KEY" -> 200 +-H "Authorization: Bearer $KEY" -> 401 +``` + +**One provider per route.** `baseUrl` is per-route and the model id is `ornith:35b` +with the tag, not `ornith`. `/v1/models` advertises `/v1/score` which does not +work — do not trust that list as a capability probe. + +Request bodies above ~10.6 KB used to fail with +`{"error":{"message":"[] is too short - 'messages'"}}`; the Kong body buffer was +raised to 16m and it is fixed. If that error ever reappears, it is the buffer, +not the client. + +Send **no `tools` array**. The controller needs none, and the reasoning route +rejects any request carrying one. + +## Steps + +1. `ChatClient::new(base_url, api_key, model)` in `mem-llm`. +2. Send the `apikey` header. Read the key from `MEM_API_KEY`, never from a + committed file. +3. `complete(system, user, max_tokens) -> Completion { text, usage, latency }`. +4. Timeout default 300s — local models are slow to first token and a cold load + can take minutes. +5. Retry on 5xx and timeout with exponential backoff, max 3. **Do not retry 4xx** + — a 400 is a malformed request and retrying it just costs three times as much. +6. On any error, include the response body in the error. The useful information + is always in the body, never the status. +7. `MEM_LLM_RECORD=` writes every request/response pair to disk, for + building fixtures without hand-writing them. + +## Acceptance + +- A real completion round-trips against the gateway. +- A 401 is reported as an auth error naming the header convention. +- A 4xx is not retried; a 5xx is. + +## Verify + +**Harness:** `wiremock` for the offline tests; one `#[ignore]` test against the +live gateway. + +**Integration test** — `tests/it_chat_client.rs`: +1. `a1_sends_apikey_header` — assert the mock received `apikey` and **no** + `authorization` header. +2. `a2_no_tools_field` — assert the serialized body has no `tools` key at all, + not merely an empty array. +3. `a3_retries_5xx` — mock 503 twice then 200; assert 3 requests and success. +4. `a4_does_not_retry_4xx` — mock 400; assert exactly 1 request and an error + carrying the body text. +5. `a5_timeout_is_configurable` — mock a 2s delay with a 1s timeout; assert a + timeout error. +6. `a6_live_smoke` — `#[ignore]`; real gateway, `qwen2.5:3b-instruct`, prompt + "reply with exactly: pong", assert the text contains `pong`. + +**Command:** `cargo test -p mem-llm chat_client` (add `-- --ignored` for a6) + +**False pass:** +- Testing only against the mock. The mock accepts whatever header you send it; + assertion 6 against the live gateway is the only thing that proves the auth + convention is right. +- Asserting `tools: []` is absent by checking `body.tools.is_empty()`. An empty + array serialized into the request is still a `tools` key, and that is what the + reasoning route rejects. Assert on the raw JSON. + +## Traps + +- Using `Authorization: Bearer`. It is what every SDK does and it 401s here. +- Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it + tripled the load and produced identical failures more slowly. +- A 60s timeout. That is the value that made `ornith` look broken when it was + merely cold. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verified facts diff --git a/tasks/M1.2-standing-query-loader.md b/tasks/M1.2-standing-query-loader.md new file mode 100644 index 0000000..5141470 --- /dev/null +++ b/tasks/M1.2-standing-query-loader.md @@ -0,0 +1,103 @@ +# M1.2 — Standing-query YAML loader + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M0.2 | + +## Goal + +Load the standing questions that give the update gate its referent, and fail at +load rather than mid-run when one is wrong. + +## Facts (inlined — no spec read needed) + +```yaml +# queries/poimen.yaml +project: poimen +roots: + - /Users/rockliang/workplace/Poimen/agent-rust # matched against session cwd +sources: [pi, claude] +queries: + - id: architecture-decisions + question: What architectural decisions were made, with reasoning and rejected alternatives? + - id: infra-root-causes + question: What infrastructure bugs were found, what was the root cause, how was it isolated? +synthesis: + question: What is the current state of this project, and what should someone know before working on it? + exit_gate: true +defaults: + memory_budget: 1024 + chunk_tokens: 5000 + exit_gate: false # L1 default — see below +``` + +**Why a question is mandatory.** The GRU-Mem memory agent is `φθ(Q, C_t, M_{t-1})` +and its update gate is defined as "does this chunk contain useful information +*about the problem*". With no `Q` the gate has no referent, and `r_update` is +undefinable — which forecloses post-training (M5) entirely. A query with an empty +question is a load error, not a warning. + +**`exit_gate: false` at L1 is deliberate.** Paper §3.3: for "what are *all* the +X" questions you cannot know evidence is sufficient without reading everything, +so the paper itself provides a without-exit-gate inference mode. L1 extraction is +that shape. The gate is still *recorded* — its signal is needed for M5. + +`query.id` is stable and frozen. It names the L1 memory, the Obsidian note, and +the log directory; renaming it orphans all three. + +## Steps + +1. `QuerySet::load(path)` in `mem-core`, `serde_yaml`. +2. Validate at load: non-empty `project`; at least one query; every `id` unique, + non-empty, and `[a-z0-9-]+`; every `question` non-empty; `memory_budget > 0`. +3. Every failure names the file, the query id, and the field. +4. `mem query validate ` prints the resolved set and exits non-zero on any + error. +5. Defaults apply per query and are overridable per query. +6. `id` collision across two files for the same project is an error. + +## Acceptance + +- An empty or missing `question` fails at load with a message naming the id. +- An id outside `[a-z0-9-]+` fails at load — it becomes a filename. +- Defaults resolve; per-query overrides win. + +## Verify + +**Harness:** table-driven over fixture YAML files, one per failure mode. + +**Integration test** — `tests/it_query_loader.rs`: +1. `a1_valid_loads` — the reference file above resolves to the expected struct. +2. `a2_empty_question_rejected` — error text contains the query id. +3. `a3_duplicate_id_rejected` — error names both occurrences. +4. `a4_bad_id_charset_rejected` — `infra/root causes` is rejected, message + mentions the filename constraint. +5. `a5_defaults_and_overrides` — a query without `exit_gate` gets `false`; one + with `true` keeps it. +6. `a6_l1_exit_gate_defaults_false` — assert explicitly, because a silent flip to + `true` truncates every extraction and looks like a model quality problem. +7. `a7_missing_question_field` — absent key behaves as empty, same error. + +**Command:** `cargo test -p mem-core query_loader` + +**False pass:** +- Testing only the happy path. Every assertion except 1 and 5 is a rejection + test, and rejection is the entire point of load-time validation. +- Asserting an error occurred without asserting the message names the offending + id. "invalid config" sends someone to read the whole file by hand. + +## Traps + +- Allowing an empty question "for now". It loads, the gate has no referent, the + model updates on nearly everything, and it reads as a bad model rather than a + bad config. +- Letting `id` contain `/` or spaces. It is a path segment in three places. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Standing queries diff --git a/tasks/M1.3-prompt-template.md b/tasks/M1.3-prompt-template.md new file mode 100644 index 0000000..aa8842a --- /dev/null +++ b/tasks/M1.3-prompt-template.md @@ -0,0 +1,120 @@ +# M1.3 — GRU-Mem prompt template + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.2 | + +## Goal + +Assemble the memory-agent prompt exactly as the paper specifies, because the +model's ability to emit parseable gates depends on the format it was aligned to. + +## Facts (inlined — no spec read needed) + +Paper Figure 10a, reproduced verbatim — this is the contract, not a starting +point to improvise on: + +``` +You are presented with a problem, a section of an article that may contain the +answer to the problem, and a previous memory. Please read the provided section +carefully. You should reason about whether the new section contains useful +information about the problem, and then update the memory with the new +information that helps to answer the problem. +Be sure to retain all relevant details from the previous memory while adding any +new, useful information. You should also carefully judge whether you have +collected enough information to answer the problem. +You should reason about whether the new section contains useful information, what +to update, and what to do next first between and . +If the new section contains useful information about the problem, you should +first generate yes. After that, update the new memory between + and . +If the new section does not contain useful information about the problem, you +should first generate no. After that, you should keep the previous +memory unchanged between and . +In the end, if you haven't collected enough information for the problem, return +continue. ONLY when enough information is collected, return +end. + {prompt} + {memory} +
{chunk}
+``` + +Substitutions for this system: `{prompt}` = the standing question, `{memory}` = +`M_{t-1}` or the literal `No previous memory` at `t=1` (the paper's own case +studies show that exact string), `{chunk}` = the rendered chunk. + +Budget, against the 32768 cap: + +``` +system + template ~400 +question ~100 +memory <=1024 +chunk <=5000 +response 2048 + ------ + ~8600 headroom is comfortable +``` + +Chunk rendering: each record as `[role] text`, records separated by a blank line. +Role labels matter — the model uses them to tell a tool result from a decision. + +## Steps + +1. `PromptBuilder` in `mem-core` producing `(system, user)`. +2. Template verbatim as above. Any deviation gets a comment saying why. +3. `t=1` renders `No previous memory` — not empty, not `null`. +4. Render chunk records as `[role] text`, blank-line separated. +5. Assert the assembled prompt fits the budget before sending; over budget is an + error naming the component that overflowed, not a truncation. +6. `mem prompt --project P --query Q --chunk N` prints the exact prompt, for + eyeballing what the model actually sees. + +## Acceptance + +- Assembled prompt matches a committed golden file byte for byte. +- `t=1` contains `No previous memory`. +- Over-budget assembly errors and names the offending component. + +## Verify + +**Harness:** golden-file comparison. The prompt is a contract; a diff in it is a +change to the contract. + +**Integration test** — `tests/it_prompt.rs`: +1. `a1_golden_t1` — first turn against `expected/prompt-t1.txt`, exact match. +2. `a2_golden_tn` — turn with a prior memory against `expected/prompt-tn.txt`. +3. `a3_no_previous_memory_literal` — assert the exact string at `t=1`. +4. `a4_all_tags_present` — ``, ``, `
` each appear + exactly once. +5. `a5_role_labels_rendered` — a chunk with all four roles renders all four + labels. +6. `a6_over_budget_errors` — a 20000-token chunk errors, message contains + `section`. +7. `a7_budget_headroom` — for the real fixture corpus, assert every assembled + prompt is under 32768 minus 2048. + +**Command:** `cargo test -p mem-core prompt` + +**False pass:** +- Asserting the prompt "contains" the question. A template that dropped the + `` instructions still contains it, and the model then emits prose the + parser cannot read. Golden-file equality is the assertion that holds. +- Skipping assertion 7 by testing only small fixtures. Budget overflow appears + at p95 chunk size, not at the median. + +## Traps + +- Improving the wording. The 3B model's gate reliability comes from this exact + format; a cleaner rewrite is an unmeasured change to the one thing M1.8 gates on. +- Rendering an empty `` at `t=1`. The paper's traces show + `No previous memory`, and an empty tag reads to the model as "memory exists and + is empty", which is a different claim. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Standing queries · paper Fig 10a diff --git a/tasks/M1.4-gate-response-parser.md b/tasks/M1.4-gate-response-parser.md new file mode 100644 index 0000000..d9fe05c --- /dev/null +++ b/tasks/M1.4-gate-response-parser.md @@ -0,0 +1,113 @@ +# M1.4 — Gate-response parser + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.3 | + +## Goal + +Turn the model's tagged output into `(U_t, M̂_t, E_t)`, strictly — because a +lenient parser silently fabricates gate decisions. + +## Facts (inlined — no spec read needed) + +Expected response shape: + +``` +... +yes|no +candidate memory, or the previous memory verbatim +continue|end +``` + +Semantics, from the paper: + +| tag | value | meaning | +|---|---|---| +| `` | `yes` | `U_t = true` — memory becomes `M̂_t` | +| `` | `no` | `U_t = false` — memory stays `M_{t-1}`, chunk discarded | +| `` | `continue` | `E_t = false` | +| `` | `end` | `E_t = true` | + +**Strict parsing is the design, matching the paper's `r_format`:** it awards 1 +only when *every* turn in the trajectory parses, 0 otherwise, "because we can not +infer whether the incorrect format is caused by the previous erroneous parsing". + +So: exactly one of each tag, properly closed, `` content exactly `yes` or +`no` after trimming, `` exactly `continue` or `end`. Anything else is a +`ParseError` naming which tag failed and carrying the raw text. + +**A parse failure must not default.** Defaulting `U_t` to `false` silently drops +evidence; defaulting to `true` pollutes memory. The loop (M1.5) decides the +retry policy; the parser only reports. + +Reasoning models emit `` natively, which can nest or repeat. Extract by +locating the *last* `` before the first ``, not by regex over the +whole body. + +## Steps + +1. `parse_gate_response(&str) -> Result` in `mem-core`. +2. `GateResponse { think: String, update_gate: bool, candidate: String, exit_gate: bool }`. +3. Reject duplicates of any tag, a missing tag, an unclosed tag, and any + ``/`` value outside the allowed set. +4. `ParseError` variants name the tag and include the raw response, truncated. +5. Trim surrounding whitespace inside tags; do not otherwise normalise — memory + text is preserved verbatim. +6. When `U_t = false`, still capture `candidate` so the log can show what the + model *would* have written. The loop ignores it; the record is diagnostic. + +## Acceptance + +- All four tags parse from a well-formed response. +- Every malformed shape errors, naming the failing tag. +- No input produces a default `U_t` or `E_t`. + +## Verify + +**Harness:** table-driven over recorded real responses plus hand-built malformed +cases. Capture real ones with `MEM_LLM_RECORD` from M1.1. + +**Integration test** — `tests/it_gate_parser.rs`: +1. `a1_wellformed_yes_continue` — `U=true`, `E=false`, candidate matches. +2. `a2_wellformed_no_end` — `U=false`, `E=true`. +3. `a3_missing_check_errors` — error names `check`. +4. `a4_duplicate_update_errors` — two `` blocks error. +5. `a5_bad_check_value_errors` — `maybe` errors, message shows the + value. +6. `a6_unclosed_tag_errors` — `` never closed. +7. `a7_nested_think` — a response with `` inside `` still finds the + right boundary. +8. `a8_no_defaults` — property test over 1000 random mutations of a valid + response: every result is either an exact parse or an error, never a + silently-defaulted `GateResponse`. +9. `a9_real_responses` — every recorded real response parses. + +**Command:** `cargo test -p mem-core gate_parser` + +**False pass:** +- A regex that finds the first `` and stops. It passes 1–2 and silently + accepts duplicates, which is assertion 4's job. +- Testing only hand-written responses. Real 3B output has whitespace, + markdown fences and stray prose the author would not think to write — + assertion 9 is the only one that sees it. +- Omitting assertion 8. A parser with `unwrap_or(false)` anywhere passes every + positive test and quietly halves the update rate. + +## Traps + +- Defaulting on parse failure. The one thing that must not happen: a fabricated + gate decision is indistinguishable from a real one in the log, and it poisons + the M5 training data at the source. +- Normalising the candidate memory (collapsing whitespace, stripping markdown). + Memory text is content; the hash and every downstream projection depend on it + being verbatim. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Architecture · paper §3.2.1 `r_format` diff --git a/tasks/M1.5-gated-loop.md b/tasks/M1.5-gated-loop.md new file mode 100644 index 0000000..fdaf6bd --- /dev/null +++ b/tasks/M1.5-gated-loop.md @@ -0,0 +1,125 @@ +# M1.5 — The gated loop + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | L — 3+ days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.4 | + +## Goal + +The recurrence itself: `U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})`, with the level as a +parameter so L2 reuses it unchanged. + +## Facts (inlined — no spec read needed) + +Paper Algorithm 1, transcribed: + +``` +t <- 1; M_0 <- None +while t <= T: + U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1}) + if U_t == True: M_t <- M̂_t # update + else: M_t <- M_{t-1} # retain, discard chunk + if use_exit_gate and E_t == True: break + t <- t + 1 +answer = ψθ(Q, M_t) +``` + +Two decisions this task must not get wrong: + +**`use_exit_gate` is a parameter, false at L1.** `E_t` is always *recorded* +regardless — its signal is the M5 training target, and on a future unbounded +stream it becomes the only termination condition. Recording a gate you do not act +on is deliberate, not dead code. + +**The memory budget is enforced by the loop, not hoped for from the model.** +`memory_budget` is 1024 tokens. If `M̂_t` exceeds it the loop does **not** silently +truncate — truncation mid-sentence corrupts the memory for every later turn. It +records a `budget_exceeded` event and retains `M_{t-1}`, treating the turn as +`U_t = false`. Memory that stops growing is recoverable; memory that is +truncated garbage is not. + +The loop is generic over the input stream, so L2 (M3.1) passes L1 memories in +place of chunks with no other change. + +## Steps + +1. `run_loop(level, query, source: impl Stream, cfg) -> RunOutcome` + in `mem-core`. +2. Per turn: build prompt (M1.3), call model (M1.1), parse (M1.4), apply the + update rule, emit events. +3. On parse error: retry the same chunk up to 2 times. Still failing, record + `parse_failed`, treat as `U_t = false`, continue. **Never** default the gate. +4. On `U_t = true`: emit an L0 `evidence` event for the chunk, then an L1 + `memory` event whose `parents` include that evidence sha plus the previous + memory's sha. +5. Enforce `memory_budget` as above. +6. Honour `E_t` only when `use_exit_gate`; always record it. +7. Emit `run_end` with `chunks_seen`, `chunks_used`, `final_memory_sha`. +8. Cancellation: a dropped future must not leave a half-written log. Emit events + only after a turn fully resolves. + +## Acceptance + +- `U_t = false` leaves memory byte-identical to the previous turn. +- `U_t = true` replaces memory and links parents. +- `exit_gate = false` processes every chunk even when `E_t = true` throughout. +- Over-budget candidate retains prior memory and records the event. +- Two parse failures then success consumes 3 calls for one chunk. + +## Verify + +**Harness:** a scripted fake `ChatClient` returning canned responses per turn, so +the whole loop runs with no network and fully determined gate sequences. + +**Integration test** — `tests/it_gated_loop.rs`: +1. `a1_retain_on_no` — scripted `no` for 5 turns; assert final memory equals + initial and `chunks_used == 0`. +2. `a2_update_on_yes` — `yes` at turn 3 only; assert memory equals turn 3's + candidate and `chunks_used == 1`. +3. `a3_exit_gate_off_reads_all` — `end` at every turn with `use_exit_gate=false`; + assert all 10 chunks processed. +4. `a4_exit_gate_on_stops` — same script, `use_exit_gate=true`; assert it stops + at turn 1. +5. `a5_exit_always_recorded` — in a3, assert 10 `gate` events carry `exit=true` + despite not acting on them. +6. `a6_budget_exceeded_retains` — candidate of 4000 tokens; assert memory + unchanged, one `budget_exceeded` event, turn counted as not-used. +7. `a7_parse_retry` — fail twice then succeed; assert 3 calls, one memory event. +8. `a8_parse_failure_is_not_an_update` — fail 3 times; assert `U_t` false, + `parse_failed` recorded, loop continues. +9. `a9_parents_linked` — every memory event's `parents` contains the evidence sha + from the same turn. +10. `a10_level_is_a_parameter` — run the identical script at L1 and L2; assert the + only difference in emitted events is the `level` field. + +**Command:** `cargo test -p mem-core gated_loop` + +**False pass:** +- Testing with a fake that always returns `yes`. Every assertion about the retain + path is skipped, and retain is the path that matters — it is what makes this a + gate rather than a summarizer. +- Asserting `chunks_used` without asserting memory bytes. A loop that updates + memory on `no` but counts correctly passes a count-only test; assertion 1 + compares bytes. +- Omitting assertion 10. Without it, L2 in M3.1 becomes a copy of this loop, and + the two drift. + +## Traps + +- Truncating over-budget memory to fit. It corrupts every subsequent turn's + input, and the damage is attributed to the model. +- Acting on `E_t` at L1 because "the model said it had enough". Paper §3.3 is + explicit that this is wrong for exhaustive questions, and it silently truncates + extraction in a way that looks like poor recall. +- Writing log events before the turn resolves. A cancelled run then leaves a + memory event with no matching gate event and `mem verify` fails on a file that + was merely interrupted. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Architecture, tier model · paper Alg 1 diff --git a/tasks/M1.6-jsonl-event-log.md b/tasks/M1.6-jsonl-event-log.md new file mode 100644 index 0000000..7e63582 --- /dev/null +++ b/tasks/M1.6-jsonl-event-log.md @@ -0,0 +1,104 @@ +# M1.6 — JSONL event log writer + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.5 | + +## Goal + +Write the authoritative record — the one artifact everything else is derived +from, and the one that must survive a crash mid-run. + +## Facts (inlined — no spec read needed) + +Path: `log///.jsonl`. Append-only, one object per line. +**Every record carries `level`.** + +```jsonl +{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."} +{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."} +{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812} +{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."} +{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":[""],"sha256":"..."} +{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."} +``` + +`evidence` appears **only** when the update gate opened. That is what makes +update-rate directly measurable from the log — `gate` records give the +denominator, `evidence` records the numerator. + +This log is authoritative: the vault and pgvector are projections rebuilt from +it. Two consequences — it is tracked in git, and it is never rewritten in place. + +A run that crashes leaves a file with no `run_end`. That is a valid, readable +state meaning "incomplete", not corruption. Readers must handle it. + +## Steps + +1. `LogWriter::open(project, query_id, run_id)` in `mem-store`, creating parents. +2. `append(event)` serializes one line and **flushes**. An unflushed buffer loses + the last turns of exactly the run you want to debug. +3. `run_id` is a ULID — lexicographically sortable by creation time, so listing + runs in order is a directory sort. +4. `LogReader` streams events back, tolerating a truncated final line. +5. `replay_memory_at(t)` reconstructs `M_t` from the events alone, proving the log + is sufficient. +6. `stats()` computes chunks seen/used and update-rate from a log file. +7. Never open in truncate mode. Append only. + +## Acceptance + +- Every emitted record has a `level`. +- `evidence` count equals the count of `gate` records with `update: true`. +- A file with no `run_end` reads cleanly and reports `incomplete`. +- `replay_memory_at(t)` matches the memory the loop held at `t`. + +## Verify + +**Harness:** the scripted loop from M1.5 writing to a temp dir, plus a corrupted +fixture. + +**Integration test** — `tests/it_event_log.rs`: +1. `a1_every_record_has_level` — parse every line, assert `level` present and in + `{L0,L1,L2}`. +2. `a2_evidence_matches_update_gates` — count `gate.update==true`, assert equal to + the `evidence` count. +3. `a3_replay_equals_live` — for every `t`, `replay_memory_at(t)` equals the + memory the loop held. This is the assertion that proves the authority model. +4. `a4_truncated_tail_reads` — chop the last line mid-object; assert all prior + events parse and the run reports `incomplete`. +5. `a5_no_run_end_is_incomplete` — a log ending after a `memory` event reports + incomplete, not an error. +6. `a6_append_only` — write, reopen, write again; assert the first events survive. +7. `a7_flush_per_event` — kill the process (or drop without close) after 3 + appends; assert 3 lines on disk. +8. `a8_run_id_sorts_by_time` — three runs, assert lexicographic order equals + chronological order. + +**Command:** `cargo test -p mem-store event_log` + +**False pass:** +- Asserting the file parses. A writer that omits `evidence` events entirely + produces a perfectly parseable log with an update-rate of zero — assertion 2 is + what catches it. +- Testing replay only at the final `t`. A writer that records only the final + memory passes that and fails assertion 3 at every intermediate turn. +- Omitting assertion 7. Buffered writes pass every test that closes the file + properly, and lose data in precisely the crash case the log exists for. + +## Traps + +- Opening with truncate. One accidental re-run erases the authoritative record, + and the projections are the only surviving copy — inverted authority. +- Gitignoring `log/`. Makes the whole "JSONL is authoritative" claim a fiction. + `agent-rust/.gitignore` has a bare `tasks` entry that untracks its whole board; + do not repeat it here. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Storage schemas, authority model diff --git a/tasks/M1.7-ingest-end-to-end.md b/tasks/M1.7-ingest-end-to-end.md new file mode 100644 index 0000000..527f5cf --- /dev/null +++ b/tasks/M1.7-ingest-end-to-end.md @@ -0,0 +1,102 @@ +# M1.7 — `mem ingest` end to end + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.6 | + +## Goal + +One command that reads a real project and produces a real log — and reports the +number that says whether the gate works. + +## Facts (inlined — no spec read needed) + +``` +mem ingest --project poimen --query infra-root-causes +mem ingest --project poimen # all queries in the set +mem ingest --project poimen --limit 50 # first 50 chunks, for iterating +mem ingest --project poimen --resume # skip chunks already in the log +``` + +Progress output, because a run that prints nothing cannot be distinguished from +one that has hung: + +``` +[ 17/412] t=17 update=yes mem=142tok 1.9s +[ 18/412] t=18 update=no mem=142tok 0.8s +... +run 01HXYZ chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s +``` + +**Update-rate is the headline number.** Tool results are ~43% of records and +mostly evidence-free; a correct gate rejects the large majority of chunks. A rate +above ~30% means the gate is not discriminating and the run is an expensive +summarizer — that is the paper's memory-explosion failure and it is what M1.8 +gates on. + +Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every +chunk costs a model call, so `--resume` is not a nicety. + +## Steps + +1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6). +2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly. +3. Report update-rate in the summary and as `--format json`. +4. `--resume` reads the existing log, finds the highest `t` with a `gate` record, + and restarts from `t+1` with that turn's memory. +5. `--limit` caps chunks processed. +6. Exit non-zero if the run did not reach `run_end`. +7. Ctrl-C finishes the in-flight turn, writes `run_end`, exits — no half-turn. + +## Acceptance + +- A real project produces a complete log with `run_end`. +- Reported update-rate equals the value computed independently from the log. +- `--resume` on a complete log is a no-op; on a partial one it continues. +- Interrupt produces a valid log. + +## Verify + +**Harness:** scripted client for determinism, plus one live `#[ignore]` run. + +**Integration test** — `tests/it_ingest.rs`: +1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event + counts match the script. +2. `a2_update_rate_matches_log` — compare the reported rate to + `LogWriter::stats()` recomputed from the file. +3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model + calls. +4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the + next call is t=11 and memory at t=11 equals the replayed memory at t=10. +5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has + `run_end`, and the last `gate` has a matching `memory`-or-not decision. +6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records. +7. `a7_live_smoke` — `#[ignore]`; real gateway, `--limit 20` on a real project; + assert `run_end` and **print** the update-rate for a human to read. + +**Command:** `cargo test -p mem-cli ingest` (add `-- --ignored` for a7) + +**False pass:** +- Asserting only that the command exits 0. A run whose gate always answers `no` + exits 0, writes a valid log, and has learned nothing — the update-rate is the + only thing that distinguishes it, which is why a7 prints it rather than + merely asserting a run happened. +- Resuming by counting lines rather than reading the highest `t` with a `gate` + record. Line counts break the moment an `evidence` record is present, i.e. as + soon as the gate ever opened. + +## Traps + +- No progress output. A 14-minute run that prints nothing is indistinguishable + from a hang, and the first instinct will be to kill it. +- Resume that replays from `t=1` with the old memory. It costs a full run and + produces a log with duplicate turns that `mem verify` will reject. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verification, P2 diff --git a/tasks/M1.8-m1-gate.md b/tasks/M1.8-m1-gate.md new file mode 100644 index 0000000..4d9614a --- /dev/null +++ b/tasks/M1.8-m1-gate.md @@ -0,0 +1,99 @@ +# M1.8 — M1 composition gate + +| Field | Value | +|---|---| +| Phase | M1 — Gated loop at L1 | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M1 | + +## Goal + +Answer the only question that matters at this stage: **did we build a gate, or an +expensive summarizer?** + +## Facts (inlined — no spec read needed) + +This gate runs against the **live gateway on a real project** and asserts +properties of the resulting log. It is the first phase that costs money in wall +clock, and the first that can fail for reasons no unit test can see. + +Two thresholds, both from the paper: + +**Update-rate < 30%.** Agent transcripts are ~43% tool results, mostly +evidence-free. Paper Figure 6 shows the failure mode directly: the ungated +MemAgent's memory climbs to its 1024-token ceiling and saturates, after which +"the accumulated noise can further impede subsequent updates". A high update-rate +is that curve starting. + +**Memory size flat, not climbing.** Plot `memory.tokens` against `t`. GRU-Mem's +curve is low and roughly flat; the ungated curve rises to the cap and stays +pinned. A monotonic climb means the gate is open too often even if the rate +looks acceptable. + +Third property, cheap and load-bearing: **the same run twice produces the same +chunk hashes**. Temperature affects the model's text, not the chunking; if chunk +shas differ between runs, identity includes something it should not (M0.2) and +every projection will churn. + +## Steps + +1. Run `mem ingest --project poimen` for all standing queries against the live + gateway. +2. Compute per query: update-rate, memory-token series, parse-failure count, + elapsed. +3. Assert the thresholds below. +4. Emit `expected/m1-gate.txt` with the summary table; commit it. Subsequent runs + diff against it, and a changed expectation is a reviewable claim. +5. Sample 20 gate decisions and have the 32B `reasoning` model audit them; report + agreement. Advisory at this gate, and the seed of M5.2's calibration. + +## Acceptance + +- Update-rate < 30% on every standing query. +- Memory tokens ≤ 1024 and not monotonically increasing. +- Parse-failure rate < 5%. +- Chunk shas stable across two runs. + +## Verify + +**Harness:** live gateway. Long-running; a nightly or on-demand job, not +per-push. + +**Integration test** — `tests/it_m1_gate.rs`, all `#[ignore]` by default: +1. `a1_update_rate_under_threshold` — per query, assert < 0.30, print actual. +2. `a2_memory_bounded` — every `memory.tokens` ≤ 1024. +3. `a3_memory_not_climbing` — fit a line to tokens vs `t`; assert the slope is + below a small positive bound. Do not assert non-increasing — a legitimately + growing memory rises early then plateaus. +4. `a4_parse_failure_rate` — `parse_failed` / turns < 0.05. +5. `a5_chunk_sha_stable` — two runs, same chunk shas in the same order. +6. `a6_evidence_traceable` — every `evidence` sha appears as a parent of some + `memory` record. +7. `a7_judge_audit` — sample 20 decisions, ask the 32B model, print agreement. + Advisory; does not fail the gate. + +**Command:** `cargo test --workspace m1_gate -- --ignored --nocapture` + +**False pass:** +- Running the gate on a tiny `--limit`. Update-rate on the first 20 chunks is + noise; the failure mode is cumulative and needs a full project. +- Asserting the rate without printing it. The number is the artifact — a run at + 29% passes and is telling you something a boolean hides. +- Asserting memory is non-increasing rather than bounded-slope. Real memory does + grow early, and a strict assertion here fails on correct behaviour, which + trains everyone to skip the gate. + +## Traps + +- Treating a high update-rate as a model-quality problem first. Check the + prompt (M1.3 golden file) and the parser (M1.4 defaults) before blaming the + 3B model — a parser with `unwrap_or(true)` produces exactly this symptom. +- Tuning the threshold to whatever the first run produced. 30% comes from the + corpus composition; moving it to accommodate a bad result deletes the gate. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verification · paper Fig 6, §4.2 diff --git a/tasks/M2.1-embeddings-client.md b/tasks/M2.1-embeddings-client.md new file mode 100644 index 0000000..4655991 --- /dev/null +++ b/tasks/M2.1-embeddings-client.md @@ -0,0 +1,86 @@ +# M2.1 — Embeddings client + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.1 | + +## Goal + +Turn text into 768-dim vectors, batched, against the gateway's TEI endpoint. + +## Facts (inlined — no spec read needed) + +``` +POST /v1/embeddings +{"model":"nomic-ai/nomic-embed-text-v2-moe","input":["..."]} +-> {"object":"list","data":[{"embedding":[...768 floats...]}],"usage":{...}} +``` + +**768 dimensions**, probed and confirmed. It is the `vector(768)` in the schema +and in the HNSW index; a model swap is a schema migration, not a config change. + +**Batch limit is 32.** Verified: 1200 inputs returned +`{"message":"batch size 1200 > maximum allowed batch size 32","code":413}`. +Chunk the input list accordingly. + +This route currently has **no auth** — no `konghq.com/plugins` annotation, so +`model-key-auth` never attaches. Send the `apikey` header anyway: the route +should be fixed, and a client that only works while auth is broken breaks when +it is fixed. + +Bodies here are large (many texts × long strings) and the Kong buffer is 16m, so +batching also keeps requests well inside it. + +## Steps + +1. `EmbeddingsClient::embed(texts: &[String]) -> Result>>` in `mem-llm`. +2. Split into batches of ≤32, preserving input order in the output. +3. Assert every returned vector is exactly 768 long; a mismatch is an error + naming the model, not a silent pad or truncate. +4. Reuse M1.1's client config: `apikey` header, retry on 5xx only, generous + timeout. +5. `embed_one` convenience wrapper. + +## Acceptance + +- 100 texts return 100 vectors in input order. +- Every vector is 768-dim. +- A dimension mismatch errors loudly. + +## Verify + +**Harness:** `wiremock` offline, one `#[ignore]` live test. + +**Integration test** — `tests/it_embeddings.rs`: +1. `a1_batches_at_32` — 100 inputs produce exactly 4 requests. +2. `a2_order_preserved` — mock returns identifiable vectors; assert output order + matches input order across batch boundaries. +3. `a3_dimension_asserted` — mock returns a 512-dim vector; assert an error + naming the model. +4. `a4_apikey_sent` — assert the header is present even though the route does not + require it. +5. `a5_live_dims` — `#[ignore]`; real gateway, assert 768. + +**Command:** `cargo test -p mem-llm embeddings` (add `-- --ignored` for a5) + +**False pass:** +- Testing with ≤32 inputs. The batching path never runs and order-across-batches + — the thing most likely to be wrong — is never exercised. +- Trusting the response order within a batch without asserting it. Assertion 2 + must use distinguishable vectors, not a length check. + +## Traps + +- Hardcoding 768 in three places. Put it in one constant that the schema + migration also references, so a model change is one edit and one migration. +- Assuming no auth is needed because it currently works without a key. That route + is missing its plugin annotation, which is a bug scheduled to be fixed. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verified facts, pgvector diff --git a/tasks/M2.2-memory-db-manifest.md b/tasks/M2.2-memory-db-manifest.md new file mode 100644 index 0000000..2f1f4a2 --- /dev/null +++ b/tasks/M2.2-memory-db-manifest.md @@ -0,0 +1,102 @@ +# M2.2 — CNPG `memory-db` + pgvector + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | homelab | +| Spec | inlined below | +| Blocks | — | + +## Goal + +A Postgres with pgvector, provisioned the way everything else in the cluster is: +through git, with no manual `psql`. + +## Facts (inlined — no spec read needed) + +**pgvector needs no custom image.** Verified on the running cluster: + +``` +$ psql -tAc "select name,default_version,installed_version + from pg_available_extensions where name='vector'" +vector|0.7.0| +``` + +on the stock `ghcr.io/cloudnative-pg/postgresql:16.2`. Available, not yet +installed — `CREATE EXTENSION` is all that is missing. + +**The operator is CNPG 1.30.0**, which supports declarative extensions on the +`Database` CRD (`kubectl explain database.spec.extensions` resolves). So the +extension is git-managed too — no manual step, consistent with the GitOps rule +that infrastructure changes flow through version control. + +Follow `k8s/infra/databases/temporal-db.yaml` exactly: 3 instances, `imageName` +pinned, `enableSuperuserAccess: false`, `storageClass: longhorn-cnpg`, +`enablePodMonitor: true`, control-plane tolerations, `podAntiAffinityType: +preferred`. + +Storage: 10Gi matches the existing clusters. At 768 dims × 4 bytes, a vector is +~3 KB; tens of thousands of nodes is well under a gigabyte, so 10Gi is generous +and consistent rather than tight. + +## Steps + +1. `k8s/infra/databases/memory-db.yaml` — `Cluster` + `Database` with + `extensions: [{name: vector, ensure: present}]`. +2. Namespace `memory`, created by the ArgoCD app that owns it. +3. Add to the owning kustomization's explicit resource list — an unlisted file is + silently dropped with no error and no drift shown. +4. Commit, push, let ArgoCD sync. **No `kubectl apply`.** +5. Verify the extension installed and the app user can create tables. +6. Record the connection string convention in the repo README; the password comes + from the CNPG-generated secret, never committed. + +## Acceptance + +- `Cluster` reaches `Cluster in healthy state` with 3 instances. +- `select extversion from pg_extension where extname='vector'` returns a version. +- ArgoCD shows the app `Synced/Healthy`. +- No manual `psql` was run to get there. + +## Verify + +**Harness:** `kubectl` and `psql` read-only checks after sync. + +**Integration test** — `verify/m2.2.sh`, output diffed against `expected/m2.2.txt`: +1. `a1_cluster_healthy` — `kubectl get cluster -n memory memory-db` reports 3/3 + ready. +2. `a2_extension_installed` — `select extname, extversion from pg_extension where + extname='vector'` returns one row. +3. `a3_declarative_not_manual` — `kubectl get database -n memory memory-db-vector + -o jsonpath='{.spec.extensions}'` shows the declaration, proving it came from + git. +4. `a4_argocd_synced` — the owning app is `Synced/Healthy`. +5. `a5_app_user_can_ddl` — as `app`, `CREATE TABLE t(v vector(768)); DROP TABLE t;` + succeeds. +6. `a6_hnsw_available` — `CREATE INDEX ... USING hnsw` on that temp table + succeeds, proving 0.7.0 has the index type the schema needs. + +**Command:** `bash verify/m2.2.sh | diff - expected/m2.2.txt` + +**False pass:** +- Checking `pg_available_extensions` instead of `pg_extension`. Available means + the files are on disk; installed means `CREATE EXTENSION` ran. The whole task + is the second one. +- Verifying after a manual `CREATE EXTENSION`. It passes and proves nothing about + the declarative path, which is the actual deliverable. Assertion 3 is the guard. + +## Traps + +- Forgetting the kustomization resource list. The file sits in git, ArgoCD reports + Synced, and the objects never exist — silent, and the failure surfaces later as + a connection error. +- Adding `prune: true` semantics without thinking about operator-created children. + CNPG creates Services, Secrets and PVCs owned by the Cluster; if ArgoCD's + tracking label propagates to them, prune fights the operator. The + `llm-serving` app already had to set `prune: false` for exactly this reason. + +--- + +Background: [DESIGN.md](../DESIGN.md) — pgvector · `k8s/infra/databases/temporal-db.yaml` diff --git a/tasks/M2.3-schema-and-migrations.md b/tasks/M2.3-schema-and-migrations.md new file mode 100644 index 0000000..b411d1b --- /dev/null +++ b/tasks/M2.3-schema-and-migrations.md @@ -0,0 +1,110 @@ +# M2.3 — Schema + sqlx migrations + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M2.2 | + +## Goal + +The tables the provenance graph lives in, with the constraints that make a +malformed graph impossible rather than merely unlikely. + +## Facts (inlined — no spec read needed) + +```sql +CREATE TABLE memory_node ( + id BIGSERIAL PRIMARY KEY, + level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')), + project TEXT NOT NULL, + query_id TEXT, -- NULL at L2 + run_id TEXT NOT NULL, + t INT NOT NULL, + source TEXT, -- set at L0 + text TEXT NOT NULL, + sha256 TEXT NOT NULL UNIQUE, -- content identity, from M0.2 + embedding vector(768) NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE TABLE memory_edge ( + child_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE, + parent_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE, + PRIMARY KEY (child_sha, parent_sha) +); +CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops); +CREATE INDEX ON memory_node (project, level); +``` + +`sha256 UNIQUE` is what makes rebuild idempotent — re-inserting identical content +is a conflict to ignore, not a duplicate row. It is also why the hash must exclude +run ids and timestamps (M0.2). + +`ON DELETE CASCADE` on both edge columns: dropping a node should not leave +dangling edges. Rebuild drops everything anyway, but a partial cleanup should not +be able to corrupt the graph. + +`query_id` is NULL at L2 by design — L2 spans queries. Enforce it: +`CHECK ((level = 'L2') = (query_id IS NULL))`. + +Cosine distance, not L2: these are normalised text embeddings and cosine is what +the model was trained for. `vector_cosine_ops` must match the operator the query +uses (`<=>`), or the index is silently ignored and every query is a seq scan. + +## Steps + +1. `migrations/0001_init.sql` with the above, plus the `query_id`/level CHECK. +2. `CREATE EXTENSION IF NOT EXISTS vector;` first — the declarative extension + (M2.2) should have run, and this makes local dev work too. +3. Wire `sqlx::migrate!()` and run at startup. +4. `sqlx prepare` for offline compile-time query checking in CI. +5. Add a `schema_version` sanity query the repo layer asserts on connect. +6. Document that changing the embedding model is a migration, because the column + width is part of the schema. + +## Acceptance + +- Migrations apply to a clean database and are idempotent. +- Inserting a duplicate `sha256` conflicts rather than duplicating. +- An `L2` row with a non-null `query_id` is rejected by the CHECK. +- The HNSW index is used by a cosine-distance query. + +## Verify + +**Harness:** a disposable database — `sqlx::test` or testcontainers with the same +image tag as production, `ghcr.io/cloudnative-pg/postgresql:16.2`. + +**Integration test** — `tests/it_schema.rs`: +1. `a1_migrate_clean` — apply to an empty database, assert both tables exist. +2. `a2_migrate_idempotent` — apply twice, assert no error. +3. `a3_sha_unique` — insert the same sha twice, assert a unique violation. +4. `a4_level_check` — `level='L3'` rejected; `level='L2'` with a `query_id` + rejected; `level='L1'` without one rejected. +5. `a5_edge_fk` — an edge referencing a missing sha is rejected. +6. `a6_cascade` — delete a node, assert its edges are gone. +7. `a7_hnsw_is_used` — `EXPLAIN` a `ORDER BY embedding <=> $1 LIMIT 10` query and + assert the plan contains `Index Scan` on the HNSW index, not `Seq Scan`. + +**Command:** `cargo test -p mem-store schema` + +**False pass:** +- Testing the schema against SQLite or plain Postgres without pgvector. It will + accept `vector(768)` as an unknown type in some configurations and every + vector assertion becomes meaningless. Use the production image. +- Omitting assertion 7. An index created with the wrong opclass exists, reports + healthy, and is never used — queries just get slower as the table grows, which + reads as a scaling problem rather than a wrong index. + +## Traps + +- `vector_l2_ops` with a `<=>` query, or the reverse. The index is silently + ignored. This is the single most common pgvector mistake. +- Making `query_id` NOT NULL because L1 always has one. L2 then cannot be stored, + and the workaround is a sentinel string that pollutes every group-by. + +--- + +Background: [DESIGN.md](../DESIGN.md) — pgvector schema diff --git a/tasks/M2.4-pgvector-repo.md b/tasks/M2.4-pgvector-repo.md new file mode 100644 index 0000000..f6ef091 --- /dev/null +++ b/tasks/M2.4-pgvector-repo.md @@ -0,0 +1,99 @@ +# M2.4 — pgvector repository + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M2.3, M2.1 | + +## Goal + +Write the projection into Postgres idempotently, so rebuild is safe to run at any +time and produces the same rows. + +## Facts (inlined — no spec read needed) + +```rust +async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>; +async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>; +async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize) + -> Result>; +async fn parents_of(&self, sha: &Sha256Hash) -> Result>; +async fn clear_project(&self, project: &ProjectId) -> Result<()>; +``` + +`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an +identical node is the same node; re-running rebuild must not duplicate or churn +rows. Same for edges on the composite key. + +`search` orders by `embedding <=> $1` — cosine distance, matching the +`vector_cosine_ops` index. Any other operator silently drops to a seq scan. + +Edges are inserted **after** both endpoints exist, or the foreign key rejects +them. Rebuild therefore has two passes: all nodes, then all edges. This is not an +optimisation; a single-pass insert fails on the first forward reference. + +Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of +rebuild — batch across nodes, not per node. + +## Steps + +1. `PgRepo::connect(url)` with a pool; run migrations on connect. +2. Implement the five methods above. +3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a + multi-row statement. +4. Two-pass write: nodes, then edges. +5. `clear_project` deletes nodes for one project; edges cascade. +6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert + to a similarity score here. The reranker (M3.2) wants the ordering, and a + lossy conversion hides ties. + +## Acceptance + +- Upserting the same node twice leaves one row. +- Edges referencing not-yet-inserted parents fail; two-pass write succeeds. +- `search` returns nearest-first and respects the level filter. +- `clear_project` removes only that project. + +## Verify + +**Harness:** disposable Postgres with the production image; a deterministic fake +embedder (hash → fixed vector) so vector assertions are exact. + +**Integration test** — `tests/it_pg_repo.rs`: +1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`. +2. `a2_two_pass_required` — single-pass insert with a forward edge reference + fails; two-pass succeeds. Proves the ordering constraint is real. +3. `a3_search_orders_by_distance` — insert three known vectors, assert returned + order matches hand-computed cosine distance. +4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only + L1. +5. `a5_project_isolation` — two projects with identical text; search one, assert + no cross-project results. +6. `a6_clear_project_scoped` — clear one, assert the other is intact and no + orphan edges remain. +7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls. +8. `a8_parents_of` — walk a two-level graph, assert the returned parents match. + +**Command:** `cargo test -p mem-store pg_repo` + +**False pass:** +- Using a random embedder. Assertion 3 becomes untestable and is usually deleted, + which removes the only check that the distance operator matches the index. +- Testing `search` with one project in the database. Assertion 5 is the only one + that catches a missing `WHERE project = $1`, and that bug leaks another + project's memory into every answer. + +## Traps + +- Converting distance to similarity in the repo. It loses precision, and the + reranker downstream wants candidates in order rather than scores. +- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13 + would do, and rebuild goes from seconds to minutes. + +--- + +Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval diff --git a/tasks/M2.5-obsidian-projector.md b/tasks/M2.5-obsidian-projector.md new file mode 100644 index 0000000..ab491b2 --- /dev/null +++ b/tasks/M2.5-obsidian-projector.md @@ -0,0 +1,113 @@ +# M2.5 — Obsidian projector + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.6 | + +## Goal + +Render the log as a vault a human reads, with the tier graph as the link graph. + +## Facts (inlined — no spec read needed) + +``` +vault// + index.md L2 synthesis, links every L1 note + .md L1, one per standing query + evidence/-.md L0, only with --emit-evidence-notes +``` + +```markdown +--- +project: poimen +level: L1 +query_id: infra-root-causes +updated: 2026-08-17 +chunks_seen: 412 +chunks_used: 17 +run_id: 01HXYZ... +--- +# Infra root causes — poimen + + + +## Provenance +- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer +``` + +**Deterministic output is the requirement, not a nicety.** M2.8 asserts that +rebuilding produces a byte-identical vault. That means: stable key order in +frontmatter, no timestamp of *generation* (only `updated` derived from the log), +sorted provenance lists, and `\n` line endings. + +`updated` comes from the run's timestamp in the log — not `now()`. A generation +timestamp makes every rebuild a diff and destroys the gate. + +L0 notes default off: 17 per query is fine, but it grows unbounded across +projects and queries. Citations inline give the same provenance without the file +count. + +Wikilinks are `[[]]`. The link target may not exist as a file +when evidence notes are off — that is fine and normal in Obsidian, and it still +shows in the graph view as an unresolved node. + +## Steps + +1. `ObsidianProjector::project(log_dir, vault_dir, opts)` in `mem-store`. +2. Read the log; take the final `memory` record per query for L1, and the L2 + record for `index.md`. +3. Frontmatter with a fixed key order; `updated` from the log. +4. Provenance section from `parents`, sorted by source then `t`. +5. `--emit-evidence-notes` writes L0 notes; default off. +6. Write with `\n`, no trailing whitespace, exactly one trailing newline. +7. A note whose L1 memory is empty is still written, with a body saying no + evidence was found — an absent file is indistinguishable from a failed run. + +## Acceptance + +- Two projections of the same log produce byte-identical files. +- Frontmatter key order is stable. +- `updated` reflects the run, not the projection. +- Every L1 note links to its L2 index and vice versa. + +## Verify + +**Harness:** a committed log fixture and a committed expected vault tree. + +**Integration test** — `tests/it_projector.rs`: +1. `a1_byte_identical_twice` — project into two temp dirs, assert every file's + bytes are equal. This is M2.8's core property, tested early. +2. `a2_no_generation_timestamp` — project, sleep 1s, project again, assert equal. + Catches `now()` leaking into output. +3. `a3_frontmatter_key_order` — assert the exact key sequence. +4. `a4_golden_tree` — diff the whole output against `expected/vault/`, empty diff. +5. `a5_empty_memory_still_writes` — a log with zero updates produces a note + saying so. +6. `a6_links_bidirectional` — every L1 note appears in `index.md` and links back. +7. `a7_evidence_notes_flag` — off by default; on, produces one note per L0 node. +8. `a8_line_endings` — no `\r`, exactly one trailing `\n`. + +**Command:** `cargo test -p mem-store projector` + +**False pass:** +- Comparing files by parsed content rather than bytes. Key reordering and + whitespace churn both pass, and both fail M2.8 later, where the cause is much + harder to find. +- Testing with a single query. Assertion 6 needs at least two L1 notes to catch a + link built from the wrong id. + +## Traps + +- `updated: {now}`. The most natural thing to write, and it makes every rebuild + dirty, which trains everyone to ignore the diff that M2.8 depends on. +- Serializing frontmatter from a `HashMap`. Iteration order is unspecified and + the output churns between runs on the same input. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Obsidian vault diff --git a/tasks/M2.6-rebuild-from-log.md b/tasks/M2.6-rebuild-from-log.md new file mode 100644 index 0000000..b1c090b --- /dev/null +++ b/tasks/M2.6-rebuild-from-log.md @@ -0,0 +1,98 @@ +# M2.6 — `mem rebuild --from-log` + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M2.4, M2.5 | + +## Goal + +Drop both projections and rebuild them from the log alone — the command that +makes "the log is authoritative" a testable claim instead of a slogan. + +## Facts (inlined — no spec read needed) + +``` +mem rebuild --from-log --project poimen # both projections +mem rebuild --from-log --project poimen --vault-only +mem rebuild --from-log --project poimen --db-only +``` + +The claim: **anything not reconstructible from the log has a hidden input, and +that is a bug.** Rebuild is the executable form of that claim. If it needs the +existing vault or database to produce correct output, something is being carried +across that is not in the record. + +Rebuild does **no model calls except embeddings**. Gate decisions, memory text and +provenance are all in the log already; re-running the controller would produce +different text and defeat the purpose. + +Order matters: clear → insert all nodes → insert all edges → project vault. Edges +before nodes violates the foreign key (M2.4). + +Embeddings are the expensive part. Cache by `sha256` so a rebuild after a vault +template change does not re-embed unchanged nodes. + +## Steps + +1. `mem rebuild --from-log --project P`. +2. Read every log file for the project, in run-id order. +3. `clear_project`, then two-pass node/edge insert, batching embeddings. +4. Project the vault (M2.5), overwriting. +5. Embedding cache keyed by sha, on disk under `.cache/`, so it survives runs. +6. Report counts: nodes by level, edges, embeddings computed vs cached. +7. Refuse to run if any log file is incomplete (no `run_end`) unless `--allow-partial` + — rebuilding from a half-run silently produces a half-memory. + +## Acceptance + +- Rebuild from an empty database and empty vault produces the full state. +- Rebuild twice produces identical database rows and identical vault bytes. +- No controller model calls occur. +- An incomplete log is refused by default. + +## Verify + +**Harness:** log fixture, disposable Postgres, temp vault. A controller client +that panics if called. + +**Integration test** — `tests/it_rebuild.rs`: +1. `a1_from_empty` — drop everything, rebuild, assert node counts per level match + the log's records. +2. `a2_idempotent_db` — rebuild twice, assert row count unchanged and no + `created_at` churn on existing rows. +3. `a3_idempotent_vault` — rebuild twice, assert vault bytes identical. +4. `a4_no_controller_calls` — inject a panicking chat client; assert rebuild + succeeds. +5. `a5_embedding_cache` — second rebuild computes zero embeddings. +6. `a6_edge_order` — a log whose first memory references a later-inserted parent + still rebuilds, proving two-pass. +7. `a7_incomplete_refused` — a log with no `run_end` exits non-zero; with + `--allow-partial` it succeeds. +8. `a8_log_is_sufficient` — delete the vault and the database entirely, rebuild, + and assert the result equals a committed golden. This is the authority claim. + +**Command:** `cargo test -p mem-cli rebuild` + +**False pass:** +- Rebuilding on top of existing state. It masks every hidden input, because the + missing piece is already there from the previous run. Assertions 1 and 8 must + start from nothing. +- Asserting row counts only. A rebuild that inserts the right number of rows with + wrong `parents` passes; assertion 6 and M2.7's edge closure are what check the + graph. + +## Traps + +- Re-running the controller during rebuild. It produces different memory text + every time, the vault never stabilises, and M2.8 can never pass. +- Caching embeddings by node id rather than content hash. Ids change between + rebuilds; hashes do not, which is the whole point of content identity. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Authority model diff --git a/tasks/M2.7-verify-edges.md b/tasks/M2.7-verify-edges.md new file mode 100644 index 0000000..40d4c94 --- /dev/null +++ b/tasks/M2.7-verify-edges.md @@ -0,0 +1,90 @@ +# M2.7 — `mem verify` — edge closure + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M2.6 | + +## Goal + +Assert the provenance graph is well-formed, so a memory with no traceable +evidence is caught rather than believed. + +## Facts (inlined — no spec read needed) + +Invariants, checked against the log and the database independently: + +1. Every L1 `memory` record has at least one L0 parent. A memory with no evidence + came from somewhere the record does not explain. +2. Every `parents` sha resolves to a node that exists. +3. Every `evidence` sha appears as a parent of at least one memory. Evidence that + nothing cites was written for no reason. +4. `evidence` count equals `gate.update == true` count (also M1.6 a2, re-checked + here across the whole project rather than one run). +5. No edge is self-referential; no cycles. +6. Levels are consistent: an L1 node's parents are L0; an L2 node's are L1. + +Invariant 6 is the one that catches a tier confusion, and it is the one most +likely to break when M3.1 adds the L2 pass — an L2 node accidentally parented to +L0 evidence would still look plausible in the vault. + +`mem verify` is a read-only diagnostic. It never repairs; repair is `mem rebuild`. + +## Steps + +1. `mem verify --project P [--db] [--log]`, defaulting to both. +2. Check invariants 1–6, collecting **all** violations rather than failing on the + first — one run should tell you everything wrong. +3. Report per violation: invariant, level, sha, run id, and the log line number. +4. Exit non-zero on any violation. +5. `--format json` for machine consumption. + +## Acceptance + +- A clean project reports zero violations, exit 0. +- Each invariant has a fixture that violates it and is detected. +- All violations are reported in one run, not just the first. + +## Verify + +**Harness:** hand-built log fixtures, one per invariant, plus a clean one. + +**Integration test** — `tests/it_verify.rs`: +1. `a1_clean_passes` — the good fixture, zero violations, exit 0. +2. `a2_orphan_memory` — L1 with empty `parents`; detected as invariant 1. +3. `a3_dangling_parent` — parent sha not present; invariant 2. +4. `a4_uncited_evidence` — evidence nothing references; invariant 3. +5. `a5_evidence_gate_mismatch` — 3 update-gates but 2 evidence records; + invariant 4. +6. `a6_cycle` — A parents B, B parents A; invariant 5. +7. `a7_level_mismatch` — L2 node parented directly to an L0 node; invariant 6. +8. `a8_reports_all` — a fixture violating three invariants at once; assert all + three appear in one run's output. +9. `a9_db_and_log_agree` — introduce a violation in the database only; assert + `--db` catches it and `--log` does not, proving the two checks are independent. + +**Command:** `cargo test -p mem-cli verify` + +**False pass:** +- Checking the database only. The log is authoritative; a log-level violation + that rebuild happens to smooth over is still a bug in the writer, and + assertion 9 is what keeps the two checks honest. +- Failing fast on the first violation. It passes every single-violation fixture + and makes assertion 8 impossible, which in practice means three rebuild cycles + to find three problems. + +## Traps + +- Treating invariant 3 as fatal. Uncited evidence is a real smell, but a + legitimate case exists: the final turn updates memory and the run is cut short + before the memory record flushes. Report it, and let the gate decide severity. +- Skipping invariant 6 because L2 does not exist yet. It is cheap now and it is + precisely what M3.1 will break. + +--- + +Background: [DESIGN.md](../DESIGN.md) — tier model, Verification diff --git a/tasks/M2.8-m2-gate.md b/tasks/M2.8-m2-gate.md new file mode 100644 index 0000000..7c12889 --- /dev/null +++ b/tasks/M2.8-m2-gate.md @@ -0,0 +1,100 @@ +# M2.8 — M2 composition gate + +| Field | Value | +|---|---| +| Phase | M2 — Projections | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M2 | + +## Goal + +Prove the authority model: the log is sufficient, and both projections are +genuinely derived. + +## Facts (inlined — no spec read needed) + +The claim under test — poimen's own principle, applied here: + +> Nothing derived is authoritative. If it cannot be dropped and rebuilt +> byte-identically, it has hidden inputs and that is a bug. + +The gate is destructive by design: it **deletes** the vault and truncates the +database, rebuilds from the log alone, and diffs. Anything that survives only +because it was already there is a hidden input, and this is the only test that +finds it. + +```sh +rm -rf vault/poimen +psql -c "delete from memory_node where project = 'poimen'" +mem rebuild --from-log --project poimen +git -C vault diff --exit-code # empty diff is the only pass +``` + +`git diff --exit-code` on a tracked vault is the assertion. It compares against +what was committed, so it also catches a projector change that was not intended. + +Run it twice: once from empty (sufficiency) and once on top of itself +(idempotence). Both must produce the same bytes. + +## Steps + +1. `verify/m2.8.sh` performing the destructive rebuild above. +2. Assert the vault diff is empty and the database node counts match the log. +3. Run `mem verify` and assert zero violations. +4. Second rebuild without clearing; assert still empty diff and unchanged row + count. +5. Assert no controller model calls (embeddings are allowed and expected). +6. Commit `expected/m2.8.txt` with the count summary; diff against it. + +## Acceptance + +- Vault rebuilt from nothing is byte-identical to the committed vault. +- Database node/edge counts match the log's records exactly. +- `mem verify` reports zero violations. +- Second rebuild changes nothing. + +## Verify + +**Harness:** disposable database, git-tracked vault, real log. Long-running; a +nightly or on-demand job. + +**Integration test** — `verify/m2.8.sh`, output diffed against `expected/m2.8.txt`: +1. `a1_vault_from_empty` — delete vault, rebuild, `git diff --exit-code` empty. +2. `a2_db_from_empty` — truncate, rebuild, counts per level equal the log's. +3. `a3_verify_clean` — `mem verify` exits 0. +4. `a4_rebuild_idempotent` — rebuild again, diff still empty, row count unchanged. +5. `a5_no_controller_calls` — assert zero calls to the chat route during rebuild + (count via the record dir from M1.1, or a proxy). +6. `a6_projection_independence` — `--vault-only` then `--db-only` produces the + same end state as a combined rebuild. +7. `a7_log_alone_suffices` — move the log to a fresh checkout with no vault and no + database, rebuild, diff against the committed vault. The strongest form of + the claim. + +**Command:** `bash verify/m2.8.sh | diff - expected/m2.8.txt` + +**False pass:** +- Running the gate without deleting the vault first. A projector that only writes + changed files produces an empty diff trivially, and the hidden input survives. +- Diffing an untracked vault. `git diff` on untracked files reports nothing, so + the assertion passes vacuously. The vault must be committed, or the script must + compare against a committed golden tree explicitly. +- Allowing controller calls "because it is easier". Rebuild then produces new + memory text each run and the gate can never pass — at which point the usual fix + is to weaken the gate. + +## Traps + +- Treating a non-empty diff as a projector bug by default. It is equally likely to + be a *hash* bug: if `sha256` includes a timestamp (M0.2), every rebuild produces + new nodes and the vault churns. Check identity before blaming rendering. +- Running against production data with the destructive script and no backup. The + log is the record; if that is intact, everything is recoverable — which is + exactly why the log must be tracked in git before this gate is first run. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Authority model, Verification diff --git a/tasks/M3.1-l2-synthesis.md b/tasks/M3.1-l2-synthesis.md new file mode 100644 index 0000000..fac2453 --- /dev/null +++ b/tasks/M3.1-l2-synthesis.md @@ -0,0 +1,102 @@ +# M3.1 — L2 synthesis pass + +| Field | Value | +|---|---| +| Phase | M3 — L2 synthesis and retrieval | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.5 | + +## Goal + +Project-level memory across the per-query memories — using the same loop, with +the exit gate switched on. + +## Facts (inlined — no spec read needed) + +``` +mem synthesize --project poimen +``` + +L2 is **not new machinery**. It is `run_loop` (M1.5) with: + +| | L1 | L2 | +|---|---|---| +| input stream | `Chunk` from sources | L1 `MemoryNode`s | +| question | per-query question | `synthesis.question` | +| `use_exit_gate` | false | **true** | +| `query_id` | set | NULL | +| parents | L0 evidence shas | L1 memory shas | + +**Why the exit gate flips on.** At L1 the input is hundreds of chunks and the +question is exhaustive ("what are *all* the X"), which is exactly the case paper +§3.3 says to run without the gate. At L2 the input is a handful of memories and +"enough evidence" is genuinely decidable, which is the case the gate was designed +for and where the paper measures its 4× speedup. + +If M1.5 was written correctly this task is mostly wiring. If it needs changes to +`run_loop`, the level was not really a parameter — and assertion a10 in M1.5 +existed to prevent exactly that. + +Ordering: L1 memories enter the stream in a stable order (query id, ascending), +so synthesis is reproducible. + +## Steps + +1. `mem synthesize --project P` reads the final L1 memory per standing query. +2. Wrap them as the loop's input stream, in sorted query-id order. +3. Run `run_loop` with `level = L2`, `use_exit_gate = true`, the synthesis + question, `query_id = None`. +4. Write to `log//_synthesis/.jsonl`. +5. `parents` on the L2 memory are the L1 memory shas consumed up to that turn. +6. Refuse to run if any standing query has no completed L1 run — synthesizing + over a partial set silently produces a partial picture. + +## Acceptance + +- No change to `run_loop` is required. +- The exit gate fires and stops early on a real project. +- L2 parents are L1 shas, never L0. +- Sorted input order makes two runs consume memories in the same sequence. + +## Verify + +**Harness:** scripted client, plus one live run. + +**Integration test** — `tests/it_l2.rs`: +1. `a1_reuses_run_loop` — assert `mem synthesize` calls the same `run_loop` + symbol; a duplicated loop is a review failure, and a `#[deny]`-style test here + is a grep asserting `fn run_loop` appears exactly once in the workspace. +2. `a2_exit_gate_on` — scripted `end` at turn 2 of 5; assert it stops at 2. +3. `a3_parents_are_l1` — every L2 parent sha resolves to an L1 node. +4. `a4_query_id_null` — the L2 record has no `query_id`. +5. `a5_stable_input_order` — two runs consume L1 memories in identical order. +6. `a6_refuses_partial` — one query with no completed run; assert non-zero exit + naming the query. +7. `a7_level_check_holds` — run `mem verify`; invariant 6 (level consistency) + passes. +8. `a8_live` — `#[ignore]`; real project, assert an L2 memory is produced and + print whether the exit gate fired and at which turn. + +**Command:** `cargo test -p mem-cli l2` (add `-- --ignored` for a8) + +**False pass:** +- Copying `run_loop` into an L2-specific function. Everything passes, the two + drift within a month, and the tier model quietly becomes two implementations. + Assertion 1 is the guard. +- Testing the exit gate with a script that never says `end`. The gate's effect is + invisible and a `use_exit_gate` that is ignored passes. + +## Traps + +- Feeding L0 evidence into L2 "for more detail". It blows the context budget and + breaks the level invariant; L2 reads memories, and if they are inadequate the + fix is at L1. +- Synthesizing over whatever L1 runs happen to exist. A missing query produces a + confident summary of an incomplete project, which is worse than no summary. + +--- + +Background: [DESIGN.md](../DESIGN.md) — tier model · paper §3.3 diff --git a/tasks/M3.2-rerank-client.md b/tasks/M3.2-rerank-client.md new file mode 100644 index 0000000..f36f360 --- /dev/null +++ b/tasks/M3.2-rerank-client.md @@ -0,0 +1,88 @@ +# M3.2 — Rerank client + +| Field | Value | +|---|---| +| Phase | M3 — L2 synthesis and retrieval | +| Size | S — under 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.1 | + +## Goal + +Reorder vector-recall candidates by actual relevance, because embedding distance +is a coarse filter. + +## Facts (inlined — no spec read needed) + +``` +POST /v1/rerank +{"query":"what is rust","texts":["rust is a language","bananas"]} +-> [{"index":0,"score":0.98176736},{"index":1,"score":0.00008833366}] +``` + +`BAAI/bge-reranker-base` via TEI. Note the **response shape is a bare array**, not +an OpenAI-style `{"data": [...]}` envelope — this route does not follow the chat +convention. + +The discrimination is real: 0.98 vs 0.00009 on that probe, four orders of +magnitude. Embedding cosine on the same pair would be far closer, which is why +recall-then-rerank beats recall alone. + +`index` refers to the position in the submitted `texts` array; results come back +**sorted by score**, so the index is the only way to map back. Do not assume +order. + +Batch limits apply as with embeddings — keep candidate lists modest (top-50 from +recall is plenty). + +## Steps + +1. `RerankClient::rerank(query, texts) -> Result>` in `mem-llm`. +2. Parse the bare array; map `index` back to the caller's items. +3. Preserve the caller's item type: take `&[T]`, return `Vec<(T, f32)>` so the + caller does not re-associate by position. +4. Same `apikey` header, retry and timeout policy as M1.1. +5. Empty input returns empty without a request. + +## Acceptance + +- Results map correctly back to input items via `index`. +- A more relevant text scores above a less relevant one on a live call. +- Empty input makes no request. + +## Verify + +**Harness:** `wiremock` offline, one `#[ignore]` live test. + +**Integration test** — `tests/it_rerank.rs`: +1. `a1_bare_array_parsed` — mock returns `[{"index":1,...},{"index":0,...}]`; + assert parsing succeeds. +2. `a2_index_mapping` — with the out-of-order mock above, assert the returned + items correspond to inputs 1 and 0 respectively, not 0 and 1. +3. `a3_empty_no_request` — empty texts; assert zero requests. +4. `a4_apikey_sent` — header present. +5. `a5_live_discriminates` — `#[ignore]`; real gateway, query "what is rust" + against `["rust is a language","bananas"]`; assert the first scores at least + 10× the second. + +**Command:** `cargo test -p mem-llm rerank` (add `-- --ignored` for a5) + +**False pass:** +- Assuming the response preserves input order. A mock that returns results in + input order passes a naive test and the live endpoint returns them sorted, + which silently mislabels every result. Assertion 2 must use an out-of-order + mock. +- Expecting an OpenAI envelope. It will fail immediately against the live route, + but a mock written to match the wrong shape hides that until integration. + +## Traps + +- Re-associating results by position instead of by `index`. The scores are right + and attached to the wrong documents — a bug that looks like poor retrieval + quality rather than a mapping error. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verified facts, retrieval diff --git a/tasks/M3.3-mem-query.md b/tasks/M3.3-mem-query.md new file mode 100644 index 0000000..365c8cf --- /dev/null +++ b/tasks/M3.3-mem-query.md @@ -0,0 +1,97 @@ +# M3.3 — `mem query` with provenance + +| Field | Value | +|---|---| +| Phase | M3 — L2 synthesis and retrieval | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M3.2, M2.4 | + +## Goal + +Ask the memory a question and get an answer that can be traced back to the +session it came from. + +## Facts (inlined — no spec read needed) + +``` +mem query "why did requests over 10KB fail?" +mem query --project poimen --levels L1,L2 --k 10 "..." +``` + +Pipeline: embed the question (M2.1) → HNSW recall top-k over `memory_node` +filtered by project and level (M2.4) → rerank the candidates (M3.2) → return with +provenance walked down through `memory_edge`. + +Default levels are **L1 and L2**, not L0. L1/L2 are synthesized answers; L0 is raw +evidence and returning it by default buries the answer in transcript. `--levels +L0` exists for "show me the actual source". + +Provenance walk: for each returned node, follow `memory_edge` to its parents and +report source and turn. One hop for L1 (to evidence), two for L2 (through L1). + +Recall wide, rerank narrow: take 50 from HNSW, rerank, return 5. Cosine distance +alone puts "bananas" close enough to matter; the reranker separated the same pair +by four orders of magnitude. + +Output is human-readable by default, `--format json` for programmatic use. + +## Steps + +1. `mem query [--project P] [--levels L] [--k N] `. +2. Embed, recall 10×k, rerank, truncate to k. +3. Walk edges to build a provenance list per hit; deduplicate sources. +4. Render: score, level, query id, the memory text, then provenance lines. +5. Project defaults to the one inferred from `$PWD`; `--project` overrides. No + project and no match is an error, not an empty result over everything. +6. `--explain` prints the recall candidates and their pre-rerank distances, for + debugging retrieval quality. + +## Acceptance + +- A question with a known answer returns the right L1 note first. +- Every hit carries at least one resolvable provenance entry. +- `--levels L0` returns evidence; the default does not. +- Reranking changes the order versus raw recall on at least one real query. + +## Verify + +**Harness:** a seeded database from a real log, deterministic fake embedder for +the offline assertions, live for quality. + +**Integration test** — `tests/it_query.rs`: +1. `a1_known_answer` — seeded with the poimen log, query "why did requests over + 10KB fail?"; assert the top hit is the `infra-root-causes` L1 node. +2. `a2_provenance_resolves` — every hit's provenance shas exist in + `memory_node`. +3. `a3_default_excludes_l0` — assert no L0 nodes in default output. +4. `a4_levels_flag` — `--levels L0` returns evidence nodes. +5. `a5_rerank_reorders` — capture pre- and post-rerank order; assert they differ + on at least one fixture query, proving the reranker is wired and not a no-op. +6. `a6_project_isolation` — two projects seeded; assert no cross-project hits. +7. `a7_no_project_errors` — unresolvable project exits non-zero. +8. `a8_l2_two_hop_provenance` — an L2 hit's provenance resolves through L1 to L0 + sources. + +**Command:** `cargo test -p mem-cli query` + +**False pass:** +- Asserting only that results are returned. A pipeline where the reranker returns + the input unchanged still returns results, and assertion 5 is the only check + that it is doing anything. +- Testing provenance existence without resolving it. A hit carrying parent shas + that do not exist in the table looks fine in the output and is useless. + +## Traps + +- Returning L0 by default. The answer is there but buried in raw transcript, and + the tool reads as low quality when it is a display default. +- Recalling exactly k then reranking. Reranking cannot recover a relevant + document that recall never returned; the width of the recall is what determines + the ceiling. + +--- + +Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval diff --git a/tasks/M3.4-m3-gate.md b/tasks/M3.4-m3-gate.md new file mode 100644 index 0000000..c3d7da8 --- /dev/null +++ b/tasks/M3.4-m3-gate.md @@ -0,0 +1,93 @@ +# M3.4 — M3 composition gate + +| Field | Value | +|---|---| +| Phase | M3 — L2 synthesis and retrieval | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M3 | + +## Goal + +Prove the whole stack answers a real question with real provenance — the first +point at which the system is useful rather than merely correct. + +## Facts (inlined — no spec read needed) + +The gate is a small set of **known-answer questions** with hand-written expected +sources, committed to the repo. Each names a fact that genuinely appears in the +ingested sessions and the session it appears in. + +Seed set, all drawn from real work in this corpus: + +| question | expected to cite | +|---|---| +| why did requests over 10KB fail? | the Kong body-buffer / `client_body_buffer_size` finding | +| why did `Authorization: Bearer` return 401? | the Kong key-auth header finding | +| what causes the 504 on a cold ornith start? | the timeouts-on-Ingress-vs-Service finding | + +This is a retrieval quality gate, so it is graded, not boolean: report +**hit rate at k=5** and **provenance precision** (fraction of cited sources that +actually contain the fact). A single failing question is information, not +necessarily a stop. + +The gate also re-runs `mem verify` including the level invariant, because M3.1 is +the change most likely to break it. + +## Steps + +1. `verify/known-answers.yaml` — question, expected node, expected source. +2. `verify/m3.4.sh` runs each through `mem query --format json`. +3. Compute hit rate at 5 and provenance precision; print both. +4. Assert the thresholds below. +5. Run `mem verify`; assert zero violations. +6. Assert L2 exists and its provenance resolves two hops. +7. Commit `expected/m3.4.txt`; diff. + +## Acceptance + +- Hit rate at k=5 ≥ 0.8 on the known-answer set. +- Provenance precision ≥ 0.9 — a citation that does not contain the fact is worse + than no citation. +- `mem verify` clean, including level consistency. +- Every L2 hit resolves to L0 sources. + +## Verify + +**Harness:** live gateway, seeded database from real logs. On-demand, not +per-push. + +**Integration test** — `verify/m3.4.sh` diffed against `expected/m3.4.txt`: +1. `a1_hit_rate` — ≥ 0.8, print actual. +2. `a2_provenance_precision` — for each cited source, fetch the L0 text and + assert it contains the expected fact substring; ≥ 0.9. +3. `a3_verify_clean` — zero violations. +4. `a4_l2_two_hop` — every L2 hit resolves through L1 to a real source. +5. `a5_rerank_contributes` — hit rate with reranking is ≥ hit rate without. If + reranking makes it worse, the wiring is wrong (probably index mapping, M3.2). +6. `a6_no_cross_project` — a question about another project returns nothing from + this one. + +**Command:** `bash verify/m3.4.sh | diff - expected/m3.4.txt` + +**False pass:** +- Writing the known-answer set after seeing what the system returns. It then + measures nothing. Write the questions and expected sources from the sessions + first, independently of any query output. +- Measuring hit rate without provenance precision. A system that returns the + right note with fabricated citations scores 1.0 on hits and is untrustworthy — + assertion 2 is the one that matters for whether anyone can act on an answer. + +## Traps + +- Tuning `k` until the hit rate passes. k=50 will hit almost everything and the + metric stops meaning anything; the gate specifies k=5 for that reason. +- Accepting a5 failing as "reranker is just not helping". It far more often means + the `index` mapping in M3.2 is wrong and scores are attached to the wrong + documents. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verification diff --git a/tasks/M4.1-skill-draft.md b/tasks/M4.1-skill-draft.md new file mode 100644 index 0000000..69c0677 --- /dev/null +++ b/tasks/M4.1-skill-draft.md @@ -0,0 +1,114 @@ +# M4.1 — `mem skill draft` + +| Field | Value | +|---|---| +| Phase | M4 — Skills | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M3.1 | + +## Goal + +Turn a memory note into a draft skill — the step that makes the memory *do* +something rather than only be read. + +## Facts (inlined — no spec read needed) + +``` +mem skill draft --from poimen/infra-root-causes +-> vault/skills/_drafts/poimen-infra-root-causes/SKILL.md +``` + +**A skill is a projection, not a level.** L0/L1/L2 are descriptive — what +happened. A skill is procedural — what to do next time. The gated loop does not +produce it: "does this chunk contain evidence for Q" has no meaning when the +output is an instruction. + +Format is free, because `SKILL.md` is YAML frontmatter plus markdown, which is +exactly an Obsidian note. Verified against a real installed skill: + +```yaml +--- +name: keyword-research +description: 'Use when the user asks to "find keywords"... Not for X — use Y.' +when_to_use: "Use when starting keyword research for a new page..." +argument-hint: " [market/language]" +--- +``` + +So the same file is a vault note and a loadable skill, with no conversion. + +**`description` is the whole game.** It is the trigger — a skill whose +description does not match how the user actually phrases the request never fires, +no matter how good the body is. Note the real example above spends half its +description on *negative* routing ("Not for X — use Y"). + +Follow the existing rubric rather than inventing one: the installed +`grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and +a four-dimension rubric — conciseness, actionability, workflow clarity, +progressive disclosure. + +**Drafts land in `_drafts/` and are never auto-loaded.** A directory, not a +frontmatter flag, because a directory cannot be accidentally globbed into +`--skill`. + +## Steps + +1. `mem skill draft --from /` reads the L1 or L2 note. +2. Prompt the model to convert descriptive memory into procedural instruction, + with the rubric's four dimensions in the prompt. +3. Emit frontmatter: `name`, `description`, `when_to_use`, plus + `generated_from: ` and `generated_at`. +4. Write to `vault/skills/_drafts/-/SKILL.md`. +5. Refuse to write outside `_drafts/`. Promotion is a human `git mv`. +6. `--dry-run` prints without writing. + +## Acceptance + +- Output parses as valid frontmatter + markdown. +- `generated_from` resolves to a real node sha. +- The file lands in `_drafts/` and nowhere else. +- Promotion is not automated anywhere in the codebase. + +## Verify + +**Harness:** seeded vault and log; scripted model client for determinism. + +**Integration test** — `tests/it_skill_draft.rs`: +1. `a1_valid_frontmatter` — parse the output; assert `name`, `description`, + `when_to_use`, `generated_from` present and non-empty. +2. `a2_generated_from_resolves` — the sha exists in `memory_node`. +3. `a3_writes_only_to_drafts` — assert the path contains `_drafts/`; attempt to + pass a path outside it and assert refusal. +4. `a4_no_promotion_path` — grep the workspace for any code writing to + `vault/skills/` that is not under `_drafts/`; assert none. Promotion must be + manual. +5. `a5_description_is_trigger_shaped` — assert `description` contains at least one + phrasing cue (a quoted user phrase or "Use when"), matching the installed + examples. +6. `a6_dry_run_writes_nothing` — assert no file created. +7. `a7_idempotent` — same input twice produces identical bytes apart from + `generated_at`. + +**Command:** `cargo test -p mem-cli skill_draft` + +**False pass:** +- Asserting the file was written without asserting *where*. The entire safety + property of this task is the location, and a draft written to `vault/skills/` + is immediately loadable. +- Accepting any non-empty `description`. A one-line restatement of the title + never triggers, so the feature appears to work and the skill never fires — + assertion 5 is a weak but real guard. + +## Traps + +- Auto-promoting "when the draft looks good". That closes the loop this design + deliberately leaves open, and there is no external verifier inside it. +- Generating the body from L0 evidence. Skills are procedure distilled from + synthesis; raw transcript produces a narrative, not an instruction. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Skills, the procedural projection diff --git a/tasks/M4.2-derived-filter.md b/tasks/M4.2-derived-filter.md new file mode 100644 index 0000000..493af2f --- /dev/null +++ b/tasks/M4.2-derived-filter.md @@ -0,0 +1,109 @@ +# M4.2 — `derived: true` ingest filter + +| Field | Value | +|---|---| +| Phase | M4 — Skills | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M4.1, M0.5 | + +## Goal + +Stop the system learning from its own output. + +## Facts (inlined — no spec read needed) + +The cycle, and it is the only one in the design: + +``` +emitted skill is loaded into a session + │ + ▼ +appears verbatim in that session's transcript + │ + ▼ +transcript is ingested as evidence + │ + ▼ +reinforces the memory that produced the skill +``` + +No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is +what actually stops it. + +Mechanism: every emitted artifact records its content hash in a manifest. During +ingest, a record whose normalised text matches a known artifact is tagged +`derived: true` and **excluded from evidence** — it is still recorded in the log +so the exclusion is visible and auditable, but the gate never sees it. + +Matching must survive the model reformatting the text slightly. Exact hash on the +whole record is too brittle: a skill quoted with different indentation would slip +through. Use a normalised shingle overlap — strip whitespace and markdown, hash +overlapping n-grams, and flag a record whose overlap with any artifact exceeds a +threshold. + +Threshold is a tradeoff and should be logged, not hidden: too low excludes +genuine discussion *about* a skill, too high lets the cycle run. + +## Steps + +1. `vault/skills/.manifest.jsonl` — one line per emitted artifact: + `{name, sha256, shingles, emitted_at}`. +2. `mem skill draft` appends to it. +3. `mem-ingest` loads the manifest and computes shingle overlap per record. +4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking. +5. Log a `derived_excluded` event with the record's provenance and the artifact + it matched, so exclusions are auditable and a false positive is findable. +6. `--no-derived-filter` to disable, for debugging only, loudly warned. +7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact. + +## Acceptance + +- A record quoting an emitted skill verbatim is excluded. +- A record quoting it with different whitespace and fences is also excluded. +- A record merely *mentioning* the skill by name is not excluded. +- Every exclusion is logged with what it matched. + +## Verify + +**Harness:** a fixture manifest with one artifact, plus records in several +paraphrase grades. + +**Integration test** — `tests/it_derived_filter.rs`: +1. `a1_verbatim_excluded` — exact copy of an artifact is excluded. +2. `a2_reformatted_excluded` — same content, different indentation and code + fences; excluded. +3. `a3_mention_not_excluded` — "I used the infra-root-causes skill" is kept. This + is the false-positive guard, and over-filtering silently starves the memory. +4. `a4_unrelated_not_excluded` — random session text is kept. +5. `a5_exclusion_logged` — assert a `derived_excluded` event naming the matched + artifact. +6. `a6_verify_catches_leak` — insert an L0 node matching an artifact directly into + the database; assert `mem verify --derived-filter` fails. +7. `a7_threshold_configurable` — assert the threshold is read from config and + appears in the run record. +8. `a8_no_manifest_is_safe` — with no manifest file, ingest proceeds and filters + nothing, rather than failing. + +**Command:** `cargo test -p mem-ingest derived_filter` + +**False pass:** +- Testing only the verbatim case. Exact-match filtering passes and the realistic + case — a model that reformats what it quotes — walks straight through. + Assertion 2 is the one that matters. +- Omitting assertion 3. A filter tuned only for recall excludes every discussion + of a topic once a skill about it exists, which quietly makes the memory worse + the more skills you write. + +## Traps + +- Filtering on record *hash*. One character of whitespace defeats it, and the + cycle runs while the filter reports itself working. +- Silent exclusion. Without assertion 5's log event, a false positive is + invisible — memory just gets thinner and nobody knows why. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Skills, Risks diff --git a/tasks/M4.3-m4-gate.md b/tasks/M4.3-m4-gate.md new file mode 100644 index 0000000..653a88f --- /dev/null +++ b/tasks/M4.3-m4-gate.md @@ -0,0 +1,87 @@ +# M4.3 — M4 composition gate + +| Field | Value | +|---|---| +| Phase | M4 — Skills | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M4 | + +## Goal + +Prove the loop stays open — that a generated skill cannot silently become +training data for the memory that generated it. + +## Facts (inlined — no spec read needed) + +Two properties, and both must hold. Either one alone is insufficient: + +1. **A draft is not loadable.** It lives in `_drafts/`, and pointing a real agent + at `vault/skills/` must not surface it. +2. **A promoted skill's text never enters evidence.** Even after a human promotes + it and it appears in a session, the derived filter keeps it out of L0. + +The gate exercises the full cycle deliberately: draft a skill, promote it, run a +session that loads it, ingest that session, and assert the skill's content did +not become evidence. + +Property 1 is tested with the real loader, not by inspecting paths — the +question is what an agent actually sees. + +## Steps + +1. Draft a skill from a real note. +2. `pi --skill vault/skills/ --list-skills` (or the equivalent enumeration); + assert the draft is absent. +3. Promote it with `git mv` out of `_drafts/`; assert it now appears. +4. Synthesize a session transcript that quotes the promoted skill, in three + grades: verbatim, reformatted, and merely referenced. +5. Ingest it; assert the first two are excluded and the third is kept. +6. Run `mem verify --derived-filter`; assert clean. +7. Commit `expected/m4.3.txt`; diff. + +## Acceptance + +- Draft absent from a real loader's skill enumeration. +- Promoted skill present. +- Verbatim and reformatted quotes excluded from evidence; a bare mention kept. +- `mem verify --derived-filter` reports zero leaks. + +## Verify + +**Harness:** the real `pi` binary for enumeration, seeded vault and log. + +**Integration test** — `verify/m4.3.sh` diffed against `expected/m4.3.txt`: +1. `a1_draft_not_loadable` — enumerate skills; assert the draft name is absent. +2. `a2_promoted_is_loadable` — after `git mv`, assert present. Proves assertion 1 + is not passing because the loader is broken. +3. `a3_verbatim_excluded` — ingest, assert no L0 node matches. +4. `a4_reformatted_excluded` — same. +5. `a5_mention_kept` — the referencing record survives as evidence. +6. `a6_verify_clean` — `mem verify --derived-filter` exits 0. +7. `a7_exclusions_auditable` — assert `derived_excluded` events name the artifact. + +**Command:** `bash verify/m4.3.sh | diff - expected/m4.3.txt` + +**False pass:** +- Asserting the draft is absent without assertion 2. If the loader silently fails + to enumerate anything, assertion 1 passes trivially and the guarantee is + untested. +- Testing exclusion only on the draft. The draft is not the risk — a *promoted* + skill is the one that actually reaches sessions, and it is the one the filter + must catch. + +## Traps + +- Automating promotion inside the gate script and leaving it there. The gate + needs to promote something to test the promoted path; make it obvious that the + script is the only place it happens, and that production has no such path. +- Reading a passing gate as "the cycle is impossible". It means the two guards + hold today. Adding a new emitted artifact type without adding it to the + manifest reopens the loop, and only assertion 6 will notice. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Skills, Risks diff --git a/tasks/M5.1-evidence-labeler.md b/tasks/M5.1-evidence-labeler.md new file mode 100644 index 0000000..2117eb7 --- /dev/null +++ b/tasks/M5.1-evidence-labeler.md @@ -0,0 +1,108 @@ +# M5.1 — `mem label` — evidence labeler + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M1.6 | + +## Goal + +Produce the per-chunk ground truth `U_t` that `r_update` needs, since this corpus +does not come with evidence labels. + +## Facts (inlined — no spec read needed) + +Paper `r_update`: + +``` +r_update_t = +1 if U_t is correct + -1 if U_t is incorrect +``` + +"Correct" means: for chunks containing evidence for `Q`, the agent should emit +`yes`; for chunks without, `no`. That requires +knowing which chunks contain evidence. + +The paper had it for free — synthetic NIAH tasks place the needle deliberately, +and HotpotQA ships supporting facts. **We have neither.** Agent transcripts have +no annotation of which turn contained the answer. + +Cheapest honest substitute: **distant supervision from the 32B model.** Ask +`reasoning` (DeepSeek-R1-Distill-Qwen-32B, vLLM) per `(question, chunk)` whether +the chunk contains evidence. It is ~10× the controller's size and sees each chunk +independently, without the memory state that might bias the 3B's decision. + +This inherits the labeler's bias, which is why M5.2 exists and must run before +anyone trains on these labels. + +Constraint: `reasoning` has a **16384 total context** and vLLM rejects +`input + max_tokens > 16384`. A 5000-token chunk plus question plus instructions +fits with room; keep `max_tokens` small (labels are one token of signal) and do +not batch chunks into one prompt. + +The labeler emits a binary label plus a short justification. Keep the +justification — it is what makes M5.2's disagreement analysis possible. + +## Steps + +1. `mem label --project P --query Q` reads chunks from the log. +2. Per chunk, prompt `reasoning`: question, chunk, "does this contain evidence for + the question? Answer yes or no, then one sentence why." +3. Send **no tools** — the reasoning route rejects any request carrying them. +4. Write `label//.jsonl`: + `{"chunk_sha":"...","t":7,"label":true,"why":"...","model":"reasoning","ts":"..."}`. +5. Resumable: skip chunks already labelled. +6. Report the label rate — the fraction of chunks the labeler calls evidence. + Compare it to the controller's update-rate from M1.7; a large gap is the + finding, not a bug. + +## Acceptance + +- Every chunk in the log gets exactly one label. +- Labels key on `chunk_sha`, so they survive re-chunking only if content is + unchanged. +- Resume skips completed work. +- Label rate is reported alongside the controller's update-rate. + +## Verify + +**Harness:** scripted client offline; one `#[ignore]` live run. + +**Integration test** — `tests/it_label.rs`: +1. `a1_one_label_per_chunk` — no duplicates, no gaps against the log's chunks. +2. `a2_keyed_by_sha` — labels reference `chunk_sha`, not `t`, so reordering the + log does not corrupt them. +3. `a3_no_tools_sent` — assert the request body has no `tools` key. +4. `a4_context_budget` — assert every labeling prompt is under + 16384 − max_tokens. +5. `a5_resume` — label, rerun, assert zero new calls. +6. `a6_justification_kept` — every label has non-empty `why`. +7. `a7_rate_reported` — the summary prints both label rate and the controller's + update-rate. +8. `a8_live` — `#[ignore]`; 20 real chunks through `reasoning`; print the labels + and justifications for a human to sanity-check. + +**Command:** `cargo test -p mem-cli label` (add `-- --ignored` for a8) + +**False pass:** +- Keying labels by `t`. A re-chunk shifts every `t`, the labels silently + misalign, and the training set is quietly wrong in a way nothing downstream can + detect. +- Dropping the justification to save space. M5.2 then has nothing to analyse and + the calibration step degenerates into a single agreement number with no way to + understand it. + +## Traps + +- Batching several chunks into one labeling prompt to save calls. The labels + become order-dependent and the 16K context is exceeded on the third chunk. +- Treating the 32B's labels as ground truth. They are a *proxy*, and M5.2 is the + task that measures how good a proxy. + +--- + +Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.1 diff --git a/tasks/M5.2-labeler-calibration.md b/tasks/M5.2-labeler-calibration.md new file mode 100644 index 0000000..1c07836 --- /dev/null +++ b/tasks/M5.2-labeler-calibration.md @@ -0,0 +1,100 @@ +# M5.2 — Labeler calibration + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M5.1 | + +## Goal + +Measure how good the proxy is, before training a policy to imitate it. + +## Facts (inlined — no spec read needed) + +M5.1's labels are distant supervision. Training on them without measuring +agreement means the policy learns the 32B model's bias and reports it as +improvement — and under a single-model deployment there is no competing variant +whose divergence would make that visible. + +Method: hand-label a stratified holdout, compare, and report more than accuracy. + +- **Sample size**: 100 chunks is enough to distinguish 0.7 from 0.9 agreement. +- **Stratify** by the labeler's own answer — 50 it called evidence, 50 it did + not. Random sampling from a corpus where ~5% is evidence gives ~5 positives, + and the positive class is the one that matters. +- **Report Cohen's κ, not raw agreement.** With a 95/5 class balance, a labeler + that always says "no" scores 95% agreement and is useless. κ corrects for + chance. +- Also report precision and recall on the positive class separately. They fail + differently: low recall silently starves memory, low precision pollutes it. + +Disagreements are the artifact. Read them; they are usually either a genuinely +ambiguous chunk or a question that is too vague — and the second is fixable at +M1.2 and worth much more than a better labeler. + +## Steps + +1. `mem label sample --project P --n 100 --stratified` writes a blind + worksheet — chunk text and question, **no** labeler answer visible. +2. Hand-label it. Record the human labels separately. +3. `mem label calibrate` joins them; reports agreement, κ, precision, recall, + and the confusion matrix. +4. Dump all disagreements with both justifications side by side. +5. Commit the holdout and the human labels; they are reusable for every future + labeler change. +6. Gate: κ ≥ 0.6 before the labels are used for training. + +## Acceptance + +- Worksheet hides the labeler's answer. +- κ, precision, recall and the confusion matrix are all reported. +- Disagreements are dumped with justifications. +- The holdout is committed and reusable. + +## Verify + +**Harness:** a synthetic labeled set with known agreement, so the statistics +themselves are testable. + +**Integration test** — `tests/it_calibration.rs`: +1. `a1_worksheet_is_blind` — assert the labeler's answer appears nowhere in the + output file. +2. `a2_stratified` — assert the sample is ~50/50 by labeler answer, not corpus + proportional. +3. `a3_kappa_correct` — feed a set with hand-computed κ; assert the reported + value matches to 3 decimals. +4. `a4_kappa_vs_accuracy` — a synthetic all-negative labeler on a 95/5 set: + assert accuracy > 0.9 **and** κ ≈ 0. This is the assertion that justifies + reporting κ at all. +5. `a5_confusion_matrix` — all four cells match hand counts. +6. `a6_disagreements_dumped` — count equals off-diagonal total; each carries both + justifications. +7. `a7_holdout_stable` — rerunning the sampler with the same seed reproduces the + same chunks. + +**Command:** `cargo test -p mem-cli calibration` + +**False pass:** +- Reporting accuracy only. On this class balance it is nearly meaningless, and it + will look excellent right up until the trained policy learns to always answer + "no". +- Hand-labeling with the labeler's answer visible. Anchoring makes agreement look + high and the whole exercise decorative — assertion 1 is a real safeguard, not + hygiene. + +## Traps + +- Sampling proportionally. A 5%-positive corpus yields five positives in a + hundred, and precision on the positive class — the number that decides whether + memory gets polluted — is estimated from five examples. +- Treating low κ as "the labeler needs a better prompt". Check the *questions* + first: an ambiguous standing question makes evidence genuinely undecidable, and + no labeler can fix that. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Risks diff --git a/tasks/M5.3-training-corpus-export.md b/tasks/M5.3-training-corpus-export.md new file mode 100644 index 0000000..c086a9e --- /dev/null +++ b/tasks/M5.3-training-corpus-export.md @@ -0,0 +1,114 @@ +# M5.3 — Training corpus export + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M5.1 | + +## Goal + +Turn the log plus labels into trajectories verl can train on — the boundary +between the Rust side and the Python side. + +## Facts (inlined — no spec read needed) + +**The JSONL log is the boundary.** Rust produces it; Python consumes it. Nothing +else crosses, which is what keeps the two halves independent. + +A training example is a **trajectory**, not a turn: the paper's advantage mixes a +trajectory-level term with a turn-level one (`Â = α·Â_traj + (1−α)·Â_turn`, +α=0.9), so turns must stay grouped by run. + +Per turn, verl needs: the exact prompt sent, the exact response, and the rewards. + +```jsonl +{"trajectory_id":"01HXYZ","turns":[ + {"t":1,"prompt":"","response":"...", + "r_update":-1,"parsed":true}, + ...], + "r_exit":-0.5,"r_format":1,"r_outcome":null} +``` + +Reward assembly, from the paper: + +- `r_update_t` = +1 if the recorded `U_t` matches M5.1's label, −1 otherwise. +- `r_exit` — one value per trajectory: `0` if `t_exit == t_last_evidence`, + `−0.75` if earlier, `−0.5` if later. `t_last_evidence` is the largest `t` whose + label is true. **Note L1 runs never exit** (exit gate off), so every L1 + trajectory is a "late" exit at −0.5 unless the exit signal is taken from the + recorded `E_t` rather than the loop's behaviour — take it from the record. +- `r_format` = 1 only if **every** turn in the trajectory parsed, 0 otherwise. + Strict, because a malformed turn may be caused by the previous one. +- `r_outcome` is null. We have no answer-correctness signal; the paper's + `is_equiv(A, Â)` has no analogue in extraction. Say so explicitly rather than + fabricating one. + +The prompt must be the **exact bytes sent**, not re-assembled. Re-assembly drifts +from what the model actually saw the moment M1.3's template changes. + +## Steps + +1. `mem export --project P --format verl --out corpus/`. +2. Join log turns to labels by `chunk_sha`. +3. Reconstruct each turn's prompt from the recorded request if `MEM_LLM_RECORD` + captured it; otherwise fail loudly rather than re-assembling. +4. Compute rewards as above; carry `r_outcome: null` through. +5. Group by run into trajectories, ordered by `t`. +6. Emit a summary: trajectories, turns, positive/negative `r_update` split, + `r_format` pass rate, `t_last_evidence` distribution. +7. Refuse to export if M5.2's κ is below the threshold or absent. + +## Acceptance + +- Turns are grouped into trajectories, ordered. +- Prompts are byte-exact recordings, never re-assembled. +- `r_format` is 0 for a trajectory with any unparsed turn. +- Export is refused without calibration. + +## Verify + +**Harness:** log fixture with known labels and a deliberately unparsed turn. + +**Integration test** — `tests/it_export.rs`: +1. `a1_trajectory_grouping` — turns grouped by run, `t` ascending, none lost. +2. `a2_r_update_signs` — matching label → +1, mismatching → −1, checked per turn. +3. `a3_r_format_strict` — a trajectory with one unparsed turn scores 0 overall, + not per turn. +4. `a4_r_exit_from_record` — assert `r_exit` derives from the recorded `E_t`, not + from whether the loop stopped. A fixture where the gate said `end` at t=5 but + the loop continued must score as an exit at 5. +5. `a5_prompt_is_recorded_bytes` — assert the exported prompt equals the recorded + request body; corrupt the recording and assert export fails rather than + silently re-assembling. +6. `a6_r_outcome_null` — assert the field is present and null, not omitted and not + zero. +7. `a7_refuses_without_calibration` — no κ file → non-zero exit naming M5.2. +8. `a8_summary_counts` — reported splits match a hand count on the fixture. + +**Command:** `cargo test -p mem-cli export` + +**False pass:** +- Re-assembling prompts at export time. Every assertion except 5 passes, and the + policy is trained on prompts the model never saw — which shows up as a training + run that will not converge, with no obvious cause. +- Applying `r_format` per turn. It looks more granular and it is wrong: the paper + is explicit that the strictness exists because a bad turn may be caused by the + previous one. +- Emitting `r_outcome: 0` instead of null. Zero is a real reward value and the + trainer will use it as signal. + +## Traps + +- Deriving `r_exit` from loop behaviour at L1. The gate is switched off there, so + every trajectory scores as a late exit and the exit signal becomes constant + noise the policy cannot learn from. +- Joining labels by `t`. Same failure as M5.1's trap — a re-chunk misaligns + everything silently. + +--- + +Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2 diff --git a/tasks/M5.4-vllm-lora-serving.md b/tasks/M5.4-vllm-lora-serving.md new file mode 100644 index 0000000..cb03d6f --- /dev/null +++ b/tasks/M5.4-vllm-lora-serving.md @@ -0,0 +1,102 @@ +# M5.4 — vLLM + `--enable-lora` + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | L — 3+ days | +| Status | ⬜ Not started | +| Flags | homelab | +| Spec | inlined below | +| Blocks | — | + +## Goal + +A serving path that can load the memory adapter — because the current one cannot. + +## Facts (inlined — no spec read needed) + +**Ollama cannot hot-swap LoRA adapters.** The controller runs on +`qwen2.5:3b-instruct` served by Ollama today, which is fine for prompted-only use +and a dead end for post-training. vLLM supports `--enable-lora` with +`--lora-modules name=path`, serving base plus adapters from one resident model. + +The pattern already exists in this cluster: the `reasoning` predictor is +`vllm/vllm-openai:v0.11.0` under KServe, adopted into ArgoCD as +`k8s/apps/llm-serving/reasoning.yaml`. Copy its shape. + +**VRAM is the constraint that makes an adapter the right answer.** One GPU, +`OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A +separate full memory model evicts something, and eviction is a weights reload +measured in tens of seconds — `ornith`'s cold start already blew a 60s gateway +timeout once. A LoRA rides on a resident base for near-zero extra VRAM. + +Two hard-won operational facts to carry over: + +1. **Kong reads timeouts from the Service, not the Ingress.** `konghq.com/read-timeout` + on an Ingress is ignored; it must reach the predictor Service, and KServe + propagates InferenceService annotations there. Getting this wrong produces a + 504 at exactly 60s on the first cold request. +2. **Readiness must mean "can serve", not "process is up."** vLLM's startup probe + needs a long `failureThreshold` — model load plus torch compile measured 108s + + 55s on the 32B. Report ready too early and the first request 504s. + +## Steps + +1. `k8s/apps/llm-serving/memory.yaml` — InferenceService, vLLM, Qwen2.5-3B-Instruct. +2. Args: `--enable-lora`, `--max-lora-rank 32`, `--max-model-len 32768`, + `--served-model-name memory`. +3. Adapter storage: a PVC or an initContainer fetching from object storage; + `--lora-modules memory-v1=/mnt/adapters/memory-v1`. +4. Kong timeout annotations on the **InferenceService** metadata so KServe + propagates them to the Service. +5. Startup probe with a generous `failureThreshold`, gated on the OpenAI + `/health` endpoint. +6. New Kong route `/v1/memory/chat/completions` in `llm-routes.yaml`, with the + `model-key-auth` plugin — in namespace `llm-serving`, since a KongPlugin + reference resolves in the annotated object's own namespace and a dangling one + fails open. +7. Commit, push, let ArgoCD sync. No `kubectl apply`. + +## Acceptance + +- Base model answers through `/v1/memory/chat/completions`. +- A named adapter is selectable via the `model` field. +- Unauthenticated requests to the new route return 401. +- Cold start does not 504. + +## Verify + +**Harness:** `kubectl` and `curl` against the live gateway after sync. + +**Integration test** — `verify/m5.4.sh` diffed against `expected/m5.4.txt`: +1. `a1_isvc_ready` — InferenceService reports Ready. +2. `a2_base_completion` — a completion naming the base model returns 200. +3. `a3_adapter_selectable` — with a dummy adapter mounted, request `model: + memory-v1`; assert 200 and that it differs from the base response. +4. `a4_auth_enforced` — no key → 401; `apikey` header → 200. +5. `a5_cold_start_no_504` — delete the pod, wait for Ready, immediately send a + request; assert 200, not 504. This is the regression test for the timeout bug. +6. `a6_timeouts_on_service` — assert `konghq.com/read-timeout` is present on the + predictor **Service**, not only the Ingress. +7. `a7_vram_headroom` — with the memory model resident alongside the others, + assert `nvidia-smi` free memory stays above a threshold. + +**Command:** `bash verify/m5.4.sh | diff - expected/m5.4.txt` + +**False pass:** +- Testing after the model is warm. The 504 bug only appears on the first request + after a restart, which is exactly the case assertion 5 forces. +- Checking the timeout annotation on the Ingress. It is ignored there, and its + presence is what makes the bug hard to see. + +## Traps + +- Adding a third resident model without checking VRAM. Something gets evicted, + and the symptom is a slow unrelated model rather than an obvious failure. +- Putting the KongPlugin in the wrong namespace. It fails **open** — the route + serves unauthenticated and looks healthy, which is how the model routes ran + with no auth for a while. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Separate weights · `k8s/apps/llm-serving/` diff --git a/tasks/M5.5-verl-training-loop.md b/tasks/M5.5-verl-training-loop.md new file mode 100644 index 0000000..7e5fca8 --- /dev/null +++ b/tasks/M5.5-verl-training-loop.md @@ -0,0 +1,120 @@ +# M5.5 — verl training loop + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | L — 3+ days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M5.3, M5.4 | + +## Goal + +Train the LoRA that makes the update and exit gates better than prompting alone. + +## Facts (inlined — no spec read needed) + +Python, using verl (`github.com/volcengine/verl`), which is what the paper used. +Lives outside the Rust workspace; the JSONL corpus is the only interface. + +Paper hyperparameters, Table 3 — start here rather than guessing: + +``` +chunk size 5000 +max prompt length 8192 +max response length 2048 +clip ratio 0.20 +learning rate 1e-6 +temperature (train) 1.0 top_p 1.0 +temperature (val) 1.0 top_p 0.7 +train batch size 128 +rollout N 16 +mini batch size 128 +LR warmup steps 20 +``` + +Advantage, paper §3.2.2 — the part most likely to be implemented wrong: + +``` +Â_traj = r_traj_g − mean over the GROUP of trajectories +Â_turn = r_update_{g,t} − mean over turns AT STEP t across groups +Â = α·Â_traj + (1−α)·Â_turn α = 0.9 +``` + +Two distinct baselines. `Â_turn` is normalised across groups **at the same `t`**, +and the group size at step `t` can differ from the trajectory group size, because +trajectories that exited early have fewer turns. + +α=0.9 is the paper's default and the ablation (Figure 8) explains why: at α=1 +there is no update-gate reward and accuracy on evidence-free chunks collapses — +the model updates indiscriminately, which is exactly the failure this whole +system exists to avoid. + +Expect instability. The paper's own limitations section says the extra rewards +"reduce training stability, requiring a smaller off-policy degree and longer +convergence time." + +## Steps + +1. `training/` directory, Python, `uv`-managed. System Python is 3.9.6; this + needs 3.11+. +2. Corpus loader for M5.3's format. +3. Configure verl for LoRA on Qwen2.5-3B-Instruct, rank 16–32. +4. Implement the three rewards and the two-baseline advantage, α configurable. +5. Log per step: update accuracy split by evidence-present and evidence-free, + exact-exit ratio, format correctness, mean response length, validation reward. +6. Hold out a validation split by **project**, not by trajectory — same-project + trajectories share vocabulary and leak. +7. Export the adapter, version it, publish where M5.4 can mount it. + +## Acceptance + +- Training runs to convergence on the validation reward. +- Both advantage terms are computed with their own baselines. +- Update accuracy on evidence-free chunks does not collapse. +- The adapter loads in M5.4's server. + +## Verify + +**Harness:** pytest for reward and advantage maths; a short training run for the +loop itself. + +**Integration test** — `training/tests/test_rewards.py`: +1. `a1_r_update_signs` — matching label +1, mismatching −1. +2. `a2_r_exit_bands` — exact 0, early −0.75, late −0.5. +3. `a3_r_format_strict` — any unparsed turn zeroes the whole trajectory. +4. `a4_traj_baseline` — `Â_traj` uses the group mean; hand-computed fixture. +5. `a5_turn_baseline_at_step_t` — `Â_turn` normalises across groups at the same + `t`; a fixture with unequal trajectory lengths must not misalign. This is the + assertion that catches the most likely implementation error. +6. `a6_alpha_mix` — α=1 yields pure trajectory advantage; α=0 pure turn. +7. `a7_alpha_1_degenerates` — train 50 steps at α=1 on a fixture; assert + evidence-free accuracy drops relative to α=0.9, reproducing the paper's + Figure 8b. +8. `a8_validation_split_by_project` — assert no project appears in both splits. +9. `a9_adapter_loads` — export, mount in M5.4, assert a completion returns 200. + +**Command:** `uv run pytest training/tests -v` + +**False pass:** +- Normalising `Â_turn` over the whole batch rather than per step `t`. It trains, + loss goes down, and the turn-level signal is diluted into noise — assertion 5 + is the only thing that catches it. +- Splitting validation by trajectory. Same-project trajectories share phrasing + and file paths, so validation reward looks excellent and generalisation is + untested. +- Skipping assertion 7 as "too slow". It is the only end-to-end evidence that the + update reward is wired to anything. + +## Traps + +- Tuning α before the rewards are verified. Every α is wrong if `r_update`'s sign + is flipped, and the symptom looks identical. +- Training on labels whose κ was never measured. The policy learns the labeler, + and there is no held-out signal that would reveal it — M5.2 exists for this and + M5.3 refuses to export without it. + +--- + +Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.2, Table 3, Fig 8 diff --git a/tasks/M5.6-m5-gate.md b/tasks/M5.6-m5-gate.md new file mode 100644 index 0000000..b606e36 --- /dev/null +++ b/tasks/M5.6-m5-gate.md @@ -0,0 +1,106 @@ +# M5.6 — M5 composition gate + +| Field | Value | +|---|---| +| Phase | M5 — Post-training | +| Size | L — 3+ days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M5 | + +## Goal + +Establish that the trained adapter is actually better than prompting — and that +"better" was measured on data it never saw. + +## Facts (inlined — no spec read needed) + +The comparison is **adapter vs. the prompted baseline**, on a held-out project, +using the same corpus and the same prompt. + +The paper's own result sets the expectation: Figure 9 shows RL helps but the +prompted workflow already works, and the gains concentrate on harder tasks. So +a modest improvement is the realistic success case; a dramatic one is a reason to +check for leakage first. + +Metrics, and why each is present: + +| metric | why | +|---|---| +| update accuracy, evidence-present | recall — does it catch evidence | +| update accuracy, evidence-free | precision — does it resist noise. **This is the one that collapses at α=1** | +| exact-exit ratio | did the exit gate learn anything | +| format correctness | the parser's job got easier or harder | +| memory token curve | the Figure 6 shape — flat, or climbing to the cap | +| wall clock per run | the paper claims up to 400% speedup with the exit gate | + +**Held-out project is non-negotiable.** Same-project trajectories share file +paths, error strings and phrasing; measuring on them is measuring memorisation. + +Report both, always: an adapter that improves evidence-present accuracy while +degrading evidence-free is worse for this system, because polluted memory +degrades every subsequent turn. + +## Steps + +1. Pick a project excluded from training. Ingest it with the prompted baseline; + record all metrics. +2. Ingest the same project with the adapter, identical prompt and chunking. +3. Compare on the M5.2 holdout labels. +4. Assert the thresholds below. +5. Plot the memory-token curve for both; commit it as the Figure 6 analogue. +6. Commit `expected/m5.6.txt`; diff. +7. If the adapter loses, keep it versioned and record why — a negative result with + the reason is worth more than a rerun with different hyperparameters. + +## Acceptance + +- Evidence-present accuracy ≥ baseline. +- Evidence-free accuracy ≥ baseline. Never traded away. +- Format correctness ≥ baseline. +- Memory token curve flat, not climbing to the cap. +- Measured on a project absent from training. + +## Verify + +**Harness:** live gateway with both models; the held-out project; M5.2's labels. + +**Integration test** — `verify/m5.6.sh` diffed against `expected/m5.6.txt`: +1. `a1_holdout_is_unseen` — assert the eval project appears in no training + trajectory. Check the corpus, not the config. +2. `a2_evidence_present_accuracy` — adapter ≥ baseline; print both. +3. `a3_evidence_free_accuracy` — adapter ≥ baseline; print both. +4. `a4_format_correctness` — adapter ≥ baseline. +5. `a5_memory_curve_flat` — slope below the M1.8 bound for both; assert the + adapter is no worse. +6. `a6_exit_ratio_reported` — print exact/early/late exit ratios. Advisory: the + exit gate is off at L1, so this measures the signal, not behaviour. +7. `a7_same_prompt` — assert both runs used a byte-identical prompt template. +8. `a8_wall_clock` — report both; no threshold, since the exit gate is off at L1. + +**Command:** `bash verify/m5.6.sh | diff - expected/m5.6.txt` + +**False pass:** +- Comparing on a project that was in training. Everything improves and none of it + generalises — assertion 1 checks the corpus rather than trusting the split + config. +- Reporting a single combined accuracy. It hides the precision/recall trade that + matters most: a model that says "yes" more often scores better on + evidence-present and pollutes memory, and the combined number can improve while + the system gets worse. +- Different prompts between runs. Any measured difference then attributes to the + adapter and is partly the prompt — assertion 7 is cheap and removes the doubt. + +## Traps + +- Retraining until the gate passes without changing anything principled. That is + fitting the gate, and the held-out project stops being held out the third time + you look at it. +- Reading a small improvement as failure. The paper's own baseline works; the + adapter's value here is as much about stability on evidence-free chunks as + headline accuracy. + +--- + +Background: [DESIGN.md](../DESIGN.md) — Verification, Risks · paper Fig 9 diff --git a/tasks/M6.1-agent-manager-db-manifest.md b/tasks/M6.1-agent-manager-db-manifest.md new file mode 100644 index 0000000..c008dad --- /dev/null +++ b/tasks/M6.1-agent-manager-db-manifest.md @@ -0,0 +1,122 @@ +# M6.1 — CNPG `agent-manager-db` manifest + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | homelab | +| Spec | inlined below | +| Blocks | — | + +## Goal + +A dedicated Postgres for agent-manager's session store, provisioned the way +everything else in the cluster is: through git, with no manual `psql`. Same +pattern this project already used for `memory-db` (M2.2), applied to a +different, unrelated app. + +## Facts (inlined — no spec read needed) + +**agent-manager is a separate repo**, not part of this Rust workspace: +`github.com/Riotpiaole/agent-manager` (fork of `github.com/YoanWai/ +agent-manager`), local checkout `~/workplace/agent-manager`, branch +`add-headless-spawn`. Its session store is `internal/store/store.go` — see +M6.2/M6.3 for the schema and query port. + +**Existing CNPG pattern, verified on the live cluster: 3 Clusters today, one +per app, each ClusterIP-only (not LAN-reachable):** + +| namespace/name | app | +|---|---| +| `cicd/forgejo-db` | Forgejo | +| `iam/authentik-db` | Authentik | +| `temporal/temporal-db` | Temporal | + +**No shared/multi-tenant DB cluster** — every app gets its own dedicated +CNPG `Cluster`. `agent-manager-db` follows the same rule; it does not join +`memory-db` (M2.2's cluster) even though both are new Postgres instances +touched by the same person around the same time. Different app, different +cluster. + +Follow `k8s/infra/databases/temporal-db.yaml` exactly, same as M2.2 did: +`imageName` pinned, `enableSuperuserAccess: false`, `storageClass: +longhorn-cnpg`, `enablePodMonitor: true`, control-plane tolerations, +`podAntiAffinityType: preferred`. + +**Instance count — open question, default to convention.** Motivation for +this whole migration is durability-of-location, not HA (single-machine +usage, not a multi-host shared-session requirement). The 3 existing +clusters are all 3-instance. Default to 3 instances for consistency with +every other app in the cluster rather than special-casing this one to 1; +revisit only if resource pressure on the homelab nodes makes it a real +tradeoff. + +Storage: session rows are tiny (`sessions`, `groups`, `settings`, +`review_*` — no blobs beyond a `snapshot TEXT` pane capture per session). +1Gi is generous; no need for `memory-db`'s 10Gi (that one holds +768-dim vectors). + +## Steps + +1. `k8s/infra/databases/agent-manager-db.yaml` — `Cluster` + `Database`, + namespace `agent-manager`, no extensions (plain relational, no + pgvector). +2. Namespace `agent-manager`, created by the ArgoCD app that owns it. +3. Add to the owning kustomization's explicit resource list — an unlisted + file is silently dropped with no error and no drift shown (the M2.2 + task file names this exact trap). +4. Commit, push to **both** Forgejo origin and the GitHub mirror — verify + which `repoURL` the eventual ArgoCD `Application` for this app actually + watches before assuming either push is the one that matters (`kong` + app, for example, tracks the GitHub mirror specifically, not Forgejo). +5. Let ArgoCD sync. **No `kubectl apply`.** +6. Verify the app user can create tables (schema arrives in M6.2, but a + throwaway `CREATE TABLE t(id text); DROP TABLE t;` proves connectivity + here). + +## Acceptance + +- `Cluster` reaches `Cluster in healthy state`. +- ArgoCD shows the app `Synced/Healthy`. +- No manual `psql` was run to get there. +- Service is ClusterIP-only — not reachable from the LAN directly (M6.4's + nginx route is the only path in). + +## Verify + +**Harness:** `kubectl` and `psql` read-only checks after sync. + +**Integration test** — `verify/m6.1.sh`, output diffed against +`expected/m6.1.txt`: +1. `a1_cluster_healthy` — `kubectl get cluster -n agent-manager + agent-manager-db` reports all instances ready. +2. `a2_clusterip_only` — `kubectl get svc -n agent-manager -o + jsonpath='{.items[*].spec.type}'` contains no `LoadBalancer` or + `NodePort`. +3. `a3_argocd_synced` — the owning app is `Synced/Healthy`. +4. `a4_app_user_can_ddl` — as `app`, `CREATE TABLE t(id text); DROP TABLE + t;` succeeds. +5. `a5_no_lan_route_yet` — connection attempt from outside the cluster + network fails at this point in the plan (M6.4 hasn't landed). + +**Command:** `bash verify/m6.1.sh | diff - expected/m6.1.txt` + +**False pass:** +- Confirming sync without checking service type. A `Cluster` can be + healthy and `Synced` while someone fat-fingered a `LoadBalancer` type + into the manifest, silently violating the "dedicated ingress, not raw + LAN IP" network-path decision this whole migration made. Assertion 2 is + the guard. + +## Traps + +- Forgetting the kustomization resource list (same trap M2.2 already + named) — file sits in git, ArgoCD reports Synced, objects never exist. +- Adding `prune: true` without accounting for CNPG-operator-created + children (Services, Secrets, PVCs). M2.2's Traps section already hit + this on `llm-serving`; same fix applies here (`prune: false`). + +--- + +Background: `k8s/infra/databases/temporal-db.yaml` · [M2.2](M2.2-memory-db-manifest.md) (same pattern, different app) diff --git a/tasks/M6.2-schema-port.md b/tasks/M6.2-schema-port.md new file mode 100644 index 0000000..60eab1e --- /dev/null +++ b/tasks/M6.2-schema-port.md @@ -0,0 +1,243 @@ +# M6.2 — Postgres schema for agent-manager sessions + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M6.1 | + +## Goal + +The sqlite schema agent-manager's `internal/store/store.go` builds up +through 14 `ALTER TABLE` migrations, collapsed into one Postgres-native +schema — with the sqlite workarounds (integer booleans, dual-encoded +timestamps) removed rather than carried over. + +## Facts (inlined — no spec read needed) + +Current schema, read directly from `internal/store/store.go` (6 tables, no +FK constraints anywhere — every relationship is enforced in Go, not SQL): + +```mermaid +erDiagram + GROUPS ||--o{ SESSIONS : "group_name (app-level, no FK)" + SESSIONS ||--o{ SESSIONS : "parent_id (self-ref, app-level, no FK)" + SESSIONS ||--o| REVIEW_TARGETS : "session_id (app-level, no FK)" + SESSIONS ||--o{ REVIEW_BASES : "session_id (app-level, no FK)" + SESSIONS ||--o| REVIEW_SCOPES : "session_id (app-level, no FK)" + SETTINGS { + text key PK + text value + } + SESSIONS { + text id PK + text name + text tool + text cwd + text group_name "app-level FK -> GROUPS.name" + text status + int archived "bool 0/1 in sqlite" + int created_at "unix nanos, dual-encoded in sqlite" + int last_status_at "unix nanos" + text agent_session_id + text pending_inputs "JSON array blob" + int pending_claimed "bool 0/1" + text launch_prompt + int sort_order + int acked "bool 0/1" + text snapshot + text worktree_repo + text worktree_branch + int agent_launched_at "unix nanos" + text retired_agent_session_id + text parent_id "app-level FK -> SESSIONS.id, self" + } + GROUPS { + text name PK + int sort_order + text path + int archived "bool 0/1" + text worktree + } + REVIEW_TARGETS { + text session_id PK + text repo_root + } + REVIEW_BASES { + text session_id PK + text repo_root PK + text base_ref + } + REVIEW_SCOPES { + text session_id PK + text scope + } +``` + +Target Postgres DDL: + +```sql +CREATE TABLE groups ( + name TEXT PRIMARY KEY, + sort_order INTEGER NOT NULL DEFAULT 0, + path TEXT NOT NULL DEFAULT '', + archived BOOLEAN NOT NULL DEFAULT FALSE, + worktree TEXT NOT NULL DEFAULT '' +); + +CREATE TABLE sessions ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + tool TEXT NOT NULL, + cwd TEXT NOT NULL, + group_name TEXT NOT NULL REFERENCES groups(name), + status TEXT NOT NULL, + archived BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL, + last_status_at TIMESTAMPTZ NOT NULL, + agent_session_id TEXT NOT NULL DEFAULT '', + pending_inputs JSONB NOT NULL DEFAULT '[]', + pending_claimed BOOLEAN NOT NULL DEFAULT FALSE, + launch_prompt TEXT NOT NULL DEFAULT '', + sort_order INTEGER NOT NULL DEFAULT 0, + acked BOOLEAN NOT NULL DEFAULT FALSE, + snapshot TEXT NOT NULL DEFAULT '', + worktree_repo TEXT NOT NULL DEFAULT '', + worktree_branch TEXT NOT NULL DEFAULT '', + agent_launched_at TIMESTAMPTZ, + retired_agent_session_id TEXT NOT NULL DEFAULT '', + parent_id TEXT REFERENCES sessions(id) +); + +CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); + +CREATE TABLE review_targets ( + session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, + repo_root TEXT NOT NULL +); + +CREATE TABLE review_bases ( + session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE, + repo_root TEXT NOT NULL, + base_ref TEXT NOT NULL, + PRIMARY KEY (session_id, repo_root) +); + +CREATE TABLE review_scopes ( + session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, + scope TEXT NOT NULL +); +``` + +**Decision: add real FKs, not app-level-only.** Current Go code manually +cascades `review_targets`/`review_bases`/`review_scopes` deletes before +deleting a `sessions` row (`Delete`, `DeleteChild` in store.go) — `ON +DELETE CASCADE` on those three removes that manual bookkeeping entirely. +`group_name -> groups(name)` and `parent_id -> sessions(id)` get FKs too, +since Postgres makes them free and they catch the exact class of bug +`validParent`/`ensureGroup` exist in Go to prevent by hand. This is a +behavior change from sqlite (constraint violation now possible on write +paths that previously just wrote garbage) — M6.3 must add error handling +for FK violations at the ~4 call sites that write `group_name`/ +`parent_id` without having already validated the reference through +`validParent`/`ensureGroup`. + +**Booleans** — `BOOLEAN`, not `INTEGER`. Removes `boolToInt()`/`!= 0` at +every read/write site in store.go. + +**Timestamps** — `TIMESTAMPTZ`, not the `encodeTime`/`decodeTime` +nanosecond-or-legacy-seconds hack (store.go:1369-1396). That hack exists +only because sqlite has no native timestamp type and old rows needed a +seconds/nanos disambiguation heuristic (`secondsCeiling`). A fresh +Postgres database has no legacy rows — the whole function pair is deleted, +not ported. `agent_launched_at` becomes nullable (`NULL` = "never +restarted") instead of the sqlite sentinel `0`. + +**`pending_inputs`** — `JSONB`, not `TEXT` holding a JSON string. Same +`json.Marshal`/`Unmarshal` round-trip in Go either way, but `JSONB` lets +Postgres validate the shape on write instead of accepting malformed JSON +that only fails on the next read. + +## Steps + +1. Write `agent-manager` migration files (this repo's `sqlx`-migration + convention doesn't apply — agent-manager is a separate Go repo; use + whatever migration tool its `add-headless-spawn` branch already has, or + a plain `.sql` file run once at `Open()` if it has none — check before + introducing a new dependency). +2. `CREATE TABLE` in FK-dependency order: `groups`, then `sessions` + (references `groups` and itself), then `settings`, + `review_targets`/`review_bases`/`review_scopes`. +3. No indexes beyond the primary keys are needed yet — `ListSessions` + filters on `archived` and orders by `group_name, sort_order, + created_at`; add a composite index only if M6.6's gate shows it's slow, + not preemptively. +4. Confirm `CREATE EXTENSION` is not needed anywhere (plain relational + schema, no pgvector) — unlike this project's own `memory-db`. + +## Acceptance + +- Schema applies to a clean `agent-manager-db` database. +- `archived`/`acked`/`pending_claimed` are `BOOLEAN`, not `INTEGER`. +- `created_at`/`last_status_at`/`agent_launched_at` are `TIMESTAMPTZ`. +- Deleting a `sessions` row cascades `review_targets`/`review_bases`/ + `review_scopes` without the Go code doing it manually. +- Inserting a session with an unknown `group_name` is rejected by the FK, + not silently written. + +## Verify + +**Harness:** disposable Postgres, same image as M6.1's `Cluster` +(`ghcr.io/cloudnative-pg/postgresql:16.2`) — or a local `postgres:16` +container for fast iteration, since agent-manager's own test suite doesn't +need the real cluster. + +**Integration test** — extend agent-manager's existing store tests +(`internal/store/*_test.go` already has `timeenc_test.go`; that file is +deleted in M6.3, its coverage folded into this schema's tests) with: +1. `a1_migrate_clean` — apply to an empty database, assert all 6 tables + exist. +2. `a2_bool_columns_are_boolean` — `information_schema.columns` reports + `boolean` for `archived`, `acked`, `pending_claimed`. +3. `a3_timestamp_columns_are_timestamptz` — same, for `created_at`, + `last_status_at`, `agent_launched_at`. +4. `a4_review_cascade` — insert a session + a `review_targets` row for it, + delete the session, assert `review_targets` is empty without a separate + `DELETE FROM review_targets` call. +5. `a5_group_fk_rejects_unknown` — insert a session with a `group_name` + that has no matching `groups` row, assert it's rejected. +6. `a6_parent_fk_self_ref` — insert two sessions where the second's + `parent_id` points at the first, assert it succeeds; point it at a + nonexistent id, assert it's rejected. + +**Command:** `go test ./internal/store/... -run TestSchema` + +**False pass:** +- Testing the schema against sqlite (leftover `modernc.org/sqlite` import) + instead of real Postgres. `BOOLEAN`/`TIMESTAMPTZ`/`JSONB`/FK-cascade + behavior all differ or are silently accepted-but-ignored by sqlite — + every assertion above becomes meaningless against the wrong engine. + +## Traps + +- Keeping `ON DELETE CASCADE` off `group_name`/`parent_id` FKs by + accident (only adding it to the `review_*` ones). A group delete or + session delete then hits an FK violation instead of the graceful + "session has terminals of its own; move them out first" error + `PlaceSession` already gives — check M6.3 preserves that message instead + of leaking a raw constraint-violation error to the CLI. +- Making `parent_id` `NOT NULL DEFAULT ''` (matching the sqlite default) + instead of nullable. `'' REFERENCES sessions(id)` is never satisfiable + except by a literal empty-string-id row, which doesn't exist — every + top-level session's insert then fails the FK. Must be `NULL` for "no + parent." + +--- + +Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch) diff --git a/tasks/M6.3-store-query-port.md b/tasks/M6.3-store-query-port.md new file mode 100644 index 0000000..50f864b --- /dev/null +++ b/tasks/M6.3-store-query-port.md @@ -0,0 +1,160 @@ +# M6.3 — store.go query port to Postgres + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | L — 3+ days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M6.2 | + +## Goal + +Every query in `internal/store/store.go` rewritten against the M6.2 schema, +sqlite dropped entirely (decided: full port, not a dual sqlite/Postgres +backend — this is single-machine usage, no standalone-without-cluster +requirement to preserve). + +## Facts (inlined — no spec read needed) + +Read directly from `internal/store/store.go` (1397 lines) on the +`add-headless-spawn` branch: + +- **65 call sites** (`db.Exec`, `db.QueryRow`, `db.Query`, `tx.Exec`, + `tx.QueryRow`) use sqlite `?` positional placeholders. Postgres + (`pgx` or `lib/pq`) needs `$1, $2, ...` — mechanical but must be done + per-statement since arg count varies 1–16 across sites. +- **Upserts are already Postgres-compatible.** All 9 upsert sites + (`SetReviewRepo`, `SetReviewBase`, `SetReviewScope`, `SetSetting`, + `ensureGroup`, `CreateGroup`, `AddGroup`, `createSession`'s group + insert, `PlaceSession`'s group insert) already use `ON + CONFLICT(...) DO UPDATE SET ... = excluded....` / `ON CONFLICT(...) DO + NOTHING`, valid as-is in Postgres. No rewrite beyond the placeholder + swap — this corrects an earlier assumption that these were sqlite + `INSERT OR REPLACE` and needed real rework. +- `boolToInt()`/`archived != 0`-style conversions at ~8 sites go away — + scan `int archived`/`int acked`/`int pendingClaimed`/`int` locals in + `ListSessions`, `Get` and replace with `bool` directly. +- `encodeTime()`/`decodeTime()` calls at every session-timestamp + read/write go away — pass `time.Time` straight through; `pgx` handles + `TIMESTAMPTZ` natively. Delete `timeenc_test.go` along with the + functions it tests (M6.2 already noted its coverage folds into the + schema tests instead). +- `db.SetMaxOpenConns(1)` + `PRAGMA journal_mode=WAL` + (store.go:76-79) gave sqlite serialized writes for free. Some call + sites lean on that implicit serialization — most notably + `createSession`'s `sort_order = (SELECT COALESCE(MAX(sort_order)+1, 0) + FROM sessions WHERE group_name = ? AND parent_id = ?)` subquery, which + races under concurrent Postgres writers with a real connection pool. + Wrap it in `SELECT ... FOR UPDATE` inside the existing transaction (the + function already opens one) rather than trusting single-writer + semantics that no longer exist. Same pattern applies to + `ReorderSession`/`SwapSessionOrder`/`ReorderGroup`/`SwapGroupOrder`'s + read-then-renumber-then-write sequences. +- FK violations are new failure modes M6.2 introduced (group/parent FKs). + `createSession` and `PlaceSession` already validate `group_name`/ + `parent_id` through `validParent`/`ensureGroup` before writing, so those + paths shouldn't hit a live constraint in practice — but wrap the actual + `INSERT`/`UPDATE` error and translate a `23503` (foreign_key_violation) + SQLSTATE into the same descriptive errors the pre-validation already + produces, so a race between the check and the write degrades to a clear + error instead of a raw driver error reaching the CLI. +- `driver: modernc.org/sqlite` import and the blank `_ "modernc.org/ + sqlite"` in `Open()` are deleted; replaced with `pgx` (`github.com/ + jackc/pgx/v5/stdlib` for `database/sql` compatibility, keeping the rest + of the file's `*sql.DB`-based code unchanged) or a native `pgx.Pool` — + pick `pgx/v5/stdlib` unless a later task needs pgx-native features + (e.g. `COPY`), since it's the smaller diff against the existing + `database/sql` code. + +## Steps + +1. Swap the driver import and `Open()`'s connection string handling + (sqlite file path -> Postgres DSN, likely from an env var or flag the + CLI already has a slot for — check `cmd/` for how `Open()` is called + today). +2. Delete `db.SetMaxOpenConns(1)` / `PRAGMA journal_mode=WAL`; size the + pool deliberately instead (`SetMaxOpenConns` to something sane for a + single-machine client, e.g. 5–10). +3. Delete `init()`'s `CREATE TABLE IF NOT EXISTS` + 14-migration list — + M6.2's migration owns schema creation now; `Open()` just connects and + optionally runs a `schema_version` sanity check. +4. Mechanically convert every `?` to `$N` across the 65 call sites, in + file order, verifying arg count against placeholder count each time + (this is where an off-by-one is easiest to introduce silently). +5. Remove `boolToInt`/`archived != 0` conversions; scan struct fields + directly as `bool`. +6. Remove `encodeTime`/`decodeTime`/`secondsCeiling`; pass `time.Time` + directly. Delete `timeenc_test.go`. +7. Add `SELECT ... FOR UPDATE` (or equivalent explicit locking) to the + sort-order read-then-write sequences named above. +8. Add FK-violation (`23503`) error translation at the write sites that + can theoretically race past their own pre-validation. +9. Run agent-manager's full existing test suite against a real Postgres + (M6.1's cluster, or local `postgres:16` for iteration) — every current + test should still pass unmodified in intent, only in backing store. + +## Acceptance + +- Zero references to `modernc.org/sqlite`, `?` placeholders, + `boolToInt`, `encodeTime`/`decodeTime` remain in `internal/store/`. +- Full existing store test suite passes against Postgres. +- Concurrent `CreateSession` calls (simulated) never produce duplicate + `sort_order` values within the same `group_name`/`parent_id`. +- A `spawn` CLI round trip (create session, update status, delete) works + end-to-end against the M6.1 cluster. + +## Verify + +**Harness:** `internal/store/*_test.go` running against the disposable +Postgres from M6.2's harness. + +**Integration test** — extend/rename the existing store test files: +1. `a1_no_sqlite_references` — `grep -r "modernc.org/sqlite" internal/` + returns nothing. +2. `a2_no_bare_placeholders` — `grep -rE '\?[,)]' internal/store/store.go` + inside SQL string literals returns nothing (manual review of any + `?` that's part of a non-SQL string, e.g. a Go format verb, to avoid a + false positive). +3. `a3_existing_suite_passes` — `go test ./internal/store/...` green + against Postgres. +4. `a4_concurrent_sort_order` — spawn N goroutines each calling + `CreateSession` into the same group concurrently; assert the resulting + `sort_order` values are a dense 0..N-1 permutation with no duplicates. +5. `a5_fk_violation_translated` — attempt `PlaceSession` with a + `parentID` deleted between the read and the write (simulate via a + second connection); assert the returned error is the existing + descriptive one, not a raw pgx driver error. +6. `a6_spawn_roundtrip` — `spawn` CLI subcommand creates a session, + `UpdateStatus`, then `Delete`; assert no error and the row is gone. + +**Command:** `go test ./internal/... -run TestStore -v` + +**False pass:** +- Running the suite against sqlite still (leftover build tag or import) + while believing it validated Postgres. Assertion 1 is the guard — + without it, the whole task could report green while the driver swap + never actually happened. +- Skipping assertion 4. A `sort_order` race is invisible in every + single-threaded test and only shows up as duplicate/out-of-order + sessions under real concurrent use, which is exactly the class of bug + removing `SetMaxOpenConns(1)` introduces. + +## Traps + +- Converting `?` to `$N` by simple find-and-replace in file order without + re-checking each statement's actual arg list — a multi-arg statement + reordered during earlier edits (e.g. `createSession`'s 15-arg INSERT) + will silently bind the wrong value to the wrong column, and Go's + `database/sql` won't catch a type-compatible mismatch (e.g. two `TEXT` + columns swapped). +- Forgetting `pending_inputs` is now `JSONB` (M6.2), not `TEXT` — the + existing `json.Marshal`/`Unmarshal` round-trip in + `encodePendingInputs`/`pendingInputState` still works unchanged (pgx + scans `JSONB` into `[]byte` the same as `TEXT`), but don't add an extra + marshal layer thinking the column type changed the Go-side contract. + +--- + +Background: [M6.2](M6.2-schema-port.md) · `internal/store/store.go`, `internal/spawn/spawn.go` (agent-manager, `add-headless-spawn` branch) diff --git a/tasks/M6.4-nginx-stream-routing.md b/tasks/M6.4-nginx-stream-routing.md new file mode 100644 index 0000000..b672cd1 --- /dev/null +++ b/tasks/M6.4-nginx-stream-routing.md @@ -0,0 +1,122 @@ +# M6.4 — nginx TCP routing to `agent-manager-db` + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | S — <1 day | +| Status | ⬜ Not started | +| Flags | homelab | +| Spec | inlined below | +| Blocks | M6.1 | + +## Goal + +A path from the Mac client running agent-manager to the cluster-internal, +ClusterIP-only `agent-manager-db` — through the shared ingress controller, +matching this homelab's existing pattern, not a raw LAN IP or a tunnel. + +## Facts (inlined — no spec read needed) + +**Decided, not open:** dedicated ingress routing through nginx, not +`kubectl port-forward`/SSH tunnel and not a MetalLB `LoadBalancer` IP. +Matches the pattern already used elsewhere in this homelab of routing +through the shared ingress controller rather than exposing raw +per-service LAN IPs. + +**Postgres is not HTTP.** The standard nginx-ingress `Ingress` resource is +HTTP(S)-oriented (host/path routing, TLS termination via SNI on 443). +Postgres speaks its own binary wire protocol on 5432. The ingress +controller needs `stream {}` block config (TCP/UDP passthrough) or a +dedicated `TCP` mode `Service`/`ConfigMap` entry — whichever the specific +nginx-ingress deployment in this cluster supports (check +`k8s/infra/ingress/` for how it's deployed and whether `tcp-services` +ConfigMap wiring already exists for anything else, since this may be the +first TCP passthrough case in the cluster). + +**No existing precedent in this homelab** — `forgejo-db`, `authentik-db`, +`temporal-db` are all consumed only by pods inside the same cluster over +their ClusterIP Service, never from outside. `agent-manager-db` is the +first case of an external (Mac) client needing to reach a CNPG cluster, +which is why this task exists as dedicated work rather than "just add a +Service." + +## Steps + +1. Confirm how nginx-ingress is deployed in this cluster (`k8s/infra/ + ingress/`) and whether it already exposes a `tcp-services` ConfigMap + or `stream {}` snippet mechanism — ingress-nginx (the community + controller) supports TCP passthrough via a `tcp-services` ConfigMap + mapping `: /:`; confirm this + is the controller in use before assuming that config shape. +2. Pick an external port for Postgres traffic (5432 is already the + in-cluster default; an external port distinct from any other exposed + service avoids collision — check what's already claimed). +3. Add the `tcp-services` (or equivalent) entry routing that external + port to `agent-manager-db-rw.agent-manager.svc.cluster.local:5432` + (CNPG's read-write Service name convention — confirm against the + actual Service name M6.1's `Cluster` generates). +4. Expose that port on the ingress controller's `Service`/`LoadBalancer` + (this is the one LAN-facing port for this whole feature — the DB + itself stays ClusterIP-only, only the ingress controller's existing + external IP gains a new port). +5. Commit, push to both remotes, verify which `repoURL` the owning + ArgoCD `Application` watches (same caveat as M6.1) before assuming a + push landed, let ArgoCD sync. +6. Test connectivity from the Mac client: `psql + postgresql://@:/ + agent_manager` (credentials from M6.5). + +## Acceptance + +- `psql` (or `pgx`) from outside the cluster reaches `agent-manager-db` + through the ingress controller's external IP/port. +- `agent-manager-db`'s own Service remains ClusterIP-only — no + `LoadBalancer`/`NodePort` added to it directly (that would defeat the + point of routing through nginx). +- TLS/auth on the connection is Postgres's own (`sslmode`, password auth) + — nginx `stream {}` passthrough does not terminate or inspect the + Postgres protocol, so it adds no auth of its own. Confirm this is + acceptable given the homelab's network boundary (LAN-only ingress + exposure, not public internet) before treating it as done. + +## Verify + +**Harness:** `psql` from the Mac client (outside the cluster network), +plus `kubectl` checks on the ingress controller's config. + +**Integration test** — `verify/m6.4.sh` diffed against +`expected/m6.4.txt`: +1. `a1_external_connects` — `psql + postgresql://app@:/agent_manager -c 'select 1'` + from the Mac client succeeds. +2. `a2_db_service_still_clusterip` — `kubectl get svc -n agent-manager + agent-manager-db-rw -o jsonpath='{.spec.type}'` is `ClusterIP`. +3. `a3_argocd_synced` — the ingress-owning app is `Synced/Healthy` after + the config change. +4. `a4_wrong_port_refused` — connecting to a random unmapped port on the + same ingress host fails (proves the mapping is port-specific, not an + accidental catch-all passthrough). + +**Command:** `bash verify/m6.4.sh | diff - expected/m6.4.txt` + +**False pass:** +- Testing connectivity from inside the cluster (e.g. `kubectl exec` into + a pod and `psql` the ClusterIP directly). That was already true before + this task and proves nothing about the ingress path — assertion 1 must + run from the actual Mac client, outside the cluster network. + +## Traps + +- Reusing port 5432 externally on the ingress controller's existing + external IP if anything else is already listening there (unlikely for + Postgres specifically, but worth a `kubectl get svc -n + -o yaml` check before assuming the port is free). +- `stream {}` / `tcp-services` config living outside the GitOps-tracked + kustomization because it's a ConfigMap edit that "felt like a quick + manual fix." Same hard rule as everything else: commit + push + + ArgoCD sync, no manual `kubectl apply` to the ingress controller's + config. + +--- + +Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `k8s/infra/ingress/` (nginx-ingress deployment, controller type to confirm) diff --git a/tasks/M6.5-credentials-secret.md b/tasks/M6.5-credentials-secret.md new file mode 100644 index 0000000..1562735 --- /dev/null +++ b/tasks/M6.5-credentials-secret.md @@ -0,0 +1,110 @@ +# M6.5 — Postgres credentials for the Mac client + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | S — <1 day | +| Status | ⬜ Not started | +| Flags | homelab | +| Spec | inlined below | +| Blocks | M6.1 | + +## Goal + +The `agent-manager` process running on the Mac gets a Postgres connection +string/credentials, managed the same ksops way the rest of this homelab +handles secrets — not a password pasted into a local config file or env +var by hand. + +## Facts (inlined — no spec read needed) + +**Existing pattern to follow: ksops-managed secret**, same as +`model-invoke-apikey` elsewhere in this homelab. CNPG itself already +generates an in-cluster Secret for the `app` user +(`agent-manager-db-app` by its usual naming convention) — the work here is +getting that credential (or a dedicated read-write user, if reusing the +CNPG-generated superuser-adjacent `app` credential isn't desired) to a +process running outside the cluster, on the Mac. + +**Two credentials touch two different trust boundaries:** +- Cluster-internal: CNPG's own generated Secret, already ksops-free + (CNPG manages it, not this repo). + - Mac client: needs that same username/password (or a separate, + narrower-scoped user) delivered to `~/workplace/agent-manager`'s + runtime config, via a ksops-encrypted file in git rather than a + manually-copied value. + +## Steps + +1. Decide: reuse CNPG's auto-generated `app` user, or create a dedicated + `agent_manager_client` role scoped to only the `sessions`/`groups`/ + `settings`/`review_*` tables (narrower blast radius if the credential + ever leaks from a Mac laptop, which is a meaningfully different threat + model than a credential that only ever lives inside the cluster). +2. If a dedicated role: add it via CNPG's declarative `Database` / + `postInitSQL` (or a one-time migration in M6.2's schema setup) rather + than a manual `psql` grant. +3. Encrypt the resulting connection string (or user/password pair) with + ksops, following the exact file layout `model-invoke-apikey` uses. +4. Commit the encrypted secret to git (safe — that's the point of ksops), + push to both remotes. +5. Wire agent-manager's config loading (check `cmd/` / existing config + file handling on the `add-headless-spawn` branch) to read the + decrypted value at runtime — decide whether decryption happens via a + `sops exec-env`-style wrapper the Mac invokes, or a decrypted file + materialized once locally and gitignored, matching whatever + `model-invoke-apikey`'s consumers already do. +6. Verify agent-manager connects using only the ksops-sourced credential + — no plaintext password anywhere in the repo or in shell history. + +## Acceptance + +- No Postgres password appears in plaintext in git, in + `~/workplace/agent-manager`'s tracked config, or was typed directly + into a `kubectl`/`psql` command during setup. +- The credential is encrypted with ksops in the same repo location/ + pattern as `model-invoke-apikey`. +- agent-manager on the Mac successfully authenticates through M6.4's + ingress route using this credential. + +## Verify + +**Harness:** manual — this is a secret-handling task, not one to automate +a fake credential through. + +**Checklist** (no `verify/*.sh`, since scripting a real credential check +means committing something that either leaks a real secret or asserts +nothing): +1. `git grep -i "password"` in both the homelab repo and + `~/workplace/agent-manager` shows only ksops-encrypted blobs or + references to environment/config loading, never a literal value. +2. The ksops secret file's structure matches `model-invoke-apikey`'s + (same encryption provider, same key layout) — a side-by-side diff of + the YAML structure (not values) confirms this. +3. `psql` from the Mac using the decrypted credential (via M6.4's route) + succeeds. +4. Revoking/rotating the credential (delete the CNPG Secret or drop the + dedicated role, if one was created) and re-running step 3 fails + cleanly — confirms the client isn't caching or falling back to + something else. + +**False pass:** +- Confirming connectivity once with a credential typed in manually during + setup, then wiring the ksops path afterward without re-testing that the + *ksops-sourced* value is what actually authenticates. Checklist item 4 + (rotate and confirm the old path is really gone) is the guard against + "it worked, but only because of leftover state." + +## Traps + +- Granting the dedicated role (if created) superuser or + database-owner-equivalent privileges "to avoid permission errors during + setup" and never narrowing it afterward — defeats the reason to create + a dedicated role at all. +- Storing the decrypted credential in a file the Mac client reads that + isn't gitignored, recreating the plaintext-secret problem one directory + away from the ksops-encrypted source of truth. + +--- + +Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `model-invoke-apikey` (ksops pattern reference, this homelab) diff --git a/tasks/M6.6-m6-gate.md b/tasks/M6.6-m6-gate.md new file mode 100644 index 0000000..a76fe1d --- /dev/null +++ b/tasks/M6.6-m6-gate.md @@ -0,0 +1,150 @@ +# M6.6 — M6 composition gate + +| Field | Value | +|---|---| +| Phase | M6 — agent-manager migration | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | all of M6 | + +## Goal + +Prove the schema, the query port, the network path, and the credentials +bind together into one working system — not four pieces that each passed +their own task in isolation. This is the property no single M6 task owns: +M6.1 proves the cluster is healthy, M6.2 proves the schema applies, M6.3 +proves the queries work against a local test database, M6.4 proves a raw +TCP connection reaches the cluster, M6.5 proves the credential decrypts — +none of them alone proves agent-manager, running for real on the Mac, +through the real ingress route, with the real ksops credential, against +the real cluster, does its actual job: track a session end-to-end without +losing or corrupting data. + +## Facts (inlined — no spec read needed) + +**"Bind together smoothly" means two concrete things here, not a vibe:** + +1. **The full network path is exercised, not simulated.** M6.3's tests run + against a disposable local Postgres — that validates the SQL, not the + route. This gate is the first (and only) task that runs agent-manager + unmodified, on the actual Mac, through M6.4's nginx `stream {}` route, + authenticating with M6.5's ksops-sourced credential, against M6.1's + real cluster. +2. **Schema conventions match the rest of this homelab, not just + "compiles."** This project's own `memory-db` (M2.x) and + `agent-manager-db` (M6.x) are two unrelated Postgres schemas landing in + the same cluster around the same time. They don't share data or a + cluster (M6.1 already ruled that out), but a reviewer scanning + `k8s/infra/databases/` should find the same shape twice: same + `Cluster`/`Database` CRD structure, same `storageClass`, same + `enableSuperuserAccess: false`, same GitOps-only provisioning + discipline. "Binds together smoothly" includes that consistency check, + not just agent-manager working in isolation. + +**What a session round trip actually touches**, so the test isn't +shallow: `CreateSession` (writes `sessions` + touches `groups` via the +`ON CONFLICT DO NOTHING` insert) -> `UpdateStatus` -> `SetAgentSessionID` +-> `SetReviewRepo` (writes `review_targets`) -> `Delete` (must cascade +`review_targets` via the FK, per M6.2's decision, with no leftover row). +That single flow crosses all 4 non-`settings` tables and exercises both +the FK-cascade decision and the placeholder-conversion correctness from +M6.3 in one pass. + +## Steps + +1. On the Mac, with agent-manager built from the fully-ported + `add-headless-spawn` branch (M6.3 complete) and configured to use + M6.4/M6.5's route and credential: run the `spawn` CLI subcommand to + create a real session. +2. Drive it through the full lifecycle above (status update, agent + session id capture, review target set, delete) using agent-manager's + own CLI/TUI, not a hand-rolled SQL script — the point is proving the + actual client works, not that Postgres accepts hand-written SQL. +3. Kill and restart agent-manager mid-lifecycle (after step 2's status + update, before delete); confirm it reconnects and reads back the same + state — proves the connection isn't accidentally caching state + client-side that masks a write that never actually landed. +4. Diff `k8s/infra/databases/agent-manager-db.yaml` against + `k8s/infra/databases/memory-db.yaml` field-by-field for the + convention-consistency check. +5. Confirm both ArgoCD Applications (agent-manager's and this project's + `memory-db`, once it exists) are tracked from the `repoURL` each + actually watches — re-verify per M6.1's caveat, since this is the + final point where a "pushed but ArgoCD never saw it" mistake would + otherwise go unnoticed until much later. +6. Commit `expected/m6.6.txt`; diff. + +## Acceptance + +- A session created, updated, and deleted through agent-manager's real + CLI on the Mac round-trips correctly through the full network path. +- A mid-lifecycle restart does not lose or duplicate state. +- Deleting the session leaves zero orphan rows in `review_targets`/ + `review_bases`/`review_scopes` (FK cascade, not app-level cleanup). +- `agent-manager-db.yaml` and `memory-db.yaml` match on every field that + isn't inherently app-specific (name, storage size). +- No manual `kubectl apply`/`psql` anywhere in the setup this gate + exercises. + +## Verify + +**Harness:** the real Mac client, the real cluster, agent-manager's own +CLI — this gate deliberately does not use a disposable/local database, +since proving the disposable path works is exactly what M6.1-M6.5 already +did. + +**Integration test** — `verify/m6.6.sh` diffed against +`expected/m6.6.txt`: +1. `a1_full_roundtrip` — create/update/set-review/delete via the real CLI; + assert no error at any step. +2. `a2_no_orphan_review_rows` — after delete, query `review_targets`/ + `review_bases`/`review_scopes` directly (from inside the cluster, as a + final-state check) for the deleted session's id; assert zero rows, + with no explicit `DELETE FROM review_*` having been issued by the CLI + (proves the FK cascade did the work, not leftover manual-cleanup code + nobody removed). +3. `a3_survives_restart` — kill agent-manager between status-update and + delete; restart; assert the status update is still visible before + proceeding to delete. +4. `a4_schema_convention_match` — diff the two `Cluster` manifests' non + app-specific fields; assert empty diff. +5. `a5_repourl_confirmed` — for each of the two ArgoCD Applications + involved, print which `repoURL` it watches and confirm it matches + which remote was actually pushed. +6. `a6_no_manual_apply_in_history` — review the shell history / session + log from M6.1 through M6.5 for a `kubectl apply` or `psql` write + command that wasn't inside an explicitly-flagged debugging exception; + assert none exist outside that exception. + +**Command:** `bash verify/m6.6.sh | diff - expected/m6.6.txt` + +**False pass:** +- Running assertion 1 against M6.3's disposable local Postgres instead of + the real cluster because it's faster/already running. That's exactly + the "four pieces that each passed in isolation" failure mode this gate + exists to catch — it must hit M6.1's actual cluster through M6.4's + actual route. +- Treating a schema diff (assertion 4) as advisory and skipping it when + short on time. A convention mismatch here is invisible today and + becomes the thing a future reviewer trips over when comparing the two + `k8s/infra/databases/*.yaml` files months later with no memory of why + they differ. + +## Traps + +- Discovering during this gate that M6.4's nginx route works from inside + the homelab LAN but not from wherever the Mac actually sits (VPN, + different subnet, etc.) — a gap none of M6.1-M6.5's narrower tests + would have caught, since this is the first task that tests from the + Mac's actual network position rather than "outside the cluster" in the + abstract. +- Fixing a gate failure by loosening the gate (e.g. deleting assertion 2 + because the cascade "mostly works") instead of fixing the underlying + FK/migration issue. Same discipline this project's other gates + (M0.8, M1.8, M2.8...) already hold to. + +--- + +Background: [M6.1](M6.1-agent-manager-db-manifest.md) · [M6.2](M6.2-schema-port.md) · [M6.3](M6.3-store-query-port.md) · [M6.4](M6.4-nginx-stream-routing.md) · [M6.5](M6.5-credentials-secret.md)