rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

(plan) system review and break down plans

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-19 09:52:07 -07:00
commit 9d28b63ff2
46 changed files with 5620 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DESIGN.md
+486
View File
@@ -0,0 +1,486 @@
# Poimen Memory — gated recurrent knowledge extraction from agent context
Target repo: `/Users/rockliang/workplace/Poimen/memory` (empty git repo, no remote, no commits).
## Context
Agent sessions accumulate faster than they can be read, and almost all of the volume is noise. One pi session in this project measured:
```
assistant 1445
toolResult 1261 <- 43%, mostly evidence-free (ls output, file reads)
user 196
+ 8 compaction events
```
Compaction already fires 8 times per session, which means context is being *discarded* rather than *retained* — the knowledge produced (root causes, decisions, gotchas) evaporates when the window rolls. Meanwhile the corpus is already project-partitioned on disk:
- `~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen--/*.jsonl` — `cwd` in the session header gives the project key
- `~/.claude/projects/<dash-encoded-cwd>/<uuid>.jsonl` — 15 MB, 7 transcripts
- `Poimen/agent-rust/tasks/artifacts/<TaskId>/` — coder/reviewer JSONL per attempt
Intended outcome: a durable, queryable project memory built by reading that history chunk-by-chunk and keeping only what answers standing questions — surfaced as an Obsidian vault a human reads and a pgvector index an agent queries.
The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the problem: the **update gate** refuses to write evidence-free chunks into memory (the 43% of toolResults), and the **exit gate** stops the scan once evidence is sufficient. The paper reports up to 400% speedup and *better* accuracy than ungated recurrent memory, because unchecked memory growth actively degrades later updates.
## Verified facts this plan depends on
| Fact | Value | How known |
|---|---|---|
| Embedding dims | **768** | probed `/v1/embeddings` with `nomic-ai/nomic-embed-text-v2-moe` |
| pgvector in CNPG | **available, v0.7.0**, not yet installed | `pg_available_extensions` on `forgejo-db-2`, stock image `ghcr.io/cloudnative-pg/postgresql:16.2` |
| CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` |
| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` |
| Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** |
| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value |
The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response.
**Side fix:** `~/.pi/agent/models.json` declares `contextWindow: 131072` for ornith. Wrong — Ollama caps at 32768, so long prompts truncate silently. Correct to 32768.
## The tier model
Memory is **levelled**, and every event in the log carries its level. The paper has one flat memory; a project knowledge base needs three.
| Level | What it is | Produced by | Bounded |
|---|---|---|---|
| **L0** | evidence chunk — the verbatim source span the update gate accepted | update gate opening at L1 | no, but sparse (~17 of 412 chunks) |
| **L1** | per-query memory — GRU-Mem's `M_t` for one standing question | gated loop over L0 chunk stream | 1024 tokens |
| **L2** | project synthesis — memory across the L1 memories of one project | gated loop over L1 memories | 1024 tokens |
**The tiering is not new machinery.** L2 is the same controller, same prompt, same two gates — run with the L1 memories as its chunk stream and a project-level question. The recurrence is the algorithm applied to its own output, so `mem-core` implements one loop and the level is a parameter. Two consequences worth having on purpose:
- **Exit gate flips by level.** Off at L1 (see below), reasonably *on* at L2, where the input is a handful of memories rather than hundreds of chunks and "enough evidence" is actually decidable.
- **Levels form a provenance graph, not a pile.** Each L1 node records the L0 nodes that produced it; each L2 node records its L1 parents. That graph *is* the Obsidian link structure and the `parent_id` edges in Postgres — one relationship expressed in both projections.
## Architecture
```
pi sessions / claude transcripts / loop.sh artifacts
│
▼ project resolver (cwd -> project id)
[ Chunk Parser ] 5000-token chunks, split on message boundaries
│
▼
┌──────────────────────────────────────────────────────┐
│ GRU-Mem Controller qwen2.5:3b-instruct │
│ <think> reason about chunk vs question │
│ <check> yes|no -> update gate U_t │
│ <update> candidate memory M̂_t │
│ <next> continue|end -> exit gate E_t │
└───────────────┬──────────────────────────────────────┘
U_t=yes │ U_t=no
┌───────┴────────┐
▼ ▼
emit L0 evidence discard chunk
M_t <- M̂_t M_t <- M_{t-1}
│
▼ L1 memory per standing query
└──────► same loop, input = L1 memories ──► L2 project synthesis
│
▼
┌──────────────────────────────────────────────────────┐
│ JSONL event log AUTHORITATIVE │
│ Obsidian vault derived projection │
│ pgvector index derived projection │
└──────────────────────────────────────────────────────┘
```
**Authority model — the load-bearing decision.** The JSONL log is the only source of truth. The vault and the vector index are projections that must be droppable and rebuildable byte-identically from the log. This is poimen's own §1 principle ("nothing derived is authoritative; if it cannot be dropped and rebuilt, it has hidden inputs and that is a bug") applied here, and it buys three things: re-embedding after a model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, and the post-training corpus is the log itself.
## Standing queries
The paper's memory agent is `φθ(Q, C_t, M_{t-1})` — **it requires a question**. The update gate is defined as "does this chunk contain useful information *about the problem*". Without `Q` the gate has no referent, and `r_update` becomes undefinable, which forecloses post-training.
So each project declares durable questions in YAML. One query = one L1 memory = one Obsidian note.
```yaml
# queries/poimen.yaml
project: poimen
sources:
- pi:--Users-rockliang-workplace-Poimen-agent-rust--
- claude:-Users-rockliang-workplace-Poimen
queries:
- id: architecture-decisions
question: What architectural decisions were made, with reasoning and rejected alternatives?
- id: infra-root-causes
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
- id: open-questions
question: What questions were raised and left unresolved?
synthesis: # the L2 pass
question: What is the current state of this project, and what should someone know before working on it?
exit_gate: true
```
**Exit gate defaults off at L1.** Paper §3.3 makes this call itself: for "what are *all* the X" questions you cannot know evidence is sufficient without reading everything, so they provide a w/o-EG inference mode. L1 extraction is exactly that shape. Keep the gate *recorded* (its signal is needed for RL) but do not act on it. Enable at L2 and for interactive retrieval, where the paper measures 4× speedup.
## Separate weights — yes, specifically a LoRA adapter
Confirming the instinct, with the reason that actually matters here:
- **Base:** Qwen2.5-3B-Instruct, already resident as `qwen2.5:3b-instruct`
- **Memory policy:** LoRA adapter, rank 16–32, ~30–60 MB
Why an adapter rather than a fine-tuned model:
1. **VRAM.** One GPU, `OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A separate full memory model evicts something, and eviction here is a weights reload measured in tens of seconds — we already watched `ornith` cold-start blow a 60s gateway timeout. A LoRA rides on the resident base for near-zero extra VRAM.
2. **Iteration.** Post-training emits a ~50 MB adapter, not a 6 GB model. Swap without redeploying.
3. **Reversibility.** A gate-behaviour regression reverts by pointing at the previous adapter.
**Infra consequence to accept up front:** Ollama cannot hot-swap LoRA adapters. Serving one means either moving the memory model to a **vLLM** InferenceService with `--enable-lora` (the `reasoning` predictor is already vLLM v0.11.0, so the pattern exists), or merging into a GGUF for Ollama and losing swappability. P1–P4 run prompted-only on the stock model, so this is deferred, not dodged.
## Repository layout
Rust workspace at `Poimen/memory`:
```
Cargo.toml workspace
crates/
mem-core/ domain types; Level enum; gate-response parser; the gated loop
mem-chunk/ RecordSource trait; chunking policy; flush triggers (see below)
mem-llm/ gateway client — chat completions, embeddings, rerank
mem-ingest/ source adapters: pi sessions, claude transcripts, loop.sh artifacts
mem-store/ JSONL log writer/reader; pgvector repo; Obsidian projector
mem-cli/ binary `mem`: ingest | synthesize | rebuild | query | verify | skill | label
memory-tasks/ the task board — INDEX.md + one file per task, tracked
queries/ standing query YAML, one file per project
vault/ Obsidian output (own git repo or gitignored)
log/ JSONL event log — authoritative, tracked
```
Crates: `sqlx` (postgres, runtime-tokio-rustls) + `pgvector` (sqlx feature), `tokenizers` for chunk sizing against the real Qwen2 tokenizer, `futures`, `serde`/`serde_json`, `clap`, `reqwest`.
Reuse rather than reinvent: `mem-llm`'s request shape and the `apikey` header convention are proven in `agent-rust/loop.sh`; response parsing mirrors its `extract_text()`.
### `mem-chunk` — its own crate, stream-shaped from day one
Chunking is split out because it is the seam where new input kinds arrive. Sources today are files with an EOF; telemetry, a live session tail, or a broker will not have one. Designing the boundary as a stream now means a future source implements a trait rather than forcing the loop to be rewritten — and it costs nothing, since the rest of the stack is already tokio (`sqlx` runtime-tokio, `reqwest`).
```rust
// mem-chunk
pub trait RecordSource {
/// Normalised records: role, text, timestamp, provenance. Sources decide
/// how to produce them; the chunker never learns about pi vs claude vs a socket.
fn records(self) -> impl Stream<Item = Result<Record>>;
}
pub struct ChunkPolicy {
pub max_tokens: usize, // 5000, paper default
pub split_on: Boundary, // never mid-message
pub flush: FlushTrigger, // see below
}
pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
```
Batch sources become streams for free via `futures::stream::iter`, so `mem-ingest` gets no more complex today.
**The one thing that genuinely differs for streams is the flush trigger.** A file chunker emits a partial chunk at EOF; a stream has no EOF, so a partially-filled chunk would sit forever. `FlushTrigger` is therefore `Tokens(n)` today and gains `OrIdle(Duration)` when a stream source lands — carrying it in the policy now means the later change is one enum variant, not a signature change through the loop.
Worth noting for whenever that happens: the **exit gate changes meaning on an unbounded source**. At L1 over a finite transcript it is switched off because "read everything" is well defined (paper §3.3). Over a live stream there is no everything, so the gate stops being an optimisation and becomes the only termination condition — which is an argument for keeping it trained even while it is switched off.
## Storage schemas
### JSONL event log — authoritative
`log/<project>/<query-id>/<run-id>.jsonl`. **Every record carries `level`.**
```jsonl
{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:2026-07-21T16-23-59_019f857d","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<L0 sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}
```
The L2 pass writes the same record types with `"level":"L2"`, `"input_level":"L1"`, and `parents` holding L1 shas. `evidence` records appear only when the update gate opened, so update-rate is directly measurable and memory at any `t` is replayable.
### pgvector — projection
One table across all levels, because retrieval wants to search them together and filter:
```sql
CREATE TABLE memory_node (
id BIGSERIAL PRIMARY KEY,
level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')),
project TEXT NOT NULL,
query_id TEXT, -- null at L2
run_id TEXT NOT NULL,
t INT NOT NULL,
source TEXT, -- set at L0
text TEXT NOT NULL,
sha256 TEXT NOT NULL UNIQUE,
embedding vector(768) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE memory_edge ( -- provenance: child <- parent
child_sha TEXT NOT NULL REFERENCES memory_node(sha256),
parent_sha TEXT NOT NULL REFERENCES memory_node(sha256),
PRIMARY KEY (child_sha, parent_sha)
);
CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON memory_node (project, level);
```
```mermaid
erDiagram
MEMORY_NODE ||--o{ MEMORY_EDGE : "child_sha -> sha256"
MEMORY_NODE ||--o{ MEMORY_EDGE : "parent_sha -> sha256"
MEMORY_NODE {
bigserial id PK
text level "L0 | L1 | L2"
text project
text query_id "NULL at L2"
text run_id
int t
text source "set at L0"
text text
text sha256 UK "content identity"
vector_768 embedding
timestamptz created_at
}
MEMORY_EDGE {
text child_sha PK,FK
text parent_sha PK,FK
}
```
One table across L0/L1/L2 (not three) — retrieval searches all levels together and filters by `level`. `memory_edge` is the provenance graph: L1 rows point back at the L0 chunks that produced them, L2 rows point back at L1 parents. `sha256 UNIQUE` is content identity (dedup key, hash excludes run id/timestamp — M0.2) and is what `memory_edge` actually references, not the surrogate `id`.
Retrieval: HNSW recall filtered by level, then `bge-reranker-base` via `/v1/rerank` for precision — that endpoint scored 0.98 vs 0.00009 on a discrimination probe, so it earns its place. Default query searches L1+L2 and walks `memory_edge` down to L0 for citations.
Infra — new CNPG cluster with the extension managed declaratively (CNPG 1.30 supports this, so **no manual `psql`**, consistent with the GitOps hard rule). Follows `k8s/infra/databases/temporal-db.yaml` exactly:
```yaml
# k8s/infra/databases/memory-db.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: memory-db, namespace: memory }
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
bootstrap: { initdb: { database: memory, owner: app, encoding: UTF8, localeCollate: C, localeCType: C } }
enableSuperuserAccess: false
storage: { size: 10Gi, storageClass: longhorn-cnpg }
monitoring: { enablePodMonitor: true }
affinity:
podAntiAffinityType: preferred
tolerations: [{ key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }]
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata: { name: memory-db-vector, namespace: memory }
spec:
name: memory
owner: app
cluster: { name: memory-db }
extensions: [{ name: vector, ensure: present }]
```
### Obsidian vault — projection
The tier graph becomes the note graph:
```
vault/poimen/
index.md <- L2 synthesis, links to every L1 note
infra-root-causes.md <- L1
architecture-decisions.md <- L1
evidence/ <- L0, optional (--emit-evidence-notes, default off)
vault/skills/
_drafts/<name>/SKILL.md <- machine-written, never auto-loaded
<name>/SKILL.md <- human-promoted, loadable
```
```markdown
---
project: poimen
level: L1
query_id: infra-root-causes
updated: 2026-08-17
chunks_seen: 412
chunks_used: 17
---
# Infra root causes — poimen
<final memory text>
## Provenance
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer
```
L0 defaults to inline citations rather than notes — 17 per query is manageable but grows unbounded across projects. The flag exists for when the graph view is worth the file count.
## Skills — the procedural projection
**A skill is a projection, not a level.** L0/L1/L2 are all *descriptive* — what happened. A skill is *procedural* — what to do next time. That is a change of modality, not a further compression, so the gated loop does not produce it: the update gate's question ("does this chunk contain evidence for Q") has no meaning when the output is an instruction.
The format is free. `SKILL.md` is YAML frontmatter plus markdown, which is exactly an Obsidian note — verified against `~/.claude/skills/seo-geo-claude-skills/research/keyword-research/SKILL.md`, whose frontmatter carries `name`, `description`, `when_to_use`, `argument-hint`. So `vault/skills/<name>/SKILL.md` is simultaneously a vault note and a loadable skill, with no conversion step:
```sh
pi --skill vault/skills/ # or set skillsPath in ~/.pi/config.json
ln -s .../vault/skills/<name> ~/.claude/skills/<name>
```
`mem skill draft --from poimen/infra-root-causes` reads an L1 or L2 note and writes a draft. Emission should follow the existing authoring rubric rather than inventing one — the installed `grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and a four-dimension rubric (conciseness, actionability, workflow clarity, progressive disclosure). Description quality is the whole game: a skill whose `description` does not match how the user actually phrases the request never fires.
**Drafts are never auto-loaded, and promotion is manual.** This is the one place the system can close a loop on itself, and the failure is subtle:
> a memory-derived skill is auto-loaded → it appears in future session transcripts → those transcripts are ingested as evidence → the memory that produced the skill is reinforced by its own output
No external verifier breaks that cycle. It is the same hazard poimen §16 names when it gates "automatic workflow mutation without human approval" by default, and it is why `_drafts/` is a separate directory rather than a frontmatter flag — a directory cannot be accidentally globbed into `--skill`.
Two mechanical guards:
1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff.
2. **Provenance marks derived text.** Every emitted skill carries `generated_from: <L2 sha>` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data.
## Phases
**P1 — Read-only spine.** `mem-ingest` implements `RecordSource` for pi sessions and Claude transcripts; `mem-chunk` chunks to 5000 tokens on message boundaries; `mem ingest --dry-run` prints the chunk plan with no model calls. Both sources are batch, but they go through the stream interface so the seam is exercised from the first commit rather than retrofitted.
**P2 — Gated loop at L1, prompted only.** `mem-llm` + gate-response parser (paper Figure 10a prompt, adapted for standing queries). Writes the JSONL log with L0 evidence and L1 memory records. Exit gate recorded, not acted on. This is the paper's "w/o RL" baseline, which Figure 9 shows already works.
**P3 — Projections.** Obsidian projector and pgvector repo, both rebuildable via `mem rebuild --from-log`. Infra commit for `memory-db`.
**P4 — L2 synthesis and retrieval.** `mem synthesize` runs the same loop over L1 memories with the exit gate on. `mem query` — embed, HNSW recall by level, rerank, return with provenance walked through `memory_edge`.
**P5 — Skill drafting.** `mem skill draft --from <note>` emits `vault/skills/_drafts/<name>/SKILL.md` with `generated_from` provenance; `mem-ingest` grows the `derived: true` exclusion filter. Promotion stays manual. Cheap to build and it is the phase that makes the memory *do* something rather than only be read.
**P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above.
## Task breakdown
Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M<phase>.<n>` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id.
```
memory-tasks/
INDEX.md board + phase order + progress mirror
M0.1-cargo-workspace.md
M0.2-domain-types.md
...
M5.6-m5-gate.md
```
Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole board there. Naming this `memory-tasks/` sidesteps that pattern — and it should be **tracked**, since the task files carry the acceptance criteria.
**M0 — Read-only spine** (no model calls anywhere in this phase)
| id | task | size | deps |
|---|---|---|---|
| M0.1 | Cargo workspace + six crate skeletons, CI builds clean | S | — |
| M0.2 | `mem-core` domain types: `Level`, `Record`, `Chunk`, `MemoryNode`, sha256 identity | S | M0.1 |
| M0.3 | `mem-chunk`: `RecordSource` trait, `ChunkPolicy`, `FlushTrigger::Tokens` | M | M0.2 |
| M0.4 | Tokenizer-backed sizing against the real Qwen2 tokenizer | M | M0.3 |
| M0.5 | `mem-ingest`: pi session adapter (`~/.pi/agent/sessions/<encoded-cwd>/*.jsonl`) | M | M0.3 |
| M0.6 | `mem-ingest`: claude transcript adapter (`~/.claude/projects/**/<uuid>.jsonl`) | S | M0.5 |
| M0.7 | `mem ingest --dry-run` — chunk plan, token histogram, source breakdown | S | M0.4, M0.6 |
| M0.8 | **M0 gate** — both adapters through one `RecordSource`, no source-specific code past the trait | M | gate |
**M1 — Gated loop at L1**
| id | task | size | deps |
|---|---|---|---|
| M1.1 | `mem-llm` chat client — `apikey` header, retry, timeout | M | M0.1 |
| M1.2 | Standing-query YAML loader + schema validation, unresolved id fails at load | M | M0.2 |
| M1.3 | GRU-Mem prompt template (paper Fig 10a), memory + chunk + question assembly | M | M1.2 |
| M1.4 | Gate-response parser: `<think>/<check>/<update>/<next>`, strict, malformed = hard error | M | M1.3 |
| M1.5 | The gated loop — `U_t` mutate-or-retain, `E_t` recorded not acted on, 1024-token budget | L | M1.4 |
| M1.6 | JSONL event log writer, `level` on every record, `parents` on memory | M | M1.5 |
| M1.7 | `mem ingest` end to end + update-rate reported on stdout | M | M1.6 |
| M1.8 | **M1 gate** — full run on `poimen`, update-rate < 30%, memory tokens flat not climbing | M | gate |
**M2 — Projections**
| id | task | size | deps |
|---|---|---|---|
| M2.1 | `mem-llm` embeddings client, 768-dim, batched | S | M1.1 |
| M2.2 | `k8s/infra/databases/memory-db.yaml` — CNPG Cluster + Database with `extensions: [vector]` | M | — |
| M2.3 | `memory_node` / `memory_edge` schema + sqlx migrations, HNSW indexes | M | M2.2 |
| M2.4 | `mem-store` pgvector repo — upsert by sha, edge insert | M | M2.3, M2.1 |
| M2.5 | Obsidian projector — frontmatter, wikilinks, L0 citations, `--emit-evidence-notes` | M | M1.6 |
| M2.6 | `mem rebuild --from-log` — drop and rebuild both projections | M | M2.4, M2.5 |
| M2.7 | `mem verify` — every L1 has ≥1 L0 parent, every parent sha resolves | S | M2.6 |
| M2.8 | **M2 gate** — rebuild is byte-identical (`git -C vault diff --exit-code` empty) | M | gate |
**M3 — L2 synthesis and retrieval**
| id | task | size | deps |
|---|---|---|---|
| M3.1 | L2 pass — same loop, input `Stream<L1 memory>`, exit gate **on** | M | M1.5 |
| M3.2 | `mem-llm` rerank client (`bge-reranker-base`) | S | M1.1 |
| M3.3 | `mem query` — embed, HNSW recall filtered by level, rerank, walk edges to L0 | M | M3.2, M2.4 |
| M3.4 | **M3 gate** — known-answer query returns the right L1 node with a real L0 citation | M | gate |
**M4 — Skills**
| id | task | size | deps |
|---|---|---|---|
| M4.1 | `mem skill draft --from <note>` → `_drafts/`, frontmatter incl. `generated_from` | M | M3.1 |
| M4.2 | `derived: true` ingest filter — emitted artifacts excluded from evidence | M | M4.1, M0.5 |
| M4.3 | **M4 gate** — draft absent from `--list-skills`; no L0 node matches an emitted artifact | M | gate |
**M5 — Post-training** (Python, separate from the Rust workspace; boundary is the JSONL)
| id | task | size | deps |
|---|---|---|---|
| M5.1 | `mem label` — 32B `reasoning` as offline evidence labeler, writes `U_t` ground truth | M | M1.6 |
| M5.2 | Labeler calibration — hand-label a holdout, measure agreement before trusting it | M | M5.1 |
| M5.3 | Training corpus export from the log to verl's expected format | M | M5.1 |
| M5.4 | vLLM InferenceService for Qwen2.5-3B with `--enable-lora` (GitOps, homelab) | L | — |
| M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 |
| M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate |
Total 38 tasks, 6 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel.
## Verification
```bash
# P1 — corpus parses, chunk plan sane, zero model calls
cargo run -p mem-cli -- ingest --project poimen --dry-run
# P2 — one query end to end
cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes
jq -r 'select(.type=="gate" and .level=="L1") | .update' log/poimen/infra-root-causes/*.jsonl | sort | uniq -c
# expect: far more false than true. Update-rate > ~30% means the gate is not
# discriminating — that is the paper's memory-explosion failure, Figure 6 is
# the reference shape.
jq -r 'select(.type=="memory") | .tokens' log/.../*.jsonl | tail -1
# expect: <= 1024 and roughly flat over t, not monotonically climbing
# levels are well-formed and edges close
jq -r '.level' log/poimen/**/*.jsonl | sort | uniq -c # L0/L1 present
cargo run -p mem-cli -- verify --project poimen
# asserts: every L1 memory has >=1 L0 parent; every parent sha exists
# P3 — projections truly derived
cargo run -p mem-cli -- rebuild --from-log --project poimen
git -C vault diff --exit-code # empty: rebuild is byte-identical
psql -c "select level, count(*) from memory_node group by level;"
# P4 — synthesis and retrieval
cargo run -p mem-cli -- synthesize --project poimen # expect exit gate to fire
cargo run -p mem-cli -- query "why did requests over 10KB fail?"
# expect: infra-root-causes L1 node, Kong body-buffer passage, L0 citation
# P5 — skill drafts land unloadable, and the cycle stays open
cargo run -p mem-cli -- skill draft --from poimen/infra-root-causes
ls vault/skills/_drafts/ # draft here, NOT in vault/skills/
pi --skill vault/skills/ --list-skills # draft must not appear
cargo run -p mem-cli -- verify --derived-filter --project poimen
# asserts: no L0 evidence node text matches an emitted skill artifact
```
The decisive P2 metric is **update-rate**, the one number distinguishing a working gate from an expensive summarizer. toolResults are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks.
## Risks
- **3B gate quality unmeasured on this corpus.** The paper evaluates on QA benchmarks with clean evidence labels; agent transcripts are messier. Mitigation: P2's update-rate is a cheap early read, and the 32B `reasoning` model can spot-audit a sample before committing to P5.
- **L2 inherits L1's errors with no path back to source.** Synthesis over memories cannot recover evidence the L1 gate wrongly discarded. `memory_edge` makes the omission *visible* (an L1 note with suspiciously few parents) but not recoverable without a re-run.
- **Self-reinforcement through skills.** The only cycle in the system: emitted skill → future session context → ingested as evidence → reinforces the memory that emitted it. Guarded by manual promotion plus the `derived: true` ingest filter, and both must hold. Audit it by checking that no L0 evidence node's text matches an emitted artifact.
- **No ground-truth evidence labels.** `r_update` needs them. Distant supervision from the 32B labeler inherits its bias; hold out a hand-labelled set to measure agreement before trusting it.
- **Vault/log divergence.** Hand edits are overwritten on rebuild. Either make the vault read-only or add an `## Notes` region the projector preserves. Decide before anyone starts editing.
- **Ollama has no LoRA path.** P5 forces the vLLM decision. Do not discover this at P5.
README.md
+226
View File
@@ -0,0 +1,226 @@
# poimen-memory
Gated recurrent memory over agent context. Reads session history chunk-by-chunk,
keeps only what answers standing questions, projects result into an Obsidian
vault and a pgvector index.
**Status: design complete, no code yet.** 37 tasks in [memory-tasks/](memory-tasks/INDEX.md),
0 done. Start at [M0.1](memory-tasks/M0.1-cargo-workspace.md).
## Problem
Agent sessions grow faster than anyone reads them, and most of the volume is
noise. One real pi session in this project:
```
assistant 1445
toolResult 1261 43% — ls output, file reads, mostly evidence-free
user 196
+ 8 compaction events
```
Compaction fires 8 times per session. Context gets *discarded*, not retained —
root causes, decisions and gotchas evaporate when window rolls.
## Mechanism
GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled
gates on a recurrent memory loop:
- **update gate** — memory only mutates when chunk contains evidence. Blocks the
memory explosion that ungated recurrent memory hits.
- **exit gate** — stop scanning once evidence sufficient.
Paper reports up to 400% speedup and *better* accuracy than ungated, because
unbounded memory growth degrades later updates.
```
sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections
```
Controller emits structured output; loop acts on it:
```
<think> reason about chunk vs question
<check> yes|no -> U_t, update or discard
<update> candidate memory M̂_t
<next> continue|end -> E_t, exit or continue
```
## Memory tiers
| Level | What | From | Bounded |
|---|---|---|---|
| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) |
| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok |
| **L2** | project synthesis | gated loop over L1 memories | 1024 tok |
L2 is not new machinery — same loop, same prompt, L1 memories as input stream.
Level is a parameter.
Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1
parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks.
## Standing queries
Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is
defined as "does this chunk contain useful information *about the problem*". No
`Q`, no gate, and `r_update` becomes undefinable, which kills post-training.
So each project declares durable questions. One query = one L1 memory = one note.
```yaml
# queries/poimen.yaml
project: poimen
roots: [/Users/rockliang/workplace/Poimen/agent-rust]
queries:
- id: infra-root-causes
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
- id: architecture-decisions
question: What architectural decisions were made, with reasoning and rejected alternatives?
synthesis:
question: What is the current state of this project, and what should someone know before working on it?
exit_gate: true
```
**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all*
the X" questions you cannot know evidence is sufficient without reading
everything. L1 extraction is that shape. At L2 input is a handful of memories and
sufficiency is decidable. Gate still *recorded* at L1 — signal needed for
post-training.
## Authority model
**JSONL log authoritative. Vault and vector index are projections.**
Anything not rebuildable byte-identically from the log has hidden inputs, and
that is a bug. Gate M2.8 enforces it destructively:
```sh
rm -rf vault/poimen
psql -c "delete from memory_node where project='poimen'"
mem rebuild --from-log --project poimen
git -C vault diff --exit-code # empty diff is the only pass
```
Buys three things: re-embedding after model change is a rebuild not a migration,
Obsidian edits cannot corrupt the record, post-training corpus is the log itself.
## Skills
A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what
happened. A skill is procedural — what to do next time. Gated loop does not
produce it.
Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian
note. So `vault/skills/<name>/SKILL.md` is both, no conversion:
```sh
pi --skill vault/skills/
ln -s .../vault/skills/<name> ~/.claude/skills/<name>
```
**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one
cycle in the design:
```
emitted skill auto-loads -> appears in future transcripts
-> ingested as evidence -> reinforces the memory that emitted it
```
No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be
globbed into `--skill`), and every artifact carries `generated_from` so ingest
tags matching chunks `derived: true` and refuses them as evidence.
## Separate weights
Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned
model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding
`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction
is a weights reload measured in tens of seconds. Adapter rides the resident base.
Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression
reverts by pointing at previous adapter.
**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora`
(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4
run prompted-only, so decision is deferred, not dodged.
## Layout
```
DESIGN.md full design, 460 lines
memory-tasks/ 37 task files + INDEX.md — tracked
crates/
mem-core/ domain types; Level; gate parser; the gated loop
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
mem-llm/ gateway client — chat, embeddings, rerank
mem-ingest/ source adapters: pi sessions, claude transcripts
mem-store/ JSONL log; pgvector repo; Obsidian projector
mem-cli/ binary `mem`
queries/ standing query YAML per project
log/ JSONL event log — authoritative, tracked
vault/ Obsidian output
```
`mem-chunk` is separate and stream-shaped from day one. Sources today are files
with an EOF; telemetry or a live tail will not have one. `RecordSource` returns
`impl Stream<Item = Record>`; batch sources become streams via
`futures::stream::iter`, so it costs nothing now and removes a rewrite later.
## Commands
```sh
mem ingest --project poimen --dry-run # chunk plan, zero model calls
mem ingest --project poimen --query infra-root-causes
mem synthesize --project poimen # L2 pass, exit gate on
mem rebuild --from-log --project poimen # drop and rebuild projections
mem verify --project poimen # provenance graph closure
mem query "why did requests over 10KB fail?"
mem skill draft --from poimen/infra-root-causes
mem label --project poimen # evidence labels for training
```
## Verified environment facts
Checked against the running cluster, not assumed:
| Fact | Value |
|---|---|
| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` |
| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) |
| pgvector | **0.7.0 available in stock CNPG image**, no custom build |
| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` |
| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config |
| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone |
| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** |
| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` |
Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response.
## Phases
Each ends in a composition gate. No phase starts until predecessor gate is green.
| | Phase | Tasks | Gate asserts |
|---|---|---|---|
| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline |
| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing |
| M2 | Projections | 8 | rebuild byte-identical from log alone |
| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 |
| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence |
| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project |
**Update-rate is the number to watch.** It is what distinguishes a gate from an
expensive summarizer. Tool results are 43% of records and mostly evidence-free,
so a correct gate rejects the large majority of chunks.
M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA)
is homelab work independent of the rest of M5.
## Reading order
1. This file
2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice
3. [DESIGN.md](DESIGN.md) — full design, schemas, risks
4. Individual task files — self-contained, no DESIGN.md read required
tasks/INDEX.md
+175
View File
@@ -0,0 +1,175 @@
# poimen-memory — task board
43 tasks — 36 build tasks plus **7 composition gates**, one per phase. One file
per task, **self-contained**: inlined design facts, executable steps, acceptance
criteria, a `Verify` section written for someone who did not build the thing, and
the traps worth naming. Reading `DESIGN.md` is not required to do a task — it is
linked as background only.
Each `Verify` section names the harness, the integration test with numbered
assertions, the command to run, and the **false pass** — the shape of test that
goes green while the feature is broken. Treat the false-pass list as part of the
acceptance criteria, not commentary.
## Ordering — declared, never derived
**Phase order is the list below. Task ids are opaque and frozen.**
`M0.3` is `M0.3` forever, in whatever phase it currently sits, because its
artifacts and cross-references key on that id. New tasks take new ids rather than
renumbering neighbours. This is the `StepId` rule applied to the board itself —
a board that renumbers to reorder has the bug it warns its own users about.
No phase starts until its predecessor's gate is green. The `gate` task at the end
of each phase **is** that gate: it proves the phase's parts compose and that its
swappable parts are genuinely swappable. Every build task is verified alone; the
gate verifies the properties no single task owns.
## Two rules this board exists to protect
**1. The JSONL log is authoritative; the vault and the vector index are
projections.** Anything that cannot be dropped and rebuilt byte-identically from
the log has hidden inputs, and that is a bug. `M2.8` is the gate that enforces it.
**2. The update gate must discriminate, not summarize.** Agent transcripts are
~43% tool results and mostly evidence-free. A gate that accepts most chunks is an
expensive summarizer that will reproduce the memory-explosion failure the whole
design exists to avoid. `M1.8` is the gate that enforces it, and **update-rate is
the single number to watch.**
## Verification practice — script first, source second
A task is verified by running its command and reading the output, then opening
the source. Reviewing the diff first is how an assertion that was quietly dropped
still gets called done: the code looks right, and nothing proves the test ran.
Numbered assertion N in a `Verify` section is test fn `aN_<slug>`. The numbering
is the contract — a test fn that does not exist should report missing rather than
be silently absent from a green summary.
`cargo test` reporting `ok` with zero tests run is not a pass.
## Progress
**Source of truth is the `Status` field in each task file.** The tables below
mirror it; a status changed here and not there is a lie.
Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
| # | Phase | Ids | Tasks | ✅ | 🟡 | ⬜ | Gate |
|---|---|---|---|---|---|---|---|
| 1 | Read-only spine | M0.x | 8 | 0 | 0 | 8 | ⬜ M0.8 |
| 2 | Gated loop at L1 | M1.x | 8 | 0 | 0 | 8 | ⬜ M1.8 |
| 3 | Projections | M2.x | 8 | 0 | 0 | 8 | ⬜ M2.8 |
| 4 | L2 synthesis + retrieval | M3.x | 4 | 0 | 0 | 4 | ⬜ M3.4 |
| 5 | Skills | M4.x | 3 | 0 | 0 | 3 | ⬜ M4.3 |
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
| | **Total** | | **43** | **0** | **0** | **43** | 0/7 green |
**Where the line is — 2026-08-18.** Nothing started. No crate exists yet: there
is no `Cargo.toml` under `memory/`, so every task below is design only. M0.1 is
the first thing that has to happen. `M2.2` (the CNPG manifest), `M5.4` (vLLM
with LoRA), and all of `M6.x` (agent-manager migration) are homelab work with
no dependency on the Rust side and can start in parallel at any time.
**M6 is a different repo, not a dependency of M0-M5.** It migrates
`github.com/Riotpiaole/agent-manager`'s session store (a separate Go CLI tool,
unrelated to this project's own memory system) from local sqlite to its own
dedicated CNPG cluster. It rides in this board because it's homelab work
happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
landing in the same cluster around the same time need to look like siblings,
not strangers — see M6.6's convention-consistency check.
## 1 — Read-only spine · M0.x
No model calls anywhere in this phase. The point is to prove the corpus parses
and chunks sanely before spending inference on it.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M0.1](M0.1-cargo-workspace.md) | Cargo workspace + crate skeletons | S | — | ⬜ |
| [M0.2](M0.2-domain-types.md) | Domain types and sha256 identity | S | — | ⬜ |
| [M0.3](M0.3-recordsource-and-chunkpolicy.md) | `RecordSource` trait + `ChunkPolicy` | M | — | ⬜ |
| [M0.4](M0.4-tokenizer-sizing.md) | Tokenizer-backed chunk sizing | M | — | ⬜ |
| [M0.5](M0.5-pi-session-adapter.md) | pi session adapter | M | — | ⬜ |
| [M0.6](M0.6-claude-transcript-adapter.md) | Claude transcript adapter | S | — | ⬜ |
| [M0.7](M0.7-ingest-dry-run.md) | `mem ingest --dry-run` | S | — | ⬜ |
| [M0.8](M0.8-m0-gate.md) | **M0 composition gate** | M | gate | ⬜ |
## 2 — Gated loop at L1 · M1.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M1.1](M1.1-llm-chat-client.md) | `mem-llm` chat client | M | — | ⬜ |
| [M1.2](M1.2-standing-query-loader.md) | Standing-query YAML loader | M | — | ⬜ |
| [M1.3](M1.3-prompt-template.md) | GRU-Mem prompt template | M | — | ⬜ |
| [M1.4](M1.4-gate-response-parser.md) | Gate-response parser | M | — | ⬜ |
| [M1.5](M1.5-gated-loop.md) | The gated loop | L | — | ⬜ |
| [M1.6](M1.6-jsonl-event-log.md) | JSONL event log writer | M | — | ⬜ |
| [M1.7](M1.7-ingest-end-to-end.md) | `mem ingest` end to end | M | — | ⬜ |
| [M1.8](M1.8-m1-gate.md) | **M1 composition gate** | M | gate | ⬜ |
## 3 — Projections · M2.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M2.1](M2.1-embeddings-client.md) | Embeddings client | S | — | ⬜ |
| [M2.2](M2.2-memory-db-manifest.md) | CNPG `memory-db` + pgvector | M | homelab | ⬜ |
| [M2.3](M2.3-schema-and-migrations.md) | Schema + sqlx migrations | M | — | ⬜ |
| [M2.4](M2.4-pgvector-repo.md) | pgvector repository | M | — | ⬜ |
| [M2.5](M2.5-obsidian-projector.md) | Obsidian projector | M | — | ⬜ |
| [M2.6](M2.6-rebuild-from-log.md) | `mem rebuild --from-log` | M | — | ⬜ |
| [M2.7](M2.7-verify-edges.md) | `mem verify` — edge closure | S | — | ⬜ |
| [M2.8](M2.8-m2-gate.md) | **M2 composition gate** | M | gate | ⬜ |
## 4 — L2 synthesis and retrieval · M3.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.1](M3.1-l2-synthesis.md) | L2 synthesis pass | M | — | ⬜ |
| [M3.2](M3.2-rerank-client.md) | Rerank client | S | — | ⬜ |
| [M3.3](M3.3-mem-query.md) | `mem query` with provenance | M | — | ⬜ |
| [M3.4](M3.4-m3-gate.md) | **M3 composition gate** | M | gate | ⬜ |
## 5 — Skills · M4.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M4.1](M4.1-skill-draft.md) | `mem skill draft` | M | — | ⬜ |
| [M4.2](M4.2-derived-filter.md) | `derived: true` ingest filter | M | — | ⬜ |
| [M4.3](M4.3-m4-gate.md) | **M4 composition gate** | M | gate | ⬜ |
## 6 — Post-training · M5.x
Python, separate from the Rust workspace. The boundary is the JSONL log.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M5.1](M5.1-evidence-labeler.md) | `mem label` — evidence labeler | M | — | ⬜ |
| [M5.2](M5.2-labeler-calibration.md) | Labeler calibration | M | — | ⬜ |
| [M5.3](M5.3-training-corpus-export.md) | Training corpus export | M | — | ⬜ |
| [M5.4](M5.4-vllm-lora-serving.md) | vLLM + `--enable-lora` | L | homelab | ⬜ |
| [M5.5](M5.5-verl-training-loop.md) | verl training loop | L | — | ⬜ |
| [M5.6](M5.6-m5-gate.md) | **M5 composition gate** | L | gate | ⬜ |
## 7 — agent-manager migration · M6.x
Separate repo (`github.com/Riotpiaole/agent-manager`, fork branch
`add-headless-spawn`), separate cluster resource, no Rust/GRU-Mem
dependency. Moves its session store off local sqlite onto a dedicated CNPG
Postgres, reachable from the Mac client through a dedicated nginx route —
durability-of-location, not a multi-host requirement.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M6.1](M6.1-agent-manager-db-manifest.md) | CNPG `agent-manager-db` manifest | M | homelab | ⬜ |
| [M6.2](M6.2-schema-port.md) | Postgres schema for agent-manager sessions | M | — | ⬜ |
| [M6.3](M6.3-store-query-port.md) | store.go query port to Postgres | L | — | ⬜ |
| [M6.4](M6.4-nginx-stream-routing.md) | nginx TCP routing to `agent-manager-db` | S | homelab | ⬜ |
| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ |
| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ |
---
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
tasks/M0.1-cargo-workspace.md
+98
View File
@@ -0,0 +1,98 @@
# M0.1 — Cargo workspace + crate skeletons
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
The six-crate workspace, building clean, with the dependency direction fixed
before any code exists to violate it.
## Facts (inlined — no spec read needed)
```
crates/
mem-core/ domain types; Level; gate-response parser; the gated loop
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
mem-llm/ gateway client — chat, embeddings, rerank
mem-ingest/ source adapters: pi sessions, claude transcripts
mem-store/ JSONL log; pgvector repo; Obsidian projector
mem-cli/ binary `mem`
```
Dependency direction, enforced from the start:
```
mem-cli -> mem-ingest, mem-store, mem-llm, mem-chunk, mem-core
mem-store -> mem-core
mem-ingest -> mem-chunk, mem-core
mem-chunk -> mem-core
mem-llm -> mem-core
mem-core -> (nothing in this workspace)
```
`mem-core` depends on no sibling. It holds the types every other crate speaks,
so a dependency out of it is a cycle waiting to happen.
Workspace deps to pin now, so versions do not drift per crate: `tokio`,
`futures`, `serde`, `serde_json`, `serde_yaml`, `anyhow`, `thiserror`,
`sha2`, `clap`, `reqwest`, `tracing`.
## Steps
1. `Cargo.toml` at the repo root with `[workspace] members = [...]` and a
`[workspace.dependencies]` block holding every shared crate version.
2. Six member crates, each declaring deps as `foo.workspace = true`.
3. `mem-cli` is the only `[[bin]]`; the rest are libraries.
4. Add `rust-toolchain.toml` pinning a version, so CI and laptop agree.
5. `.gitignore`: `target/`, `vault/` — but **not** `log/` and **not**
`memory-tasks/`. The log is authoritative and the board carries acceptance
criteria; both are tracked.
6. Wire a CI job running `cargo build --workspace` and `cargo clippy --workspace
-- -D warnings`.
## Acceptance
- `cargo build --workspace` succeeds from a clean checkout.
- `cargo clippy --workspace -- -D warnings` is clean.
- `mem-core` has zero intra-workspace dependencies.
## Verify
**Harness:** cargo itself, plus a dependency assertion that does not trust the
manifests to be read by a human.
**Integration test** — `tests/it_workspace.rs` in the root:
1. `a1_all_members_build` — shell out to `cargo build --workspace`, assert exit 0.
2. `a2_mem_core_has_no_sibling_deps` — parse `crates/mem-core/Cargo.toml`, assert
no dependency name starts with `mem-`.
3. `a3_dependency_direction` — parse every member manifest, build the edge set,
assert it is a subset of the table above and that the graph is acyclic.
4. `a4_log_and_tasks_are_tracked` — assert `.gitignore` matches neither
`log/` nor `memory-tasks/`.
**Command:** `cargo test --workspace workspace`
**False pass:**
- Asserting the graph is acyclic **only**. Acyclic permits `mem-core ->
mem-store`, which is backwards and still acyclic. Assertion 3 must check the
edge set against the table, not just for cycles.
- A CI job that runs `cargo build` in the root without `--workspace`. It builds
the virtual manifest and can miss a member that does not compile.
## Traps
- Per-crate dependency versions instead of `workspace.dependencies`. They drift,
and two versions of `serde` in one tree is a confusing type error much later.
- Gitignoring `log/`. It is the authoritative record; ignoring it makes the whole
authority model a fiction.
---
Background: [DESIGN.md](../DESIGN.md) — Repository layout
tasks/M0.2-domain-types.md
+115
View File
@@ -0,0 +1,115 @@
# M0.2 — Domain types and sha256 identity
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.1 |
## Goal
The vocabulary every other crate speaks, and the content-hash identity the whole
provenance graph hangs on.
## Facts (inlined — no spec read needed)
```rust
/// Closed. L0 evidence, L1 per-query memory, L2 project synthesis.
#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum Level { L0, L1, L2 }
/// A normalised unit from any source. Adapters produce these; nothing
/// downstream learns whether it came from pi, claude, or a socket.
pub struct Record {
pub role: Role, // User | Assistant | ToolResult | System
pub text: String,
pub timestamp: OffsetDateTime,
pub provenance: Provenance, // source id + offset within it
}
/// One or more Records, under the token budget, never split mid-Record.
pub struct Chunk {
pub t: u32, // 1-based turn index within a run
pub records: Vec<Record>,
pub tokens: usize,
pub sha256: Sha256Hash,
}
pub struct MemoryNode {
pub level: Level,
pub project: ProjectId,
pub query_id: Option<QueryId>, // None at L2
pub run_id: RunId,
pub t: u32,
pub text: String,
pub sha256: Sha256Hash,
pub parents: Vec<Sha256Hash>,
}
```
**Identity is the content hash, not a counter.** `sha256` is computed over the
canonical serialization of the semantic content — for `Chunk`, the concatenated
record texts and their provenance; for `MemoryNode`, `(level, project, query_id,
text)`. It must **not** include the timestamp or the run id, or re-running the
same input produces different hashes and `mem rebuild` stops being idempotent.
Newtypes with no `Default`: `ProjectId`, `QueryId`, `RunId`, `Sha256Hash`. A
placeholder that type-checks is invisible — a hardcoded `"current"` compiles,
passes tests, and makes every downstream result unattributable.
## Steps
1. Declare `Level`, `Role`, `Record`, `Provenance`, `Chunk`, `MemoryNode`.
2. Declare the newtypes. None derives `Default`. None has `From<String>` without
validation.
3. `fn content_hash(&self) -> Sha256Hash` on `Chunk` and `MemoryNode`, over a
canonical byte encoding that excludes timestamps and run ids.
4. `Level` serializes as the literal strings `"L0" | "L1" | "L2"` — the JSONL and
the SQL `CHECK` constraint both depend on that spelling.
5. Round-trip serde tests for every type.
## Acceptance
- Two `Chunk`s built from identical records in different runs hash identically.
- Changing one character of any record text changes the hash.
- `Level` round-trips through JSON as `"L0"`, not `0` and not `"l0"`.
## Verify
**Harness:** unit tests in `mem-core`, plus a hash-stability fixture committed
as bytes.
**Integration test** — `tests/it_identity.rs`:
1. `a1_same_content_same_hash` — build the same chunk twice with different
`RunId` and timestamps, assert equal hashes.
2. `a2_text_change_changes_hash` — flip one byte, assert the hash differs.
3. `a3_level_wire_format` — `serde_json::to_string(&Level::L0) == "\"L0\""`.
4. `a4_hash_stability_across_versions` — hash a committed fixture record set,
assert it equals a hash literal written into the test. This catches a
canonicalization change that would silently orphan every stored node.
5. `a5_newtypes_have_no_default` — compile-fail test (`trybuild`) asserting
`ProjectId::default()` does not compile.
**Command:** `cargo test -p mem-core identity`
**False pass:**
- Asserting only that hashing is deterministic *within one process*. A hash that
includes the timestamp is deterministic per run and still breaks rebuild.
Assertion 1 must vary the run id and timestamp deliberately.
- Omitting assertion 4. Without a committed expected hash, any future
canonicalization change passes every other test and silently invalidates the
database.
## Traps
- Including `run_id` or `created_at` in the hash. Rebuild then produces new nodes
every time and `memory_edge` accumulates orphans.
- Deriving `Default` on an id newtype "for tests". That default reaches
production and every row attributes to the same fake project.
---
Background: [DESIGN.md](../DESIGN.md) — The tier model, Storage schemas
tasks/M0.3-recordsource-and-chunkpolicy.md
+106
View File
@@ -0,0 +1,106 @@
# M0.3 — `RecordSource` trait + `ChunkPolicy`
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.2 |
## Goal
The seam where new input kinds arrive, shaped as a stream from the first commit
so a future streaming source implements a trait instead of forcing a rewrite.
## Facts (inlined — no spec read needed)
```rust
pub trait RecordSource {
/// Sources decide how to produce records; the chunker never learns
/// whether they came from pi, claude, or a socket.
fn records(self) -> impl Stream<Item = Result<Record>>;
}
pub struct ChunkPolicy {
pub max_tokens: usize, // 5000 — GRU-Mem paper default
pub split_on: Boundary, // Boundary::Record — never mid-Record
pub flush: FlushTrigger,
}
pub enum FlushTrigger {
Tokens(usize),
// OrIdle(Duration) lands with the first streaming source. Carrying the
// enum now means that change is one variant, not a signature change
// threaded through the loop.
}
pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
```
Why a stream when both current sources are files: sources today have an EOF;
telemetry, a live session tail, or a broker will not. Batch sources become
streams for free via `futures::stream::iter`, so this costs nothing today and
removes a rewrite later. The rest of the stack is already tokio.
**A single `Record` larger than `max_tokens` is not an error.** Tool results can
be enormous. It becomes a chunk of one, over budget, and the chunker records that
it did — silently truncating would destroy evidence, and silently dropping would
lose it.
## Steps
1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
record would exceed `max_tokens`, then yields.
3. Never split a `Record`. An oversized single record yields alone, with
`Chunk::over_budget = true`.
4. `t` is 1-based and contiguous across the whole stream.
5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
chunking is testable with no I/O and no model.
6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
a `CharsOverFour` stub is enough here.
## Acceptance
- Chunk boundaries never fall inside a `Record`.
- Chunk `t` values are 1-based, contiguous, no gaps.
- An oversized single record yields one chunk flagged `over_budget`.
- Concatenating all chunks' records reproduces the input sequence exactly.
## Verify
**Harness:** `VecSource` + the stub counter. No files, no network.
**Integration test** — `tests/it_chunking.rs`:
1. `a1_no_record_is_split` — for every chunk, every record equals some input
record byte for byte.
2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
input sequence, same order, same length.
3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
one record and `over_budget = true`.
5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
appears whole in the output, not truncated and not dropped.
6. `a6_empty_source` — zero records yields zero chunks, no panic.
**Command:** `cargo test -p mem-chunk chunking`
**False pass:**
- Testing only with uniformly small records. The budget logic is never exercised
and the oversized path never runs. Assertion 5 is the guard.
- Asserting chunk count rather than losslessness. A chunker that drops the final
partial chunk produces a plausible count and loses the tail — assertion 2 is
what catches it.
## Traps
- Truncating an oversized record to fit the budget. That is silent evidence
destruction, and the update gate will later be blamed for missing it.
- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
every test here, and defeats the entire reason this crate is separate.
---
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one
tasks/M0.4-tokenizer-sizing.md
+92
View File
@@ -0,0 +1,92 @@
# M0.4 — Tokenizer-backed chunk sizing
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.3 |
## Goal
Count tokens with the tokenizer the serving model actually uses, so a chunk that
fits locally also fits at the gateway.
## Facts (inlined — no spec read needed)
The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
Budget arithmetic, and why an approximation is not good enough:
```
Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side)
chunk 5000
prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024)
response 2048
```
A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
Undercount and the gateway rejects the request; overcount and chunks are smaller
than they need to be, which multiplies the number of model calls.
`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
runtime: a build that needs the network is a build that fails offline, and a
tokenizer that changes under you silently re-chunks the entire corpus.
## Steps
1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
3. Assert at load that the vendored file's hash matches the recorded one.
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
`qwen2.5:3b-instruct`.
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
for cross-checking against the gateway's reported `prompt_tokens`.
## Acceptance
- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
real records.
- Loading with a modified tokenizer file fails loudly, not silently.
## Verify
**Harness:** the vendored tokenizer plus recorded gateway responses. The
cross-check against the live gateway is a separate, network-gated test.
**Integration test** — `tests/it_tokens.rs`:
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
(ASCII, CJK, code, JSON, emoji), asserted exactly.
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
error naming the file.
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
`usage.prompt_tokens` to the local count minus the measured template overhead;
assert within 2%.
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
locally-counted size exceeds the budget.
**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
**False pass:**
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
this corpus contains both — the caveman skill text alone is bilingual.
- Comparing the local count to itself via a helper that calls the same function.
Assertion 1's expected values must be recorded from the tokenizer once and
written as literals.
## Traps
- Downloading the tokenizer at runtime. Offline builds break, and a silent
upstream change re-chunks everything, which changes every chunk hash, which
orphans every stored node.
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
ornith and is wrong; do not take a client-side config as the source of truth.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`
tasks/M0.5-pi-session-adapter.md
+131
View File
@@ -0,0 +1,131 @@
# M0.5 — pi session adapter
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.3 |
## Goal
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
from the path without guessing.
## Facts (inlined — no spec read needed)
Layout, verified on this machine:
```
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
└─ cwd with / replaced by -, wrapped in leading and trailing --
```
Record types observed in a real 2902-message session:
```
message 2902
model_change 112
thinking_level_change 16
compaction 8
session 1 <- always first line
```
Shapes:
```jsonc
// first line
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
// message
{"type":"message","id":"..","parentId":"..","timestamp":"..",
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
```
Role distribution in that same session — this is the whole reason the update gate
exists:
```
assistant 1445
toolResult 1261 43%, mostly evidence-free
user 196
```
`cwd` in the `session` header is authoritative for the project key. The directory
name is a lossy encoding (a real `-` in a path is indistinguishable from a
separator) — **parse `cwd`, do not decode the directory name.**
`content` is not always a string. Assistant messages carry content blocks; tool
results carry structured payloads. Normalise to text, and keep the block type in
`Provenance` so a later filter can act on it.
## Steps
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
2. Read the first line, require `type == "session"`, take `cwd` as the project
key. A file whose first line is not a session header is an error naming the
file, not a skip.
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
`toolResult -> Role::ToolResult`.
5. Flatten `content` to text for all shapes; preserve the original block type in
`Provenance`.
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
emit it as `Role::System` with the marker text, because a compaction boundary
is where context was lost and that is worth seeing in the log.
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
8. A malformed line is a counted, reported skip — never a panic. These files are
appended to by a live process and the last line may be a partial write.
## Acceptance
- Project key comes from `cwd`, matching for a path containing a literal `-`.
- All three roles are emitted with correct counts on a real session.
- A truncated final line is skipped with a warning, not a panic.
- Compaction events appear in the record stream.
## Verify
**Harness:** two committed fixtures — one small hand-built session covering every
record type and content shape, one real session copied verbatim (secrets
scrubbed) for volume.
**Integration test** — `tests/it_pi_source.rs`:
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
key is `/tmp/my-project`, proving the directory name was not decoded.
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
3. `a3_content_shapes` — string content, block-array content, and structured tool
result all flatten to non-empty text.
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
prior records and reports exactly one skip.
5. `a5_missing_header` — file whose first line is a `message`; assert an error
naming the path.
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
record before reading the whole file (assert peak allocation, or instrument
reads).
**Command:** `cargo test -p mem-ingest pi_source`
**False pass:**
- Testing only against the hand-built fixture. It will contain the content shapes
you thought of, which is the set you already handle. Assertion 2 against a real
session is what finds the rest.
- Asserting "no panic" on malformed input without asserting the *count* of
skips. A source that silently drops every line panics never and ingests
nothing.
## Traps
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
is ambiguous the moment a path component contains `-`, which is common.
- Treating `content` as `String`. It parses for user messages and fails on
assistant blocks, so the bug appears to be about assistants specifically and
wastes an afternoon.
- Materializing the file. These reach tens of MB and the trait is a stream for a
reason.
---
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
tasks/M0.6-claude-transcript-adapter.md
+92
View File
@@ -0,0 +1,92 @@
# M0.6 — Claude transcript adapter
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.5 |
## Goal
The second `RecordSource`, which is the one that proves the trait is real.
## Facts (inlined — no spec read needed)
```
~/.claude/projects/-Users-rockliang-workplace-Poimen/<uuid>.jsonl
└─ cwd, / replaced by -, no wrapping dashes (differs from pi)
```
15 MB across 7 transcripts on this machine. Record types observed:
```
attachment 150 queue-operation 138 assistant 134 user 81
file-history-snapshot 69 system 69 ai-title 21 last-prompt 21 mode 21
```
Fields, read from any line: `sessionId`, `cwd`, `gitBranch`.
Typed records dispatch on `type`; the ones that matter here:
- `user` / `assistant` — content under `message.content`
- `system` with `subtype == "api_error"` — a failed turn, worth keeping
- `summary` — carries `summary`
- everything else — ignore
The filename stem is a UUID. The encoding differs from pi (no wrapping `--`),
which is exactly why the project key must come from the `cwd` **field**, as in
M0.5, and not from the directory name.
## Steps
1. Implement `ClaudeTranscriptSource` in `mem-ingest`, implementing `RecordSource`.
2. Take `cwd` from the first line carrying it; error if no line does.
3. Emit records for `user`, `assistant`, and `system`+`api_error`.
4. Flatten `message.content` across its shapes, as in M0.5.
5. `Provenance` = `claude:<uuid>` plus line offset.
6. Reuse the content-flattening and malformed-line handling from M0.5 — extract
them into a shared helper rather than copying, since divergence between two
flatteners is a bug that only shows up on one source.
## Acceptance
- Both sources satisfy `RecordSource` with no changes to `mem-chunk`.
- Project keys from pi and claude for the same directory resolve to the same
`ProjectId`.
- `api_error` system records survive into the stream.
## Verify
**Harness:** a committed transcript fixture plus a real one, secrets scrubbed.
**Integration test** — `tests/it_claude_source.rs`:
1. `a1_project_from_cwd_field` — assert the key comes from `cwd`, not the
directory name, using a fixture where they would differ.
2. `a2_same_project_across_sources` — a pi session and a claude transcript for
the same directory yield an equal `ProjectId`. This is the assertion that
makes cross-source memory possible at all.
3. `a3_role_mapping` — user/assistant/api_error appear; `attachment`,
`queue-operation`, `file-history-snapshot` do not.
4. `a4_shared_flattener` — the same content-block fixture flattens identically
through both sources (call both, compare strings).
**Command:** `cargo test -p mem-ingest claude_source`
**False pass:**
- Testing each source in isolation. The point of this task is that they agree;
assertions 2 and 4 are the only ones that check it, and both are cross-source.
## Traps
- Copying the flattener instead of sharing it. The two will drift, and the
resulting bug looks like "claude transcripts lose tool output" rather than
"there are two flatteners".
- Assuming the pi encoding. Claude has no wrapping `--`, so a decoder written
against pi silently produces a different project key for the same directory —
and cross-source memory quietly splits in two.
---
Background: [DESIGN.md](../DESIGN.md) — Context
tasks/M0.7-ingest-dry-run.md
+94
View File
@@ -0,0 +1,94 @@
# M0.7 — `mem ingest --dry-run`
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.4, M0.6 |
## Goal
See the chunk plan for a real project before spending a single model call on it.
## Facts (inlined — no spec read needed)
`--dry-run` makes **zero network calls**. That is the property, not a side
effect: this is the last checkpoint before inference, and its value is telling
you the corpus is sane while feedback is still free.
Output shape:
```
project /Users/rockliang/workplace/Poimen/agent-rust
sources pi:4 files claude:2 files
records 3118 (assistant 1445 toolResult 1261 user 196 system 216)
chunks 412 over-budget 3
tokens min 84 p50 4870 p95 5000 max 11204
```
`over-budget` counts single records exceeding the chunk budget (M0.3). A nonzero
count is expected — large tool results — and is worth surfacing because it
predicts requests the gateway may reject.
Numbers to sanity-check against: a real pi session in this project has 2902
messages with the role split above, and toolResult being ~43% is the signal that
the update gate has something to discriminate.
## Steps
1. `mem ingest --project <path-or-key> --dry-run`.
2. Resolve sources: scan both source roots for directories whose `cwd` matches
the project. Report which files matched.
3. Stream records through `mem-chunk` with the real tokenizer; accumulate stats
without retaining chunk bodies.
4. Print the table above. Machine-readable variant behind `--format json`.
5. `--limit <n>` to stop after n chunks, for iterating on a large project.
6. Exit non-zero if zero sources matched — a silent empty plan reads like success.
## Acceptance
- No network syscall occurs during `--dry-run`.
- Stats are computed streaming; memory does not scale with corpus size.
- Zero matched sources exits non-zero with a message naming the project key.
## Verify
**Harness:** the fixtures from M0.5/M0.6, plus a network guard.
**Integration test** — `tests/it_dry_run.rs`:
1. `a1_no_network` — run the command with outbound TCP blocked (inject a
`reqwest` client that panics on use, or set an unroutable proxy); assert exit 0.
2. `a2_counts_match_sources` — record and role counts equal the sum of what the
two adapters yield independently.
3. `a3_chunk_count_matches_chunker` — the reported chunk count equals
`chunks(...).count()` computed separately.
4. `a4_over_budget_reported` — fixture with one oversized record; assert
`over-budget 1`.
5. `a5_empty_project_fails` — unknown project key exits non-zero, message names
the key.
6. `a6_constant_memory` — run against a 50 MB fixture; assert peak RSS stays
under a bound well below file size.
**Command:** `cargo test -p mem-cli dry_run`
**False pass:**
- Asserting the command exits 0 and printing looks right. A dry run that matched
no sources also exits 0 and prints a tidy table of zeros — assertion 5 is what
separates them.
- Computing stats by collecting chunks into a `Vec` first. Every assertion here
passes and assertion 6 is the only one that fails, which is why it is present.
## Traps
- Making `--dry-run` construct the LLM client "but not call it". Construction
reads credentials and can fail; the guarantee is no network, and the cheapest
way to keep it is to not build the client at all.
- Reporting p50 chunk size only. The max is the interesting number — it predicts
which requests will be rejected downstream.
---
Background: [DESIGN.md](../DESIGN.md) — Verification, P1
tasks/M0.8-m0-gate.md
+87
View File
@@ -0,0 +1,87 @@
# M0.8 — M0 composition gate
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M0 |
## Goal
Prove the parts compose and that the swappable part is genuinely swappable —
the properties no single M0 task owns.
## Facts (inlined — no spec read needed)
The claim this phase makes: **a new input kind is added by implementing
`RecordSource`, and nothing downstream changes.** If that is false, the crate
split bought nothing and the streaming work later will be a rewrite.
The gate proves it by adding a third source that resembles nothing already
supported, and asserting the rest of the pipeline is untouched.
Second claim: the whole phase is free. No model calls, so the gate is a CI job
that runs on every push without a gateway or credentials.
## Steps
1. Implement `SyntheticSource` in the test tree only — generates records from a
seed, has no file format, and deliberately produces a record larger than the
chunk budget and a run of empty-text records.
2. Assert `chunks()` handles it with **no change** to `mem-chunk`.
3. Assert dependency direction still holds (M0.1 assertion 3) after three sources
exist — this is when someone is tempted to reach backwards.
4. Run `mem ingest --dry-run` over all three sources for one project and diff the
summary against a committed expected file.
5. Wire the whole thing as a required CI job.
## Acceptance
- Three sources, one `RecordSource`, zero source-specific branches in `mem-chunk`
or `mem-cli`.
- Dry-run summary diffs empty against the committed expectation.
- CI job passes with no network and no credentials.
## Verify
**Harness:** committed expected-output file, diffed. The script's output is the
review artifact.
**Integration test** — `tests/it_m0_gate.rs`:
1. `a1_third_source_needs_no_downstream_change` — `SyntheticSource` flows through
`chunks()`; assert `git diff --stat crates/mem-chunk crates/mem-cli` is empty
for the commit that added it (enforced by a CI step, not by the test binary).
2. `a2_no_source_specific_branches` — grep `crates/mem-chunk` and `crates/mem-cli`
for the strings `pi:`, `claude:`, `sessionId`, `toolResult`; assert none
appear outside `mem-ingest`.
3. `a3_dry_run_golden` — run the dry run over all three sources, diff against
`expected/m0-gate.txt`; empty diff is the only pass.
4. `a4_offline` — the entire gate runs with networking disabled.
5. `a5_lossless_end_to_end` — records in equals records out, across all three
sources composed.
**Command:** `cargo test --workspace m0_gate`
**False pass:**
- Adding `SyntheticSource` in a way that mirrors the pi format. It then exercises
the same code path and proves nothing about generality. It must have no file,
no header line, and an oversized record.
- Assertion 3 passing because the expected file was regenerated in the same
commit. A changed `expected/` file in a diff is a claim that the contract
changed, and must be reviewed as one.
## Traps
- Skipping assertion 2 because "obviously there are no source-specific branches".
There will be — the first `if provenance.starts_with("pi:")` gets added to fix
a real bug and is entirely reasonable in isolation.
- Letting the gate need credentials. The value of an offline gate is that it runs
on every push; the moment it needs a gateway key it becomes a nightly job that
nobody watches.
---
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, Verification
tasks/M1.1-llm-chat-client.md
+107
View File
@@ -0,0 +1,107 @@
# M1.1 — `mem-llm` chat client
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.1 |
## Goal
Talk to the homelab gateway, with the two non-obvious details that cost a day to
find already baked in.
## Facts (inlined — no spec read needed)
```
base https://api.riotpiao.com/v1
route POST /v1/qwen/chat/completions qwen2.5:3b-instruct
POST /v1/ornith/chat/completions ornith:35b
POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools)
```
**Auth is `apikey:`, not `Authorization: Bearer`.** Kong's `key-auth` compares
the whole header value against the stored key, so the OpenAI SDK convention
returns 401. Verified:
```
-H "apikey: $KEY" -> 200
-H "Authorization: $KEY" -> 200
-H "Authorization: Bearer $KEY" -> 401
```
**One provider per route.** `baseUrl` is per-route and the model id is `ornith:35b`
with the tag, not `ornith`. `/v1/models` advertises `/v1/score` which does not
work — do not trust that list as a capability probe.
Request bodies above ~10.6 KB used to fail with
`{"error":{"message":"[] is too short - 'messages'"}}`; the Kong body buffer was
raised to 16m and it is fixed. If that error ever reappears, it is the buffer,
not the client.
Send **no `tools` array**. The controller needs none, and the reasoning route
rejects any request carrying one.
## Steps
1. `ChatClient::new(base_url, api_key, model)` in `mem-llm`.
2. Send the `apikey` header. Read the key from `MEM_API_KEY`, never from a
committed file.
3. `complete(system, user, max_tokens) -> Completion { text, usage, latency }`.
4. Timeout default 300s — local models are slow to first token and a cold load
can take minutes.
5. Retry on 5xx and timeout with exponential backoff, max 3. **Do not retry 4xx**
— a 400 is a malformed request and retrying it just costs three times as much.
6. On any error, include the response body in the error. The useful information
is always in the body, never the status.
7. `MEM_LLM_RECORD=<dir>` writes every request/response pair to disk, for
building fixtures without hand-writing them.
## Acceptance
- A real completion round-trips against the gateway.
- A 401 is reported as an auth error naming the header convention.
- A 4xx is not retried; a 5xx is.
## Verify
**Harness:** `wiremock` for the offline tests; one `#[ignore]` test against the
live gateway.
**Integration test** — `tests/it_chat_client.rs`:
1. `a1_sends_apikey_header` — assert the mock received `apikey` and **no**
`authorization` header.
2. `a2_no_tools_field` — assert the serialized body has no `tools` key at all,
not merely an empty array.
3. `a3_retries_5xx` — mock 503 twice then 200; assert 3 requests and success.
4. `a4_does_not_retry_4xx` — mock 400; assert exactly 1 request and an error
carrying the body text.
5. `a5_timeout_is_configurable` — mock a 2s delay with a 1s timeout; assert a
timeout error.
6. `a6_live_smoke` — `#[ignore]`; real gateway, `qwen2.5:3b-instruct`, prompt
"reply with exactly: pong", assert the text contains `pong`.
**Command:** `cargo test -p mem-llm chat_client` (add `-- --ignored` for a6)
**False pass:**
- Testing only against the mock. The mock accepts whatever header you send it;
assertion 6 against the live gateway is the only thing that proves the auth
convention is right.
- Asserting `tools: []` is absent by checking `body.tools.is_empty()`. An empty
array serialized into the request is still a `tools` key, and that is what the
reasoning route rejects. Assert on the raw JSON.
## Traps
- Using `Authorization: Bearer`. It is what every SDK does and it 401s here.
- Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it
tripled the load and produced identical failures more slowly.
- A 60s timeout. That is the value that made `ornith` look broken when it was
merely cold.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts
tasks/M1.2-standing-query-loader.md
+103
View File
@@ -0,0 +1,103 @@
# M1.2 — Standing-query YAML loader
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.2 |
## Goal
Load the standing questions that give the update gate its referent, and fail at
load rather than mid-run when one is wrong.
## Facts (inlined — no spec read needed)
```yaml
# queries/poimen.yaml
project: poimen
roots:
- /Users/rockliang/workplace/Poimen/agent-rust # matched against session cwd
sources: [pi, claude]
queries:
- id: architecture-decisions
question: What architectural decisions were made, with reasoning and rejected alternatives?
- id: infra-root-causes
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
synthesis:
question: What is the current state of this project, and what should someone know before working on it?
exit_gate: true
defaults:
memory_budget: 1024
chunk_tokens: 5000
exit_gate: false # L1 default — see below
```
**Why a question is mandatory.** The GRU-Mem memory agent is `φθ(Q, C_t, M_{t-1})`
and its update gate is defined as "does this chunk contain useful information
*about the problem*". With no `Q` the gate has no referent, and `r_update` is
undefinable — which forecloses post-training (M5) entirely. A query with an empty
question is a load error, not a warning.
**`exit_gate: false` at L1 is deliberate.** Paper §3.3: for "what are *all* the
X" questions you cannot know evidence is sufficient without reading everything,
so the paper itself provides a without-exit-gate inference mode. L1 extraction is
that shape. The gate is still *recorded* — its signal is needed for M5.
`query.id` is stable and frozen. It names the L1 memory, the Obsidian note, and
the log directory; renaming it orphans all three.
## Steps
1. `QuerySet::load(path)` in `mem-core`, `serde_yaml`.
2. Validate at load: non-empty `project`; at least one query; every `id` unique,
non-empty, and `[a-z0-9-]+`; every `question` non-empty; `memory_budget > 0`.
3. Every failure names the file, the query id, and the field.
4. `mem query validate <file>` prints the resolved set and exits non-zero on any
error.
5. Defaults apply per query and are overridable per query.
6. `id` collision across two files for the same project is an error.
## Acceptance
- An empty or missing `question` fails at load with a message naming the id.
- An id outside `[a-z0-9-]+` fails at load — it becomes a filename.
- Defaults resolve; per-query overrides win.
## Verify
**Harness:** table-driven over fixture YAML files, one per failure mode.
**Integration test** — `tests/it_query_loader.rs`:
1. `a1_valid_loads` — the reference file above resolves to the expected struct.
2. `a2_empty_question_rejected` — error text contains the query id.
3. `a3_duplicate_id_rejected` — error names both occurrences.
4. `a4_bad_id_charset_rejected` — `infra/root causes` is rejected, message
mentions the filename constraint.
5. `a5_defaults_and_overrides` — a query without `exit_gate` gets `false`; one
with `true` keeps it.
6. `a6_l1_exit_gate_defaults_false` — assert explicitly, because a silent flip to
`true` truncates every extraction and looks like a model quality problem.
7. `a7_missing_question_field` — absent key behaves as empty, same error.
**Command:** `cargo test -p mem-core query_loader`
**False pass:**
- Testing only the happy path. Every assertion except 1 and 5 is a rejection
test, and rejection is the entire point of load-time validation.
- Asserting an error occurred without asserting the message names the offending
id. "invalid config" sends someone to read the whole file by hand.
## Traps
- Allowing an empty question "for now". It loads, the gate has no referent, the
model updates on nearly everything, and it reads as a bad model rather than a
bad config.
- Letting `id` contain `/` or spaces. It is a path segment in three places.
---
Background: [DESIGN.md](../DESIGN.md) — Standing queries
tasks/M1.3-prompt-template.md
+120
View File
@@ -0,0 +1,120 @@
# M1.3 — GRU-Mem prompt template
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.2 |
## Goal
Assemble the memory-agent prompt exactly as the paper specifies, because the
model's ability to emit parseable gates depends on the format it was aligned to.
## Facts (inlined — no spec read needed)
Paper Figure 10a, reproduced verbatim — this is the contract, not a starting
point to improvise on:
```
You are presented with a problem, a section of an article that may contain the
answer to the problem, and a previous memory. Please read the provided section
carefully. You should reason about whether the new section contains useful
information about the problem, and then update the memory with the new
information that helps to answer the problem.
Be sure to retain all relevant details from the previous memory while adding any
new, useful information. You should also carefully judge whether you have
collected enough information to answer the problem.
You should reason about whether the new section contains useful information, what
to update, and what to do next first between <think> and </think>.
If the new section contains useful information about the problem, you should
first generate <check>yes</check>. After that, update the new memory between
<update> and </update>.
If the new section does not contain useful information about the problem, you
should first generate <check>no</check>. After that, you should keep the previous
memory unchanged between <update> and </update>.
In the end, if you haven't collected enough information for the problem, return
<next>continue</next>. ONLY when enough information is collected, return
<next>end</next>.
<problem> {prompt} </problem>
<memory> {memory} </memory>
<section> {chunk} </section>
```
Substitutions for this system: `{prompt}` = the standing question, `{memory}` =
`M_{t-1}` or the literal `No previous memory` at `t=1` (the paper's own case
studies show that exact string), `{chunk}` = the rendered chunk.
Budget, against the 32768 cap:
```
system + template ~400
question ~100
memory <=1024
chunk <=5000
response 2048
------
~8600 headroom is comfortable
```
Chunk rendering: each record as `[role] text`, records separated by a blank line.
Role labels matter — the model uses them to tell a tool result from a decision.
## Steps
1. `PromptBuilder` in `mem-core` producing `(system, user)`.
2. Template verbatim as above. Any deviation gets a comment saying why.
3. `t=1` renders `No previous memory` — not empty, not `null`.
4. Render chunk records as `[role] text`, blank-line separated.
5. Assert the assembled prompt fits the budget before sending; over budget is an
error naming the component that overflowed, not a truncation.
6. `mem prompt --project P --query Q --chunk N` prints the exact prompt, for
eyeballing what the model actually sees.
## Acceptance
- Assembled prompt matches a committed golden file byte for byte.
- `t=1` contains `No previous memory`.
- Over-budget assembly errors and names the offending component.
## Verify
**Harness:** golden-file comparison. The prompt is a contract; a diff in it is a
change to the contract.
**Integration test** — `tests/it_prompt.rs`:
1. `a1_golden_t1` — first turn against `expected/prompt-t1.txt`, exact match.
2. `a2_golden_tn` — turn with a prior memory against `expected/prompt-tn.txt`.
3. `a3_no_previous_memory_literal` — assert the exact string at `t=1`.
4. `a4_all_tags_present` — `<problem>`, `<memory>`, `<section>` each appear
exactly once.
5. `a5_role_labels_rendered` — a chunk with all four roles renders all four
labels.
6. `a6_over_budget_errors` — a 20000-token chunk errors, message contains
`section`.
7. `a7_budget_headroom` — for the real fixture corpus, assert every assembled
prompt is under 32768 minus 2048.
**Command:** `cargo test -p mem-core prompt`
**False pass:**
- Asserting the prompt "contains" the question. A template that dropped the
`<check>` instructions still contains it, and the model then emits prose the
parser cannot read. Golden-file equality is the assertion that holds.
- Skipping assertion 7 by testing only small fixtures. Budget overflow appears
at p95 chunk size, not at the median.
## Traps
- Improving the wording. The 3B model's gate reliability comes from this exact
format; a cleaner rewrite is an unmeasured change to the one thing M1.8 gates on.
- Rendering an empty `<memory></memory>` at `t=1`. The paper's traces show
`No previous memory`, and an empty tag reads to the model as "memory exists and
is empty", which is a different claim.
---
Background: [DESIGN.md](../DESIGN.md) — Standing queries · paper Fig 10a
tasks/M1.4-gate-response-parser.md
+113
View File
@@ -0,0 +1,113 @@
# M1.4 — Gate-response parser
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.3 |
## Goal
Turn the model's tagged output into `(U_t, M̂_t, E_t)`, strictly — because a
lenient parser silently fabricates gate decisions.
## Facts (inlined — no spec read needed)
Expected response shape:
```
<think>...</think>
<check>yes|no</check>
<update>candidate memory, or the previous memory verbatim</update>
<next>continue|end</next>
```
Semantics, from the paper:
| tag | value | meaning |
|---|---|---|
| `<check>` | `yes` | `U_t = true` — memory becomes `M̂_t` |
| `<check>` | `no` | `U_t = false` — memory stays `M_{t-1}`, chunk discarded |
| `<next>` | `continue` | `E_t = false` |
| `<next>` | `end` | `E_t = true` |
**Strict parsing is the design, matching the paper's `r_format`:** it awards 1
only when *every* turn in the trajectory parses, 0 otherwise, "because we can not
infer whether the incorrect format is caused by the previous erroneous parsing".
So: exactly one of each tag, properly closed, `<check>` content exactly `yes` or
`no` after trimming, `<next>` exactly `continue` or `end`. Anything else is a
`ParseError` naming which tag failed and carrying the raw text.
**A parse failure must not default.** Defaulting `U_t` to `false` silently drops
evidence; defaulting to `true` pollutes memory. The loop (M1.5) decides the
retry policy; the parser only reports.
Reasoning models emit `<think>` natively, which can nest or repeat. Extract by
locating the *last* `</think>` before the first `<check>`, not by regex over the
whole body.
## Steps
1. `parse_gate_response(&str) -> Result<GateResponse, ParseError>` in `mem-core`.
2. `GateResponse { think: String, update_gate: bool, candidate: String, exit_gate: bool }`.
3. Reject duplicates of any tag, a missing tag, an unclosed tag, and any
`<check>`/`<next>` value outside the allowed set.
4. `ParseError` variants name the tag and include the raw response, truncated.
5. Trim surrounding whitespace inside tags; do not otherwise normalise — memory
text is preserved verbatim.
6. When `U_t = false`, still capture `candidate` so the log can show what the
model *would* have written. The loop ignores it; the record is diagnostic.
## Acceptance
- All four tags parse from a well-formed response.
- Every malformed shape errors, naming the failing tag.
- No input produces a default `U_t` or `E_t`.
## Verify
**Harness:** table-driven over recorded real responses plus hand-built malformed
cases. Capture real ones with `MEM_LLM_RECORD` from M1.1.
**Integration test** — `tests/it_gate_parser.rs`:
1. `a1_wellformed_yes_continue` — `U=true`, `E=false`, candidate matches.
2. `a2_wellformed_no_end` — `U=false`, `E=true`.
3. `a3_missing_check_errors` — error names `check`.
4. `a4_duplicate_update_errors` — two `<update>` blocks error.
5. `a5_bad_check_value_errors` — `<check>maybe</check>` errors, message shows the
value.
6. `a6_unclosed_tag_errors` — `<update>` never closed.
7. `a7_nested_think` — a response with `<think>` inside `<think>` still finds the
right boundary.
8. `a8_no_defaults` — property test over 1000 random mutations of a valid
response: every result is either an exact parse or an error, never a
silently-defaulted `GateResponse`.
9. `a9_real_responses` — every recorded real response parses.
**Command:** `cargo test -p mem-core gate_parser`
**False pass:**
- A regex that finds the first `<check>` and stops. It passes 1–2 and silently
accepts duplicates, which is assertion 4's job.
- Testing only hand-written responses. Real 3B output has whitespace,
markdown fences and stray prose the author would not think to write —
assertion 9 is the only one that sees it.
- Omitting assertion 8. A parser with `unwrap_or(false)` anywhere passes every
positive test and quietly halves the update rate.
## Traps
- Defaulting on parse failure. The one thing that must not happen: a fabricated
gate decision is indistinguishable from a real one in the log, and it poisons
the M5 training data at the source.
- Normalising the candidate memory (collapsing whitespace, stripping markdown).
Memory text is content; the hash and every downstream projection depend on it
being verbatim.
---
Background: [DESIGN.md](../DESIGN.md) — Architecture · paper §3.2.1 `r_format`
tasks/M1.5-gated-loop.md
+125
View File
@@ -0,0 +1,125 @@
# M1.5 — The gated loop
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | L — 3+ days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.4 |
## Goal
The recurrence itself: `U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})`, with the level as a
parameter so L2 reuses it unchanged.
## Facts (inlined — no spec read needed)
Paper Algorithm 1, transcribed:
```
t <- 1; M_0 <- None
while t <= T:
U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})
if U_t == True: M_t <- M̂_t # update
else: M_t <- M_{t-1} # retain, discard chunk
if use_exit_gate and E_t == True: break
t <- t + 1
answer = ψθ(Q, M_t)
```
Two decisions this task must not get wrong:
**`use_exit_gate` is a parameter, false at L1.** `E_t` is always *recorded*
regardless — its signal is the M5 training target, and on a future unbounded
stream it becomes the only termination condition. Recording a gate you do not act
on is deliberate, not dead code.
**The memory budget is enforced by the loop, not hoped for from the model.**
`memory_budget` is 1024 tokens. If `M̂_t` exceeds it the loop does **not** silently
truncate — truncation mid-sentence corrupts the memory for every later turn. It
records a `budget_exceeded` event and retains `M_{t-1}`, treating the turn as
`U_t = false`. Memory that stops growing is recoverable; memory that is
truncated garbage is not.
The loop is generic over the input stream, so L2 (M3.1) passes L1 memories in
place of chunks with no other change.
## Steps
1. `run_loop(level, query, source: impl Stream<Item = Chunk>, cfg) -> RunOutcome`
in `mem-core`.
2. Per turn: build prompt (M1.3), call model (M1.1), parse (M1.4), apply the
update rule, emit events.
3. On parse error: retry the same chunk up to 2 times. Still failing, record
`parse_failed`, treat as `U_t = false`, continue. **Never** default the gate.
4. On `U_t = true`: emit an L0 `evidence` event for the chunk, then an L1
`memory` event whose `parents` include that evidence sha plus the previous
memory's sha.
5. Enforce `memory_budget` as above.
6. Honour `E_t` only when `use_exit_gate`; always record it.
7. Emit `run_end` with `chunks_seen`, `chunks_used`, `final_memory_sha`.
8. Cancellation: a dropped future must not leave a half-written log. Emit events
only after a turn fully resolves.
## Acceptance
- `U_t = false` leaves memory byte-identical to the previous turn.
- `U_t = true` replaces memory and links parents.
- `exit_gate = false` processes every chunk even when `E_t = true` throughout.
- Over-budget candidate retains prior memory and records the event.
- Two parse failures then success consumes 3 calls for one chunk.
## Verify
**Harness:** a scripted fake `ChatClient` returning canned responses per turn, so
the whole loop runs with no network and fully determined gate sequences.
**Integration test** — `tests/it_gated_loop.rs`:
1. `a1_retain_on_no` — scripted `no` for 5 turns; assert final memory equals
initial and `chunks_used == 0`.
2. `a2_update_on_yes` — `yes` at turn 3 only; assert memory equals turn 3's
candidate and `chunks_used == 1`.
3. `a3_exit_gate_off_reads_all` — `end` at every turn with `use_exit_gate=false`;
assert all 10 chunks processed.
4. `a4_exit_gate_on_stops` — same script, `use_exit_gate=true`; assert it stops
at turn 1.
5. `a5_exit_always_recorded` — in a3, assert 10 `gate` events carry `exit=true`
despite not acting on them.
6. `a6_budget_exceeded_retains` — candidate of 4000 tokens; assert memory
unchanged, one `budget_exceeded` event, turn counted as not-used.
7. `a7_parse_retry` — fail twice then succeed; assert 3 calls, one memory event.
8. `a8_parse_failure_is_not_an_update` — fail 3 times; assert `U_t` false,
`parse_failed` recorded, loop continues.
9. `a9_parents_linked` — every memory event's `parents` contains the evidence sha
from the same turn.
10. `a10_level_is_a_parameter` — run the identical script at L1 and L2; assert the
only difference in emitted events is the `level` field.
**Command:** `cargo test -p mem-core gated_loop`
**False pass:**
- Testing with a fake that always returns `yes`. Every assertion about the retain
path is skipped, and retain is the path that matters — it is what makes this a
gate rather than a summarizer.
- Asserting `chunks_used` without asserting memory bytes. A loop that updates
memory on `no` but counts correctly passes a count-only test; assertion 1
compares bytes.
- Omitting assertion 10. Without it, L2 in M3.1 becomes a copy of this loop, and
the two drift.
## Traps
- Truncating over-budget memory to fit. It corrupts every subsequent turn's
input, and the damage is attributed to the model.
- Acting on `E_t` at L1 because "the model said it had enough". Paper §3.3 is
explicit that this is wrong for exhaustive questions, and it silently truncates
extraction in a way that looks like poor recall.
- Writing log events before the turn resolves. A cancelled run then leaves a
memory event with no matching gate event and `mem verify` fails on a file that
was merely interrupted.
---
Background: [DESIGN.md](../DESIGN.md) — Architecture, tier model · paper Alg 1
tasks/M1.6-jsonl-event-log.md
+104
View File
@@ -0,0 +1,104 @@
# M1.6 — JSONL event log writer
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.5 |
## Goal
Write the authoritative record — the one artifact everything else is derived
from, and the one that must survive a crash mid-run.
## Facts (inlined — no spec read needed)
Path: `log/<project>/<query-id>/<run-id>.jsonl`. Append-only, one object per line.
**Every record carries `level`.**
```jsonl
{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}
```
`evidence` appears **only** when the update gate opened. That is what makes
update-rate directly measurable from the log — `gate` records give the
denominator, `evidence` records the numerator.
This log is authoritative: the vault and pgvector are projections rebuilt from
it. Two consequences — it is tracked in git, and it is never rewritten in place.
A run that crashes leaves a file with no `run_end`. That is a valid, readable
state meaning "incomplete", not corruption. Readers must handle it.
## Steps
1. `LogWriter::open(project, query_id, run_id)` in `mem-store`, creating parents.
2. `append(event)` serializes one line and **flushes**. An unflushed buffer loses
the last turns of exactly the run you want to debug.
3. `run_id` is a ULID — lexicographically sortable by creation time, so listing
runs in order is a directory sort.
4. `LogReader` streams events back, tolerating a truncated final line.
5. `replay_memory_at(t)` reconstructs `M_t` from the events alone, proving the log
is sufficient.
6. `stats()` computes chunks seen/used and update-rate from a log file.
7. Never open in truncate mode. Append only.
## Acceptance
- Every emitted record has a `level`.
- `evidence` count equals the count of `gate` records with `update: true`.
- A file with no `run_end` reads cleanly and reports `incomplete`.
- `replay_memory_at(t)` matches the memory the loop held at `t`.
## Verify
**Harness:** the scripted loop from M1.5 writing to a temp dir, plus a corrupted
fixture.
**Integration test** — `tests/it_event_log.rs`:
1. `a1_every_record_has_level` — parse every line, assert `level` present and in
`{L0,L1,L2}`.
2. `a2_evidence_matches_update_gates` — count `gate.update==true`, assert equal to
the `evidence` count.
3. `a3_replay_equals_live` — for every `t`, `replay_memory_at(t)` equals the
memory the loop held. This is the assertion that proves the authority model.
4. `a4_truncated_tail_reads` — chop the last line mid-object; assert all prior
events parse and the run reports `incomplete`.
5. `a5_no_run_end_is_incomplete` — a log ending after a `memory` event reports
incomplete, not an error.
6. `a6_append_only` — write, reopen, write again; assert the first events survive.
7. `a7_flush_per_event` — kill the process (or drop without close) after 3
appends; assert 3 lines on disk.
8. `a8_run_id_sorts_by_time` — three runs, assert lexicographic order equals
chronological order.
**Command:** `cargo test -p mem-store event_log`
**False pass:**
- Asserting the file parses. A writer that omits `evidence` events entirely
produces a perfectly parseable log with an update-rate of zero — assertion 2 is
what catches it.
- Testing replay only at the final `t`. A writer that records only the final
memory passes that and fails assertion 3 at every intermediate turn.
- Omitting assertion 7. Buffered writes pass every test that closes the file
properly, and lose data in precisely the crash case the log exists for.
## Traps
- Opening with truncate. One accidental re-run erases the authoritative record,
and the projections are the only surviving copy — inverted authority.
- Gitignoring `log/`. Makes the whole "JSONL is authoritative" claim a fiction.
`agent-rust/.gitignore` has a bare `tasks` entry that untracks its whole board;
do not repeat it here.
---
Background: [DESIGN.md](../DESIGN.md) — Storage schemas, authority model
tasks/M1.7-ingest-end-to-end.md
+102
View File
@@ -0,0 +1,102 @@
# M1.7 — `mem ingest` end to end
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |
## Goal
One command that reads a real project and produces a real log — and reports the
number that says whether the gate works.
## Facts (inlined — no spec read needed)
```
mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen # all queries in the set
mem ingest --project poimen --limit 50 # first 50 chunks, for iterating
mem ingest --project poimen --resume # skip chunks already in the log
```
Progress output, because a run that prints nothing cannot be distinguished from
one that has hung:
```
[ 17/412] t=17 update=yes mem=142tok 1.9s
[ 18/412] t=18 update=no mem=142tok 0.8s
...
run 01HXYZ chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
```
**Update-rate is the headline number.** Tool results are ~43% of records and
mostly evidence-free; a correct gate rejects the large majority of chunks. A rate
above ~30% means the gate is not discriminating and the run is an expensive
summarizer — that is the paper's memory-explosion failure and it is what M1.8
gates on.
Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every
chunk costs a model call, so `--resume` is not a nicety.
## Steps
1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
3. Report update-rate in the summary and as `--format json`.
4. `--resume` reads the existing log, finds the highest `t` with a `gate` record,
and restarts from `t+1` with that turn's memory.
5. `--limit` caps chunks processed.
6. Exit non-zero if the run did not reach `run_end`.
7. Ctrl-C finishes the in-flight turn, writes `run_end`, exits — no half-turn.
## Acceptance
- A real project produces a complete log with `run_end`.
- Reported update-rate equals the value computed independently from the log.
- `--resume` on a complete log is a no-op; on a partial one it continues.
- Interrupt produces a valid log.
## Verify
**Harness:** scripted client for determinism, plus one live `#[ignore]` run.
**Integration test** — `tests/it_ingest.rs`:
1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event
counts match the script.
2. `a2_update_rate_matches_log` — compare the reported rate to
`LogWriter::stats()` recomputed from the file.
3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model
calls.
4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the
next call is t=11 and memory at t=11 equals the replayed memory at t=10.
5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has
`run_end`, and the last `gate` has a matching `memory`-or-not decision.
6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records.
7. `a7_live_smoke` — `#[ignore]`; real gateway, `--limit 20` on a real project;
assert `run_end` and **print** the update-rate for a human to read.
**Command:** `cargo test -p mem-cli ingest` (add `-- --ignored` for a7)
**False pass:**
- Asserting only that the command exits 0. A run whose gate always answers `no`
exits 0, writes a valid log, and has learned nothing — the update-rate is the
only thing that distinguishes it, which is why a7 prints it rather than
merely asserting a run happened.
- Resuming by counting lines rather than reading the highest `t` with a `gate`
record. Line counts break the moment an `evidence` record is present, i.e. as
soon as the gate ever opened.
## Traps
- No progress output. A 14-minute run that prints nothing is indistinguishable
from a hang, and the first instinct will be to kill it.
- Resume that replays from `t=1` with the old memory. It costs a full run and
produces a log with duplicate turns that `mem verify` will reject.
---
Background: [DESIGN.md](../DESIGN.md) — Verification, P2
tasks/M1.8-m1-gate.md
+99
View File
@@ -0,0 +1,99 @@
# M1.8 — M1 composition gate
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M1 |
## Goal
Answer the only question that matters at this stage: **did we build a gate, or an
expensive summarizer?**
## Facts (inlined — no spec read needed)
This gate runs against the **live gateway on a real project** and asserts
properties of the resulting log. It is the first phase that costs money in wall
clock, and the first that can fail for reasons no unit test can see.
Two thresholds, both from the paper:
**Update-rate < 30%.** Agent transcripts are ~43% tool results, mostly
evidence-free. Paper Figure 6 shows the failure mode directly: the ungated
MemAgent's memory climbs to its 1024-token ceiling and saturates, after which
"the accumulated noise can further impede subsequent updates". A high update-rate
is that curve starting.
**Memory size flat, not climbing.** Plot `memory.tokens` against `t`. GRU-Mem's
curve is low and roughly flat; the ungated curve rises to the cap and stays
pinned. A monotonic climb means the gate is open too often even if the rate
looks acceptable.
Third property, cheap and load-bearing: **the same run twice produces the same
chunk hashes**. Temperature affects the model's text, not the chunking; if chunk
shas differ between runs, identity includes something it should not (M0.2) and
every projection will churn.
## Steps
1. Run `mem ingest --project poimen` for all standing queries against the live
gateway.
2. Compute per query: update-rate, memory-token series, parse-failure count,
elapsed.
3. Assert the thresholds below.
4. Emit `expected/m1-gate.txt` with the summary table; commit it. Subsequent runs
diff against it, and a changed expectation is a reviewable claim.
5. Sample 20 gate decisions and have the 32B `reasoning` model audit them; report
agreement. Advisory at this gate, and the seed of M5.2's calibration.
## Acceptance
- Update-rate < 30% on every standing query.
- Memory tokens ≤ 1024 and not monotonically increasing.
- Parse-failure rate < 5%.
- Chunk shas stable across two runs.
## Verify
**Harness:** live gateway. Long-running; a nightly or on-demand job, not
per-push.
**Integration test** — `tests/it_m1_gate.rs`, all `#[ignore]` by default:
1. `a1_update_rate_under_threshold` — per query, assert < 0.30, print actual.
2. `a2_memory_bounded` — every `memory.tokens` ≤ 1024.
3. `a3_memory_not_climbing` — fit a line to tokens vs `t`; assert the slope is
below a small positive bound. Do not assert non-increasing — a legitimately
growing memory rises early then plateaus.
4. `a4_parse_failure_rate` — `parse_failed` / turns < 0.05.
5. `a5_chunk_sha_stable` — two runs, same chunk shas in the same order.
6. `a6_evidence_traceable` — every `evidence` sha appears as a parent of some
`memory` record.
7. `a7_judge_audit` — sample 20 decisions, ask the 32B model, print agreement.
Advisory; does not fail the gate.
**Command:** `cargo test --workspace m1_gate -- --ignored --nocapture`
**False pass:**
- Running the gate on a tiny `--limit`. Update-rate on the first 20 chunks is
noise; the failure mode is cumulative and needs a full project.
- Asserting the rate without printing it. The number is the artifact — a run at
29% passes and is telling you something a boolean hides.
- Asserting memory is non-increasing rather than bounded-slope. Real memory does
grow early, and a strict assertion here fails on correct behaviour, which
trains everyone to skip the gate.
## Traps
- Treating a high update-rate as a model-quality problem first. Check the
prompt (M1.3 golden file) and the parser (M1.4 defaults) before blaming the
3B model — a parser with `unwrap_or(true)` produces exactly this symptom.
- Tuning the threshold to whatever the first run produced. 30% comes from the
corpus composition; moving it to accommodate a bad result deletes the gate.
---
Background: [DESIGN.md](../DESIGN.md) — Verification · paper Fig 6, §4.2
tasks/M2.1-embeddings-client.md
+86
View File
@@ -0,0 +1,86 @@
# M2.1 — Embeddings client
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.1 |
## Goal
Turn text into 768-dim vectors, batched, against the gateway's TEI endpoint.
## Facts (inlined — no spec read needed)
```
POST /v1/embeddings
{"model":"nomic-ai/nomic-embed-text-v2-moe","input":["..."]}
-> {"object":"list","data":[{"embedding":[...768 floats...]}],"usage":{...}}
```
**768 dimensions**, probed and confirmed. It is the `vector(768)` in the schema
and in the HNSW index; a model swap is a schema migration, not a config change.
**Batch limit is 32.** Verified: 1200 inputs returned
`{"message":"batch size 1200 > maximum allowed batch size 32","code":413}`.
Chunk the input list accordingly.
This route currently has **no auth** — no `konghq.com/plugins` annotation, so
`model-key-auth` never attaches. Send the `apikey` header anyway: the route
should be fixed, and a client that only works while auth is broken breaks when
it is fixed.
Bodies here are large (many texts × long strings) and the Kong buffer is 16m, so
batching also keeps requests well inside it.
## Steps
1. `EmbeddingsClient::embed(texts: &[String]) -> Result<Vec<Vec<f32>>>` in `mem-llm`.
2. Split into batches of ≤32, preserving input order in the output.
3. Assert every returned vector is exactly 768 long; a mismatch is an error
naming the model, not a silent pad or truncate.
4. Reuse M1.1's client config: `apikey` header, retry on 5xx only, generous
timeout.
5. `embed_one` convenience wrapper.
## Acceptance
- 100 texts return 100 vectors in input order.
- Every vector is 768-dim.
- A dimension mismatch errors loudly.
## Verify
**Harness:** `wiremock` offline, one `#[ignore]` live test.
**Integration test** — `tests/it_embeddings.rs`:
1. `a1_batches_at_32` — 100 inputs produce exactly 4 requests.
2. `a2_order_preserved` — mock returns identifiable vectors; assert output order
matches input order across batch boundaries.
3. `a3_dimension_asserted` — mock returns a 512-dim vector; assert an error
naming the model.
4. `a4_apikey_sent` — assert the header is present even though the route does not
require it.
5. `a5_live_dims` — `#[ignore]`; real gateway, assert 768.
**Command:** `cargo test -p mem-llm embeddings` (add `-- --ignored` for a5)
**False pass:**
- Testing with ≤32 inputs. The batching path never runs and order-across-batches
— the thing most likely to be wrong — is never exercised.
- Trusting the response order within a batch without asserting it. Assertion 2
must use distinguishable vectors, not a length check.
## Traps
- Hardcoding 768 in three places. Put it in one constant that the schema
migration also references, so a model change is one edit and one migration.
- Assuming no auth is needed because it currently works without a key. That route
is missing its plugin annotation, which is a bug scheduled to be fixed.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts, pgvector
tasks/M2.2-memory-db-manifest.md
+102
View File
@@ -0,0 +1,102 @@
# M2.2 — CNPG `memory-db` + pgvector
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | homelab |
| Spec | inlined below |
| Blocks | — |
## Goal
A Postgres with pgvector, provisioned the way everything else in the cluster is:
through git, with no manual `psql`.
## Facts (inlined — no spec read needed)
**pgvector needs no custom image.** Verified on the running cluster:
```
$ psql -tAc "select name,default_version,installed_version
from pg_available_extensions where name='vector'"
vector|0.7.0|
```
on the stock `ghcr.io/cloudnative-pg/postgresql:16.2`. Available, not yet
installed — `CREATE EXTENSION` is all that is missing.
**The operator is CNPG 1.30.0**, which supports declarative extensions on the
`Database` CRD (`kubectl explain database.spec.extensions` resolves). So the
extension is git-managed too — no manual step, consistent with the GitOps rule
that infrastructure changes flow through version control.
Follow `k8s/infra/databases/temporal-db.yaml` exactly: 3 instances, `imageName`
pinned, `enableSuperuserAccess: false`, `storageClass: longhorn-cnpg`,
`enablePodMonitor: true`, control-plane tolerations, `podAntiAffinityType:
preferred`.
Storage: 10Gi matches the existing clusters. At 768 dims × 4 bytes, a vector is
~3 KB; tens of thousands of nodes is well under a gigabyte, so 10Gi is generous
and consistent rather than tight.
## Steps
1. `k8s/infra/databases/memory-db.yaml` — `Cluster` + `Database` with
`extensions: [{name: vector, ensure: present}]`.
2. Namespace `memory`, created by the ArgoCD app that owns it.
3. Add to the owning kustomization's explicit resource list — an unlisted file is
silently dropped with no error and no drift shown.
4. Commit, push, let ArgoCD sync. **No `kubectl apply`.**
5. Verify the extension installed and the app user can create tables.
6. Record the connection string convention in the repo README; the password comes
from the CNPG-generated secret, never committed.
## Acceptance
- `Cluster` reaches `Cluster in healthy state` with 3 instances.
- `select extversion from pg_extension where extname='vector'` returns a version.
- ArgoCD shows the app `Synced/Healthy`.
- No manual `psql` was run to get there.
## Verify
**Harness:** `kubectl` and `psql` read-only checks after sync.
**Integration test** — `verify/m2.2.sh`, output diffed against `expected/m2.2.txt`:
1. `a1_cluster_healthy` — `kubectl get cluster -n memory memory-db` reports 3/3
ready.
2. `a2_extension_installed` — `select extname, extversion from pg_extension where
extname='vector'` returns one row.
3. `a3_declarative_not_manual` — `kubectl get database -n memory memory-db-vector
-o jsonpath='{.spec.extensions}'` shows the declaration, proving it came from
git.
4. `a4_argocd_synced` — the owning app is `Synced/Healthy`.
5. `a5_app_user_can_ddl` — as `app`, `CREATE TABLE t(v vector(768)); DROP TABLE t;`
succeeds.
6. `a6_hnsw_available` — `CREATE INDEX ... USING hnsw` on that temp table
succeeds, proving 0.7.0 has the index type the schema needs.
**Command:** `bash verify/m2.2.sh | diff - expected/m2.2.txt`
**False pass:**
- Checking `pg_available_extensions` instead of `pg_extension`. Available means
the files are on disk; installed means `CREATE EXTENSION` ran. The whole task
is the second one.
- Verifying after a manual `CREATE EXTENSION`. It passes and proves nothing about
the declarative path, which is the actual deliverable. Assertion 3 is the guard.
## Traps
- Forgetting the kustomization resource list. The file sits in git, ArgoCD reports
Synced, and the objects never exist — silent, and the failure surfaces later as
a connection error.
- Adding `prune: true` semantics without thinking about operator-created children.
CNPG creates Services, Secrets and PVCs owned by the Cluster; if ArgoCD's
tracking label propagates to them, prune fights the operator. The
`llm-serving` app already had to set `prune: false` for exactly this reason.
---
Background: [DESIGN.md](../DESIGN.md) — pgvector · `k8s/infra/databases/temporal-db.yaml`
tasks/M2.3-schema-and-migrations.md
+110
View File
@@ -0,0 +1,110 @@
# M2.3 — Schema + sqlx migrations
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M2.2 |
## Goal
The tables the provenance graph lives in, with the constraints that make a
malformed graph impossible rather than merely unlikely.
## Facts (inlined — no spec read needed)
```sql
CREATE TABLE memory_node (
id BIGSERIAL PRIMARY KEY,
level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')),
project TEXT NOT NULL,
query_id TEXT, -- NULL at L2
run_id TEXT NOT NULL,
t INT NOT NULL,
source TEXT, -- set at L0
text TEXT NOT NULL,
sha256 TEXT NOT NULL UNIQUE, -- content identity, from M0.2
embedding vector(768) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE memory_edge (
child_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
parent_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
PRIMARY KEY (child_sha, parent_sha)
);
CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON memory_node (project, level);
```
`sha256 UNIQUE` is what makes rebuild idempotent — re-inserting identical content
is a conflict to ignore, not a duplicate row. It is also why the hash must exclude
run ids and timestamps (M0.2).
`ON DELETE CASCADE` on both edge columns: dropping a node should not leave
dangling edges. Rebuild drops everything anyway, but a partial cleanup should not
be able to corrupt the graph.
`query_id` is NULL at L2 by design — L2 spans queries. Enforce it:
`CHECK ((level = 'L2') = (query_id IS NULL))`.
Cosine distance, not L2: these are normalised text embeddings and cosine is what
the model was trained for. `vector_cosine_ops` must match the operator the query
uses (`<=>`), or the index is silently ignored and every query is a seq scan.
## Steps
1. `migrations/0001_init.sql` with the above, plus the `query_id`/level CHECK.
2. `CREATE EXTENSION IF NOT EXISTS vector;` first — the declarative extension
(M2.2) should have run, and this makes local dev work too.
3. Wire `sqlx::migrate!()` and run at startup.
4. `sqlx prepare` for offline compile-time query checking in CI.
5. Add a `schema_version` sanity query the repo layer asserts on connect.
6. Document that changing the embedding model is a migration, because the column
width is part of the schema.
## Acceptance
- Migrations apply to a clean database and are idempotent.
- Inserting a duplicate `sha256` conflicts rather than duplicating.
- An `L2` row with a non-null `query_id` is rejected by the CHECK.
- The HNSW index is used by a cosine-distance query.
## Verify
**Harness:** a disposable database — `sqlx::test` or testcontainers with the same
image tag as production, `ghcr.io/cloudnative-pg/postgresql:16.2`.
**Integration test** — `tests/it_schema.rs`:
1. `a1_migrate_clean` — apply to an empty database, assert both tables exist.
2. `a2_migrate_idempotent` — apply twice, assert no error.
3. `a3_sha_unique` — insert the same sha twice, assert a unique violation.
4. `a4_level_check` — `level='L3'` rejected; `level='L2'` with a `query_id`
rejected; `level='L1'` without one rejected.
5. `a5_edge_fk` — an edge referencing a missing sha is rejected.
6. `a6_cascade` — delete a node, assert its edges are gone.
7. `a7_hnsw_is_used` — `EXPLAIN` a `ORDER BY embedding <=> $1 LIMIT 10` query and
assert the plan contains `Index Scan` on the HNSW index, not `Seq Scan`.
**Command:** `cargo test -p mem-store schema`
**False pass:**
- Testing the schema against SQLite or plain Postgres without pgvector. It will
accept `vector(768)` as an unknown type in some configurations and every
vector assertion becomes meaningless. Use the production image.
- Omitting assertion 7. An index created with the wrong opclass exists, reports
healthy, and is never used — queries just get slower as the table grows, which
reads as a scaling problem rather than a wrong index.
## Traps
- `vector_l2_ops` with a `<=>` query, or the reverse. The index is silently
ignored. This is the single most common pgvector mistake.
- Making `query_id` NOT NULL because L1 always has one. L2 then cannot be stored,
and the workaround is a sentinel string that pollutes every group-by.
---
Background: [DESIGN.md](../DESIGN.md) — pgvector schema
tasks/M2.4-pgvector-repo.md
+99
View File
@@ -0,0 +1,99 @@
# M2.4 — pgvector repository
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M2.3, M2.1 |
## Goal
Write the projection into Postgres idempotently, so rebuild is safe to run at any
time and produces the same rows.
## Facts (inlined — no spec read needed)
```rust
async fn upsert_node(&self, node: &MemoryNode, embedding: &[f32]) -> Result<()>;
async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
async fn search(&self, q: &[f32], levels: &[Level], project: &ProjectId, k: usize)
-> Result<Vec<ScoredNode>>;
async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
async fn clear_project(&self, project: &ProjectId) -> Result<()>;
```
`upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
identical node is the same node; re-running rebuild must not duplicate or churn
rows. Same for edges on the composite key.
`search` orders by `embedding <=> $1` — cosine distance, matching the
`vector_cosine_ops` index. Any other operator silently drops to a seq scan.
Edges are inserted **after** both endpoints exist, or the foreign key rejects
them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
optimisation; a single-pass insert fails on the first forward reference.
Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
rebuild — batch across nodes, not per node.
## Steps
1. `PgRepo::connect(url)` with a pool; run migrations on connect.
2. Implement the five methods above.
3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
multi-row statement.
4. Two-pass write: nodes, then edges.
5. `clear_project` deletes nodes for one project; edges cascade.
6. Return `ScoredNode { node, distance }` — keep the raw distance, do not convert
to a similarity score here. The reranker (M3.2) wants the ordering, and a
lossy conversion hides ties.
## Acceptance
- Upserting the same node twice leaves one row.
- Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
- `search` returns nearest-first and respects the level filter.
- `clear_project` removes only that project.
## Verify
**Harness:** disposable Postgres with the production image; a deterministic fake
embedder (hash → fixed vector) so vector assertions are exact.
**Integration test** — `tests/it_pg_repo.rs`:
1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
2. `a2_two_pass_required` — single-pass insert with a forward edge reference
fails; two-pass succeeds. Proves the ordering constraint is real.
3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
order matches hand-computed cosine distance.
4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
L1.
5. `a5_project_isolation` — two projects with identical text; search one, assert
no cross-project results.
6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
orphan edges remain.
7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.
**Command:** `cargo test -p mem-store pg_repo`
**False pass:**
- Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
which removes the only check that the distance operator matches the index.
- Testing `search` with one project in the database. Assertion 5 is the only one
that catches a missing `WHERE project = $1`, and that bug leaks another
project's memory into every answer.
## Traps
- Converting distance to similarity in the repo. It loses precision, and the
reranker downstream wants candidates in order rather than scores.
- Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
would do, and rebuild goes from seconds to minutes.
---
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
tasks/M2.5-obsidian-projector.md
+113
View File
@@ -0,0 +1,113 @@
# M2.5 — Obsidian projector
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |
## Goal
Render the log as a vault a human reads, with the tier graph as the link graph.
## Facts (inlined — no spec read needed)
```
vault/<project>/
index.md L2 synthesis, links every L1 note
<query-id>.md L1, one per standing query
evidence/<source>-<t>.md L0, only with --emit-evidence-notes
```
```markdown
---
project: poimen
level: L1
query_id: infra-root-causes
updated: 2026-08-17
chunks_seen: 412
chunks_used: 17
run_id: 01HXYZ...
---
# Infra root causes — poimen
<final memory text, verbatim>
## Provenance
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer
```
**Deterministic output is the requirement, not a nicety.** M2.8 asserts that
rebuilding produces a byte-identical vault. That means: stable key order in
frontmatter, no timestamp of *generation* (only `updated` derived from the log),
sorted provenance lists, and `\n` line endings.
`updated` comes from the run's timestamp in the log — not `now()`. A generation
timestamp makes every rebuild a diff and destroys the gate.
L0 notes default off: 17 per query is fine, but it grows unbounded across
projects and queries. Citations inline give the same provenance without the file
count.
Wikilinks are `[[<source-note-name>]]`. The link target may not exist as a file
when evidence notes are off — that is fine and normal in Obsidian, and it still
shows in the graph view as an unresolved node.
## Steps
1. `ObsidianProjector::project(log_dir, vault_dir, opts)` in `mem-store`.
2. Read the log; take the final `memory` record per query for L1, and the L2
record for `index.md`.
3. Frontmatter with a fixed key order; `updated` from the log.
4. Provenance section from `parents`, sorted by source then `t`.
5. `--emit-evidence-notes` writes L0 notes; default off.
6. Write with `\n`, no trailing whitespace, exactly one trailing newline.
7. A note whose L1 memory is empty is still written, with a body saying no
evidence was found — an absent file is indistinguishable from a failed run.
## Acceptance
- Two projections of the same log produce byte-identical files.
- Frontmatter key order is stable.
- `updated` reflects the run, not the projection.
- Every L1 note links to its L2 index and vice versa.
## Verify
**Harness:** a committed log fixture and a committed expected vault tree.
**Integration test** — `tests/it_projector.rs`:
1. `a1_byte_identical_twice` — project into two temp dirs, assert every file's
bytes are equal. This is M2.8's core property, tested early.
2. `a2_no_generation_timestamp` — project, sleep 1s, project again, assert equal.
Catches `now()` leaking into output.
3. `a3_frontmatter_key_order` — assert the exact key sequence.
4. `a4_golden_tree` — diff the whole output against `expected/vault/`, empty diff.
5. `a5_empty_memory_still_writes` — a log with zero updates produces a note
saying so.
6. `a6_links_bidirectional` — every L1 note appears in `index.md` and links back.
7. `a7_evidence_notes_flag` — off by default; on, produces one note per L0 node.
8. `a8_line_endings` — no `\r`, exactly one trailing `\n`.
**Command:** `cargo test -p mem-store projector`
**False pass:**
- Comparing files by parsed content rather than bytes. Key reordering and
whitespace churn both pass, and both fail M2.8 later, where the cause is much
harder to find.
- Testing with a single query. Assertion 6 needs at least two L1 notes to catch a
link built from the wrong id.
## Traps
- `updated: {now}`. The most natural thing to write, and it makes every rebuild
dirty, which trains everyone to ignore the diff that M2.8 depends on.
- Serializing frontmatter from a `HashMap`. Iteration order is unspecified and
the output churns between runs on the same input.
---
Background: [DESIGN.md](../DESIGN.md) — Obsidian vault
tasks/M2.6-rebuild-from-log.md
+98
View File
@@ -0,0 +1,98 @@
# M2.6 — `mem rebuild --from-log`
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M2.4, M2.5 |
## Goal
Drop both projections and rebuild them from the log alone — the command that
makes "the log is authoritative" a testable claim instead of a slogan.
## Facts (inlined — no spec read needed)
```
mem rebuild --from-log --project poimen # both projections
mem rebuild --from-log --project poimen --vault-only
mem rebuild --from-log --project poimen --db-only
```
The claim: **anything not reconstructible from the log has a hidden input, and
that is a bug.** Rebuild is the executable form of that claim. If it needs the
existing vault or database to produce correct output, something is being carried
across that is not in the record.
Rebuild does **no model calls except embeddings**. Gate decisions, memory text and
provenance are all in the log already; re-running the controller would produce
different text and defeat the purpose.
Order matters: clear → insert all nodes → insert all edges → project vault. Edges
before nodes violates the foreign key (M2.4).
Embeddings are the expensive part. Cache by `sha256` so a rebuild after a vault
template change does not re-embed unchanged nodes.
## Steps
1. `mem rebuild --from-log --project P`.
2. Read every log file for the project, in run-id order.
3. `clear_project`, then two-pass node/edge insert, batching embeddings.
4. Project the vault (M2.5), overwriting.
5. Embedding cache keyed by sha, on disk under `.cache/`, so it survives runs.
6. Report counts: nodes by level, edges, embeddings computed vs cached.
7. Refuse to run if any log file is incomplete (no `run_end`) unless `--allow-partial`
— rebuilding from a half-run silently produces a half-memory.
## Acceptance
- Rebuild from an empty database and empty vault produces the full state.
- Rebuild twice produces identical database rows and identical vault bytes.
- No controller model calls occur.
- An incomplete log is refused by default.
## Verify
**Harness:** log fixture, disposable Postgres, temp vault. A controller client
that panics if called.
**Integration test** — `tests/it_rebuild.rs`:
1. `a1_from_empty` — drop everything, rebuild, assert node counts per level match
the log's records.
2. `a2_idempotent_db` — rebuild twice, assert row count unchanged and no
`created_at` churn on existing rows.
3. `a3_idempotent_vault` — rebuild twice, assert vault bytes identical.
4. `a4_no_controller_calls` — inject a panicking chat client; assert rebuild
succeeds.
5. `a5_embedding_cache` — second rebuild computes zero embeddings.
6. `a6_edge_order` — a log whose first memory references a later-inserted parent
still rebuilds, proving two-pass.
7. `a7_incomplete_refused` — a log with no `run_end` exits non-zero; with
`--allow-partial` it succeeds.
8. `a8_log_is_sufficient` — delete the vault and the database entirely, rebuild,
and assert the result equals a committed golden. This is the authority claim.
**Command:** `cargo test -p mem-cli rebuild`
**False pass:**
- Rebuilding on top of existing state. It masks every hidden input, because the
missing piece is already there from the previous run. Assertions 1 and 8 must
start from nothing.
- Asserting row counts only. A rebuild that inserts the right number of rows with
wrong `parents` passes; assertion 6 and M2.7's edge closure are what check the
graph.
## Traps
- Re-running the controller during rebuild. It produces different memory text
every time, the vault never stabilises, and M2.8 can never pass.
- Caching embeddings by node id rather than content hash. Ids change between
rebuilds; hashes do not, which is the whole point of content identity.
---
Background: [DESIGN.md](../DESIGN.md) — Authority model
tasks/M2.7-verify-edges.md
+90
View File
@@ -0,0 +1,90 @@
# M2.7 — `mem verify` — edge closure
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M2.6 |
## Goal
Assert the provenance graph is well-formed, so a memory with no traceable
evidence is caught rather than believed.
## Facts (inlined — no spec read needed)
Invariants, checked against the log and the database independently:
1. Every L1 `memory` record has at least one L0 parent. A memory with no evidence
came from somewhere the record does not explain.
2. Every `parents` sha resolves to a node that exists.
3. Every `evidence` sha appears as a parent of at least one memory. Evidence that
nothing cites was written for no reason.
4. `evidence` count equals `gate.update == true` count (also M1.6 a2, re-checked
here across the whole project rather than one run).
5. No edge is self-referential; no cycles.
6. Levels are consistent: an L1 node's parents are L0; an L2 node's are L1.
Invariant 6 is the one that catches a tier confusion, and it is the one most
likely to break when M3.1 adds the L2 pass — an L2 node accidentally parented to
L0 evidence would still look plausible in the vault.
`mem verify` is a read-only diagnostic. It never repairs; repair is `mem rebuild`.
## Steps
1. `mem verify --project P [--db] [--log]`, defaulting to both.
2. Check invariants 1–6, collecting **all** violations rather than failing on the
first — one run should tell you everything wrong.
3. Report per violation: invariant, level, sha, run id, and the log line number.
4. Exit non-zero on any violation.
5. `--format json` for machine consumption.
## Acceptance
- A clean project reports zero violations, exit 0.
- Each invariant has a fixture that violates it and is detected.
- All violations are reported in one run, not just the first.
## Verify
**Harness:** hand-built log fixtures, one per invariant, plus a clean one.
**Integration test** — `tests/it_verify.rs`:
1. `a1_clean_passes` — the good fixture, zero violations, exit 0.
2. `a2_orphan_memory` — L1 with empty `parents`; detected as invariant 1.
3. `a3_dangling_parent` — parent sha not present; invariant 2.
4. `a4_uncited_evidence` — evidence nothing references; invariant 3.
5. `a5_evidence_gate_mismatch` — 3 update-gates but 2 evidence records;
invariant 4.
6. `a6_cycle` — A parents B, B parents A; invariant 5.
7. `a7_level_mismatch` — L2 node parented directly to an L0 node; invariant 6.
8. `a8_reports_all` — a fixture violating three invariants at once; assert all
three appear in one run's output.
9. `a9_db_and_log_agree` — introduce a violation in the database only; assert
`--db` catches it and `--log` does not, proving the two checks are independent.
**Command:** `cargo test -p mem-cli verify`
**False pass:**
- Checking the database only. The log is authoritative; a log-level violation
that rebuild happens to smooth over is still a bug in the writer, and
assertion 9 is what keeps the two checks honest.
- Failing fast on the first violation. It passes every single-violation fixture
and makes assertion 8 impossible, which in practice means three rebuild cycles
to find three problems.
## Traps
- Treating invariant 3 as fatal. Uncited evidence is a real smell, but a
legitimate case exists: the final turn updates memory and the run is cut short
before the memory record flushes. Report it, and let the gate decide severity.
- Skipping invariant 6 because L2 does not exist yet. It is cheap now and it is
precisely what M3.1 will break.
---
Background: [DESIGN.md](../DESIGN.md) — tier model, Verification
tasks/M2.8-m2-gate.md
+100
View File
@@ -0,0 +1,100 @@
# M2.8 — M2 composition gate
| Field | Value |
|---|---|
| Phase | M2 — Projections |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M2 |
## Goal
Prove the authority model: the log is sufficient, and both projections are
genuinely derived.
## Facts (inlined — no spec read needed)
The claim under test — poimen's own principle, applied here:
> Nothing derived is authoritative. If it cannot be dropped and rebuilt
> byte-identically, it has hidden inputs and that is a bug.
The gate is destructive by design: it **deletes** the vault and truncates the
database, rebuilds from the log alone, and diffs. Anything that survives only
because it was already there is a hidden input, and this is the only test that
finds it.
```sh
rm -rf vault/poimen
psql -c "delete from memory_node where project = 'poimen'"
mem rebuild --from-log --project poimen
git -C vault diff --exit-code # empty diff is the only pass
```
`git diff --exit-code` on a tracked vault is the assertion. It compares against
what was committed, so it also catches a projector change that was not intended.
Run it twice: once from empty (sufficiency) and once on top of itself
(idempotence). Both must produce the same bytes.
## Steps
1. `verify/m2.8.sh` performing the destructive rebuild above.
2. Assert the vault diff is empty and the database node counts match the log.
3. Run `mem verify` and assert zero violations.
4. Second rebuild without clearing; assert still empty diff and unchanged row
count.
5. Assert no controller model calls (embeddings are allowed and expected).
6. Commit `expected/m2.8.txt` with the count summary; diff against it.
## Acceptance
- Vault rebuilt from nothing is byte-identical to the committed vault.
- Database node/edge counts match the log's records exactly.
- `mem verify` reports zero violations.
- Second rebuild changes nothing.
## Verify
**Harness:** disposable database, git-tracked vault, real log. Long-running; a
nightly or on-demand job.
**Integration test** — `verify/m2.8.sh`, output diffed against `expected/m2.8.txt`:
1. `a1_vault_from_empty` — delete vault, rebuild, `git diff --exit-code` empty.
2. `a2_db_from_empty` — truncate, rebuild, counts per level equal the log's.
3. `a3_verify_clean` — `mem verify` exits 0.
4. `a4_rebuild_idempotent` — rebuild again, diff still empty, row count unchanged.
5. `a5_no_controller_calls` — assert zero calls to the chat route during rebuild
(count via the record dir from M1.1, or a proxy).
6. `a6_projection_independence` — `--vault-only` then `--db-only` produces the
same end state as a combined rebuild.
7. `a7_log_alone_suffices` — move the log to a fresh checkout with no vault and no
database, rebuild, diff against the committed vault. The strongest form of
the claim.
**Command:** `bash verify/m2.8.sh | diff - expected/m2.8.txt`
**False pass:**
- Running the gate without deleting the vault first. A projector that only writes
changed files produces an empty diff trivially, and the hidden input survives.
- Diffing an untracked vault. `git diff` on untracked files reports nothing, so
the assertion passes vacuously. The vault must be committed, or the script must
compare against a committed golden tree explicitly.
- Allowing controller calls "because it is easier". Rebuild then produces new
memory text each run and the gate can never pass — at which point the usual fix
is to weaken the gate.
## Traps
- Treating a non-empty diff as a projector bug by default. It is equally likely to
be a *hash* bug: if `sha256` includes a timestamp (M0.2), every rebuild produces
new nodes and the vault churns. Check identity before blaming rendering.
- Running against production data with the destructive script and no backup. The
log is the record; if that is intact, everything is recoverable — which is
exactly why the log must be tracked in git before this gate is first run.
---
Background: [DESIGN.md](../DESIGN.md) — Authority model, Verification
tasks/M3.1-l2-synthesis.md
+102
View File
@@ -0,0 +1,102 @@
# M3.1 — L2 synthesis pass
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.5 |
## Goal
Project-level memory across the per-query memories — using the same loop, with
the exit gate switched on.
## Facts (inlined — no spec read needed)
```
mem synthesize --project poimen
```
L2 is **not new machinery**. It is `run_loop` (M1.5) with:
| | L1 | L2 |
|---|---|---|
| input stream | `Chunk` from sources | L1 `MemoryNode`s |
| question | per-query question | `synthesis.question` |
| `use_exit_gate` | false | **true** |
| `query_id` | set | NULL |
| parents | L0 evidence shas | L1 memory shas |
**Why the exit gate flips on.** At L1 the input is hundreds of chunks and the
question is exhaustive ("what are *all* the X"), which is exactly the case paper
§3.3 says to run without the gate. At L2 the input is a handful of memories and
"enough evidence" is genuinely decidable, which is the case the gate was designed
for and where the paper measures its 4× speedup.
If M1.5 was written correctly this task is mostly wiring. If it needs changes to
`run_loop`, the level was not really a parameter — and assertion a10 in M1.5
existed to prevent exactly that.
Ordering: L1 memories enter the stream in a stable order (query id, ascending),
so synthesis is reproducible.
## Steps
1. `mem synthesize --project P` reads the final L1 memory per standing query.
2. Wrap them as the loop's input stream, in sorted query-id order.
3. Run `run_loop` with `level = L2`, `use_exit_gate = true`, the synthesis
question, `query_id = None`.
4. Write to `log/<project>/_synthesis/<run-id>.jsonl`.
5. `parents` on the L2 memory are the L1 memory shas consumed up to that turn.
6. Refuse to run if any standing query has no completed L1 run — synthesizing
over a partial set silently produces a partial picture.
## Acceptance
- No change to `run_loop` is required.
- The exit gate fires and stops early on a real project.
- L2 parents are L1 shas, never L0.
- Sorted input order makes two runs consume memories in the same sequence.
## Verify
**Harness:** scripted client, plus one live run.
**Integration test** — `tests/it_l2.rs`:
1. `a1_reuses_run_loop` — assert `mem synthesize` calls the same `run_loop`
symbol; a duplicated loop is a review failure, and a `#[deny]`-style test here
is a grep asserting `fn run_loop` appears exactly once in the workspace.
2. `a2_exit_gate_on` — scripted `end` at turn 2 of 5; assert it stops at 2.
3. `a3_parents_are_l1` — every L2 parent sha resolves to an L1 node.
4. `a4_query_id_null` — the L2 record has no `query_id`.
5. `a5_stable_input_order` — two runs consume L1 memories in identical order.
6. `a6_refuses_partial` — one query with no completed run; assert non-zero exit
naming the query.
7. `a7_level_check_holds` — run `mem verify`; invariant 6 (level consistency)
passes.
8. `a8_live` — `#[ignore]`; real project, assert an L2 memory is produced and
print whether the exit gate fired and at which turn.
**Command:** `cargo test -p mem-cli l2` (add `-- --ignored` for a8)
**False pass:**
- Copying `run_loop` into an L2-specific function. Everything passes, the two
drift within a month, and the tier model quietly becomes two implementations.
Assertion 1 is the guard.
- Testing the exit gate with a script that never says `end`. The gate's effect is
invisible and a `use_exit_gate` that is ignored passes.
## Traps
- Feeding L0 evidence into L2 "for more detail". It blows the context budget and
breaks the level invariant; L2 reads memories, and if they are inadequate the
fix is at L1.
- Synthesizing over whatever L1 runs happen to exist. A missing query produces a
confident summary of an incomplete project, which is worse than no summary.
---
Background: [DESIGN.md](../DESIGN.md) — tier model · paper §3.3
tasks/M3.2-rerank-client.md
+88
View File
@@ -0,0 +1,88 @@
# M3.2 — Rerank client
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.1 |
## Goal
Reorder vector-recall candidates by actual relevance, because embedding distance
is a coarse filter.
## Facts (inlined — no spec read needed)
```
POST /v1/rerank
{"query":"what is rust","texts":["rust is a language","bananas"]}
-> [{"index":0,"score":0.98176736},{"index":1,"score":0.00008833366}]
```
`BAAI/bge-reranker-base` via TEI. Note the **response shape is a bare array**, not
an OpenAI-style `{"data": [...]}` envelope — this route does not follow the chat
convention.
The discrimination is real: 0.98 vs 0.00009 on that probe, four orders of
magnitude. Embedding cosine on the same pair would be far closer, which is why
recall-then-rerank beats recall alone.
`index` refers to the position in the submitted `texts` array; results come back
**sorted by score**, so the index is the only way to map back. Do not assume
order.
Batch limits apply as with embeddings — keep candidate lists modest (top-50 from
recall is plenty).
## Steps
1. `RerankClient::rerank(query, texts) -> Result<Vec<Scored>>` in `mem-llm`.
2. Parse the bare array; map `index` back to the caller's items.
3. Preserve the caller's item type: take `&[T]`, return `Vec<(T, f32)>` so the
caller does not re-associate by position.
4. Same `apikey` header, retry and timeout policy as M1.1.
5. Empty input returns empty without a request.
## Acceptance
- Results map correctly back to input items via `index`.
- A more relevant text scores above a less relevant one on a live call.
- Empty input makes no request.
## Verify
**Harness:** `wiremock` offline, one `#[ignore]` live test.
**Integration test** — `tests/it_rerank.rs`:
1. `a1_bare_array_parsed` — mock returns `[{"index":1,...},{"index":0,...}]`;
assert parsing succeeds.
2. `a2_index_mapping` — with the out-of-order mock above, assert the returned
items correspond to inputs 1 and 0 respectively, not 0 and 1.
3. `a3_empty_no_request` — empty texts; assert zero requests.
4. `a4_apikey_sent` — header present.
5. `a5_live_discriminates` — `#[ignore]`; real gateway, query "what is rust"
against `["rust is a language","bananas"]`; assert the first scores at least
10× the second.
**Command:** `cargo test -p mem-llm rerank` (add `-- --ignored` for a5)
**False pass:**
- Assuming the response preserves input order. A mock that returns results in
input order passes a naive test and the live endpoint returns them sorted,
which silently mislabels every result. Assertion 2 must use an out-of-order
mock.
- Expecting an OpenAI envelope. It will fail immediately against the live route,
but a mock written to match the wrong shape hides that until integration.
## Traps
- Re-associating results by position instead of by `index`. The scores are right
and attached to the wrong documents — a bug that looks like poor retrieval
quality rather than a mapping error.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts, retrieval
tasks/M3.3-mem-query.md
+97
View File
@@ -0,0 +1,97 @@
# M3.3 — `mem query` with provenance
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.2, M2.4 |
## Goal
Ask the memory a question and get an answer that can be traced back to the
session it came from.
## Facts (inlined — no spec read needed)
```
mem query "why did requests over 10KB fail?"
mem query --project poimen --levels L1,L2 --k 10 "..."
```
Pipeline: embed the question (M2.1) → HNSW recall top-k over `memory_node`
filtered by project and level (M2.4) → rerank the candidates (M3.2) → return with
provenance walked down through `memory_edge`.
Default levels are **L1 and L2**, not L0. L1/L2 are synthesized answers; L0 is raw
evidence and returning it by default buries the answer in transcript. `--levels
L0` exists for "show me the actual source".
Provenance walk: for each returned node, follow `memory_edge` to its parents and
report source and turn. One hop for L1 (to evidence), two for L2 (through L1).
Recall wide, rerank narrow: take 50 from HNSW, rerank, return 5. Cosine distance
alone puts "bananas" close enough to matter; the reranker separated the same pair
by four orders of magnitude.
Output is human-readable by default, `--format json` for programmatic use.
## Steps
1. `mem query [--project P] [--levels L] [--k N] <question>`.
2. Embed, recall 10×k, rerank, truncate to k.
3. Walk edges to build a provenance list per hit; deduplicate sources.
4. Render: score, level, query id, the memory text, then provenance lines.
5. Project defaults to the one inferred from `$PWD`; `--project` overrides. No
project and no match is an error, not an empty result over everything.
6. `--explain` prints the recall candidates and their pre-rerank distances, for
debugging retrieval quality.
## Acceptance
- A question with a known answer returns the right L1 note first.
- Every hit carries at least one resolvable provenance entry.
- `--levels L0` returns evidence; the default does not.
- Reranking changes the order versus raw recall on at least one real query.
## Verify
**Harness:** a seeded database from a real log, deterministic fake embedder for
the offline assertions, live for quality.
**Integration test** — `tests/it_query.rs`:
1. `a1_known_answer` — seeded with the poimen log, query "why did requests over
10KB fail?"; assert the top hit is the `infra-root-causes` L1 node.
2. `a2_provenance_resolves` — every hit's provenance shas exist in
`memory_node`.
3. `a3_default_excludes_l0` — assert no L0 nodes in default output.
4. `a4_levels_flag` — `--levels L0` returns evidence nodes.
5. `a5_rerank_reorders` — capture pre- and post-rerank order; assert they differ
on at least one fixture query, proving the reranker is wired and not a no-op.
6. `a6_project_isolation` — two projects seeded; assert no cross-project hits.
7. `a7_no_project_errors` — unresolvable project exits non-zero.
8. `a8_l2_two_hop_provenance` — an L2 hit's provenance resolves through L1 to L0
sources.
**Command:** `cargo test -p mem-cli query`
**False pass:**
- Asserting only that results are returned. A pipeline where the reranker returns
the input unchanged still returns results, and assertion 5 is the only check
that it is doing anything.
- Testing provenance existence without resolving it. A hit carrying parent shas
that do not exist in the table looks fine in the output and is useless.
## Traps
- Returning L0 by default. The answer is there but buried in raw transcript, and
the tool reads as low quality when it is a display default.
- Recalling exactly k then reranking. Reranking cannot recover a relevant
document that recall never returned; the width of the recall is what determines
the ceiling.
---
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
tasks/M3.4-m3-gate.md
+93
View File
@@ -0,0 +1,93 @@
# M3.4 — M3 composition gate
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M3 |
## Goal
Prove the whole stack answers a real question with real provenance — the first
point at which the system is useful rather than merely correct.
## Facts (inlined — no spec read needed)
The gate is a small set of **known-answer questions** with hand-written expected
sources, committed to the repo. Each names a fact that genuinely appears in the
ingested sessions and the session it appears in.
Seed set, all drawn from real work in this corpus:
| question | expected to cite |
|---|---|
| why did requests over 10KB fail? | the Kong body-buffer / `client_body_buffer_size` finding |
| why did `Authorization: Bearer` return 401? | the Kong key-auth header finding |
| what causes the 504 on a cold ornith start? | the timeouts-on-Ingress-vs-Service finding |
This is a retrieval quality gate, so it is graded, not boolean: report
**hit rate at k=5** and **provenance precision** (fraction of cited sources that
actually contain the fact). A single failing question is information, not
necessarily a stop.
The gate also re-runs `mem verify` including the level invariant, because M3.1 is
the change most likely to break it.
## Steps
1. `verify/known-answers.yaml` — question, expected node, expected source.
2. `verify/m3.4.sh` runs each through `mem query --format json`.
3. Compute hit rate at 5 and provenance precision; print both.
4. Assert the thresholds below.
5. Run `mem verify`; assert zero violations.
6. Assert L2 exists and its provenance resolves two hops.
7. Commit `expected/m3.4.txt`; diff.
## Acceptance
- Hit rate at k=5 ≥ 0.8 on the known-answer set.
- Provenance precision ≥ 0.9 — a citation that does not contain the fact is worse
than no citation.
- `mem verify` clean, including level consistency.
- Every L2 hit resolves to L0 sources.
## Verify
**Harness:** live gateway, seeded database from real logs. On-demand, not
per-push.
**Integration test** — `verify/m3.4.sh` diffed against `expected/m3.4.txt`:
1. `a1_hit_rate` — ≥ 0.8, print actual.
2. `a2_provenance_precision` — for each cited source, fetch the L0 text and
assert it contains the expected fact substring; ≥ 0.9.
3. `a3_verify_clean` — zero violations.
4. `a4_l2_two_hop` — every L2 hit resolves through L1 to a real source.
5. `a5_rerank_contributes` — hit rate with reranking is ≥ hit rate without. If
reranking makes it worse, the wiring is wrong (probably index mapping, M3.2).
6. `a6_no_cross_project` — a question about another project returns nothing from
this one.
**Command:** `bash verify/m3.4.sh | diff - expected/m3.4.txt`
**False pass:**
- Writing the known-answer set after seeing what the system returns. It then
measures nothing. Write the questions and expected sources from the sessions
first, independently of any query output.
- Measuring hit rate without provenance precision. A system that returns the
right note with fabricated citations scores 1.0 on hits and is untrustworthy —
assertion 2 is the one that matters for whether anyone can act on an answer.
## Traps
- Tuning `k` until the hit rate passes. k=50 will hit almost everything and the
metric stops meaning anything; the gate specifies k=5 for that reason.
- Accepting a5 failing as "reranker is just not helping". It far more often means
the `index` mapping in M3.2 is wrong and scores are attached to the wrong
documents.
---
Background: [DESIGN.md](../DESIGN.md) — Verification
tasks/M4.1-skill-draft.md
+114
View File
@@ -0,0 +1,114 @@
# M4.1 — `mem skill draft`
| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.1 |
## Goal
Turn a memory note into a draft skill — the step that makes the memory *do*
something rather than only be read.
## Facts (inlined — no spec read needed)
```
mem skill draft --from poimen/infra-root-causes
-> vault/skills/_drafts/poimen-infra-root-causes/SKILL.md
```
**A skill is a projection, not a level.** L0/L1/L2 are descriptive — what
happened. A skill is procedural — what to do next time. The gated loop does not
produce it: "does this chunk contain evidence for Q" has no meaning when the
output is an instruction.
Format is free, because `SKILL.md` is YAML frontmatter plus markdown, which is
exactly an Obsidian note. Verified against a real installed skill:
```yaml
---
name: keyword-research
description: 'Use when the user asks to "find keywords"... Not for X — use Y.'
when_to_use: "Use when starting keyword research for a new page..."
argument-hint: "<topic or seed keyword> [market/language]"
---
```
So the same file is a vault note and a loadable skill, with no conversion.
**`description` is the whole game.** It is the trigger — a skill whose
description does not match how the user actually phrases the request never fires,
no matter how good the body is. Note the real example above spends half its
description on *negative* routing ("Not for X — use Y").
Follow the existing rubric rather than inventing one: the installed
`grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and
a four-dimension rubric — conciseness, actionability, workflow clarity,
progressive disclosure.
**Drafts land in `_drafts/` and are never auto-loaded.** A directory, not a
frontmatter flag, because a directory cannot be accidentally globbed into
`--skill`.
## Steps
1. `mem skill draft --from <project>/<query-id>` reads the L1 or L2 note.
2. Prompt the model to convert descriptive memory into procedural instruction,
with the rubric's four dimensions in the prompt.
3. Emit frontmatter: `name`, `description`, `when_to_use`, plus
`generated_from: <sha>` and `generated_at`.
4. Write to `vault/skills/_drafts/<project>-<query-id>/SKILL.md`.
5. Refuse to write outside `_drafts/`. Promotion is a human `git mv`.
6. `--dry-run` prints without writing.
## Acceptance
- Output parses as valid frontmatter + markdown.
- `generated_from` resolves to a real node sha.
- The file lands in `_drafts/` and nowhere else.
- Promotion is not automated anywhere in the codebase.
## Verify
**Harness:** seeded vault and log; scripted model client for determinism.
**Integration test** — `tests/it_skill_draft.rs`:
1. `a1_valid_frontmatter` — parse the output; assert `name`, `description`,
`when_to_use`, `generated_from` present and non-empty.
2. `a2_generated_from_resolves` — the sha exists in `memory_node`.
3. `a3_writes_only_to_drafts` — assert the path contains `_drafts/`; attempt to
pass a path outside it and assert refusal.
4. `a4_no_promotion_path` — grep the workspace for any code writing to
`vault/skills/` that is not under `_drafts/`; assert none. Promotion must be
manual.
5. `a5_description_is_trigger_shaped` — assert `description` contains at least one
phrasing cue (a quoted user phrase or "Use when"), matching the installed
examples.
6. `a6_dry_run_writes_nothing` — assert no file created.
7. `a7_idempotent` — same input twice produces identical bytes apart from
`generated_at`.
**Command:** `cargo test -p mem-cli skill_draft`
**False pass:**
- Asserting the file was written without asserting *where*. The entire safety
property of this task is the location, and a draft written to `vault/skills/`
is immediately loadable.
- Accepting any non-empty `description`. A one-line restatement of the title
never triggers, so the feature appears to work and the skill never fires —
assertion 5 is a weak but real guard.
## Traps
- Auto-promoting "when the draft looks good". That closes the loop this design
deliberately leaves open, and there is no external verifier inside it.
- Generating the body from L0 evidence. Skills are procedure distilled from
synthesis; raw transcript produces a narrative, not an instruction.
---
Background: [DESIGN.md](../DESIGN.md) — Skills, the procedural projection
tasks/M4.2-derived-filter.md
+109
View File
@@ -0,0 +1,109 @@
# M4.2 — `derived: true` ingest filter
| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M4.1, M0.5 |
## Goal
Stop the system learning from its own output.
## Facts (inlined — no spec read needed)
The cycle, and it is the only one in the design:
```
emitted skill is loaded into a session
│
▼
appears verbatim in that session's transcript
│
▼
transcript is ingested as evidence
│
▼
reinforces the memory that produced the skill
```
No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
what actually stops it.
Mechanism: every emitted artifact records its content hash in a manifest. During
ingest, a record whose normalised text matches a known artifact is tagged
`derived: true` and **excluded from evidence** — it is still recorded in the log
so the exclusion is visible and auditable, but the gate never sees it.
Matching must survive the model reformatting the text slightly. Exact hash on the
whole record is too brittle: a skill quoted with different indentation would slip
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
threshold.
Threshold is a tradeoff and should be logged, not hidden: too low excludes
genuine discussion *about* a skill, too high lets the cycle run.
## Steps
1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
`{name, sha256, shingles, emitted_at}`.
2. `mem skill draft` appends to it.
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
5. Log a `derived_excluded` event with the record's provenance and the artifact
it matched, so exclusions are auditable and a false positive is findable.
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.
## Acceptance
- A record quoting an emitted skill verbatim is excluded.
- A record quoting it with different whitespace and fences is also excluded.
- A record merely *mentioning* the skill by name is not excluded.
- Every exclusion is logged with what it matched.
## Verify
**Harness:** a fixture manifest with one artifact, plus records in several
paraphrase grades.
**Integration test** — `tests/it_derived_filter.rs`:
1. `a1_verbatim_excluded` — exact copy of an artifact is excluded.
2. `a2_reformatted_excluded` — same content, different indentation and code
fences; excluded.
3. `a3_mention_not_excluded` — "I used the infra-root-causes skill" is kept. This
is the false-positive guard, and over-filtering silently starves the memory.
4. `a4_unrelated_not_excluded` — random session text is kept.
5. `a5_exclusion_logged` — assert a `derived_excluded` event naming the matched
artifact.
6. `a6_verify_catches_leak` — insert an L0 node matching an artifact directly into
the database; assert `mem verify --derived-filter` fails.
7. `a7_threshold_configurable` — assert the threshold is read from config and
appears in the run record.
8. `a8_no_manifest_is_safe` — with no manifest file, ingest proceeds and filters
nothing, rather than failing.
**Command:** `cargo test -p mem-ingest derived_filter`
**False pass:**
- Testing only the verbatim case. Exact-match filtering passes and the realistic
case — a model that reformats what it quotes — walks straight through.
Assertion 2 is the one that matters.
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
of a topic once a skill about it exists, which quietly makes the memory worse
the more skills you write.
## Traps
- Filtering on record *hash*. One character of whitespace defeats it, and the
cycle runs while the filter reports itself working.
- Silent exclusion. Without assertion 5's log event, a false positive is
invisible — memory just gets thinner and nobody knows why.
---
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
tasks/M4.3-m4-gate.md
+87
View File
@@ -0,0 +1,87 @@
# M4.3 — M4 composition gate
| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M4 |
## Goal
Prove the loop stays open — that a generated skill cannot silently become
training data for the memory that generated it.
## Facts (inlined — no spec read needed)
Two properties, and both must hold. Either one alone is insufficient:
1. **A draft is not loadable.** It lives in `_drafts/`, and pointing a real agent
at `vault/skills/` must not surface it.
2. **A promoted skill's text never enters evidence.** Even after a human promotes
it and it appears in a session, the derived filter keeps it out of L0.
The gate exercises the full cycle deliberately: draft a skill, promote it, run a
session that loads it, ingest that session, and assert the skill's content did
not become evidence.
Property 1 is tested with the real loader, not by inspecting paths — the
question is what an agent actually sees.
## Steps
1. Draft a skill from a real note.
2. `pi --skill vault/skills/ --list-skills` (or the equivalent enumeration);
assert the draft is absent.
3. Promote it with `git mv` out of `_drafts/`; assert it now appears.
4. Synthesize a session transcript that quotes the promoted skill, in three
grades: verbatim, reformatted, and merely referenced.
5. Ingest it; assert the first two are excluded and the third is kept.
6. Run `mem verify --derived-filter`; assert clean.
7. Commit `expected/m4.3.txt`; diff.
## Acceptance
- Draft absent from a real loader's skill enumeration.
- Promoted skill present.
- Verbatim and reformatted quotes excluded from evidence; a bare mention kept.
- `mem verify --derived-filter` reports zero leaks.
## Verify
**Harness:** the real `pi` binary for enumeration, seeded vault and log.
**Integration test** — `verify/m4.3.sh` diffed against `expected/m4.3.txt`:
1. `a1_draft_not_loadable` — enumerate skills; assert the draft name is absent.
2. `a2_promoted_is_loadable` — after `git mv`, assert present. Proves assertion 1
is not passing because the loader is broken.
3. `a3_verbatim_excluded` — ingest, assert no L0 node matches.
4. `a4_reformatted_excluded` — same.
5. `a5_mention_kept` — the referencing record survives as evidence.
6. `a6_verify_clean` — `mem verify --derived-filter` exits 0.
7. `a7_exclusions_auditable` — assert `derived_excluded` events name the artifact.
**Command:** `bash verify/m4.3.sh | diff - expected/m4.3.txt`
**False pass:**
- Asserting the draft is absent without assertion 2. If the loader silently fails
to enumerate anything, assertion 1 passes trivially and the guarantee is
untested.
- Testing exclusion only on the draft. The draft is not the risk — a *promoted*
skill is the one that actually reaches sessions, and it is the one the filter
must catch.
## Traps
- Automating promotion inside the gate script and leaving it there. The gate
needs to promote something to test the promoted path; make it obvious that the
script is the only place it happens, and that production has no such path.
- Reading a passing gate as "the cycle is impossible". It means the two guards
hold today. Adding a new emitted artifact type without adding it to the
manifest reopens the loop, and only assertion 6 will notice.
---
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
tasks/M5.1-evidence-labeler.md
+108
View File
@@ -0,0 +1,108 @@
# M5.1 — `mem label` — evidence labeler
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |
## Goal
Produce the per-chunk ground truth `U_t` that `r_update` needs, since this corpus
does not come with evidence labels.
## Facts (inlined — no spec read needed)
Paper `r_update`:
```
r_update_t = +1 if U_t is correct
-1 if U_t is incorrect
```
"Correct" means: for chunks containing evidence for `Q`, the agent should emit
`<check>yes</check>`; for chunks without, `<check>no</check>`. That requires
knowing which chunks contain evidence.
The paper had it for free — synthetic NIAH tasks place the needle deliberately,
and HotpotQA ships supporting facts. **We have neither.** Agent transcripts have
no annotation of which turn contained the answer.
Cheapest honest substitute: **distant supervision from the 32B model.** Ask
`reasoning` (DeepSeek-R1-Distill-Qwen-32B, vLLM) per `(question, chunk)` whether
the chunk contains evidence. It is ~10× the controller's size and sees each chunk
independently, without the memory state that might bias the 3B's decision.
This inherits the labeler's bias, which is why M5.2 exists and must run before
anyone trains on these labels.
Constraint: `reasoning` has a **16384 total context** and vLLM rejects
`input + max_tokens > 16384`. A 5000-token chunk plus question plus instructions
fits with room; keep `max_tokens` small (labels are one token of signal) and do
not batch chunks into one prompt.
The labeler emits a binary label plus a short justification. Keep the
justification — it is what makes M5.2's disagreement analysis possible.
## Steps
1. `mem label --project P --query Q` reads chunks from the log.
2. Per chunk, prompt `reasoning`: question, chunk, "does this contain evidence for
the question? Answer yes or no, then one sentence why."
3. Send **no tools** — the reasoning route rejects any request carrying them.
4. Write `label/<project>/<query-id>.jsonl`:
`{"chunk_sha":"...","t":7,"label":true,"why":"...","model":"reasoning","ts":"..."}`.
5. Resumable: skip chunks already labelled.
6. Report the label rate — the fraction of chunks the labeler calls evidence.
Compare it to the controller's update-rate from M1.7; a large gap is the
finding, not a bug.
## Acceptance
- Every chunk in the log gets exactly one label.
- Labels key on `chunk_sha`, so they survive re-chunking only if content is
unchanged.
- Resume skips completed work.
- Label rate is reported alongside the controller's update-rate.
## Verify
**Harness:** scripted client offline; one `#[ignore]` live run.
**Integration test** — `tests/it_label.rs`:
1. `a1_one_label_per_chunk` — no duplicates, no gaps against the log's chunks.
2. `a2_keyed_by_sha` — labels reference `chunk_sha`, not `t`, so reordering the
log does not corrupt them.
3. `a3_no_tools_sent` — assert the request body has no `tools` key.
4. `a4_context_budget` — assert every labeling prompt is under
16384 − max_tokens.
5. `a5_resume` — label, rerun, assert zero new calls.
6. `a6_justification_kept` — every label has non-empty `why`.
7. `a7_rate_reported` — the summary prints both label rate and the controller's
update-rate.
8. `a8_live` — `#[ignore]`; 20 real chunks through `reasoning`; print the labels
and justifications for a human to sanity-check.
**Command:** `cargo test -p mem-cli label` (add `-- --ignored` for a8)
**False pass:**
- Keying labels by `t`. A re-chunk shifts every `t`, the labels silently
misalign, and the training set is quietly wrong in a way nothing downstream can
detect.
- Dropping the justification to save space. M5.2 then has nothing to analyse and
the calibration step degenerates into a single agreement number with no way to
understand it.
## Traps
- Batching several chunks into one labeling prompt to save calls. The labels
become order-dependent and the 16K context is exceeded on the third chunk.
- Treating the 32B's labels as ground truth. They are a *proxy*, and M5.2 is the
task that measures how good a proxy.
---
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.1
tasks/M5.2-labeler-calibration.md
+100
View File
@@ -0,0 +1,100 @@
# M5.2 — Labeler calibration
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M5.1 |
## Goal
Measure how good the proxy is, before training a policy to imitate it.
## Facts (inlined — no spec read needed)
M5.1's labels are distant supervision. Training on them without measuring
agreement means the policy learns the 32B model's bias and reports it as
improvement — and under a single-model deployment there is no competing variant
whose divergence would make that visible.
Method: hand-label a stratified holdout, compare, and report more than accuracy.
- **Sample size**: 100 chunks is enough to distinguish 0.7 from 0.9 agreement.
- **Stratify** by the labeler's own answer — 50 it called evidence, 50 it did
not. Random sampling from a corpus where ~5% is evidence gives ~5 positives,
and the positive class is the one that matters.
- **Report Cohen's κ, not raw agreement.** With a 95/5 class balance, a labeler
that always says "no" scores 95% agreement and is useless. κ corrects for
chance.
- Also report precision and recall on the positive class separately. They fail
differently: low recall silently starves memory, low precision pollutes it.
Disagreements are the artifact. Read them; they are usually either a genuinely
ambiguous chunk or a question that is too vague — and the second is fixable at
M1.2 and worth much more than a better labeler.
## Steps
1. `mem label sample --project P --n 100 --stratified` writes a blind
worksheet — chunk text and question, **no** labeler answer visible.
2. Hand-label it. Record the human labels separately.
3. `mem label calibrate` joins them; reports agreement, κ, precision, recall,
and the confusion matrix.
4. Dump all disagreements with both justifications side by side.
5. Commit the holdout and the human labels; they are reusable for every future
labeler change.
6. Gate: κ ≥ 0.6 before the labels are used for training.
## Acceptance
- Worksheet hides the labeler's answer.
- κ, precision, recall and the confusion matrix are all reported.
- Disagreements are dumped with justifications.
- The holdout is committed and reusable.
## Verify
**Harness:** a synthetic labeled set with known agreement, so the statistics
themselves are testable.
**Integration test** — `tests/it_calibration.rs`:
1. `a1_worksheet_is_blind` — assert the labeler's answer appears nowhere in the
output file.
2. `a2_stratified` — assert the sample is ~50/50 by labeler answer, not corpus
proportional.
3. `a3_kappa_correct` — feed a set with hand-computed κ; assert the reported
value matches to 3 decimals.
4. `a4_kappa_vs_accuracy` — a synthetic all-negative labeler on a 95/5 set:
assert accuracy > 0.9 **and** κ ≈ 0. This is the assertion that justifies
reporting κ at all.
5. `a5_confusion_matrix` — all four cells match hand counts.
6. `a6_disagreements_dumped` — count equals off-diagonal total; each carries both
justifications.
7. `a7_holdout_stable` — rerunning the sampler with the same seed reproduces the
same chunks.
**Command:** `cargo test -p mem-cli calibration`
**False pass:**
- Reporting accuracy only. On this class balance it is nearly meaningless, and it
will look excellent right up until the trained policy learns to always answer
"no".
- Hand-labeling with the labeler's answer visible. Anchoring makes agreement look
high and the whole exercise decorative — assertion 1 is a real safeguard, not
hygiene.
## Traps
- Sampling proportionally. A 5%-positive corpus yields five positives in a
hundred, and precision on the positive class — the number that decides whether
memory gets polluted — is estimated from five examples.
- Treating low κ as "the labeler needs a better prompt". Check the *questions*
first: an ambiguous standing question makes evidence genuinely undecidable, and
no labeler can fix that.
---
Background: [DESIGN.md](../DESIGN.md) — Risks
tasks/M5.3-training-corpus-export.md
+114
View File
@@ -0,0 +1,114 @@
# M5.3 — Training corpus export
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M5.1 |
## Goal
Turn the log plus labels into trajectories verl can train on — the boundary
between the Rust side and the Python side.
## Facts (inlined — no spec read needed)
**The JSONL log is the boundary.** Rust produces it; Python consumes it. Nothing
else crosses, which is what keeps the two halves independent.
A training example is a **trajectory**, not a turn: the paper's advantage mixes a
trajectory-level term with a turn-level one (`Â = α·Â_traj + (1−α)·Â_turn`,
α=0.9), so turns must stay grouped by run.
Per turn, verl needs: the exact prompt sent, the exact response, and the rewards.
```jsonl
{"trajectory_id":"01HXYZ","turns":[
{"t":1,"prompt":"<full assembled prompt>","response":"<think>...</next>",
"r_update":-1,"parsed":true},
...],
"r_exit":-0.5,"r_format":1,"r_outcome":null}
```
Reward assembly, from the paper:
- `r_update_t` = +1 if the recorded `U_t` matches M5.1's label, −1 otherwise.
- `r_exit` — one value per trajectory: `0` if `t_exit == t_last_evidence`,
`−0.75` if earlier, `−0.5` if later. `t_last_evidence` is the largest `t` whose
label is true. **Note L1 runs never exit** (exit gate off), so every L1
trajectory is a "late" exit at −0.5 unless the exit signal is taken from the
recorded `E_t` rather than the loop's behaviour — take it from the record.
- `r_format` = 1 only if **every** turn in the trajectory parsed, 0 otherwise.
Strict, because a malformed turn may be caused by the previous one.
- `r_outcome` is null. We have no answer-correctness signal; the paper's
`is_equiv(A, Â)` has no analogue in extraction. Say so explicitly rather than
fabricating one.
The prompt must be the **exact bytes sent**, not re-assembled. Re-assembly drifts
from what the model actually saw the moment M1.3's template changes.
## Steps
1. `mem export --project P --format verl --out corpus/`.
2. Join log turns to labels by `chunk_sha`.
3. Reconstruct each turn's prompt from the recorded request if `MEM_LLM_RECORD`
captured it; otherwise fail loudly rather than re-assembling.
4. Compute rewards as above; carry `r_outcome: null` through.
5. Group by run into trajectories, ordered by `t`.
6. Emit a summary: trajectories, turns, positive/negative `r_update` split,
`r_format` pass rate, `t_last_evidence` distribution.
7. Refuse to export if M5.2's κ is below the threshold or absent.
## Acceptance
- Turns are grouped into trajectories, ordered.
- Prompts are byte-exact recordings, never re-assembled.
- `r_format` is 0 for a trajectory with any unparsed turn.
- Export is refused without calibration.
## Verify
**Harness:** log fixture with known labels and a deliberately unparsed turn.
**Integration test** — `tests/it_export.rs`:
1. `a1_trajectory_grouping` — turns grouped by run, `t` ascending, none lost.
2. `a2_r_update_signs` — matching label → +1, mismatching → −1, checked per turn.
3. `a3_r_format_strict` — a trajectory with one unparsed turn scores 0 overall,
not per turn.
4. `a4_r_exit_from_record` — assert `r_exit` derives from the recorded `E_t`, not
from whether the loop stopped. A fixture where the gate said `end` at t=5 but
the loop continued must score as an exit at 5.
5. `a5_prompt_is_recorded_bytes` — assert the exported prompt equals the recorded
request body; corrupt the recording and assert export fails rather than
silently re-assembling.
6. `a6_r_outcome_null` — assert the field is present and null, not omitted and not
zero.
7. `a7_refuses_without_calibration` — no κ file → non-zero exit naming M5.2.
8. `a8_summary_counts` — reported splits match a hand count on the fixture.
**Command:** `cargo test -p mem-cli export`
**False pass:**
- Re-assembling prompts at export time. Every assertion except 5 passes, and the
policy is trained on prompts the model never saw — which shows up as a training
run that will not converge, with no obvious cause.
- Applying `r_format` per turn. It looks more granular and it is wrong: the paper
is explicit that the strictness exists because a bad turn may be caused by the
previous one.
- Emitting `r_outcome: 0` instead of null. Zero is a real reward value and the
trainer will use it as signal.
## Traps
- Deriving `r_exit` from loop behaviour at L1. The gate is switched off there, so
every trajectory scores as a late exit and the exit signal becomes constant
noise the policy cannot learn from.
- Joining labels by `t`. Same failure as M5.1's trap — a re-chunk misaligns
everything silently.
---
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2
tasks/M5.4-vllm-lora-serving.md
+102
View File
@@ -0,0 +1,102 @@
# M5.4 — vLLM + `--enable-lora`
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | L — 3+ days |
| Status | ⬜ Not started |
| Flags | homelab |
| Spec | inlined below |
| Blocks | — |
## Goal
A serving path that can load the memory adapter — because the current one cannot.
## Facts (inlined — no spec read needed)
**Ollama cannot hot-swap LoRA adapters.** The controller runs on
`qwen2.5:3b-instruct` served by Ollama today, which is fine for prompted-only use
and a dead end for post-training. vLLM supports `--enable-lora` with
`--lora-modules name=path`, serving base plus adapters from one resident model.
The pattern already exists in this cluster: the `reasoning` predictor is
`vllm/vllm-openai:v0.11.0` under KServe, adopted into ArgoCD as
`k8s/apps/llm-serving/reasoning.yaml`. Copy its shape.
**VRAM is the constraint that makes an adapter the right answer.** One GPU,
`OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A
separate full memory model evicts something, and eviction is a weights reload
measured in tens of seconds — `ornith`'s cold start already blew a 60s gateway
timeout once. A LoRA rides on a resident base for near-zero extra VRAM.
Two hard-won operational facts to carry over:
1. **Kong reads timeouts from the Service, not the Ingress.** `konghq.com/read-timeout`
on an Ingress is ignored; it must reach the predictor Service, and KServe
propagates InferenceService annotations there. Getting this wrong produces a
504 at exactly 60s on the first cold request.
2. **Readiness must mean "can serve", not "process is up."** vLLM's startup probe
needs a long `failureThreshold` — model load plus torch compile measured 108s
+ 55s on the 32B. Report ready too early and the first request 504s.
## Steps
1. `k8s/apps/llm-serving/memory.yaml` — InferenceService, vLLM, Qwen2.5-3B-Instruct.
2. Args: `--enable-lora`, `--max-lora-rank 32`, `--max-model-len 32768`,
`--served-model-name memory`.
3. Adapter storage: a PVC or an initContainer fetching from object storage;
`--lora-modules memory-v1=/mnt/adapters/memory-v1`.
4. Kong timeout annotations on the **InferenceService** metadata so KServe
propagates them to the Service.
5. Startup probe with a generous `failureThreshold`, gated on the OpenAI
`/health` endpoint.
6. New Kong route `/v1/memory/chat/completions` in `llm-routes.yaml`, with the
`model-key-auth` plugin — in namespace `llm-serving`, since a KongPlugin
reference resolves in the annotated object's own namespace and a dangling one
fails open.
7. Commit, push, let ArgoCD sync. No `kubectl apply`.
## Acceptance
- Base model answers through `/v1/memory/chat/completions`.
- A named adapter is selectable via the `model` field.
- Unauthenticated requests to the new route return 401.
- Cold start does not 504.
## Verify
**Harness:** `kubectl` and `curl` against the live gateway after sync.
**Integration test** — `verify/m5.4.sh` diffed against `expected/m5.4.txt`:
1. `a1_isvc_ready` — InferenceService reports Ready.
2. `a2_base_completion` — a completion naming the base model returns 200.
3. `a3_adapter_selectable` — with a dummy adapter mounted, request `model:
memory-v1`; assert 200 and that it differs from the base response.
4. `a4_auth_enforced` — no key → 401; `apikey` header → 200.
5. `a5_cold_start_no_504` — delete the pod, wait for Ready, immediately send a
request; assert 200, not 504. This is the regression test for the timeout bug.
6. `a6_timeouts_on_service` — assert `konghq.com/read-timeout` is present on the
predictor **Service**, not only the Ingress.
7. `a7_vram_headroom` — with the memory model resident alongside the others,
assert `nvidia-smi` free memory stays above a threshold.
**Command:** `bash verify/m5.4.sh | diff - expected/m5.4.txt`
**False pass:**
- Testing after the model is warm. The 504 bug only appears on the first request
after a restart, which is exactly the case assertion 5 forces.
- Checking the timeout annotation on the Ingress. It is ignored there, and its
presence is what makes the bug hard to see.
## Traps
- Adding a third resident model without checking VRAM. Something gets evicted,
and the symptom is a slow unrelated model rather than an obvious failure.
- Putting the KongPlugin in the wrong namespace. It fails **open** — the route
serves unauthenticated and looks healthy, which is how the model routes ran
with no auth for a while.
---
Background: [DESIGN.md](../DESIGN.md) — Separate weights · `k8s/apps/llm-serving/`
tasks/M5.5-verl-training-loop.md
+120
View File
@@ -0,0 +1,120 @@
# M5.5 — verl training loop
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | L — 3+ days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M5.3, M5.4 |
## Goal
Train the LoRA that makes the update and exit gates better than prompting alone.
## Facts (inlined — no spec read needed)
Python, using verl (`github.com/volcengine/verl`), which is what the paper used.
Lives outside the Rust workspace; the JSONL corpus is the only interface.
Paper hyperparameters, Table 3 — start here rather than guessing:
```
chunk size 5000
max prompt length 8192
max response length 2048
clip ratio 0.20
learning rate 1e-6
temperature (train) 1.0 top_p 1.0
temperature (val) 1.0 top_p 0.7
train batch size 128
rollout N 16
mini batch size 128
LR warmup steps 20
```
Advantage, paper §3.2.2 — the part most likely to be implemented wrong:
```
Â_traj = r_traj_g − mean over the GROUP of trajectories
Â_turn = r_update_{g,t} − mean over turns AT STEP t across groups
 = α·Â_traj + (1−α)·Â_turn α = 0.9
```
Two distinct baselines. `Â_turn` is normalised across groups **at the same `t`**,
and the group size at step `t` can differ from the trajectory group size, because
trajectories that exited early have fewer turns.
α=0.9 is the paper's default and the ablation (Figure 8) explains why: at α=1
there is no update-gate reward and accuracy on evidence-free chunks collapses —
the model updates indiscriminately, which is exactly the failure this whole
system exists to avoid.
Expect instability. The paper's own limitations section says the extra rewards
"reduce training stability, requiring a smaller off-policy degree and longer
convergence time."
## Steps
1. `training/` directory, Python, `uv`-managed. System Python is 3.9.6; this
needs 3.11+.
2. Corpus loader for M5.3's format.
3. Configure verl for LoRA on Qwen2.5-3B-Instruct, rank 16–32.
4. Implement the three rewards and the two-baseline advantage, α configurable.
5. Log per step: update accuracy split by evidence-present and evidence-free,
exact-exit ratio, format correctness, mean response length, validation reward.
6. Hold out a validation split by **project**, not by trajectory — same-project
trajectories share vocabulary and leak.
7. Export the adapter, version it, publish where M5.4 can mount it.
## Acceptance
- Training runs to convergence on the validation reward.
- Both advantage terms are computed with their own baselines.
- Update accuracy on evidence-free chunks does not collapse.
- The adapter loads in M5.4's server.
## Verify
**Harness:** pytest for reward and advantage maths; a short training run for the
loop itself.
**Integration test** — `training/tests/test_rewards.py`:
1. `a1_r_update_signs` — matching label +1, mismatching −1.
2. `a2_r_exit_bands` — exact 0, early −0.75, late −0.5.
3. `a3_r_format_strict` — any unparsed turn zeroes the whole trajectory.
4. `a4_traj_baseline` — `Â_traj` uses the group mean; hand-computed fixture.
5. `a5_turn_baseline_at_step_t` — `Â_turn` normalises across groups at the same
`t`; a fixture with unequal trajectory lengths must not misalign. This is the
assertion that catches the most likely implementation error.
6. `a6_alpha_mix` — α=1 yields pure trajectory advantage; α=0 pure turn.
7. `a7_alpha_1_degenerates` — train 50 steps at α=1 on a fixture; assert
evidence-free accuracy drops relative to α=0.9, reproducing the paper's
Figure 8b.
8. `a8_validation_split_by_project` — assert no project appears in both splits.
9. `a9_adapter_loads` — export, mount in M5.4, assert a completion returns 200.
**Command:** `uv run pytest training/tests -v`
**False pass:**
- Normalising `Â_turn` over the whole batch rather than per step `t`. It trains,
loss goes down, and the turn-level signal is diluted into noise — assertion 5
is the only thing that catches it.
- Splitting validation by trajectory. Same-project trajectories share phrasing
and file paths, so validation reward looks excellent and generalisation is
untested.
- Skipping assertion 7 as "too slow". It is the only end-to-end evidence that the
update reward is wired to anything.
## Traps
- Tuning α before the rewards are verified. Every α is wrong if `r_update`'s sign
is flipped, and the symptom looks identical.
- Training on labels whose κ was never measured. The policy learns the labeler,
and there is no held-out signal that would reveal it — M5.2 exists for this and
M5.3 refuses to export without it.
---
Background: [DESIGN.md](../DESIGN.md) — P6 · paper §3.2.2, Table 3, Fig 8
tasks/M5.6-m5-gate.md
+106
View File
@@ -0,0 +1,106 @@
# M5.6 — M5 composition gate
| Field | Value |
|---|---|
| Phase | M5 — Post-training |
| Size | L — 3+ days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M5 |
## Goal
Establish that the trained adapter is actually better than prompting — and that
"better" was measured on data it never saw.
## Facts (inlined — no spec read needed)
The comparison is **adapter vs. the prompted baseline**, on a held-out project,
using the same corpus and the same prompt.
The paper's own result sets the expectation: Figure 9 shows RL helps but the
prompted workflow already works, and the gains concentrate on harder tasks. So
a modest improvement is the realistic success case; a dramatic one is a reason to
check for leakage first.
Metrics, and why each is present:
| metric | why |
|---|---|
| update accuracy, evidence-present | recall — does it catch evidence |
| update accuracy, evidence-free | precision — does it resist noise. **This is the one that collapses at α=1** |
| exact-exit ratio | did the exit gate learn anything |
| format correctness | the parser's job got easier or harder |
| memory token curve | the Figure 6 shape — flat, or climbing to the cap |
| wall clock per run | the paper claims up to 400% speedup with the exit gate |
**Held-out project is non-negotiable.** Same-project trajectories share file
paths, error strings and phrasing; measuring on them is measuring memorisation.
Report both, always: an adapter that improves evidence-present accuracy while
degrading evidence-free is worse for this system, because polluted memory
degrades every subsequent turn.
## Steps
1. Pick a project excluded from training. Ingest it with the prompted baseline;
record all metrics.
2. Ingest the same project with the adapter, identical prompt and chunking.
3. Compare on the M5.2 holdout labels.
4. Assert the thresholds below.
5. Plot the memory-token curve for both; commit it as the Figure 6 analogue.
6. Commit `expected/m5.6.txt`; diff.
7. If the adapter loses, keep it versioned and record why — a negative result with
the reason is worth more than a rerun with different hyperparameters.
## Acceptance
- Evidence-present accuracy ≥ baseline.
- Evidence-free accuracy ≥ baseline. Never traded away.
- Format correctness ≥ baseline.
- Memory token curve flat, not climbing to the cap.
- Measured on a project absent from training.
## Verify
**Harness:** live gateway with both models; the held-out project; M5.2's labels.
**Integration test** — `verify/m5.6.sh` diffed against `expected/m5.6.txt`:
1. `a1_holdout_is_unseen` — assert the eval project appears in no training
trajectory. Check the corpus, not the config.
2. `a2_evidence_present_accuracy` — adapter ≥ baseline; print both.
3. `a3_evidence_free_accuracy` — adapter ≥ baseline; print both.
4. `a4_format_correctness` — adapter ≥ baseline.
5. `a5_memory_curve_flat` — slope below the M1.8 bound for both; assert the
adapter is no worse.
6. `a6_exit_ratio_reported` — print exact/early/late exit ratios. Advisory: the
exit gate is off at L1, so this measures the signal, not behaviour.
7. `a7_same_prompt` — assert both runs used a byte-identical prompt template.
8. `a8_wall_clock` — report both; no threshold, since the exit gate is off at L1.
**Command:** `bash verify/m5.6.sh | diff - expected/m5.6.txt`
**False pass:**
- Comparing on a project that was in training. Everything improves and none of it
generalises — assertion 1 checks the corpus rather than trusting the split
config.
- Reporting a single combined accuracy. It hides the precision/recall trade that
matters most: a model that says "yes" more often scores better on
evidence-present and pollutes memory, and the combined number can improve while
the system gets worse.
- Different prompts between runs. Any measured difference then attributes to the
adapter and is partly the prompt — assertion 7 is cheap and removes the doubt.
## Traps
- Retraining until the gate passes without changing anything principled. That is
fitting the gate, and the held-out project stops being held out the third time
you look at it.
- Reading a small improvement as failure. The paper's own baseline works; the
adapter's value here is as much about stability on evidence-free chunks as
headline accuracy.
---
Background: [DESIGN.md](../DESIGN.md) — Verification, Risks · paper Fig 9
tasks/M6.1-agent-manager-db-manifest.md
+122
View File
@@ -0,0 +1,122 @@
# M6.1 — CNPG `agent-manager-db` manifest
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | homelab |
| Spec | inlined below |
| Blocks | — |
## Goal
A dedicated Postgres for agent-manager's session store, provisioned the way
everything else in the cluster is: through git, with no manual `psql`. Same
pattern this project already used for `memory-db` (M2.2), applied to a
different, unrelated app.
## Facts (inlined — no spec read needed)
**agent-manager is a separate repo**, not part of this Rust workspace:
`github.com/Riotpiaole/agent-manager` (fork of `github.com/YoanWai/
agent-manager`), local checkout `~/workplace/agent-manager`, branch
`add-headless-spawn`. Its session store is `internal/store/store.go` — see
M6.2/M6.3 for the schema and query port.
**Existing CNPG pattern, verified on the live cluster: 3 Clusters today, one
per app, each ClusterIP-only (not LAN-reachable):**
| namespace/name | app |
|---|---|
| `cicd/forgejo-db` | Forgejo |
| `iam/authentik-db` | Authentik |
| `temporal/temporal-db` | Temporal |
**No shared/multi-tenant DB cluster** — every app gets its own dedicated
CNPG `Cluster`. `agent-manager-db` follows the same rule; it does not join
`memory-db` (M2.2's cluster) even though both are new Postgres instances
touched by the same person around the same time. Different app, different
cluster.
Follow `k8s/infra/databases/temporal-db.yaml` exactly, same as M2.2 did:
`imageName` pinned, `enableSuperuserAccess: false`, `storageClass:
longhorn-cnpg`, `enablePodMonitor: true`, control-plane tolerations,
`podAntiAffinityType: preferred`.
**Instance count — open question, default to convention.** Motivation for
this whole migration is durability-of-location, not HA (single-machine
usage, not a multi-host shared-session requirement). The 3 existing
clusters are all 3-instance. Default to 3 instances for consistency with
every other app in the cluster rather than special-casing this one to 1;
revisit only if resource pressure on the homelab nodes makes it a real
tradeoff.
Storage: session rows are tiny (`sessions`, `groups`, `settings`,
`review_*` — no blobs beyond a `snapshot TEXT` pane capture per session).
1Gi is generous; no need for `memory-db`'s 10Gi (that one holds
768-dim vectors).
## Steps
1. `k8s/infra/databases/agent-manager-db.yaml` — `Cluster` + `Database`,
namespace `agent-manager`, no extensions (plain relational, no
pgvector).
2. Namespace `agent-manager`, created by the ArgoCD app that owns it.
3. Add to the owning kustomization's explicit resource list — an unlisted
file is silently dropped with no error and no drift shown (the M2.2
task file names this exact trap).
4. Commit, push to **both** Forgejo origin and the GitHub mirror — verify
which `repoURL` the eventual ArgoCD `Application` for this app actually
watches before assuming either push is the one that matters (`kong`
app, for example, tracks the GitHub mirror specifically, not Forgejo).
5. Let ArgoCD sync. **No `kubectl apply`.**
6. Verify the app user can create tables (schema arrives in M6.2, but a
throwaway `CREATE TABLE t(id text); DROP TABLE t;` proves connectivity
here).
## Acceptance
- `Cluster` reaches `Cluster in healthy state`.
- ArgoCD shows the app `Synced/Healthy`.
- No manual `psql` was run to get there.
- Service is ClusterIP-only — not reachable from the LAN directly (M6.4's
nginx route is the only path in).
## Verify
**Harness:** `kubectl` and `psql` read-only checks after sync.
**Integration test** — `verify/m6.1.sh`, output diffed against
`expected/m6.1.txt`:
1. `a1_cluster_healthy` — `kubectl get cluster -n agent-manager
agent-manager-db` reports all instances ready.
2. `a2_clusterip_only` — `kubectl get svc -n agent-manager -o
jsonpath='{.items[*].spec.type}'` contains no `LoadBalancer` or
`NodePort`.
3. `a3_argocd_synced` — the owning app is `Synced/Healthy`.
4. `a4_app_user_can_ddl` — as `app`, `CREATE TABLE t(id text); DROP TABLE
t;` succeeds.
5. `a5_no_lan_route_yet` — connection attempt from outside the cluster
network fails at this point in the plan (M6.4 hasn't landed).
**Command:** `bash verify/m6.1.sh | diff - expected/m6.1.txt`
**False pass:**
- Confirming sync without checking service type. A `Cluster` can be
healthy and `Synced` while someone fat-fingered a `LoadBalancer` type
into the manifest, silently violating the "dedicated ingress, not raw
LAN IP" network-path decision this whole migration made. Assertion 2 is
the guard.
## Traps
- Forgetting the kustomization resource list (same trap M2.2 already
named) — file sits in git, ArgoCD reports Synced, objects never exist.
- Adding `prune: true` without accounting for CNPG-operator-created
children (Services, Secrets, PVCs). M2.2's Traps section already hit
this on `llm-serving`; same fix applies here (`prune: false`).
---
Background: `k8s/infra/databases/temporal-db.yaml` · [M2.2](M2.2-memory-db-manifest.md) (same pattern, different app)
tasks/M6.2-schema-port.md
+243
View File
@@ -0,0 +1,243 @@
# M6.2 — Postgres schema for agent-manager sessions
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M6.1 |
## Goal
The sqlite schema agent-manager's `internal/store/store.go` builds up
through 14 `ALTER TABLE` migrations, collapsed into one Postgres-native
schema — with the sqlite workarounds (integer booleans, dual-encoded
timestamps) removed rather than carried over.
## Facts (inlined — no spec read needed)
Current schema, read directly from `internal/store/store.go` (6 tables, no
FK constraints anywhere — every relationship is enforced in Go, not SQL):
```mermaid
erDiagram
GROUPS ||--o{ SESSIONS : "group_name (app-level, no FK)"
SESSIONS ||--o{ SESSIONS : "parent_id (self-ref, app-level, no FK)"
SESSIONS ||--o| REVIEW_TARGETS : "session_id (app-level, no FK)"
SESSIONS ||--o{ REVIEW_BASES : "session_id (app-level, no FK)"
SESSIONS ||--o| REVIEW_SCOPES : "session_id (app-level, no FK)"
SETTINGS {
text key PK
text value
}
SESSIONS {
text id PK
text name
text tool
text cwd
text group_name "app-level FK -> GROUPS.name"
text status
int archived "bool 0/1 in sqlite"
int created_at "unix nanos, dual-encoded in sqlite"
int last_status_at "unix nanos"
text agent_session_id
text pending_inputs "JSON array blob"
int pending_claimed "bool 0/1"
text launch_prompt
int sort_order
int acked "bool 0/1"
text snapshot
text worktree_repo
text worktree_branch
int agent_launched_at "unix nanos"
text retired_agent_session_id
text parent_id "app-level FK -> SESSIONS.id, self"
}
GROUPS {
text name PK
int sort_order
text path
int archived "bool 0/1"
text worktree
}
REVIEW_TARGETS {
text session_id PK
text repo_root
}
REVIEW_BASES {
text session_id PK
text repo_root PK
text base_ref
}
REVIEW_SCOPES {
text session_id PK
text scope
}
```
Target Postgres DDL:
```sql
CREATE TABLE groups (
name TEXT PRIMARY KEY,
sort_order INTEGER NOT NULL DEFAULT 0,
path TEXT NOT NULL DEFAULT '',
archived BOOLEAN NOT NULL DEFAULT FALSE,
worktree TEXT NOT NULL DEFAULT ''
);
CREATE TABLE sessions (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
tool TEXT NOT NULL,
cwd TEXT NOT NULL,
group_name TEXT NOT NULL REFERENCES groups(name),
status TEXT NOT NULL,
archived BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ NOT NULL,
last_status_at TIMESTAMPTZ NOT NULL,
agent_session_id TEXT NOT NULL DEFAULT '',
pending_inputs JSONB NOT NULL DEFAULT '[]',
pending_claimed BOOLEAN NOT NULL DEFAULT FALSE,
launch_prompt TEXT NOT NULL DEFAULT '',
sort_order INTEGER NOT NULL DEFAULT 0,
acked BOOLEAN NOT NULL DEFAULT FALSE,
snapshot TEXT NOT NULL DEFAULT '',
worktree_repo TEXT NOT NULL DEFAULT '',
worktree_branch TEXT NOT NULL DEFAULT '',
agent_launched_at TIMESTAMPTZ,
retired_agent_session_id TEXT NOT NULL DEFAULT '',
parent_id TEXT REFERENCES sessions(id)
);
CREATE TABLE settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
CREATE TABLE review_targets (
session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE,
repo_root TEXT NOT NULL
);
CREATE TABLE review_bases (
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
repo_root TEXT NOT NULL,
base_ref TEXT NOT NULL,
PRIMARY KEY (session_id, repo_root)
);
CREATE TABLE review_scopes (
session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE,
scope TEXT NOT NULL
);
```
**Decision: add real FKs, not app-level-only.** Current Go code manually
cascades `review_targets`/`review_bases`/`review_scopes` deletes before
deleting a `sessions` row (`Delete`, `DeleteChild` in store.go) — `ON
DELETE CASCADE` on those three removes that manual bookkeeping entirely.
`group_name -> groups(name)` and `parent_id -> sessions(id)` get FKs too,
since Postgres makes them free and they catch the exact class of bug
`validParent`/`ensureGroup` exist in Go to prevent by hand. This is a
behavior change from sqlite (constraint violation now possible on write
paths that previously just wrote garbage) — M6.3 must add error handling
for FK violations at the ~4 call sites that write `group_name`/
`parent_id` without having already validated the reference through
`validParent`/`ensureGroup`.
**Booleans** — `BOOLEAN`, not `INTEGER`. Removes `boolToInt()`/`!= 0` at
every read/write site in store.go.
**Timestamps** — `TIMESTAMPTZ`, not the `encodeTime`/`decodeTime`
nanosecond-or-legacy-seconds hack (store.go:1369-1396). That hack exists
only because sqlite has no native timestamp type and old rows needed a
seconds/nanos disambiguation heuristic (`secondsCeiling`). A fresh
Postgres database has no legacy rows — the whole function pair is deleted,
not ported. `agent_launched_at` becomes nullable (`NULL` = "never
restarted") instead of the sqlite sentinel `0`.
**`pending_inputs`** — `JSONB`, not `TEXT` holding a JSON string. Same
`json.Marshal`/`Unmarshal` round-trip in Go either way, but `JSONB` lets
Postgres validate the shape on write instead of accepting malformed JSON
that only fails on the next read.
## Steps
1. Write `agent-manager` migration files (this repo's `sqlx`-migration
convention doesn't apply — agent-manager is a separate Go repo; use
whatever migration tool its `add-headless-spawn` branch already has, or
a plain `.sql` file run once at `Open()` if it has none — check before
introducing a new dependency).
2. `CREATE TABLE` in FK-dependency order: `groups`, then `sessions`
(references `groups` and itself), then `settings`,
`review_targets`/`review_bases`/`review_scopes`.
3. No indexes beyond the primary keys are needed yet — `ListSessions`
filters on `archived` and orders by `group_name, sort_order,
created_at`; add a composite index only if M6.6's gate shows it's slow,
not preemptively.
4. Confirm `CREATE EXTENSION` is not needed anywhere (plain relational
schema, no pgvector) — unlike this project's own `memory-db`.
## Acceptance
- Schema applies to a clean `agent-manager-db` database.
- `archived`/`acked`/`pending_claimed` are `BOOLEAN`, not `INTEGER`.
- `created_at`/`last_status_at`/`agent_launched_at` are `TIMESTAMPTZ`.
- Deleting a `sessions` row cascades `review_targets`/`review_bases`/
`review_scopes` without the Go code doing it manually.
- Inserting a session with an unknown `group_name` is rejected by the FK,
not silently written.
## Verify
**Harness:** disposable Postgres, same image as M6.1's `Cluster`
(`ghcr.io/cloudnative-pg/postgresql:16.2`) — or a local `postgres:16`
container for fast iteration, since agent-manager's own test suite doesn't
need the real cluster.
**Integration test** — extend agent-manager's existing store tests
(`internal/store/*_test.go` already has `timeenc_test.go`; that file is
deleted in M6.3, its coverage folded into this schema's tests) with:
1. `a1_migrate_clean` — apply to an empty database, assert all 6 tables
exist.
2. `a2_bool_columns_are_boolean` — `information_schema.columns` reports
`boolean` for `archived`, `acked`, `pending_claimed`.
3. `a3_timestamp_columns_are_timestamptz` — same, for `created_at`,
`last_status_at`, `agent_launched_at`.
4. `a4_review_cascade` — insert a session + a `review_targets` row for it,
delete the session, assert `review_targets` is empty without a separate
`DELETE FROM review_targets` call.
5. `a5_group_fk_rejects_unknown` — insert a session with a `group_name`
that has no matching `groups` row, assert it's rejected.
6. `a6_parent_fk_self_ref` — insert two sessions where the second's
`parent_id` points at the first, assert it succeeds; point it at a
nonexistent id, assert it's rejected.
**Command:** `go test ./internal/store/... -run TestSchema`
**False pass:**
- Testing the schema against sqlite (leftover `modernc.org/sqlite` import)
instead of real Postgres. `BOOLEAN`/`TIMESTAMPTZ`/`JSONB`/FK-cascade
behavior all differ or are silently accepted-but-ignored by sqlite —
every assertion above becomes meaningless against the wrong engine.
## Traps
- Keeping `ON DELETE CASCADE` off `group_name`/`parent_id` FKs by
accident (only adding it to the `review_*` ones). A group delete or
session delete then hits an FK violation instead of the graceful
"session has terminals of its own; move them out first" error
`PlaceSession` already gives — check M6.3 preserves that message instead
of leaking a raw constraint-violation error to the CLI.
- Making `parent_id` `NOT NULL DEFAULT ''` (matching the sqlite default)
instead of nullable. `'' REFERENCES sessions(id)` is never satisfiable
except by a literal empty-string-id row, which doesn't exist — every
top-level session's insert then fails the FK. Must be `NULL` for "no
parent."
---
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
tasks/M6.3-store-query-port.md
+160
View File
@@ -0,0 +1,160 @@
# M6.3 — store.go query port to Postgres
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | L — 3+ days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M6.2 |
## Goal
Every query in `internal/store/store.go` rewritten against the M6.2 schema,
sqlite dropped entirely (decided: full port, not a dual sqlite/Postgres
backend — this is single-machine usage, no standalone-without-cluster
requirement to preserve).
## Facts (inlined — no spec read needed)
Read directly from `internal/store/store.go` (1397 lines) on the
`add-headless-spawn` branch:
- **65 call sites** (`db.Exec`, `db.QueryRow`, `db.Query`, `tx.Exec`,
`tx.QueryRow`) use sqlite `?` positional placeholders. Postgres
(`pgx` or `lib/pq`) needs `$1, $2, ...` — mechanical but must be done
per-statement since arg count varies 1–16 across sites.
- **Upserts are already Postgres-compatible.** All 9 upsert sites
(`SetReviewRepo`, `SetReviewBase`, `SetReviewScope`, `SetSetting`,
`ensureGroup`, `CreateGroup`, `AddGroup`, `createSession`'s group
insert, `PlaceSession`'s group insert) already use `ON
CONFLICT(...) DO UPDATE SET ... = excluded....` / `ON CONFLICT(...) DO
NOTHING`, valid as-is in Postgres. No rewrite beyond the placeholder
swap — this corrects an earlier assumption that these were sqlite
`INSERT OR REPLACE` and needed real rework.
- `boolToInt()`/`archived != 0`-style conversions at ~8 sites go away —
scan `int archived`/`int acked`/`int pendingClaimed`/`int` locals in
`ListSessions`, `Get` and replace with `bool` directly.
- `encodeTime()`/`decodeTime()` calls at every session-timestamp
read/write go away — pass `time.Time` straight through; `pgx` handles
`TIMESTAMPTZ` natively. Delete `timeenc_test.go` along with the
functions it tests (M6.2 already noted its coverage folds into the
schema tests instead).
- `db.SetMaxOpenConns(1)` + `PRAGMA journal_mode=WAL`
(store.go:76-79) gave sqlite serialized writes for free. Some call
sites lean on that implicit serialization — most notably
`createSession`'s `sort_order = (SELECT COALESCE(MAX(sort_order)+1, 0)
FROM sessions WHERE group_name = ? AND parent_id = ?)` subquery, which
races under concurrent Postgres writers with a real connection pool.
Wrap it in `SELECT ... FOR UPDATE` inside the existing transaction (the
function already opens one) rather than trusting single-writer
semantics that no longer exist. Same pattern applies to
`ReorderSession`/`SwapSessionOrder`/`ReorderGroup`/`SwapGroupOrder`'s
read-then-renumber-then-write sequences.
- FK violations are new failure modes M6.2 introduced (group/parent FKs).
`createSession` and `PlaceSession` already validate `group_name`/
`parent_id` through `validParent`/`ensureGroup` before writing, so those
paths shouldn't hit a live constraint in practice — but wrap the actual
`INSERT`/`UPDATE` error and translate a `23503` (foreign_key_violation)
SQLSTATE into the same descriptive errors the pre-validation already
produces, so a race between the check and the write degrades to a clear
error instead of a raw driver error reaching the CLI.
- `driver: modernc.org/sqlite` import and the blank `_ "modernc.org/
sqlite"` in `Open()` are deleted; replaced with `pgx` (`github.com/
jackc/pgx/v5/stdlib` for `database/sql` compatibility, keeping the rest
of the file's `*sql.DB`-based code unchanged) or a native `pgx.Pool` —
pick `pgx/v5/stdlib` unless a later task needs pgx-native features
(e.g. `COPY`), since it's the smaller diff against the existing
`database/sql` code.
## Steps
1. Swap the driver import and `Open()`'s connection string handling
(sqlite file path -> Postgres DSN, likely from an env var or flag the
CLI already has a slot for — check `cmd/` for how `Open()` is called
today).
2. Delete `db.SetMaxOpenConns(1)` / `PRAGMA journal_mode=WAL`; size the
pool deliberately instead (`SetMaxOpenConns` to something sane for a
single-machine client, e.g. 5–10).
3. Delete `init()`'s `CREATE TABLE IF NOT EXISTS` + 14-migration list —
M6.2's migration owns schema creation now; `Open()` just connects and
optionally runs a `schema_version` sanity check.
4. Mechanically convert every `?` to `$N` across the 65 call sites, in
file order, verifying arg count against placeholder count each time
(this is where an off-by-one is easiest to introduce silently).
5. Remove `boolToInt`/`archived != 0` conversions; scan struct fields
directly as `bool`.
6. Remove `encodeTime`/`decodeTime`/`secondsCeiling`; pass `time.Time`
directly. Delete `timeenc_test.go`.
7. Add `SELECT ... FOR UPDATE` (or equivalent explicit locking) to the
sort-order read-then-write sequences named above.
8. Add FK-violation (`23503`) error translation at the write sites that
can theoretically race past their own pre-validation.
9. Run agent-manager's full existing test suite against a real Postgres
(M6.1's cluster, or local `postgres:16` for iteration) — every current
test should still pass unmodified in intent, only in backing store.
## Acceptance
- Zero references to `modernc.org/sqlite`, `?` placeholders,
`boolToInt`, `encodeTime`/`decodeTime` remain in `internal/store/`.
- Full existing store test suite passes against Postgres.
- Concurrent `CreateSession` calls (simulated) never produce duplicate
`sort_order` values within the same `group_name`/`parent_id`.
- A `spawn` CLI round trip (create session, update status, delete) works
end-to-end against the M6.1 cluster.
## Verify
**Harness:** `internal/store/*_test.go` running against the disposable
Postgres from M6.2's harness.
**Integration test** — extend/rename the existing store test files:
1. `a1_no_sqlite_references` — `grep -r "modernc.org/sqlite" internal/`
returns nothing.
2. `a2_no_bare_placeholders` — `grep -rE '\?[,)]' internal/store/store.go`
inside SQL string literals returns nothing (manual review of any
`?` that's part of a non-SQL string, e.g. a Go format verb, to avoid a
false positive).
3. `a3_existing_suite_passes` — `go test ./internal/store/...` green
against Postgres.
4. `a4_concurrent_sort_order` — spawn N goroutines each calling
`CreateSession` into the same group concurrently; assert the resulting
`sort_order` values are a dense 0..N-1 permutation with no duplicates.
5. `a5_fk_violation_translated` — attempt `PlaceSession` with a
`parentID` deleted between the read and the write (simulate via a
second connection); assert the returned error is the existing
descriptive one, not a raw pgx driver error.
6. `a6_spawn_roundtrip` — `spawn` CLI subcommand creates a session,
`UpdateStatus`, then `Delete`; assert no error and the row is gone.
**Command:** `go test ./internal/... -run TestStore -v`
**False pass:**
- Running the suite against sqlite still (leftover build tag or import)
while believing it validated Postgres. Assertion 1 is the guard —
without it, the whole task could report green while the driver swap
never actually happened.
- Skipping assertion 4. A `sort_order` race is invisible in every
single-threaded test and only shows up as duplicate/out-of-order
sessions under real concurrent use, which is exactly the class of bug
removing `SetMaxOpenConns(1)` introduces.
## Traps
- Converting `?` to `$N` by simple find-and-replace in file order without
re-checking each statement's actual arg list — a multi-arg statement
reordered during earlier edits (e.g. `createSession`'s 15-arg INSERT)
will silently bind the wrong value to the wrong column, and Go's
`database/sql` won't catch a type-compatible mismatch (e.g. two `TEXT`
columns swapped).
- Forgetting `pending_inputs` is now `JSONB` (M6.2), not `TEXT` — the
existing `json.Marshal`/`Unmarshal` round-trip in
`encodePendingInputs`/`pendingInputState` still works unchanged (pgx
scans `JSONB` into `[]byte` the same as `TEXT`), but don't add an extra
marshal layer thinking the column type changed the Go-side contract.
---
Background: [M6.2](M6.2-schema-port.md) · `internal/store/store.go`, `internal/spawn/spawn.go` (agent-manager, `add-headless-spawn` branch)
tasks/M6.4-nginx-stream-routing.md
+122
View File
@@ -0,0 +1,122 @@
# M6.4 — nginx TCP routing to `agent-manager-db`
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | S — <1 day |
| Status | ⬜ Not started |
| Flags | homelab |
| Spec | inlined below |
| Blocks | M6.1 |
## Goal
A path from the Mac client running agent-manager to the cluster-internal,
ClusterIP-only `agent-manager-db` — through the shared ingress controller,
matching this homelab's existing pattern, not a raw LAN IP or a tunnel.
## Facts (inlined — no spec read needed)
**Decided, not open:** dedicated ingress routing through nginx, not
`kubectl port-forward`/SSH tunnel and not a MetalLB `LoadBalancer` IP.
Matches the pattern already used elsewhere in this homelab of routing
through the shared ingress controller rather than exposing raw
per-service LAN IPs.
**Postgres is not HTTP.** The standard nginx-ingress `Ingress` resource is
HTTP(S)-oriented (host/path routing, TLS termination via SNI on 443).
Postgres speaks its own binary wire protocol on 5432. The ingress
controller needs `stream {}` block config (TCP/UDP passthrough) or a
dedicated `TCP` mode `Service`/`ConfigMap` entry — whichever the specific
nginx-ingress deployment in this cluster supports (check
`k8s/infra/ingress/` for how it's deployed and whether `tcp-services`
ConfigMap wiring already exists for anything else, since this may be the
first TCP passthrough case in the cluster).
**No existing precedent in this homelab** — `forgejo-db`, `authentik-db`,
`temporal-db` are all consumed only by pods inside the same cluster over
their ClusterIP Service, never from outside. `agent-manager-db` is the
first case of an external (Mac) client needing to reach a CNPG cluster,
which is why this task exists as dedicated work rather than "just add a
Service."
## Steps
1. Confirm how nginx-ingress is deployed in this cluster (`k8s/infra/
ingress/`) and whether it already exposes a `tcp-services` ConfigMap
or `stream {}` snippet mechanism — ingress-nginx (the community
controller) supports TCP passthrough via a `tcp-services` ConfigMap
mapping `<external-port>: <namespace>/<service>:<port>`; confirm this
is the controller in use before assuming that config shape.
2. Pick an external port for Postgres traffic (5432 is already the
in-cluster default; an external port distinct from any other exposed
service avoids collision — check what's already claimed).
3. Add the `tcp-services` (or equivalent) entry routing that external
port to `agent-manager-db-rw.agent-manager.svc.cluster.local:5432`
(CNPG's read-write Service name convention — confirm against the
actual Service name M6.1's `Cluster` generates).
4. Expose that port on the ingress controller's `Service`/`LoadBalancer`
(this is the one LAN-facing port for this whole feature — the DB
itself stays ClusterIP-only, only the ingress controller's existing
external IP gains a new port).
5. Commit, push to both remotes, verify which `repoURL` the owning
ArgoCD `Application` watches (same caveat as M6.1) before assuming a
push landed, let ArgoCD sync.
6. Test connectivity from the Mac client: `psql
postgresql://<user>@<homelab-ingress-host>:<external-port>/
agent_manager` (credentials from M6.5).
## Acceptance
- `psql` (or `pgx`) from outside the cluster reaches `agent-manager-db`
through the ingress controller's external IP/port.
- `agent-manager-db`'s own Service remains ClusterIP-only — no
`LoadBalancer`/`NodePort` added to it directly (that would defeat the
point of routing through nginx).
- TLS/auth on the connection is Postgres's own (`sslmode`, password auth)
— nginx `stream {}` passthrough does not terminate or inspect the
Postgres protocol, so it adds no auth of its own. Confirm this is
acceptable given the homelab's network boundary (LAN-only ingress
exposure, not public internet) before treating it as done.
## Verify
**Harness:** `psql` from the Mac client (outside the cluster network),
plus `kubectl` checks on the ingress controller's config.
**Integration test** — `verify/m6.4.sh` diffed against
`expected/m6.4.txt`:
1. `a1_external_connects` — `psql
postgresql://app@<ingress-host>:<port>/agent_manager -c 'select 1'`
from the Mac client succeeds.
2. `a2_db_service_still_clusterip` — `kubectl get svc -n agent-manager
agent-manager-db-rw -o jsonpath='{.spec.type}'` is `ClusterIP`.
3. `a3_argocd_synced` — the ingress-owning app is `Synced/Healthy` after
the config change.
4. `a4_wrong_port_refused` — connecting to a random unmapped port on the
same ingress host fails (proves the mapping is port-specific, not an
accidental catch-all passthrough).
**Command:** `bash verify/m6.4.sh | diff - expected/m6.4.txt`
**False pass:**
- Testing connectivity from inside the cluster (e.g. `kubectl exec` into
a pod and `psql` the ClusterIP directly). That was already true before
this task and proves nothing about the ingress path — assertion 1 must
run from the actual Mac client, outside the cluster network.
## Traps
- Reusing port 5432 externally on the ingress controller's existing
external IP if anything else is already listening there (unlikely for
Postgres specifically, but worth a `kubectl get svc -n <ingress-ns>
<ingress-svc> -o yaml` check before assuming the port is free).
- `stream {}` / `tcp-services` config living outside the GitOps-tracked
kustomization because it's a ConfigMap edit that "felt like a quick
manual fix." Same hard rule as everything else: commit + push +
ArgoCD sync, no manual `kubectl apply` to the ingress controller's
config.
---
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `k8s/infra/ingress/` (nginx-ingress deployment, controller type to confirm)
tasks/M6.5-credentials-secret.md
+110
View File
@@ -0,0 +1,110 @@
# M6.5 — Postgres credentials for the Mac client
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | S — <1 day |
| Status | ⬜ Not started |
| Flags | homelab |
| Spec | inlined below |
| Blocks | M6.1 |
## Goal
The `agent-manager` process running on the Mac gets a Postgres connection
string/credentials, managed the same ksops way the rest of this homelab
handles secrets — not a password pasted into a local config file or env
var by hand.
## Facts (inlined — no spec read needed)
**Existing pattern to follow: ksops-managed secret**, same as
`model-invoke-apikey` elsewhere in this homelab. CNPG itself already
generates an in-cluster Secret for the `app` user
(`agent-manager-db-app` by its usual naming convention) — the work here is
getting that credential (or a dedicated read-write user, if reusing the
CNPG-generated superuser-adjacent `app` credential isn't desired) to a
process running outside the cluster, on the Mac.
**Two credentials touch two different trust boundaries:**
- Cluster-internal: CNPG's own generated Secret, already ksops-free
(CNPG manages it, not this repo).
- Mac client: needs that same username/password (or a separate,
narrower-scoped user) delivered to `~/workplace/agent-manager`'s
runtime config, via a ksops-encrypted file in git rather than a
manually-copied value.
## Steps
1. Decide: reuse CNPG's auto-generated `app` user, or create a dedicated
`agent_manager_client` role scoped to only the `sessions`/`groups`/
`settings`/`review_*` tables (narrower blast radius if the credential
ever leaks from a Mac laptop, which is a meaningfully different threat
model than a credential that only ever lives inside the cluster).
2. If a dedicated role: add it via CNPG's declarative `Database` /
`postInitSQL` (or a one-time migration in M6.2's schema setup) rather
than a manual `psql` grant.
3. Encrypt the resulting connection string (or user/password pair) with
ksops, following the exact file layout `model-invoke-apikey` uses.
4. Commit the encrypted secret to git (safe — that's the point of ksops),
push to both remotes.
5. Wire agent-manager's config loading (check `cmd/` / existing config
file handling on the `add-headless-spawn` branch) to read the
decrypted value at runtime — decide whether decryption happens via a
`sops exec-env`-style wrapper the Mac invokes, or a decrypted file
materialized once locally and gitignored, matching whatever
`model-invoke-apikey`'s consumers already do.
6. Verify agent-manager connects using only the ksops-sourced credential
— no plaintext password anywhere in the repo or in shell history.
## Acceptance
- No Postgres password appears in plaintext in git, in
`~/workplace/agent-manager`'s tracked config, or was typed directly
into a `kubectl`/`psql` command during setup.
- The credential is encrypted with ksops in the same repo location/
pattern as `model-invoke-apikey`.
- agent-manager on the Mac successfully authenticates through M6.4's
ingress route using this credential.
## Verify
**Harness:** manual — this is a secret-handling task, not one to automate
a fake credential through.
**Checklist** (no `verify/*.sh`, since scripting a real credential check
means committing something that either leaks a real secret or asserts
nothing):
1. `git grep -i "password"` in both the homelab repo and
`~/workplace/agent-manager` shows only ksops-encrypted blobs or
references to environment/config loading, never a literal value.
2. The ksops secret file's structure matches `model-invoke-apikey`'s
(same encryption provider, same key layout) — a side-by-side diff of
the YAML structure (not values) confirms this.
3. `psql` from the Mac using the decrypted credential (via M6.4's route)
succeeds.
4. Revoking/rotating the credential (delete the CNPG Secret or drop the
dedicated role, if one was created) and re-running step 3 fails
cleanly — confirms the client isn't caching or falling back to
something else.
**False pass:**
- Confirming connectivity once with a credential typed in manually during
setup, then wiring the ksops path afterward without re-testing that the
*ksops-sourced* value is what actually authenticates. Checklist item 4
(rotate and confirm the old path is really gone) is the guard against
"it worked, but only because of leftover state."
## Traps
- Granting the dedicated role (if created) superuser or
database-owner-equivalent privileges "to avoid permission errors during
setup" and never narrowing it afterward — defeats the reason to create
a dedicated role at all.
- Storing the decrypted credential in a file the Mac client reads that
isn't gitignored, recreating the plaintext-secret problem one directory
away from the ksops-encrypted source of truth.
---
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · `model-invoke-apikey` (ksops pattern reference, this homelab)
tasks/M6.6-m6-gate.md
+150
View File
@@ -0,0 +1,150 @@
# M6.6 — M6 composition gate
| Field | Value |
|---|---|
| Phase | M6 — agent-manager migration |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M6 |
## Goal
Prove the schema, the query port, the network path, and the credentials
bind together into one working system — not four pieces that each passed
their own task in isolation. This is the property no single M6 task owns:
M6.1 proves the cluster is healthy, M6.2 proves the schema applies, M6.3
proves the queries work against a local test database, M6.4 proves a raw
TCP connection reaches the cluster, M6.5 proves the credential decrypts —
none of them alone proves agent-manager, running for real on the Mac,
through the real ingress route, with the real ksops credential, against
the real cluster, does its actual job: track a session end-to-end without
losing or corrupting data.
## Facts (inlined — no spec read needed)
**"Bind together smoothly" means two concrete things here, not a vibe:**
1. **The full network path is exercised, not simulated.** M6.3's tests run
against a disposable local Postgres — that validates the SQL, not the
route. This gate is the first (and only) task that runs agent-manager
unmodified, on the actual Mac, through M6.4's nginx `stream {}` route,
authenticating with M6.5's ksops-sourced credential, against M6.1's
real cluster.
2. **Schema conventions match the rest of this homelab, not just
"compiles."** This project's own `memory-db` (M2.x) and
`agent-manager-db` (M6.x) are two unrelated Postgres schemas landing in
the same cluster around the same time. They don't share data or a
cluster (M6.1 already ruled that out), but a reviewer scanning
`k8s/infra/databases/` should find the same shape twice: same
`Cluster`/`Database` CRD structure, same `storageClass`, same
`enableSuperuserAccess: false`, same GitOps-only provisioning
discipline. "Binds together smoothly" includes that consistency check,
not just agent-manager working in isolation.
**What a session round trip actually touches**, so the test isn't
shallow: `CreateSession` (writes `sessions` + touches `groups` via the
`ON CONFLICT DO NOTHING` insert) -> `UpdateStatus` -> `SetAgentSessionID`
-> `SetReviewRepo` (writes `review_targets`) -> `Delete` (must cascade
`review_targets` via the FK, per M6.2's decision, with no leftover row).
That single flow crosses all 4 non-`settings` tables and exercises both
the FK-cascade decision and the placeholder-conversion correctness from
M6.3 in one pass.
## Steps
1. On the Mac, with agent-manager built from the fully-ported
`add-headless-spawn` branch (M6.3 complete) and configured to use
M6.4/M6.5's route and credential: run the `spawn` CLI subcommand to
create a real session.
2. Drive it through the full lifecycle above (status update, agent
session id capture, review target set, delete) using agent-manager's
own CLI/TUI, not a hand-rolled SQL script — the point is proving the
actual client works, not that Postgres accepts hand-written SQL.
3. Kill and restart agent-manager mid-lifecycle (after step 2's status
update, before delete); confirm it reconnects and reads back the same
state — proves the connection isn't accidentally caching state
client-side that masks a write that never actually landed.
4. Diff `k8s/infra/databases/agent-manager-db.yaml` against
`k8s/infra/databases/memory-db.yaml` field-by-field for the
convention-consistency check.
5. Confirm both ArgoCD Applications (agent-manager's and this project's
`memory-db`, once it exists) are tracked from the `repoURL` each
actually watches — re-verify per M6.1's caveat, since this is the
final point where a "pushed but ArgoCD never saw it" mistake would
otherwise go unnoticed until much later.
6. Commit `expected/m6.6.txt`; diff.
## Acceptance
- A session created, updated, and deleted through agent-manager's real
CLI on the Mac round-trips correctly through the full network path.
- A mid-lifecycle restart does not lose or duplicate state.
- Deleting the session leaves zero orphan rows in `review_targets`/
`review_bases`/`review_scopes` (FK cascade, not app-level cleanup).
- `agent-manager-db.yaml` and `memory-db.yaml` match on every field that
isn't inherently app-specific (name, storage size).
- No manual `kubectl apply`/`psql` anywhere in the setup this gate
exercises.
## Verify
**Harness:** the real Mac client, the real cluster, agent-manager's own
CLI — this gate deliberately does not use a disposable/local database,
since proving the disposable path works is exactly what M6.1-M6.5 already
did.
**Integration test** — `verify/m6.6.sh` diffed against
`expected/m6.6.txt`:
1. `a1_full_roundtrip` — create/update/set-review/delete via the real CLI;
assert no error at any step.
2. `a2_no_orphan_review_rows` — after delete, query `review_targets`/
`review_bases`/`review_scopes` directly (from inside the cluster, as a
final-state check) for the deleted session's id; assert zero rows,
with no explicit `DELETE FROM review_*` having been issued by the CLI
(proves the FK cascade did the work, not leftover manual-cleanup code
nobody removed).
3. `a3_survives_restart` — kill agent-manager between status-update and
delete; restart; assert the status update is still visible before
proceeding to delete.
4. `a4_schema_convention_match` — diff the two `Cluster` manifests' non
app-specific fields; assert empty diff.
5. `a5_repourl_confirmed` — for each of the two ArgoCD Applications
involved, print which `repoURL` it watches and confirm it matches
which remote was actually pushed.
6. `a6_no_manual_apply_in_history` — review the shell history / session
log from M6.1 through M6.5 for a `kubectl apply` or `psql` write
command that wasn't inside an explicitly-flagged debugging exception;
assert none exist outside that exception.
**Command:** `bash verify/m6.6.sh | diff - expected/m6.6.txt`
**False pass:**
- Running assertion 1 against M6.3's disposable local Postgres instead of
the real cluster because it's faster/already running. That's exactly
the "four pieces that each passed in isolation" failure mode this gate
exists to catch — it must hit M6.1's actual cluster through M6.4's
actual route.
- Treating a schema diff (assertion 4) as advisory and skipping it when
short on time. A convention mismatch here is invisible today and
becomes the thing a future reviewer trips over when comparing the two
`k8s/infra/databases/*.yaml` files months later with no memory of why
they differ.
## Traps
- Discovering during this gate that M6.4's nginx route works from inside
the homelab LAN but not from wherever the Mac actually sits (VPN,
different subnet, etc.) — a gap none of M6.1-M6.5's narrower tests
would have caught, since this is the first task that tests from the
Mac's actual network position rather than "outside the cluster" in the
abstract.
- Fixing a gate failure by loosening the gate (e.g. deleting assertion 2
because the cascade "mostly works") instead of fixing the underlying
FK/migration issue. Same discipline this project's other gates
(M0.8, M1.8, M2.8...) already hold to.
---
Background: [M6.1](M6.1-agent-manager-db-manifest.md) · [M6.2](M6.2-schema-port.md) · [M6.3](M6.3-store-query-port.md) · [M6.4](M6.4-nginx-stream-routing.md) · [M6.5](M6.5-credentials-secret.md)