commit c1ada4161712ae1299ab9fae046d8e7364c7ab02 Author: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed Aug 19 09:52:07 2026 -0700 (plan) system review and break down plans diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..eaa2d4f --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,486 @@ +# Poimen Memory — gated recurrent knowledge extraction from agent context + +Target repo: `/Users/rockliang/workplace/Poimen/memory` (empty git repo, no remote, no commits). + +## Context + +Agent sessions accumulate faster than they can be read, and almost all of the volume is noise. One pi session in this project measured: + +``` +assistant 1445 +toolResult 1261 <- 43%, mostly evidence-free (ls output, file reads) +user 196 ++ 8 compaction events +``` + +Compaction already fires 8 times per session, which means context is being *discarded* rather than *retained* — the knowledge produced (root causes, decisions, gotchas) evaporates when the window rolls. Meanwhile the corpus is already project-partitioned on disk: + +- `~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen--/*.jsonl` — `cwd` in the session header gives the project key +- `~/.claude/projects//.jsonl` — 15 MB, 7 transcripts +- `Poimen/agent-rust/tasks/artifacts//` — coder/reviewer JSONL per attempt + +Intended outcome: a durable, queryable project memory built by reading that history chunk-by-chunk and keeping only what answers standing questions — surfaced as an Obsidian vault a human reads and a pgvector index an agent queries. + +The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the problem: the **update gate** refuses to write evidence-free chunks into memory (the 43% of toolResults), and the **exit gate** stops the scan once evidence is sufficient. The paper reports up to 400% speedup and *better* accuracy than ungated recurrent memory, because unchecked memory growth actively degrades later updates. + +## Verified facts this plan depends on + +| Fact | Value | How known | +|---|---|---| +| Embedding dims | **768** | probed `/v1/embeddings` with `nomic-ai/nomic-embed-text-v2-moe` | +| pgvector in CNPG | **available, v0.7.0**, not yet installed | `pg_available_extensions` on `forgejo-db-2`, stock image `ghcr.io/cloudnative-pg/postgresql:16.2` | +| CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` | +| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` | +| Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** | +| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value | + +The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response. + +**Side fix:** `~/.pi/agent/models.json` declares `contextWindow: 131072` for ornith. Wrong — Ollama caps at 32768, so long prompts truncate silently. Correct to 32768. + +## The tier model + +Memory is **levelled**, and every event in the log carries its level. The paper has one flat memory; a project knowledge base needs three. + +| Level | What it is | Produced by | Bounded | +|---|---|---|---| +| **L0** | evidence chunk — the verbatim source span the update gate accepted | update gate opening at L1 | no, but sparse (~17 of 412 chunks) | +| **L1** | per-query memory — GRU-Mem's `M_t` for one standing question | gated loop over L0 chunk stream | 1024 tokens | +| **L2** | project synthesis — memory across the L1 memories of one project | gated loop over L1 memories | 1024 tokens | + +**The tiering is not new machinery.** L2 is the same controller, same prompt, same two gates — run with the L1 memories as its chunk stream and a project-level question. The recurrence is the algorithm applied to its own output, so `mem-core` implements one loop and the level is a parameter. Two consequences worth having on purpose: + +- **Exit gate flips by level.** Off at L1 (see below), reasonably *on* at L2, where the input is a handful of memories rather than hundreds of chunks and "enough evidence" is actually decidable. +- **Levels form a provenance graph, not a pile.** Each L1 node records the L0 nodes that produced it; each L2 node records its L1 parents. That graph *is* the Obsidian link structure and the `parent_id` edges in Postgres — one relationship expressed in both projections. + +## Architecture + +``` +pi sessions / claude transcripts / loop.sh artifacts + │ + ▼ project resolver (cwd -> project id) + [ Chunk Parser ] 5000-token chunks, split on message boundaries + │ + ▼ +┌──────────────────────────────────────────────────────┐ +│ GRU-Mem Controller qwen2.5:3b-instruct │ +│ reason about chunk vs question │ +│ yes|no -> update gate U_t │ +│ candidate memory M̂_t │ +│ continue|end -> exit gate E_t │ +└───────────────┬──────────────────────────────────────┘ + U_t=yes │ U_t=no + ┌───────┴────────┐ + ▼ ▼ + emit L0 evidence discard chunk + M_t <- M̂_t M_t <- M_{t-1} + │ + ▼ L1 memory per standing query + └──────► same loop, input = L1 memories ──► L2 project synthesis + │ + ▼ +┌──────────────────────────────────────────────────────┐ +│ JSONL event log AUTHORITATIVE │ +│ Obsidian vault derived projection │ +│ pgvector index derived projection │ +└──────────────────────────────────────────────────────┘ +``` + +**Authority model — the load-bearing decision.** The JSONL log is the only source of truth. The vault and the vector index are projections that must be droppable and rebuildable byte-identically from the log. This is poimen's own §1 principle ("nothing derived is authoritative; if it cannot be dropped and rebuilt, it has hidden inputs and that is a bug") applied here, and it buys three things: re-embedding after a model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, and the post-training corpus is the log itself. + +## Standing queries + +The paper's memory agent is `φθ(Q, C_t, M_{t-1})` — **it requires a question**. The update gate is defined as "does this chunk contain useful information *about the problem*". Without `Q` the gate has no referent, and `r_update` becomes undefinable, which forecloses post-training. + +So each project declares durable questions in YAML. One query = one L1 memory = one Obsidian note. + +```yaml +# queries/poimen.yaml +project: poimen +sources: + - pi:--Users-rockliang-workplace-Poimen-agent-rust-- + - claude:-Users-rockliang-workplace-Poimen +queries: + - id: architecture-decisions + question: What architectural decisions were made, with reasoning and rejected alternatives? + - id: infra-root-causes + question: What infrastructure bugs were found, what was the root cause, how was it isolated? + - id: open-questions + question: What questions were raised and left unresolved? +synthesis: # the L2 pass + question: What is the current state of this project, and what should someone know before working on it? + exit_gate: true +``` + +**Exit gate defaults off at L1.** Paper §3.3 makes this call itself: for "what are *all* the X" questions you cannot know evidence is sufficient without reading everything, so they provide a w/o-EG inference mode. L1 extraction is exactly that shape. Keep the gate *recorded* (its signal is needed for RL) but do not act on it. Enable at L2 and for interactive retrieval, where the paper measures 4× speedup. + +## Separate weights — yes, specifically a LoRA adapter + +Confirming the instinct, with the reason that actually matters here: + +- **Base:** Qwen2.5-3B-Instruct, already resident as `qwen2.5:3b-instruct` +- **Memory policy:** LoRA adapter, rank 16–32, ~30–60 MB + +Why an adapter rather than a fine-tuned model: + +1. **VRAM.** One GPU, `OLLAMA_MAX_LOADED_MODELS=2`, currently holding `ornith:35b` + `qwen2.5:3b`. A separate full memory model evicts something, and eviction here is a weights reload measured in tens of seconds — we already watched `ornith` cold-start blow a 60s gateway timeout. A LoRA rides on the resident base for near-zero extra VRAM. +2. **Iteration.** Post-training emits a ~50 MB adapter, not a 6 GB model. Swap without redeploying. +3. **Reversibility.** A gate-behaviour regression reverts by pointing at the previous adapter. + +**Infra consequence to accept up front:** Ollama cannot hot-swap LoRA adapters. Serving one means either moving the memory model to a **vLLM** InferenceService with `--enable-lora` (the `reasoning` predictor is already vLLM v0.11.0, so the pattern exists), or merging into a GGUF for Ollama and losing swappability. P1–P4 run prompted-only on the stock model, so this is deferred, not dodged. + +## Repository layout + +Rust workspace at `Poimen/memory`: + +``` +Cargo.toml workspace +crates/ + mem-core/ domain types; Level enum; gate-response parser; the gated loop + mem-chunk/ RecordSource trait; chunking policy; flush triggers (see below) + mem-llm/ gateway client — chat completions, embeddings, rerank + mem-ingest/ source adapters: pi sessions, claude transcripts, loop.sh artifacts + mem-store/ JSONL log writer/reader; pgvector repo; Obsidian projector + mem-cli/ binary `mem`: ingest | synthesize | rebuild | query | verify | skill | label +memory-tasks/ the task board — INDEX.md + one file per task, tracked +queries/ standing query YAML, one file per project +vault/ Obsidian output (own git repo or gitignored) +log/ JSONL event log — authoritative, tracked +``` + +Crates: `sqlx` (postgres, runtime-tokio-rustls) + `pgvector` (sqlx feature), `tokenizers` for chunk sizing against the real Qwen2 tokenizer, `futures`, `serde`/`serde_json`, `clap`, `reqwest`. + +Reuse rather than reinvent: `mem-llm`'s request shape and the `apikey` header convention are proven in `agent-rust/loop.sh`; response parsing mirrors its `extract_text()`. + +### `mem-chunk` — its own crate, stream-shaped from day one + +Chunking is split out because it is the seam where new input kinds arrive. Sources today are files with an EOF; telemetry, a live session tail, or a broker will not have one. Designing the boundary as a stream now means a future source implements a trait rather than forcing the loop to be rewritten — and it costs nothing, since the rest of the stack is already tokio (`sqlx` runtime-tokio, `reqwest`). + +```rust +// mem-chunk +pub trait RecordSource { + /// Normalised records: role, text, timestamp, provenance. Sources decide + /// how to produce them; the chunker never learns about pi vs claude vs a socket. + fn records(self) -> impl Stream>; +} + +pub struct ChunkPolicy { + pub max_tokens: usize, // 5000, paper default + pub split_on: Boundary, // never mid-message + pub flush: FlushTrigger, // see below +} + +pub fn chunks(src: S, p: ChunkPolicy) -> impl Stream; +``` + +Batch sources become streams for free via `futures::stream::iter`, so `mem-ingest` gets no more complex today. + +**The one thing that genuinely differs for streams is the flush trigger.** A file chunker emits a partial chunk at EOF; a stream has no EOF, so a partially-filled chunk would sit forever. `FlushTrigger` is therefore `Tokens(n)` today and gains `OrIdle(Duration)` when a stream source lands — carrying it in the policy now means the later change is one enum variant, not a signature change through the loop. + +Worth noting for whenever that happens: the **exit gate changes meaning on an unbounded source**. At L1 over a finite transcript it is switched off because "read everything" is well defined (paper §3.3). Over a live stream there is no everything, so the gate stops being an optimisation and becomes the only termination condition — which is an argument for keeping it trained even while it is switched off. + +## Storage schemas + +### JSONL event log — authoritative + +`log///.jsonl`. **Every record carries `level`.** + +```jsonl +{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."} +{"type":"chunk","level":"L0","t":1,"source":"pi:2026-07-21T16-23-59_019f857d","span":[0,42],"sha256":"..."} +{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812} +{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."} +{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":[""],"sha256":"..."} +{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."} +``` + +The L2 pass writes the same record types with `"level":"L2"`, `"input_level":"L1"`, and `parents` holding L1 shas. `evidence` records appear only when the update gate opened, so update-rate is directly measurable and memory at any `t` is replayable. + +### pgvector — projection + +One table across all levels, because retrieval wants to search them together and filter: + +```sql +CREATE TABLE memory_node ( + id BIGSERIAL PRIMARY KEY, + level TEXT NOT NULL CHECK (level IN ('L0','L1','L2')), + project TEXT NOT NULL, + query_id TEXT, -- null at L2 + run_id TEXT NOT NULL, + t INT NOT NULL, + source TEXT, -- set at L0 + text TEXT NOT NULL, + sha256 TEXT NOT NULL UNIQUE, + embedding vector(768) NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE TABLE memory_edge ( -- provenance: child <- parent + child_sha TEXT NOT NULL REFERENCES memory_node(sha256), + parent_sha TEXT NOT NULL REFERENCES memory_node(sha256), + PRIMARY KEY (child_sha, parent_sha) +); +CREATE INDEX ON memory_node USING hnsw (embedding vector_cosine_ops); +CREATE INDEX ON memory_node (project, level); +``` + +```mermaid +erDiagram + MEMORY_NODE ||--o{ MEMORY_EDGE : "child_sha -> sha256" + MEMORY_NODE ||--o{ MEMORY_EDGE : "parent_sha -> sha256" + + MEMORY_NODE { + bigserial id PK + text level "L0 | L1 | L2" + text project + text query_id "NULL at L2" + text run_id + int t + text source "set at L0" + text text + text sha256 UK "content identity" + vector_768 embedding + timestamptz created_at + } + + MEMORY_EDGE { + text child_sha PK,FK + text parent_sha PK,FK + } +``` + +One table across L0/L1/L2 (not three) — retrieval searches all levels together and filters by `level`. `memory_edge` is the provenance graph: L1 rows point back at the L0 chunks that produced them, L2 rows point back at L1 parents. `sha256 UNIQUE` is content identity (dedup key, hash excludes run id/timestamp — M0.2) and is what `memory_edge` actually references, not the surrogate `id`. + +Retrieval: HNSW recall filtered by level, then `bge-reranker-base` via `/v1/rerank` for precision — that endpoint scored 0.98 vs 0.00009 on a discrimination probe, so it earns its place. Default query searches L1+L2 and walks `memory_edge` down to L0 for citations. + +Infra — new CNPG cluster with the extension managed declaratively (CNPG 1.30 supports this, so **no manual `psql`**, consistent with the GitOps hard rule). Follows `k8s/infra/databases/temporal-db.yaml` exactly: + +```yaml +# k8s/infra/databases/memory-db.yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: { name: memory-db, namespace: memory } +spec: + instances: 3 + imageName: ghcr.io/cloudnative-pg/postgresql:16.2 + bootstrap: { initdb: { database: memory, owner: app, encoding: UTF8, localeCollate: C, localeCType: C } } + enableSuperuserAccess: false + storage: { size: 10Gi, storageClass: longhorn-cnpg } + monitoring: { enablePodMonitor: true } + affinity: + podAntiAffinityType: preferred + tolerations: [{ key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }] +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Database +metadata: { name: memory-db-vector, namespace: memory } +spec: + name: memory + owner: app + cluster: { name: memory-db } + extensions: [{ name: vector, ensure: present }] +``` + +### Obsidian vault — projection + +The tier graph becomes the note graph: + +``` +vault/poimen/ + index.md <- L2 synthesis, links to every L1 note + infra-root-causes.md <- L1 + architecture-decisions.md <- L1 + evidence/ <- L0, optional (--emit-evidence-notes, default off) +vault/skills/ + _drafts//SKILL.md <- machine-written, never auto-loaded + /SKILL.md <- human-promoted, loadable +``` + +```markdown +--- +project: poimen +level: L1 +query_id: infra-root-causes +updated: 2026-08-17 +chunks_seen: 412 +chunks_used: 17 +--- +# Infra root causes — poimen + + + +## Provenance +- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer +``` + +L0 defaults to inline citations rather than notes — 17 per query is manageable but grows unbounded across projects. The flag exists for when the graph view is worth the file count. + +## Skills — the procedural projection + +**A skill is a projection, not a level.** L0/L1/L2 are all *descriptive* — what happened. A skill is *procedural* — what to do next time. That is a change of modality, not a further compression, so the gated loop does not produce it: the update gate's question ("does this chunk contain evidence for Q") has no meaning when the output is an instruction. + +The format is free. `SKILL.md` is YAML frontmatter plus markdown, which is exactly an Obsidian note — verified against `~/.claude/skills/seo-geo-claude-skills/research/keyword-research/SKILL.md`, whose frontmatter carries `name`, `description`, `when_to_use`, `argument-hint`. So `vault/skills//SKILL.md` is simultaneously a vault note and a loadable skill, with no conversion step: + +```sh +pi --skill vault/skills/ # or set skillsPath in ~/.pi/config.json +ln -s .../vault/skills/ ~/.claude/skills/ +``` + +`mem skill draft --from poimen/infra-root-causes` reads an L1 or L2 note and writes a draft. Emission should follow the existing authoring rubric rather than inventing one — the installed `grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and a four-dimension rubric (conciseness, actionability, workflow clarity, progressive disclosure). Description quality is the whole game: a skill whose `description` does not match how the user actually phrases the request never fires. + +**Drafts are never auto-loaded, and promotion is manual.** This is the one place the system can close a loop on itself, and the failure is subtle: + +> a memory-derived skill is auto-loaded → it appears in future session transcripts → those transcripts are ingested as evidence → the memory that produced the skill is reinforced by its own output + +No external verifier breaks that cycle. It is the same hazard poimen §16 names when it gates "automatic workflow mutation without human approval" by default, and it is why `_drafts/` is a separate directory rather than a frontmatter flag — a directory cannot be accidentally globbed into `--skill`. + +Two mechanical guards: + +1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff. +2. **Provenance marks derived text.** Every emitted skill carries `generated_from: ` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data. + +## Phases + +**P1 — Read-only spine.** `mem-ingest` implements `RecordSource` for pi sessions and Claude transcripts; `mem-chunk` chunks to 5000 tokens on message boundaries; `mem ingest --dry-run` prints the chunk plan with no model calls. Both sources are batch, but they go through the stream interface so the seam is exercised from the first commit rather than retrofitted. + +**P2 — Gated loop at L1, prompted only.** `mem-llm` + gate-response parser (paper Figure 10a prompt, adapted for standing queries). Writes the JSONL log with L0 evidence and L1 memory records. Exit gate recorded, not acted on. This is the paper's "w/o RL" baseline, which Figure 9 shows already works. + +**P3 — Projections.** Obsidian projector and pgvector repo, both rebuildable via `mem rebuild --from-log`. Infra commit for `memory-db`. + +**P4 — L2 synthesis and retrieval.** `mem synthesize` runs the same loop over L1 memories with the exit gate on. `mem query` — embed, HNSW recall by level, rerank, return with provenance walked through `memory_edge`. + +**P5 — Skill drafting.** `mem skill draft --from ` emits `vault/skills/_drafts//SKILL.md` with `generated_from` provenance; `mem-ingest` grows the `derived: true` exclusion filter. Promotion stays manual. Cheap to build and it is the phase that makes the memory *do* something rather than only be read. + +**P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above. + +## Task breakdown + +Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M.` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id. + +``` +memory-tasks/ + INDEX.md board + phase order + progress mirror + M0.1-cargo-workspace.md + M0.2-domain-types.md + ... + M5.6-m5-gate.md +``` + +Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole board there. Naming this `memory-tasks/` sidesteps that pattern — and it should be **tracked**, since the task files carry the acceptance criteria. + +**M0 — Read-only spine** (no model calls anywhere in this phase) + +| id | task | size | deps | +|---|---|---|---| +| M0.1 | Cargo workspace + six crate skeletons, CI builds clean | S | — | +| M0.2 | `mem-core` domain types: `Level`, `Record`, `Chunk`, `MemoryNode`, sha256 identity | S | M0.1 | +| M0.3 | `mem-chunk`: `RecordSource` trait, `ChunkPolicy`, `FlushTrigger::Tokens` | M | M0.2 | +| M0.4 | Tokenizer-backed sizing against the real Qwen2 tokenizer | M | M0.3 | +| M0.5 | `mem-ingest`: pi session adapter (`~/.pi/agent/sessions//*.jsonl`) | M | M0.3 | +| M0.6 | `mem-ingest`: claude transcript adapter (`~/.claude/projects/**/.jsonl`) | S | M0.5 | +| M0.7 | `mem ingest --dry-run` — chunk plan, token histogram, source breakdown | S | M0.4, M0.6 | +| M0.8 | **M0 gate** — both adapters through one `RecordSource`, no source-specific code past the trait | M | gate | + +**M1 — Gated loop at L1** + +| id | task | size | deps | +|---|---|---|---| +| M1.1 | `mem-llm` chat client — `apikey` header, retry, timeout | M | M0.1 | +| M1.2 | Standing-query YAML loader + schema validation, unresolved id fails at load | M | M0.2 | +| M1.3 | GRU-Mem prompt template (paper Fig 10a), memory + chunk + question assembly | M | M1.2 | +| M1.4 | Gate-response parser: `///`, strict, malformed = hard error | M | M1.3 | +| M1.5 | The gated loop — `U_t` mutate-or-retain, `E_t` recorded not acted on, 1024-token budget | L | M1.4 | +| M1.6 | JSONL event log writer, `level` on every record, `parents` on memory | M | M1.5 | +| M1.7 | `mem ingest` end to end + update-rate reported on stdout | M | M1.6 | +| M1.8 | **M1 gate** — full run on `poimen`, update-rate < 30%, memory tokens flat not climbing | M | gate | + +**M2 — Projections** + +| id | task | size | deps | +|---|---|---|---| +| M2.1 | `mem-llm` embeddings client, 768-dim, batched | S | M1.1 | +| M2.2 | `k8s/infra/databases/memory-db.yaml` — CNPG Cluster + Database with `extensions: [vector]` | M | — | +| M2.3 | `memory_node` / `memory_edge` schema + sqlx migrations, HNSW indexes | M | M2.2 | +| M2.4 | `mem-store` pgvector repo — upsert by sha, edge insert | M | M2.3, M2.1 | +| M2.5 | Obsidian projector — frontmatter, wikilinks, L0 citations, `--emit-evidence-notes` | M | M1.6 | +| M2.6 | `mem rebuild --from-log` — drop and rebuild both projections | M | M2.4, M2.5 | +| M2.7 | `mem verify` — every L1 has ≥1 L0 parent, every parent sha resolves | S | M2.6 | +| M2.8 | **M2 gate** — rebuild is byte-identical (`git -C vault diff --exit-code` empty) | M | gate | + +**M3 — L2 synthesis and retrieval** + +| id | task | size | deps | +|---|---|---|---| +| M3.1 | L2 pass — same loop, input `Stream`, exit gate **on** | M | M1.5 | +| M3.2 | `mem-llm` rerank client (`bge-reranker-base`) | S | M1.1 | +| M3.3 | `mem query` — embed, HNSW recall filtered by level, rerank, walk edges to L0 | M | M3.2, M2.4 | +| M3.4 | **M3 gate** — known-answer query returns the right L1 node with a real L0 citation | M | gate | + +**M4 — Skills** + +| id | task | size | deps | +|---|---|---|---| +| M4.1 | `mem skill draft --from ` → `_drafts/`, frontmatter incl. `generated_from` | M | M3.1 | +| M4.2 | `derived: true` ingest filter — emitted artifacts excluded from evidence | M | M4.1, M0.5 | +| M4.3 | **M4 gate** — draft absent from `--list-skills`; no L0 node matches an emitted artifact | M | gate | + +**M5 — Post-training** (Python, separate from the Rust workspace; boundary is the JSONL) + +| id | task | size | deps | +|---|---|---|---| +| M5.1 | `mem label` — 32B `reasoning` as offline evidence labeler, writes `U_t` ground truth | M | M1.6 | +| M5.2 | Labeler calibration — hand-label a holdout, measure agreement before trusting it | M | M5.1 | +| M5.3 | Training corpus export from the log to verl's expected format | M | M5.1 | +| M5.4 | vLLM InferenceService for Qwen2.5-3B with `--enable-lora` (GitOps, homelab) | L | — | +| M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 | +| M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate | + +Total 38 tasks, 6 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. + +## Verification + +```bash +# P1 — corpus parses, chunk plan sane, zero model calls +cargo run -p mem-cli -- ingest --project poimen --dry-run + +# P2 — one query end to end +cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes +jq -r 'select(.type=="gate" and .level=="L1") | .update' log/poimen/infra-root-causes/*.jsonl | sort | uniq -c +# expect: far more false than true. Update-rate > ~30% means the gate is not +# discriminating — that is the paper's memory-explosion failure, Figure 6 is +# the reference shape. +jq -r 'select(.type=="memory") | .tokens' log/.../*.jsonl | tail -1 +# expect: <= 1024 and roughly flat over t, not monotonically climbing + +# levels are well-formed and edges close +jq -r '.level' log/poimen/**/*.jsonl | sort | uniq -c # L0/L1 present +cargo run -p mem-cli -- verify --project poimen +# asserts: every L1 memory has >=1 L0 parent; every parent sha exists + +# P3 — projections truly derived +cargo run -p mem-cli -- rebuild --from-log --project poimen +git -C vault diff --exit-code # empty: rebuild is byte-identical +psql -c "select level, count(*) from memory_node group by level;" + +# P4 — synthesis and retrieval +cargo run -p mem-cli -- synthesize --project poimen # expect exit gate to fire +cargo run -p mem-cli -- query "why did requests over 10KB fail?" +# expect: infra-root-causes L1 node, Kong body-buffer passage, L0 citation + +# P5 — skill drafts land unloadable, and the cycle stays open +cargo run -p mem-cli -- skill draft --from poimen/infra-root-causes +ls vault/skills/_drafts/ # draft here, NOT in vault/skills/ +pi --skill vault/skills/ --list-skills # draft must not appear +cargo run -p mem-cli -- verify --derived-filter --project poimen +# asserts: no L0 evidence node text matches an emitted skill artifact +``` + +The decisive P2 metric is **update-rate**, the one number distinguishing a working gate from an expensive summarizer. toolResults are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks. + +## Risks + +- **3B gate quality unmeasured on this corpus.** The paper evaluates on QA benchmarks with clean evidence labels; agent transcripts are messier. Mitigation: P2's update-rate is a cheap early read, and the 32B `reasoning` model can spot-audit a sample before committing to P5. +- **L2 inherits L1's errors with no path back to source.** Synthesis over memories cannot recover evidence the L1 gate wrongly discarded. `memory_edge` makes the omission *visible* (an L1 note with suspiciously few parents) but not recoverable without a re-run. +- **Self-reinforcement through skills.** The only cycle in the system: emitted skill → future session context → ingested as evidence → reinforces the memory that emitted it. Guarded by manual promotion plus the `derived: true` ingest filter, and both must hold. Audit it by checking that no L0 evidence node's text matches an emitted artifact. +- **No ground-truth evidence labels.** `r_update` needs them. Distant supervision from the 32B labeler inherits its bias; hold out a hand-labelled set to measure agreement before trusting it. +- **Vault/log divergence.** Hand edits are overwritten on rebuild. Either make the vault read-only or add an `## Notes` region the projector preserves. Decide before anyone starts editing. +- **Ollama has no LoRA path.** P5 forces the vLLM decision. Do not discover this at P5. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4e7e773 --- /dev/null +++ b/README.md @@ -0,0 +1,226 @@ +# poimen-memory + +Gated recurrent memory over agent context. Reads session history chunk-by-chunk, +keeps only what answers standing questions, projects result into an Obsidian +vault and a pgvector index. + +**Status: design complete, no code yet.** 37 tasks in [memory-tasks/](memory-tasks/INDEX.md), +0 done. Start at [M0.1](memory-tasks/M0.1-cargo-workspace.md). + +## Problem + +Agent sessions grow faster than anyone reads them, and most of the volume is +noise. One real pi session in this project: + +``` +assistant 1445 +toolResult 1261 43% — ls output, file reads, mostly evidence-free +user 196 ++ 8 compaction events +``` + +Compaction fires 8 times per session. Context gets *discarded*, not retained — +root causes, decisions and gotchas evaporate when window rolls. + +## Mechanism + +GRU-Mem ([arXiv 2602.10560](https://arxiv.org/abs/2602.10560)). Two text-controlled +gates on a recurrent memory loop: + +- **update gate** — memory only mutates when chunk contains evidence. Blocks the + memory explosion that ungated recurrent memory hits. +- **exit gate** — stop scanning once evidence sufficient. + +Paper reports up to 400% speedup and *better* accuracy than ungated, because +unbounded memory growth degrades later updates. + +``` +sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections +``` + +Controller emits structured output; loop acts on it: + +``` + reason about chunk vs question + yes|no -> U_t, update or discard + candidate memory M̂_t + continue|end -> E_t, exit or continue +``` + +## Memory tiers + +| Level | What | From | Bounded | +|---|---|---|---| +| **L0** | evidence chunk, verbatim | update gate opening | no, but sparse (~17 of 412) | +| **L1** | per-query memory, `M_t` | gated loop over chunks | 1024 tok | +| **L2** | project synthesis | gated loop over L1 memories | 1024 tok | + +L2 is not new machinery — same loop, same prompt, L1 memories as input stream. +Level is a parameter. + +Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1 +parents. Same relation becomes both `memory_edge` rows and Obsidian wikilinks. + +## Standing queries + +Update gate needs a referent. Paper's agent is `φθ(Q, C_t, M_{t-1})` — gate is +defined as "does this chunk contain useful information *about the problem*". No +`Q`, no gate, and `r_update` becomes undefinable, which kills post-training. + +So each project declares durable questions. One query = one L1 memory = one note. + +```yaml +# queries/poimen.yaml +project: poimen +roots: [/Users/rockliang/workplace/Poimen/agent-rust] +queries: + - id: infra-root-causes + question: What infrastructure bugs were found, what was the root cause, how was it isolated? + - id: architecture-decisions + question: What architectural decisions were made, with reasoning and rejected alternatives? +synthesis: + question: What is the current state of this project, and what should someone know before working on it? + exit_gate: true +``` + +**Exit gate off at L1, on at L2.** Paper §3.3 makes this call: for "what are *all* +the X" questions you cannot know evidence is sufficient without reading +everything. L1 extraction is that shape. At L2 input is a handful of memories and +sufficiency is decidable. Gate still *recorded* at L1 — signal needed for +post-training. + +## Authority model + +**JSONL log authoritative. Vault and vector index are projections.** + +Anything not rebuildable byte-identically from the log has hidden inputs, and +that is a bug. Gate M2.8 enforces it destructively: + +```sh +rm -rf vault/poimen +psql -c "delete from memory_node where project='poimen'" +mem rebuild --from-log --project poimen +git -C vault diff --exit-code # empty diff is the only pass +``` + +Buys three things: re-embedding after model change is a rebuild not a migration, +Obsidian edits cannot corrupt the record, post-training corpus is the log itself. + +## Skills + +A skill is a **projection, not a level**. L0/L1/L2 are descriptive — what +happened. A skill is procedural — what to do next time. Gated loop does not +produce it. + +Format free: `SKILL.md` is YAML frontmatter + markdown, which is an Obsidian +note. So `vault/skills//SKILL.md` is both, no conversion: + +```sh +pi --skill vault/skills/ +ln -s .../vault/skills/ ~/.claude/skills/ +``` + +**Drafts land in `_drafts/`, promotion is a human `git mv`.** This is the one +cycle in the design: + +``` +emitted skill auto-loads -> appears in future transcripts +-> ingested as evidence -> reinforces the memory that emitted it +``` + +No external verifier breaks it. Two guards: `_drafts/` is a directory (cannot be +globbed into `--skill`), and every artifact carries `generated_from` so ingest +tags matching chunks `derived: true` and refuses them as evidence. + +## Separate weights + +Memory policy is a **LoRA adapter** on Qwen2.5-3B-Instruct, not a fine-tuned +model. Reason is VRAM: one GPU, `OLLAMA_MAX_LOADED_MODELS=2`, already holding +`ornith:35b` + `qwen2.5:3b`. Separate full model evicts something, and eviction +is a weights reload measured in tens of seconds. Adapter rides the resident base. + +Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression +reverts by pointing at previous adapter. + +**Ollama cannot hot-swap LoRA.** Serving one needs vLLM with `--enable-lora` +(pattern already exists — `reasoning` predictor is vLLM v0.11.0). Phases M0–M4 +run prompted-only, so decision is deferred, not dodged. + +## Layout + +``` +DESIGN.md full design, 460 lines +memory-tasks/ 37 task files + INDEX.md — tracked +crates/ + mem-core/ domain types; Level; gate parser; the gated loop + mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger + mem-llm/ gateway client — chat, embeddings, rerank + mem-ingest/ source adapters: pi sessions, claude transcripts + mem-store/ JSONL log; pgvector repo; Obsidian projector + mem-cli/ binary `mem` +queries/ standing query YAML per project +log/ JSONL event log — authoritative, tracked +vault/ Obsidian output +``` + +`mem-chunk` is separate and stream-shaped from day one. Sources today are files +with an EOF; telemetry or a live tail will not have one. `RecordSource` returns +`impl Stream`; batch sources become streams via +`futures::stream::iter`, so it costs nothing now and removes a rewrite later. + +## Commands + +```sh +mem ingest --project poimen --dry-run # chunk plan, zero model calls +mem ingest --project poimen --query infra-root-causes +mem synthesize --project poimen # L2 pass, exit gate on +mem rebuild --from-log --project poimen # drop and rebuild projections +mem verify --project poimen # provenance graph closure +mem query "why did requests over 10KB fail?" +mem skill draft --from poimen/infra-root-causes +mem label --project poimen # evidence labels for training +``` + +## Verified environment facts + +Checked against the running cluster, not assumed: + +| Fact | Value | +|---|---| +| Embedding dims | **768**, `nomic-ai/nomic-embed-text-v2-moe` | +| Embedding batch limit | **32** (`batch size 1200 > maximum allowed batch size 32`) | +| pgvector | **0.7.0 available in stock CNPG image**, no custom build | +| CNPG operator | **1.30.0**, declarative `Database.spec.extensions` | +| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) — cluster-side, overrides client config | +| Controller | `qwen2.5:3b-instruct` — paper's exact 3B backbone | +| Gateway auth | `apikey:` header. `Authorization: Bearer` returns **401** | +| Rerank response | bare array, not `{"data":[...]}`; sorted by score, map back via `index` | + +Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response. + +## Phases + +Each ends in a composition gate. No phase starts until predecessor gate is green. + +| | Phase | Tasks | Gate asserts | +|---|---|---|---| +| M0 | Read-only spine | 8 | third source needs no downstream change; runs offline | +| M1 | Gated loop at L1 | 8 | **update-rate < 30%**, memory flat not climbing | +| M2 | Projections | 8 | rebuild byte-identical from log alone | +| M3 | L2 + retrieval | 4 | hit rate ≥ 0.8, provenance precision ≥ 0.9 | +| M4 | Skills | 3 | draft not loadable; promoted skill never becomes evidence | +| M5 | Post-training | 6 | adapter beats prompted baseline on held-out project | + +**Update-rate is the number to watch.** It is what distinguishes a gate from an +expensive summarizer. Tool results are 43% of records and mostly evidence-free, +so a correct gate rejects the large majority of chunks. + +M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA) +is homelab work independent of the rest of M5. + +## Reading order + +1. This file +2. [memory-tasks/INDEX.md](memory-tasks/INDEX.md) — board, ordering rules, verification practice +3. [DESIGN.md](DESIGN.md) — full design, schemas, risks +4. Individual task files — self-contained, no DESIGN.md read required