poimen-memory

Gated recurrent memory over agent context. Reads session history chunk-by-chunk, keeps only what answers standing questions, projects result into an Obsidian vault and a pgvector index.

Status: design complete, no code yet. 37 tasks in memory-tasks/, 0 done. Start at M0.1.

Problem

Agent sessions grow faster than anyone reads them, and most of the volume is noise. One real pi session in this project:

assistant   1445
toolResult  1261     43% — ls output, file reads, mostly evidence-free
user         196
+ 8 compaction events

Compaction fires 8 times per session. Context gets discarded, not retained — root causes, decisions and gotchas evaporate when window rolls.

Mechanism

GRU-Mem (arXiv 2602.10560). Two text-controlled gates on a recurrent memory loop:

• update gate — memory only mutates when chunk contains evidence. Blocks the memory explosion that ungated recurrent memory hits.
• exit gate — stop scanning once evidence sufficient.

Paper reports up to 400% speedup and better accuracy than ungated, because unbounded memory growth degrades later updates.

sessions ─> chunk (5000 tok) ─> controller ─> gates ─> memory ─> projections

Controller emits structured output; loop acts on it:

<think>   reason about chunk vs question
<check>   yes|no        -> U_t, update or discard
<update>  candidate memory M̂_t
<next>    continue|end  -> E_t, exit or continue

Memory tiers

Level	What	From	Bounded
L0	evidence chunk, verbatim	update gate opening	no, but sparse (~17 of 412)
L1	per-query memory, M_t	gated loop over chunks	1024 tok
L2	project synthesis	gated loop over L1 memories	1024 tok

L2 is not new machinery — same loop, same prompt, L1 memories as input stream. Level is a parameter.

Tiers form a provenance graph. Each L1 records its L0 parents, each L2 its L1 parents. Same relation becomes both memory_edge rows and Obsidian wikilinks.

Standing queries

Update gate needs a referent. Paper's agent is φθ(Q, C_t, M_{t-1}) — gate is defined as "does this chunk contain useful information about the problem". No Q, no gate, and r_update becomes undefinable, which kills post-training.

So each project declares durable questions. One query = one L1 memory = one note.

# queries/poimen.yaml
project: poimen
roots: [/Users/rockliang/workplace/Poimen/agent-rust]
queries:
  - id: infra-root-causes
    question: What infrastructure bugs were found, what was the root cause, how was it isolated?
  - id: architecture-decisions
    question: What architectural decisions were made, with reasoning and rejected alternatives?
synthesis:
  question: What is the current state of this project, and what should someone know before working on it?
  exit_gate: true

Exit gate off at L1, on at L2. Paper §3.3 makes this call: for "what are all the X" questions you cannot know evidence is sufficient without reading everything. L1 extraction is that shape. At L2 input is a handful of memories and sufficiency is decidable. Gate still recorded at L1 — signal needed for post-training.

Authority model

JSONL log authoritative. Vault and vector index are projections.

Anything not rebuildable byte-identically from the log has hidden inputs, and that is a bug. Gate M2.8 enforces it destructively:

rm -rf vault/poimen
psql -c "delete from memory_node where project='poimen'"
mem rebuild --from-log --project poimen
git -C vault diff --exit-code      # empty diff is the only pass

Buys three things: re-embedding after model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, post-training corpus is the log itself.

Skills

A skill is a projection, not a level. L0/L1/L2 are descriptive — what happened. A skill is procedural — what to do next time. Gated loop does not produce it.

Format free: SKILL.md is YAML frontmatter + markdown, which is an Obsidian note. So vault/skills/<name>/SKILL.md is both, no conversion:

pi --skill vault/skills/
ln -s .../vault/skills/<name> ~/.claude/skills/<name>

Drafts land in _drafts/, promotion is a human git mv. This is the one cycle in the design:

emitted skill auto-loads -> appears in future transcripts
-> ingested as evidence -> reinforces the memory that emitted it

No external verifier breaks it. Two guards: _drafts/ is a directory (cannot be globbed into --skill), and every artifact carries generated_from so ingest tags matching chunks derived: true and refuses them as evidence.

Separate weights

Memory policy is a LoRA adapter on Qwen2.5-3B-Instruct, not a fine-tuned model. Reason is VRAM: one GPU, OLLAMA_MAX_LOADED_MODELS=2, already holding ornith:35b + qwen2.5:3b. Separate full model evicts something, and eviction is a weights reload measured in tens of seconds. Adapter rides the resident base.

Also: post-training emits ~50 MB, not 6 GB. Swap without redeploy. Regression reverts by pointing at previous adapter.

Ollama cannot hot-swap LoRA. Serving one needs vLLM with --enable-lora (pattern already exists — reasoning predictor is vLLM v0.11.0). Phases M0–M4 run prompted-only, so decision is deferred, not dodged.

Layout

DESIGN.md            full design, 460 lines
memory-tasks/        37 task files + INDEX.md — tracked
crates/
  mem-core/          domain types; Level; gate parser; the gated loop
  mem-chunk/         RecordSource trait; ChunkPolicy; FlushTrigger
  mem-llm/           gateway client — chat, embeddings, rerank
  mem-ingest/        source adapters: pi sessions, claude transcripts
  mem-store/         JSONL log; pgvector repo; Obsidian projector
  mem-cli/           binary `mem`
queries/             standing query YAML per project
log/                 JSONL event log — authoritative, tracked
vault/               Obsidian output

mem-chunk is separate and stream-shaped from day one. Sources today are files with an EOF; telemetry or a live tail will not have one. RecordSource returns impl Stream<Item = Record>; batch sources become streams via futures::stream::iter, so it costs nothing now and removes a rewrite later.

Commands

mem ingest --project poimen --dry-run          # chunk plan, zero model calls
mem ingest --project poimen --query infra-root-causes
mem synthesize --project poimen                # L2 pass, exit gate on
mem rebuild --from-log --project poimen        # drop and rebuild projections
mem verify --project poimen                    # provenance graph closure
mem query "why did requests over 10KB fail?"
mem skill draft --from poimen/infra-root-causes
mem label --project poimen                     # evidence labels for training

Verified environment facts

Checked against the running cluster, not assumed:

Fact	Value
Embedding dims	768, nomic-ai/nomic-embed-text-v2-moe
Embedding batch limit	32 (batch size 1200 > maximum allowed batch size 32)
pgvector	0.7.0 available in stock CNPG image, no custom build
CNPG operator	1.30.0, declarative Database.spec.extensions
Ollama context cap	32768 (OLLAMA_CONTEXT_LENGTH) — cluster-side, overrides client config
Controller	qwen2.5:3b-instruct — paper's exact 3B backbone
Gateway auth	apikey: header. Authorization: Bearer returns 401
Rerank response	bare array, not {"data":[...]}; sorted by score, map back via index

Budget fits the 32K cap: 5000 chunk + ~3200 prompt/memory + 2048 response.

Phases

Each ends in a composition gate. No phase starts until predecessor gate is green.

	Phase	Tasks	Gate asserts
M0	Read-only spine	8	third source needs no downstream change; runs offline
M1	Gated loop at L1	8	update-rate < 30%, memory flat not climbing
M2	Projections	8	rebuild byte-identical from log alone
M3	L2 + retrieval	4	hit rate ≥ 0.8, provenance precision ≥ 0.9
M4	Skills	3	draft not loadable; promoted skill never becomes evidence
M5	Post-training	6	adapter beats prompted baseline on held-out project

Update-rate is the number to watch. It is what distinguishes a gate from an expensive summarizer. Tool results are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks.

M0 and M2.2 need no model access and can start immediately. M5.4 (vLLM + LoRA) is homelab work independent of the rest of M5.

Reading order

1. This file
2. memory-tasks/INDEX.md — board, ordering rules, verification practice
3. DESIGN.md — full design, schemas, risks
4. Individual task files — self-contained, no DESIGN.md read required

Trigger build run 130

CI trigger