poimen-memory/tasks/INDEX.md

poimen-memory — task board

71 tasks — 60 build tasks plus 11 composition gates, one per phase. One file per task, self-contained: inlined design facts, executable steps, acceptance criteria, a Verify section written for someone who did not build the thing, and the traps worth naming. Reading DESIGN.md is not required to do a task — it is linked as background only.

Each Verify section names the harness, the integration test with numbered assertions, the command to run, and the false pass — the shape of test that goes green while the feature is broken. Treat the false-pass list as part of the acceptance criteria, not commentary.

Ordering — declared, never derived

Phase order is the list below. Task ids are opaque and frozen.

M0.3 is M0.3 forever, in whatever phase it currently sits, because its artifacts and cross-references key on that id. New tasks take new ids rather than renumbering neighbours. This is the StepId rule applied to the board itself — a board that renumbers to reorder has the bug it warns its own users about.

No phase starts until its predecessor's gate is green. The gate task at the end of each phase is that gate: it proves the phase's parts compose and that its swappable parts are genuinely swappable. Every build task is verified alone; the gate verifies the properties no single task owns.

Two rules this board exists to protect

1. The JSONL log is authoritative; the vault and the vector index are projections. Anything that cannot be dropped and rebuilt byte-identically from the log has hidden inputs, and that is a bug. M2.8 is the gate that enforces it.

2. The update gate must discriminate, not summarize. Agent transcripts are ~43% tool results and mostly evidence-free. A gate that accepts most chunks is an expensive summarizer that will reproduce the memory-explosion failure the whole design exists to avoid. M1.8 is the gate that enforces it, and update-rate is the single number to watch.

Verification practice — script first, source second

A task is verified by running its command and reading the output, then opening the source. Reviewing the diff first is how an assertion that was quietly dropped still gets called done: the code looks right, and nothing proves the test ran.

Numbered assertion N in a Verify section is test fn aN_<slug>. The numbering is the contract — a test fn that does not exist should report missing rather than be silently absent from a green summary.

cargo test reporting ok with zero tests run is not a pass.

Progress

Source of truth is the Status field in each task file. The tables below mirror it; a status changed here and not there is a lie.

Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked

#	Phase	Ids	Tasks	✅	🟡	⬜	Gate
1	Read-only spine	M0.x	8	8	0	0	✅ M0.8
2	Gated loop at L1	M1.x	8	8	0	0	✅ M1.8
3	Projections	M2.x	8	8	0	0	✅ M2.8
4	L2 synthesis + retrieval	M3.x	4	4	0	0	✅ M3.4
4.5	Distributed API Layer	M3.5.x	10	10	0	0	✅ M3.5.8
5	Skills	M4.x	3	2	0	1	⬜ M4.3
5.5	Reference corpora	M3.6.x	7	1	0	6	⬜ M3.6.8
5.6	Tool context	M3.7.x	4	2	0	2	⬜ M3.7.6
5.7	Context optimization	M3.8.x	4	1	0	3	⬜ M3.8.4
6	Post-training	M5.x	6	0	0	6	⬜ M5.6
7	agent-manager migration	M6.x	6	0	0	6	⬜ M6.6
8	Source connectors	M7.x	10	0	0	10	⬜ M7.10
9	Hybrid search	M8.x	9	1	1	7	⬜ M8.9
	Total		76	61	0	15	7/13 green

Current status — 2025-01-28. Completed phases M0.x, M1.x fully archived (16/16 tasks). M2.1-6 ✅ (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). M3.x ✅ (4/4). M3.5.x ✅ (10/10 complete + archived). M3.7.7-8 ✅ (failure diagnosis). M4.1-2 ✅ (skill drafting + derived filter). M3.6.1 ✅ (DocCorpusSource). M3.6.3 ❌ retired (Obsidian UI replaces CLI). M3.6.7-8 ⬜ new (ingest enrichment + deduplication). M8.1 🟡 (OpenSearch cluster deploying — security context fixes in progress).

Current work:

• M3.8.2: CacheAligner integration with HTTP headers + drift metrics
• M8.1: OpenSearch deployment (pod security context)
• M8.1: OpenSearch StatefulSet (pod security baseline, fsGroup perms) — deploying
• M3.7.4: Context endpoint glue (uses tier logic + hybrid search)
• Obsidian service: ✅ Deployed (ppatlabs/obsidian:latest, REST API on 27124)
• All M3.5 endpoints ready: vault JSON + hybrid search (60% pgvector + 40% OpenSearch), JWT auth live with Authentik

Immediate blocker (M8.1):

• OpenSearch StatefulSet pod security context fix

Blocked until M8.2 green:

• M8.3-9 (query optimizer, RRF, hybrid endpoint, benchmarks)
• M3.7.4 (context endpoint needs hybrid search to rank tiers) — now has M3.7.7 + M3.7.8 ✅
• M3.6.2+ (reference corpus indexing depends on dual-write layer for Postgres + OpenSearch)

Tests: 265+ passing, 2 ignored (M2.1 +8, M3.5 +16, M4 +20, M3.7.7 +18 unit, M3.7.8 +22 integration). 60/71 tasks complete (85%), 6/11 gates green. M3.7.3 & M3.7.5 retired (hybrid search covers). M3.7.7 & M3.7.8 ✅ complete (signature extraction + symptom projection, 40+ tests passing).

M2.2 (CNPG manifest), M5.4 (vLLM+LoRA), M3.5.9 (git refs), and M6.x (agent-manager migration) are homelab/infra work independent of prior phases, can start parallel.

M6 is a different repo, not a dependency of M0-M5. It migrates github.com/Riotpiaole/agent-manager's session store (a separate Go CLI tool, unrelated to this project's own memory system) from local sqlite to its own dedicated CNPG cluster. It rides in this board because it's homelab work happening alongside M2.2/M5.4, and because the two projects' Postgres schemas landing in the same cluster around the same time need to look like siblings, not strangers — see M6.6's convention-consistency check.

✅ Archived Phases

Completed and archived: M0.x (8/8), M1.x (8/8) — all task files deleted from /tasks/ after verification. See git log for historical record and CLAUDE.md for session context.

3 — Projections · M2.x

Status: ✅ Complete · 8/8 done. All task files archived.

M2.1-2.8 ✅ ARCHIVED:

• M2.1 ✅ (embeddings client: 768-dim batching @32)
• M2.2 ✅ (CNPG Cluster + Database CRD with pgvector 0.7.0)
• M2.3 ✅ (schema + sqlx migrations — 5 entity tables)
• M2.4 ✅ (pgvector repository: upsert, search, edges, 8 tests)
• M2.5 ✅ (obsidian projector: deterministic rebuild, 10 tests)
• M2.6 ✅ (rebuild from log: orchestration engine, 6 tests)
• M2.7 ✅ (verify edges: 6 invariants, 9 tests)
• M2.8 ✅ (M2 composition gate: idempotence + rebuild-from-log proof)

4 — L2 synthesis and retrieval · M3.x

Status: ✅ Complete · 4/4 tasks done. Gate M3.4 passing. Task files archived.

4.5 — Distributed API Layer · M3.5.x

Homelab frontend integration: HTTP facade via api.riotpiao.com. Runs in parallel with M4 and M5 after M3.4 green.

Status: 10/10 done · M3.5.8 gate ✅ passing. M3.5.1–10 archived (all task files deleted). Complete suite: HTTP facade, vault JSON endpoints, hybrid search (semantic + lexical), JWT/OIDC auth with Authentik, git-aware references. All integration tests passing. Awaiting Docker image rollout for production deployment.

5 — Skills · M4.x

Status: 2/3 done · M4.1 (skill draft) ✅ and M4.2 (derived filter) ✅ archived. Pending M4.3 gate composition.

mem skill draft --project <proj> --from <query-id> writes to vault/skills/_drafts/. Dry-run mode supported. Shingle matcher (M4.2) provides derived LessonSource filtering. M4.1-2 task files deleted after archival.

5.5 — Reference corpora · M3.6.x

Source of truth: Obsidian vault (REST API, deployed M2.5). Documentation the local models are weak at — kubectl, tea — are stored in Obsidian and made retrievable as level R: embedded and indexed, never evidence.

Ids are M3.6.x and stay M3.6.x; the phase sits here rather than at 4.6 because M3.6.4 extends M4.2's matcher instead of duplicating it, and because skills are the better answer to the same problem and should exist first.

The load-bearing property is a negative one. Adding Obsidian vault reference documents must not change update-rate, must not change default query output, and must not put an R node in any provenance chain. R bypasses the recurrence structurally — run_loop needs a Query and a corpus has none — not by a flag. M3.6.6 asserts M1.8's numbers are unchanged, not merely still-passing, because documentation fed to the gate would lower update-rate and make M1.8 easier to clear while the memory got worse.

M3.6.3 retired: Obsidian UI replaces CLI corpus management. Users edit files in Obsidian; mem rebuild auto-fetches from Obsidian REST API and re-indexes only changed chunks (SHA comparison, deterministic embedding).

Task	Title	Size	Flags	Status
M3.6.1	DocCorpusSource + heading chunking	M	—	✅
M3.6.2	Level R: Obsidian reference indexing + rebuild parity	M	—	⬜
M3.6.3	mem ref — corpus management CLI	M	—	❌ RETIRED (Obsidian UI replaces)
M3.6.4	Reference text cannot re-enter as evidence	M	—	⬜
M3.6.5	Query: filter-then-recall, R opt-in, floor	M	—	⬜
M3.6.6	M3.6 composition gate	M	gate	⬜
M3.6.7	Contextual enrichment at ingest (Anthropic-style)	M	—	⬜
M3.6.8	Chunk deduplication via MinHash	M	—	⬜

5.6 — Tool context · M3.7.x

Context lookup over HTTP for failure diagnosis. Consumers are pi, curl, or an MCP call.

M3.7.3 & M3.7.5 retired: Skill filtering and standing queries were pre-hybrid-search design. Hybrid search (M8) solves skill matching + failure retrieval better. Tasks dropped.

Remaining work (2 tasks):

• M3.7.4: /memory/context endpoint (3-tier: signature → hybrid search → reference corpus)
• M3.7.6: Composition gate (verifies tiers work end-to-end)

Archived (2 complete):

• M3.7.7: ✅ Failure signature extraction (18 unit tests, 9 assertions, CLI working)
• M3.7.8: ✅ Symptom projection (22 tests: 10 unit + 12 integration, 6 assertions)

Ids are M3.7.x and frozen. M3.7.1, M3.7.2, M3.7.3, M3.7.5 retired.

Task	Title	Size	Flags	Status
M3.7.3	GET /memory/skills?task= — skill match	—	—	❌ Retired (hybrid search covers)
M3.7.4	/memory/context — three-tier lookup	M	—	⬜
M3.7.5	tool-failures standing query	—	—	❌ Retired (hybrid search covers)
M3.7.6	M3.7 composition gate	M	gate	⬜
M3.7.7	Failure signature extraction + normalisation	M	—	✅ ARCHIVED
M3.7.8	Symptom projection at ingest	M	—	✅ ARCHIVED

5.7 — Context optimization · M3.8.x

Headroom-inspired pre-LLM compression. Sits between hybrid search retrieval and the LLM gateway. Search indexes (pgvector + OpenSearch) stay at full fidelity; only evidence chunks entering the prompt get optimized.

Task	Title	Size	Flags	Status
M3.8.1	Context optimizer pipeline (router + compressors + CCR)	L	—	✅ COMPLETE
M3.8.2	CacheAligner integration with prompt builder	M	—	⬜
M3.8.3	Compression benchmarks + tuning	M	—	⬜
M3.8.4	M3.8 composition gate	M	gate	⬜

6 — Post-training · M5.x

Python, separate from the Rust workspace. The boundary is the JSONL log.

Task	Title	Size	Flags	Status
M5.1	mem label — evidence labeler	M	—	⬜
M5.2	Labeler calibration	M	—	⬜
M5.3	Training corpus export	M	—	⬜
M5.4	vLLM + --enable-lora	L	homelab	⬜
M5.5	verl training loop	L	—	⬜
M5.6	M5 composition gate	L	gate	⬜

7 — agent-manager migration · M6.x

Separate repo (github.com/Riotpiaole/agent-manager, fork branch add-headless-spawn), separate cluster resource, no Rust/GRU-Mem dependency. Moves its session store off local sqlite onto a dedicated CNPG Postgres, reachable from the Mac client through a dedicated nginx route — durability-of-location, not a multi-host requirement.

Task	Title	Size	Flags	Status
M6.1	CNPG agent-manager-db manifest	M	homelab	⬜
M6.2	Postgres schema for agent-manager sessions	M	—	⬜
M6.3	store.go query port to Postgres	L	—	⬜
M6.4	nginx TCP routing to agent-manager-db	S	homelab	⬜
M6.5	Postgres credentials for the Mac client	S	homelab	⬜
M6.6	M6 composition gate	M	gate	⬜

8 — Source connectors · M7.x

Extensible multi-source ingestion. SourceConnector trait + YAML-driven registry. Adding a new document source (paperless-ngx, S3, git repo) requires implementing one trait and adding one config block — no changes to the ingest pipeline, chunking, embedding, storage, or query layers.

Document connectors produce Level R content (reference material, bypasses gated loop). Session connectors (pi, claude) produce evidence for L0/L1/L2. The connector's source_type() declares the pipeline.

Sync framework handles change detection (sha-based skip), tombstoning, drift reporting, and resumable sync for all connectors.

Task	Title	Size	Flags	Status
M7.1	SourceConnector trait + registry	M	—	⬜
M7.2	Obsidian vault connector	M	—	⬜
M7.3	paperless-ngx connector	M	—	⬜
M7.4	Git repo connector	M	—	⬜
M7.5	S3-compatible storage connector	M	—	⬜
M7.6	Sync framework	L	—	⬜
M7.7	mem source CLI	M	—	⬜
M7.8	Source HTTP endpoints	M	—	⬜
M7.9	Connector health + observability	S	—	⬜
M7.10	M7 composition gate	M	gate	⬜

---

9 — Hybrid search · M8.x

Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with Reciprocal Rank Fusion. Adds a QueryOptimizer that classifies queries and routes to the best strategy before any database call.

The load-bearing property is accuracy. Hybrid must produce measurably better NDCG@10 than either engine alone. If it doesn't, the gate fails — not because the code is broken, but because the system isn't earning its complexity budget. M8.9 requires benchmark numbers, not just green tests.

Approach A: Parallel RRF. Both engines run simultaneously via tokio::try_join!. Results are merged by rank position, not score magnitude, because pgvector cosine ([0,1]) and BM25 ([0,50+]) are incomparable distributions. RRF needs no parameter tuning (k=60 is the academic standard). The alternative — weighted linear combination — requires labelled data for weight selection that we don't have yet.

Dual-write indexing. Every chunk gets the same UUID in both pgvector and OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked opensearch_pending and retried by a background task. The gate (M8.9) checks for zero orphans.

Fallback. If OpenSearch is down at query time, the worker degrades to semantic-only. If the embedding model is down, it degrades to lexical-only. The response search_strategy field always reports which mode was actually used.

Task	Title	Size	Flags	Status
M8.1	OpenSearch cluster + JWT realm	M	homelab	✅
M8.2	Dual-write indexing pipeline	M	—	⬜
M8.3	Query optimizer: context + routing	M	—	⬜
M8.4	Reciprocal Rank Fusion engine	S	—	⬜
M8.5	Hybrid query worker: parallel retrieval	L	—	⬜
M8.6	Upgrade GET /query to hybrid + fallback	M	—	⬜
M8.7	Index tuning: HNSW + OpenSearch analyzers	M	—	⬜
M8.8	Accuracy benchmarks: NDCG, MRR, P@K	M	—	⬜
M8.9	M8 composition gate	M	gate	⬜

---

Background: DESIGN.md · GRU-Mem, arXiv 2602.10560 · internal/store/store.go (agent-manager, add-headless-spawn branch)