poimen-memory/tasks/INDEX.md

# poimen-memory — task board

64 tasks — 54 build tasks plus **10 composition gates**, one per phase. One file
per task, **self-contained**: inlined design facts, executable steps, acceptance
criteria, a `Verify` section written for someone who did not build the thing, and
the traps worth naming. Reading `DESIGN.md` is not required to do a task — it is
linked as background only.

Each `Verify` section names the harness, the integration test with numbered
assertions, the command to run, and the **false pass** — the shape of test that
goes green while the feature is broken. Treat the false-pass list as part of the
acceptance criteria, not commentary.

## Ordering — declared, never derived

**Phase order is the list below. Task ids are opaque and frozen.**

`M0.3` is `M0.3` forever, in whatever phase it currently sits, because its
artifacts and cross-references key on that id. New tasks take new ids rather than
renumbering neighbours. This is the `StepId` rule applied to the board itself —
a board that renumbers to reorder has the bug it warns its own users about.

No phase starts until its predecessor's gate is green. The `gate` task at the end
of each phase **is** that gate: it proves the phase's parts compose and that its
swappable parts are genuinely swappable. Every build task is verified alone; the
gate verifies the properties no single task owns.

## Two rules this board exists to protect

**1. The JSONL log is authoritative; the vault and the vector index are
projections.** Anything that cannot be dropped and rebuilt byte-identically from
the log has hidden inputs, and that is a bug. `M2.8` is the gate that enforces it.

**2. The update gate must discriminate, not summarize.** Agent transcripts are
~43% tool results and mostly evidence-free. A gate that accepts most chunks is an
expensive summarizer that will reproduce the memory-explosion failure the whole
design exists to avoid. `M1.8` is the gate that enforces it, and **update-rate is
the single number to watch.**

## Verification practice — script first, source second

A task is verified by running its command and reading the output, then opening
the source. Reviewing the diff first is how an assertion that was quietly dropped
still gets called done: the code looks right, and nothing proves the test ran.

Numbered assertion N in a `Verify` section is test fn `aN_<slug>`. The numbering
is the contract — a test fn that does not exist should report missing rather than
be silently absent from a green summary.

`cargo test` reporting `ok` with zero tests run is not a pass.

## Progress

**Source of truth is the `Status` field in each task file.** The tables below
mirror it; a status changed here and not there is a lie.

Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked

| # | Phase | Ids | Tasks | ✅ | 🟡 | ⬜ | Gate |
|---|---|---|---|---|---|---|---|
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
| 3 | Projections | M2.x | 8 | 8 | 0 | 0 | ✅ M2.8 |
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
| 4.5 | Distributed API Layer | M3.5.x | 10 | 10 | 0 | 0 | ✅ M3.5.8 |
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
| 5.6 | Tool context | M3.7.x | 5 | 0 | 1 | 4 | ⬜ M3.7.6 |
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
| 9 | Hybrid search | M8.x | 9 | 1 | 1 | 7 | ⬜ M8.9 |
| | **Total** | | **72** | **57** | **1** | **14** | 6/11 green |

**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).

**Current work:**
- M8.1: OpenSearch StatefulSet (pod security baseline, fsGroup perms) — 2 commits pending (9a07659, 611f4d8)
- M3.7.4: `/memory/context` 3-tier lookup endpoint (starts after M3.6 Phase 1 + M8.2)
- Obsidian service: ✅ Deployed (ppatlabs/obsidian:latest, REST API on 27124)
- All M3.5 endpoints ready: vault JSON + hybrid search (60% pgvector + 40% OpenSearch), JWT auth live with Authentik

**Blocked until M8.1 green:**
- M8.2 (dual-write indexing) — needs OpenSearch cluster ready
- M8.3-9 (query optimizer, RRF, hybrid endpoint, benchmarks) — M8.2 blocker
- M3.7.4 context endpoint uses hybrid search results — M8.2 blocker

**Tests: 247 passing, 2 ignored** (M2.1 +8, M3.5 +16, M4 +20). **57/72 tasks complete (79%)**, **6/11 gates green**. M3.7.5 retired (redundant with hybrid search).

`M2.2` (CNPG manifest), `M5.4` (vLLM+LoRA), `M3.5.9` (git refs), and
`M6.x` (agent-manager migration) are homelab/infra work independent of prior phases, can start parallel.

**M6 is a different repo, not a dependency of M0-M5.** It migrates
`github.com/Riotpiaole/agent-manager`'s session store (a separate Go CLI tool,
unrelated to this project's own memory system) from local sqlite to its own
dedicated CNPG cluster. It rides in this board because it's homelab work
happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
landing in the same cluster around the same time need to look like siblings,
not strangers — see M6.6's convention-consistency check.

## ✅ Archived Phases

Completed and archived: **M0.x (8/8)**, **M1.x (8/8)** — all task files deleted from `/tasks/` after verification. See git log for historical record and `CLAUDE.md` for session context.

## 3 — Projections · M2.x

**Status:** ✅ Complete · 8/8 done. All task files archived.

M2.1-2.8 ✅ ARCHIVED: 
- M2.1 ✅ (embeddings client: 768-dim batching @32)
- M2.2 ✅ (CNPG Cluster + Database CRD with pgvector 0.7.0)
- M2.3 ✅ (schema + sqlx migrations — 5 entity tables)
- M2.4 ✅ (pgvector repository: upsert, search, edges, 8 tests)
- M2.5 ✅ (obsidian projector: deterministic rebuild, 10 tests)
- M2.6 ✅ (rebuild from log: orchestration engine, 6 tests)
- M2.7 ✅ (verify edges: 6 invariants, 9 tests)
- M2.8 ✅ (M2 composition gate: idempotence + rebuild-from-log proof)

## 4 — L2 synthesis and retrieval · M3.x

**Status:** ✅ Complete · 4/4 tasks done. Gate M3.4 passing. Task files archived.

## 4.5 — Distributed API Layer · M3.5.x

Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parallel with M4 and M5 after M3.4 green.

**Status:** 10/10 done · M3.5.8 gate ✅ passing. M3.5.1–10 archived (all task files deleted). Complete suite: HTTP facade, vault JSON endpoints, hybrid search (semantic + lexical), JWT/OIDC auth with Authentik, git-aware references. All integration tests passing. Awaiting Docker image rollout for production deployment.

## 5 — Skills · M4.x

**Status:** 2/3 done · M4.1 (skill draft) ✅ and M4.2 (derived filter) ✅ archived. Pending M4.3 gate composition.

`mem skill draft --project <proj> --from <query-id>` writes to `vault/skills/_drafts/`. Dry-run mode supported. Shingle matcher (M4.2) provides derived LessonSource filtering. M4.1-2 task files deleted after archival.

## 5.5 — Reference corpora · M3.6.x

Documentation the local models are weak at — `kubectl`, `tea` — made retrievable
as level **R**: embedded and indexed, never evidence. Ids are `M3.6.x` and stay
`M3.6.x`; the phase sits here rather than at 4.6 because [M3.6.4](M3.6.4-reference-cycle-guard.md)
extends M4.2's matcher instead of duplicating it, and because skills are the
better answer to the same problem and should exist first.

**The load-bearing property is a negative one.** Adding a corpus must not change
update-rate, must not change default query output, and must not put an R node in
any provenance chain. R bypasses the recurrence structurally — `run_loop` needs a
`Query` and a corpus has none — not by a flag. [M3.6.6](M3.6.6-m3.6-gate.md)
asserts M1.8's numbers are *unchanged*, not merely still-passing, because
documentation fed to the gate would lower update-rate and make M1.8 easier to
clear while the memory got worse.

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| M3.6.1 | `DocCorpusSource` + heading chunking | M | — | ✅ |
| [M3.6.2](M3.6.2-level-r-storage.md) | Level R: log, index, vault, rebuild parity | M | — | ⬜ |
| [M3.6.3](M3.6.3-mem-ref-cli.md) | `mem ref` — replace-on-change corpus management | M | — | ⬜ |
| [M3.6.4](M3.6.4-reference-cycle-guard.md) | Reference text cannot re-enter as evidence | M | — | ⬜ |
| [M3.6.5](M3.6.5-query-levels-and-floor.md) | Query: filter-then-recall, R opt-in, floor | M | — | ⬜ |
| [M3.6.6](M3.6.6-m3.6-gate.md) | **M3.6 composition gate** | M | gate | ⬜ |

## 5.6 — Tool context · M3.7.x

Context lookup over HTTP for failure diagnosis and tool troubleshooting.
Consumers are `pi`, curl, or an MCP call.

**Three-tier lookup:**
1. **Tier 1 (Exact):** Normalised failure signature hash → past solutions
2. **Tier 2 (Semantic):** Hybrid search over skill/evidence corpus → similar cases
3. **Tier 3 (Reference):** R corpus (documentation) → general guidance

Hybrid search (M8) now covers tiers 1 & 2. Tier 3 enabled by M3.6 (reference corpora).

**M3.7.5 deferred:** Standing query for tool-failures was pre-hybrid-search design.
Now redundant — hybrid search solves the same problem better. Task retired.

**Key remaining work:**
- M3.7.7: Failure signature extraction (normalize timestamps, paths)
- M3.7.8: Symptom projection (query → normalized form for matching)
- M3.7.4: HTTP endpoint glue (tier logic + response shape)

Ids are `M3.7.x` and frozen. `M3.7.1`, `M3.7.2`, `M3.7.5` retired (not needed).

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.7.3](M3.7.3-skill-matching.md) | `GET /memory/skills?task=` — match a subset | S | — | ⬜ |
| [M3.7.4](M3.7.4-context-endpoint.md) | `/memory/context` — three-tier lookup | M | — | 🟡 |
| M3.7.5 | `tool-failures` standing query | — | — | ❌ Retired (hybrid search covers) |
| [M3.7.6](M3.7.6-m3.7-gate.md) | **M3.7 composition gate** | M | gate | ⬜ |
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 |
| [M3.7.8](M3.7.8-symptom-projection.md) | Symptom projection at ingest | M | — | ⬜ |

## 6 — Post-training · M5.x

Python, separate from the Rust workspace. The boundary is the JSONL log.

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M5.1](M5.1-evidence-labeler.md) | `mem label` — evidence labeler | M | — | ⬜ |
| [M5.2](M5.2-labeler-calibration.md) | Labeler calibration | M | — | ⬜ |
| [M5.3](M5.3-training-corpus-export.md) | Training corpus export | M | — | ⬜ |
| [M5.4](M5.4-vllm-lora-serving.md) | vLLM + `--enable-lora` | L | homelab | ⬜ |
| [M5.5](M5.5-verl-training-loop.md) | verl training loop | L | — | ⬜ |
| [M5.6](M5.6-m5-gate.md) | **M5 composition gate** | L | gate | ⬜ |

## 7 — agent-manager migration · M6.x

Separate repo (`github.com/Riotpiaole/agent-manager`, fork branch
`add-headless-spawn`), separate cluster resource, no Rust/GRU-Mem
dependency. Moves its session store off local sqlite onto a dedicated CNPG
Postgres, reachable from the Mac client through a dedicated nginx route —
durability-of-location, not a multi-host requirement.

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M6.1](M6.1-agent-manager-db-manifest.md) | CNPG `agent-manager-db` manifest | M | homelab | ⬜ |
| [M6.2](M6.2-schema-port.md) | Postgres schema for agent-manager sessions | M | — | ⬜ |
| [M6.3](M6.3-store-query-port.md) | store.go query port to Postgres | L | — | ⬜ |
| [M6.4](M6.4-nginx-stream-routing.md) | nginx TCP routing to `agent-manager-db` | S | homelab | ⬜ |
| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ |
| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ |

## 8 — Source connectors · M7.x

Extensible multi-source ingestion. `SourceConnector` trait + YAML-driven registry.
Adding a new document source (paperless-ngx, S3, git repo) requires implementing
one trait and adding one config block — no changes to the ingest pipeline, chunking,
embedding, storage, or query layers.

**Document connectors** produce Level R content (reference material, bypasses gated
loop). **Session connectors** (pi, claude) produce evidence for L0/L1/L2. The
connector's `source_type()` declares the pipeline.

Sync framework handles change detection (sha-based skip), tombstoning, drift
reporting, and resumable sync for all connectors.

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M7.1](M7.1-source-connector-trait.md) | `SourceConnector` trait + registry | M | — | ⬜ |
| [M7.2](M7.2-obsidian-connector.md) | Obsidian vault connector | M | — | ⬜ |
| [M7.3](M7.3-paperless-connector.md) | paperless-ngx connector | M | — | ⬜ |
| [M7.4](M7.4-git-repo-connector.md) | Git repo connector | M | — | ⬜ |
| [M7.5](M7.5-s3-connector.md) | S3-compatible storage connector | M | — | ⬜ |
| [M7.6](M7.6-sync-framework.md) | Sync framework | L | — | ⬜ |
| [M7.7](M7.7-source-cli.md) | `mem source` CLI | M | — | ⬜ |
| [M7.8](M7.8-source-http-endpoints.md) | Source HTTP endpoints | M | — | ⬜ |
| [M7.9](M7.9-connector-health-monitoring.md) | Connector health + observability | S | — | ⬜ |
| [M7.10](M7.10-m7-gate.md) | **M7 composition gate** | M | gate | ⬜ |

---

## 9 — Hybrid search · M8.x

Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with
Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and
routes to the best strategy before any database call.

**The load-bearing property is accuracy.** Hybrid must produce measurably better
NDCG@10 than either engine alone. If it doesn't, the gate fails — not because
the code is broken, but because the system isn't earning its complexity budget.
M8.9 requires benchmark numbers, not just green tests.

**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`.
Results are merged by rank position, not score magnitude, because pgvector cosine
(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no
parameter tuning (`k=60` is the academic standard). The alternative — weighted
linear combination — requires labelled data for weight selection that we don't
have yet.

**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and
OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked
`opensearch_pending` and retried by a background task. The gate (M8.9) checks
for zero orphans.

**Fallback.** If OpenSearch is down at query time, the worker degrades to
semantic-only. If the embedding model is down, it degrades to lexical-only.
The response `search_strategy` field always reports which mode was actually used.

| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | ✅ |
| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | — | ⬜ |
| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | — | ⬜ |
| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | — | ⬜ |
| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | — | ⬜ |
| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | — | ⬜ |
| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | — | ⬜ |
| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | — | ⬜ |
| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | ⬜ |

---

Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)