poimen-memory/IMPLEMENTATION-ROADMAP.md

Implementation Roadmap: M3 → M5

Current Status

• M0-M2: ✅ COMPLETE (104 tests passing, rerank client done)
• M3.1: ✅ DONE (L2 synthesis code exists, M1.5 refactored)
• M3.2: ✅ DONE (rerank client, 5 tests passing)
• M3.3–M5.6: ⏳ READY TO START

---

Critical Path: M3.3 → M3.4 → M4.3 → M5.6

M3.3 — mem query (M, 1–3 days)

Status: Ready; QueryWorker stub exists

Scope:

add Query command to CLI
  ├─ --project <name>    (default: infer from $PWD)
  ├─ --levels <L0|L1|L2> (default: L1,L2)
  ├─ --k <n>             (default: 5, recall 10×k, rerank to k)
  ├─ --format <text|json>(default: text)
  └─ --explain           (show recall candidates before reranking)

Pipeline:

embed question
  ↓
HNSW recall (top 50)
  ↓
rerank (using RerankClient)
  ↓
top 5 results
  ↓
walk memory_edge for provenance
  ↓
render (human-readable + JSON)

Files to modify:

• crates/mem-cli/src/main.rs — add Query to Commands enum
• crates/mem-cli/src/query_worker.rs — wire reranker, add edge walking
• tests/it_query.rs — 8 integration tests (seeded DB, deterministic)

Acceptance: known-answer query, provenance resolves, level filtering works, reranking changes order

---

M3.4 — M3 Gate (S, 1 day)

Status: Blocked on M3.1–M3.3

Scope:

Unit test: M3.1 + M3.2 + M3.3 compose correctly
  ├─ synthesize project-level memory (M3.1)
  ├─ query returns reranked results (M3.3 + M3.2)
  └─ provenance walks all 3 levels (L0 ← L1 ← L2)

Acceptance: known-answer query on seeded poimen corpus returns correct answer first

---

M4.1 — mem skill draft (M, 1–3 days)

Status: 60% done (lesson.rs: 871 lines)

Scope:

mem skill draft --from poimen/infra-root-causes
  ├─ Read L1/L2 node from pgvector
  ├─ LLM-assisted conversion: descriptive → procedural
  ├─ Generate SKILL.md in vault/skills/_drafts/
  ├─ Frontmatter: name, description, generated_from: <sha>
  └─ Output: vault/skills/_drafts/<name>/SKILL.md (read-only)

Files to modify:

• crates/mem-cli/src/main.rs — add Skill(Draft) to Commands
• crates/mem-cli/src/skill_draft.rs — new, implement draft logic
• tests/it_skill_draft.rs — 7 tests (generation, format, provenance)

Acceptance: draft generated correctly, carries generated_from metadata, never overwrites promoted skill

---

M4.2 — Cycle Guard (M, 1–3 days)

Status: Not started

Scope:

Stop emitted skill → session → ingest → reinforcement cycle
  ├─ Manifest file: hash of all emitted artifacts
  ├─ During ingest: match chunks against manifest (shingle overlap)
  ├─ Tag matching chunks: derived: true
  └─ Gate never sees derived chunks (evidence filtered)

Files to modify:

• crates/mem-core/src/derived_filter.rs — new, implement shingle matching
• crates/mem-ingest/src/ — wire filter into ingest pipeline
• tests/it_derived_filter.rs — verify cycle cannot form

Acceptance: promoted skill ingested and filtered (never becomes evidence), cycle stays open

---

M4.3 — M4 Gate (S, 1 day)

Status: Blocked on M4.1–M4.2

Scope:

Verify loop stays open
  1. Draft skill generated
  2. Human promotes to vault/skills/<name>
  3. Session loads promoted skill
  4. Session ingested
  5. Skill text still tagged derived: true (never evidence)

---

M5.1–M5.3 — Labeling & Corpus Export (M×3, 3 days)

Status: Not started

Scope:

M5.1: Use 32B reasoning model to label chunks
  ├─ Input: JSONL log from M1.8 runs
  ├─ Output: U_t ground truth ("does chunk answer Q?")
  └─ Store in new JSONL with label field

M5.2: Hand-label holdout, measure calibration
  ├─ Label ~100 examples manually
  ├─ Compare vs 32B labels
  ├─ Measure Cohen's κ
  └─ Proceed only if κ > 0.75

M5.3: Export to verl format
  ├─ Input: labeled JSONL
  ├─ Output: verl dataset (prompt, response, label, reward)
  └─ Split: train/val/test

---

M5.4 — vLLM InferenceService (L, 3+ days, CAN RUN IN PARALLEL)

Status: Not started, independent

Scope:

Deploy vLLM with --enable-lora (Ollama can't hot-swap LoRA)
  ├─ K8s manifest: InferenceService, vLLM v0.11+
  ├─ Model: Qwen2.5-3B base
  ├─ Endpoint: /v1/completions with LoRA adapter
  └─ Homelab deployment

Acceptance: vLLM serving Qwen2.5-3B with LoRA support

---

M5.5 — verl Training Loop (L, 3+ days)

Status: Not started, depends on M5.3 + M5.4

Scope:

Train LoRA adapter using verl
  ├─ Input: labeled corpus from M5.3
  ├─ Base model: Qwen2.5-3B (resident in Ollama)
  ├─ Adapter: LoRA, rank 16–32
  ├─ Rewards:
  │  ├─ r_update: +1 correct gate, −1 wrong
  │  ├─ r_exit: 0 correct, −0.5 late, −0.75 early
  │  ├─ r_format: strict (gate response parseable)
  │  └─ α=0.9 mixing
  └─ Output: adapter.safetensors (~50 MB)

---

M5.6 — M5 Gate (M, 1 day)

Status: Blocked on M5.1–M5.5

Scope:

Adapter beats prompted baseline
  ├─ Held-out project: measure update accuracy
  ├─ Prompted baseline: stock Qwen2.5-3B
  ├─ Adapter baseline: trained LoRA
  ├─ Assert: adapter accuracy > prompted
  └─ Assert: LoRA < 60 MB

---

Implementation Sequence

Week 1: M3 (core retrieval)

Mon: M3.3 (mem query) — embed, recall, rerank, edge walking
Tue: M3.3 continued — CLI, output formatting
Wed: M3.4 gate — compose M3 pieces, verify end-to-end

Week 2: M4 (skills)

Thu: M4.1 (skill draft) — complete lesson.rs, add LLM conversion
Fri: M4.2 (cycle guard) — shingle matching, derived filter
     M4.3 gate — full cycle test

Week 3: M5 (post-training, parallel tracks)

Mon: M5.1 (labeling) — 32B labeler, ground truth extraction
Tue: M5.2 (calibration) — hand-label holdout, κ measurement
Wed: M5.3 (corpus export) — JSONL → verl format
     **M5.4 vLLM (parallel)** — K8s manifest, deploy
Thu: M5.5 (training) — verl loop, reward shaping
Fri: M5.6 gate — verify adapter beats baseline

---

Parallel Tracks (can start anytime)

• M3.5 (HTTP API): Already mostly done (gate green)
• M3.6 (Reference corpora): 6 tasks, independent
• M3.7 (Tool context): 6 tasks, 60% done (lesson.rs)
• M6 (Agent-manager): Different repo, independent

---

Code structure ready

crates/
  mem-core/
    ├── gate_parser.rs        ✅ (M1)
    ├── gated_loop.rs         ✅ (M1)
    └── derived_filter.rs     ⏳ (M4.2, to create)
  
  mem-llm/
    ├── chat.rs               ✅ (M1)
    ├── embeddings.rs         ✅ (M2)
    └── rerank.rs             ✅ (M3.2, tests pass)
  
  mem-cli/
    ├── main.rs               ⏳ (add Query, Skill(Draft))
    ├── ingest_worker.rs      ✅ (M1-M2)
    ├── query_worker.rs       🟡 (stub exists, needs completion)
    ├── skill_draft.rs        ⏳ (to create)
    └── http_server.rs        ✅ (M3.5)
  
  mem-store/
    └── vector_store.rs       ✅ (HNSW, edge walking)

tests/
  ├── it_rerank.rs           ✅ (4/5 tests pass, 1 ignored)
  ├── it_query.rs            ⏳ (to create, 8 tests)
  ├── it_skill_draft.rs      ⏳ (to create, 7 tests)
  └── it_derived_filter.rs   ⏳ (to create)

---

Gate progression

M3.4 gate ✅
  ├─ M3.1 ✅ (L2 synthesis)
  ├─ M3.2 ✅ (rerank)
  └─ M3.3 ⏳ (query)

M4.3 gate ⏳
  ├─ M4.1 🟡 (skill draft, 60% done)
  └─ M4.2 ⏳ (cycle guard)

M5.6 gate ⏳
  ├─ M5.1 ⏳ (labeling)
  ├─ M5.2 ⏳ (calibration)
  ├─ M5.3 ⏳ (corpus export)
  ├─ M5.4 ⏳ (vLLM, can run parallel)
  └─ M5.5 ⏳ (training)

---

Next: Start M3.3

Ready to implement. Blocking: none (M3.2 ✅, M2.4 ✅, M2.1 ✅)

See tasks/M3.3-mem-query.md for full spec.