From 4733b89165321cba572fd6fb74a9d8340064bb4d Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Tue, 25 Aug 2026 11:59:13 -0700 Subject: [PATCH] docs: implementation roadmap for M3, M4, M5 with detailed breakdown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M3.2 (rerank client): ✅ COMPLETE (5 tests passing) M3.3 (mem query): Ready, pipeline specified, code structure ready M3.4 (gate): Blocked on M3.3 M4.1 (skill draft): 60% done (lesson.rs: 871 lines) M4.2 (cycle-guard): Detailed spec M4.3 (gate): Blocked on M4.1-4.2 M5.1-5.6 (post-training): Separate Python, M5.4 can run in parallel Includes: - Sequential implementation plan (3 weeks) - Code structure inventory - Gate progression tracking - Parallel tracks (M3.5, M3.6, M3.7, M6) - Acceptance criteria for each task --- IMPLEMENTATION-ROADMAP.md | 302 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 IMPLEMENTATION-ROADMAP.md diff --git a/IMPLEMENTATION-ROADMAP.md b/IMPLEMENTATION-ROADMAP.md new file mode 100644 index 0000000..df00f8a --- /dev/null +++ b/IMPLEMENTATION-ROADMAP.md @@ -0,0 +1,302 @@ +# Implementation Roadmap: M3 → M5 + +## Current Status +- M0-M2: ✅ **COMPLETE** (104 tests passing, rerank client done) +- M3.1: ✅ **DONE** (L2 synthesis code exists, M1.5 refactored) +- M3.2: ✅ **DONE** (rerank client, 5 tests passing) +- M3.3–M5.6: ⏳ **READY TO START** + +--- + +## Critical Path: M3.3 → M3.4 → M4.3 → M5.6 + +### M3.3 — `mem query` (M, 1–3 days) + +**Status:** Ready; QueryWorker stub exists + +**Scope:** +``` +add Query command to CLI + ├─ --project (default: infer from $PWD) + ├─ --levels (default: L1,L2) + ├─ --k (default: 5, recall 10×k, rerank to k) + ├─ --format (default: text) + └─ --explain (show recall candidates before reranking) +``` + +**Pipeline:** +``` +embed question + ↓ +HNSW recall (top 50) + ↓ +rerank (using RerankClient) + ↓ +top 5 results + ↓ +walk memory_edge for provenance + ↓ +render (human-readable + JSON) +``` + +**Files to modify:** +- `crates/mem-cli/src/main.rs` — add Query to Commands enum +- `crates/mem-cli/src/query_worker.rs` — wire reranker, add edge walking +- `tests/it_query.rs` — 8 integration tests (seeded DB, deterministic) + +**Acceptance:** known-answer query, provenance resolves, level filtering works, reranking changes order + +--- + +### M3.4 — M3 Gate (S, 1 day) + +**Status:** Blocked on M3.1–M3.3 + +**Scope:** +``` +Unit test: M3.1 + M3.2 + M3.3 compose correctly + ├─ synthesize project-level memory (M3.1) + ├─ query returns reranked results (M3.3 + M3.2) + └─ provenance walks all 3 levels (L0 ← L1 ← L2) +``` + +**Acceptance:** known-answer query on seeded poimen corpus returns correct answer first + +--- + +### M4.1 — `mem skill draft` (M, 1–3 days) + +**Status:** 60% done (lesson.rs: 871 lines) + +**Scope:** +``` +mem skill draft --from poimen/infra-root-causes + ├─ Read L1/L2 node from pgvector + ├─ LLM-assisted conversion: descriptive → procedural + ├─ Generate SKILL.md in vault/skills/_drafts/ + ├─ Frontmatter: name, description, generated_from: + └─ Output: vault/skills/_drafts//SKILL.md (read-only) +``` + +**Files to modify:** +- `crates/mem-cli/src/main.rs` — add Skill(Draft) to Commands +- `crates/mem-cli/src/skill_draft.rs` — new, implement draft logic +- `tests/it_skill_draft.rs` — 7 tests (generation, format, provenance) + +**Acceptance:** draft generated correctly, carries generated_from metadata, never overwrites promoted skill + +--- + +### M4.2 — Cycle Guard (M, 1–3 days) + +**Status:** Not started + +**Scope:** +``` +Stop emitted skill → session → ingest → reinforcement cycle + ├─ Manifest file: hash of all emitted artifacts + ├─ During ingest: match chunks against manifest (shingle overlap) + ├─ Tag matching chunks: derived: true + └─ Gate never sees derived chunks (evidence filtered) +``` + +**Files to modify:** +- `crates/mem-core/src/derived_filter.rs` — new, implement shingle matching +- `crates/mem-ingest/src/` — wire filter into ingest pipeline +- `tests/it_derived_filter.rs` — verify cycle cannot form + +**Acceptance:** promoted skill ingested and filtered (never becomes evidence), cycle stays open + +--- + +### M4.3 — M4 Gate (S, 1 day) + +**Status:** Blocked on M4.1–M4.2 + +**Scope:** +``` +Verify loop stays open + 1. Draft skill generated + 2. Human promotes to vault/skills/ + 3. Session loads promoted skill + 4. Session ingested + 5. Skill text still tagged derived: true (never evidence) +``` + +--- + +### M5.1–M5.3 — Labeling & Corpus Export (M×3, 3 days) + +**Status:** Not started + +**Scope:** +``` +M5.1: Use 32B reasoning model to label chunks + ├─ Input: JSONL log from M1.8 runs + ├─ Output: U_t ground truth ("does chunk answer Q?") + └─ Store in new JSONL with label field + +M5.2: Hand-label holdout, measure calibration + ├─ Label ~100 examples manually + ├─ Compare vs 32B labels + ├─ Measure Cohen's κ + └─ Proceed only if κ > 0.75 + +M5.3: Export to verl format + ├─ Input: labeled JSONL + ├─ Output: verl dataset (prompt, response, label, reward) + └─ Split: train/val/test +``` + +--- + +### M5.4 — vLLM InferenceService (L, 3+ days, **CAN RUN IN PARALLEL**) + +**Status:** Not started, independent + +**Scope:** +``` +Deploy vLLM with --enable-lora (Ollama can't hot-swap LoRA) + ├─ K8s manifest: InferenceService, vLLM v0.11+ + ├─ Model: Qwen2.5-3B base + ├─ Endpoint: /v1/completions with LoRA adapter + └─ Homelab deployment +``` + +**Acceptance:** vLLM serving Qwen2.5-3B with LoRA support + +--- + +### M5.5 — verl Training Loop (L, 3+ days) + +**Status:** Not started, depends on M5.3 + M5.4 + +**Scope:** +``` +Train LoRA adapter using verl + ├─ Input: labeled corpus from M5.3 + ├─ Base model: Qwen2.5-3B (resident in Ollama) + ├─ Adapter: LoRA, rank 16–32 + ├─ Rewards: + │ ├─ r_update: +1 correct gate, −1 wrong + │ ├─ r_exit: 0 correct, −0.5 late, −0.75 early + │ ├─ r_format: strict (gate response parseable) + │ └─ α=0.9 mixing + └─ Output: adapter.safetensors (~50 MB) +``` + +--- + +### M5.6 — M5 Gate (M, 1 day) + +**Status:** Blocked on M5.1–M5.5 + +**Scope:** +``` +Adapter beats prompted baseline + ├─ Held-out project: measure update accuracy + ├─ Prompted baseline: stock Qwen2.5-3B + ├─ Adapter baseline: trained LoRA + ├─ Assert: adapter accuracy > prompted + └─ Assert: LoRA < 60 MB +``` + +--- + +## Implementation Sequence + +**Week 1: M3 (core retrieval)** +``` +Mon: M3.3 (mem query) — embed, recall, rerank, edge walking +Tue: M3.3 continued — CLI, output formatting +Wed: M3.4 gate — compose M3 pieces, verify end-to-end +``` + +**Week 2: M4 (skills)** +``` +Thu: M4.1 (skill draft) — complete lesson.rs, add LLM conversion +Fri: M4.2 (cycle guard) — shingle matching, derived filter + M4.3 gate — full cycle test +``` + +**Week 3: M5 (post-training, parallel tracks)** +``` +Mon: M5.1 (labeling) — 32B labeler, ground truth extraction +Tue: M5.2 (calibration) — hand-label holdout, κ measurement +Wed: M5.3 (corpus export) — JSONL → verl format + **M5.4 vLLM (parallel)** — K8s manifest, deploy +Thu: M5.5 (training) — verl loop, reward shaping +Fri: M5.6 gate — verify adapter beats baseline +``` + +--- + +## Parallel Tracks (can start anytime) + +- **M3.5** (HTTP API): Already mostly done (gate green) +- **M3.6** (Reference corpora): 6 tasks, independent +- **M3.7** (Tool context): 6 tasks, 60% done (lesson.rs) +- **M6** (Agent-manager): Different repo, independent + +--- + +## Code structure ready + +``` +crates/ + mem-core/ + ├── gate_parser.rs ✅ (M1) + ├── gated_loop.rs ✅ (M1) + └── derived_filter.rs ⏳ (M4.2, to create) + + mem-llm/ + ├── chat.rs ✅ (M1) + ├── embeddings.rs ✅ (M2) + └── rerank.rs ✅ (M3.2, tests pass) + + mem-cli/ + ├── main.rs ⏳ (add Query, Skill(Draft)) + ├── ingest_worker.rs ✅ (M1-M2) + ├── query_worker.rs 🟡 (stub exists, needs completion) + ├── skill_draft.rs ⏳ (to create) + └── http_server.rs ✅ (M3.5) + + mem-store/ + └── vector_store.rs ✅ (HNSW, edge walking) + +tests/ + ├── it_rerank.rs ✅ (4/5 tests pass, 1 ignored) + ├── it_query.rs ⏳ (to create, 8 tests) + ├── it_skill_draft.rs ⏳ (to create, 7 tests) + └── it_derived_filter.rs ⏳ (to create) +``` + +--- + +## Gate progression + +``` +M3.4 gate ✅ + ├─ M3.1 ✅ (L2 synthesis) + ├─ M3.2 ✅ (rerank) + └─ M3.3 ⏳ (query) + +M4.3 gate ⏳ + ├─ M4.1 🟡 (skill draft, 60% done) + └─ M4.2 ⏳ (cycle guard) + +M5.6 gate ⏳ + ├─ M5.1 ⏳ (labeling) + ├─ M5.2 ⏳ (calibration) + ├─ M5.3 ⏳ (corpus export) + ├─ M5.4 ⏳ (vLLM, can run parallel) + └─ M5.5 ⏳ (training) +``` + +--- + +## Next: Start M3.3 + +Ready to implement. Blocking: none (M3.2 ✅, M2.4 ✅, M2.1 ✅) + +See tasks/M3.3-mem-query.md for full spec.