Story Crater Bot
4733b89165
M3.2 (rerank client): ✅ COMPLETE (5 tests passing)
M3.3 (mem query): Ready, pipeline specified, code structure ready
M3.4 (gate): Blocked on M3.3
M4.1 (skill draft): 60% done (lesson.rs: 871 lines)
M4.2 (cycle-guard): Detailed spec
M4.3 (gate): Blocked on M4.1-4.2
M5.1-5.6 (post-training): Separate Python, M5.4 can run in parallel
Includes:
- Sequential implementation plan (3 weeks)
- Code structure inventory
- Gate progression tracking
- Parallel tracks (M3.5, M3.6, M3.7, M6)
- Acceptance criteria for each task
303 lines
7.8 KiB
Markdown
303 lines
7.8 KiB
Markdown
# Implementation Roadmap: M3 → M5
|
||
|
||
## Current Status
|
||
- M0-M2: ✅ **COMPLETE** (104 tests passing, rerank client done)
|
||
- M3.1: ✅ **DONE** (L2 synthesis code exists, M1.5 refactored)
|
||
- M3.2: ✅ **DONE** (rerank client, 5 tests passing)
|
||
- M3.3–M5.6: ⏳ **READY TO START**
|
||
|
||
---
|
||
|
||
## Critical Path: M3.3 → M3.4 → M4.3 → M5.6
|
||
|
||
### M3.3 — `mem query` (M, 1–3 days)
|
||
|
||
**Status:** Ready; QueryWorker stub exists
|
||
|
||
**Scope:**
|
||
```
|
||
add Query command to CLI
|
||
├─ --project <name> (default: infer from $PWD)
|
||
├─ --levels <L0|L1|L2> (default: L1,L2)
|
||
├─ --k <n> (default: 5, recall 10×k, rerank to k)
|
||
├─ --format <text|json>(default: text)
|
||
└─ --explain (show recall candidates before reranking)
|
||
```
|
||
|
||
**Pipeline:**
|
||
```
|
||
embed question
|
||
↓
|
||
HNSW recall (top 50)
|
||
↓
|
||
rerank (using RerankClient)
|
||
↓
|
||
top 5 results
|
||
↓
|
||
walk memory_edge for provenance
|
||
↓
|
||
render (human-readable + JSON)
|
||
```
|
||
|
||
**Files to modify:**
|
||
- `crates/mem-cli/src/main.rs` — add Query to Commands enum
|
||
- `crates/mem-cli/src/query_worker.rs` — wire reranker, add edge walking
|
||
- `tests/it_query.rs` — 8 integration tests (seeded DB, deterministic)
|
||
|
||
**Acceptance:** known-answer query, provenance resolves, level filtering works, reranking changes order
|
||
|
||
---
|
||
|
||
### M3.4 — M3 Gate (S, 1 day)
|
||
|
||
**Status:** Blocked on M3.1–M3.3
|
||
|
||
**Scope:**
|
||
```
|
||
Unit test: M3.1 + M3.2 + M3.3 compose correctly
|
||
├─ synthesize project-level memory (M3.1)
|
||
├─ query returns reranked results (M3.3 + M3.2)
|
||
└─ provenance walks all 3 levels (L0 ← L1 ← L2)
|
||
```
|
||
|
||
**Acceptance:** known-answer query on seeded poimen corpus returns correct answer first
|
||
|
||
---
|
||
|
||
### M4.1 — `mem skill draft` (M, 1–3 days)
|
||
|
||
**Status:** 60% done (lesson.rs: 871 lines)
|
||
|
||
**Scope:**
|
||
```
|
||
mem skill draft --from poimen/infra-root-causes
|
||
├─ Read L1/L2 node from pgvector
|
||
├─ LLM-assisted conversion: descriptive → procedural
|
||
├─ Generate SKILL.md in vault/skills/_drafts/
|
||
├─ Frontmatter: name, description, generated_from: <sha>
|
||
└─ Output: vault/skills/_drafts/<name>/SKILL.md (read-only)
|
||
```
|
||
|
||
**Files to modify:**
|
||
- `crates/mem-cli/src/main.rs` — add Skill(Draft) to Commands
|
||
- `crates/mem-cli/src/skill_draft.rs` — new, implement draft logic
|
||
- `tests/it_skill_draft.rs` — 7 tests (generation, format, provenance)
|
||
|
||
**Acceptance:** draft generated correctly, carries generated_from metadata, never overwrites promoted skill
|
||
|
||
---
|
||
|
||
### M4.2 — Cycle Guard (M, 1–3 days)
|
||
|
||
**Status:** Not started
|
||
|
||
**Scope:**
|
||
```
|
||
Stop emitted skill → session → ingest → reinforcement cycle
|
||
├─ Manifest file: hash of all emitted artifacts
|
||
├─ During ingest: match chunks against manifest (shingle overlap)
|
||
├─ Tag matching chunks: derived: true
|
||
└─ Gate never sees derived chunks (evidence filtered)
|
||
```
|
||
|
||
**Files to modify:**
|
||
- `crates/mem-core/src/derived_filter.rs` — new, implement shingle matching
|
||
- `crates/mem-ingest/src/` — wire filter into ingest pipeline
|
||
- `tests/it_derived_filter.rs` — verify cycle cannot form
|
||
|
||
**Acceptance:** promoted skill ingested and filtered (never becomes evidence), cycle stays open
|
||
|
||
---
|
||
|
||
### M4.3 — M4 Gate (S, 1 day)
|
||
|
||
**Status:** Blocked on M4.1–M4.2
|
||
|
||
**Scope:**
|
||
```
|
||
Verify loop stays open
|
||
1. Draft skill generated
|
||
2. Human promotes to vault/skills/<name>
|
||
3. Session loads promoted skill
|
||
4. Session ingested
|
||
5. Skill text still tagged derived: true (never evidence)
|
||
```
|
||
|
||
---
|
||
|
||
### M5.1–M5.3 — Labeling & Corpus Export (M×3, 3 days)
|
||
|
||
**Status:** Not started
|
||
|
||
**Scope:**
|
||
```
|
||
M5.1: Use 32B reasoning model to label chunks
|
||
├─ Input: JSONL log from M1.8 runs
|
||
├─ Output: U_t ground truth ("does chunk answer Q?")
|
||
└─ Store in new JSONL with label field
|
||
|
||
M5.2: Hand-label holdout, measure calibration
|
||
├─ Label ~100 examples manually
|
||
├─ Compare vs 32B labels
|
||
├─ Measure Cohen's κ
|
||
└─ Proceed only if κ > 0.75
|
||
|
||
M5.3: Export to verl format
|
||
├─ Input: labeled JSONL
|
||
├─ Output: verl dataset (prompt, response, label, reward)
|
||
└─ Split: train/val/test
|
||
```
|
||
|
||
---
|
||
|
||
### M5.4 — vLLM InferenceService (L, 3+ days, **CAN RUN IN PARALLEL**)
|
||
|
||
**Status:** Not started, independent
|
||
|
||
**Scope:**
|
||
```
|
||
Deploy vLLM with --enable-lora (Ollama can't hot-swap LoRA)
|
||
├─ K8s manifest: InferenceService, vLLM v0.11+
|
||
├─ Model: Qwen2.5-3B base
|
||
├─ Endpoint: /v1/completions with LoRA adapter
|
||
└─ Homelab deployment
|
||
```
|
||
|
||
**Acceptance:** vLLM serving Qwen2.5-3B with LoRA support
|
||
|
||
---
|
||
|
||
### M5.5 — verl Training Loop (L, 3+ days)
|
||
|
||
**Status:** Not started, depends on M5.3 + M5.4
|
||
|
||
**Scope:**
|
||
```
|
||
Train LoRA adapter using verl
|
||
├─ Input: labeled corpus from M5.3
|
||
├─ Base model: Qwen2.5-3B (resident in Ollama)
|
||
├─ Adapter: LoRA, rank 16–32
|
||
├─ Rewards:
|
||
│ ├─ r_update: +1 correct gate, −1 wrong
|
||
│ ├─ r_exit: 0 correct, −0.5 late, −0.75 early
|
||
│ ├─ r_format: strict (gate response parseable)
|
||
│ └─ α=0.9 mixing
|
||
└─ Output: adapter.safetensors (~50 MB)
|
||
```
|
||
|
||
---
|
||
|
||
### M5.6 — M5 Gate (M, 1 day)
|
||
|
||
**Status:** Blocked on M5.1–M5.5
|
||
|
||
**Scope:**
|
||
```
|
||
Adapter beats prompted baseline
|
||
├─ Held-out project: measure update accuracy
|
||
├─ Prompted baseline: stock Qwen2.5-3B
|
||
├─ Adapter baseline: trained LoRA
|
||
├─ Assert: adapter accuracy > prompted
|
||
└─ Assert: LoRA < 60 MB
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation Sequence
|
||
|
||
**Week 1: M3 (core retrieval)**
|
||
```
|
||
Mon: M3.3 (mem query) — embed, recall, rerank, edge walking
|
||
Tue: M3.3 continued — CLI, output formatting
|
||
Wed: M3.4 gate — compose M3 pieces, verify end-to-end
|
||
```
|
||
|
||
**Week 2: M4 (skills)**
|
||
```
|
||
Thu: M4.1 (skill draft) — complete lesson.rs, add LLM conversion
|
||
Fri: M4.2 (cycle guard) — shingle matching, derived filter
|
||
M4.3 gate — full cycle test
|
||
```
|
||
|
||
**Week 3: M5 (post-training, parallel tracks)**
|
||
```
|
||
Mon: M5.1 (labeling) — 32B labeler, ground truth extraction
|
||
Tue: M5.2 (calibration) — hand-label holdout, κ measurement
|
||
Wed: M5.3 (corpus export) — JSONL → verl format
|
||
**M5.4 vLLM (parallel)** — K8s manifest, deploy
|
||
Thu: M5.5 (training) — verl loop, reward shaping
|
||
Fri: M5.6 gate — verify adapter beats baseline
|
||
```
|
||
|
||
---
|
||
|
||
## Parallel Tracks (can start anytime)
|
||
|
||
- **M3.5** (HTTP API): Already mostly done (gate green)
|
||
- **M3.6** (Reference corpora): 6 tasks, independent
|
||
- **M3.7** (Tool context): 6 tasks, 60% done (lesson.rs)
|
||
- **M6** (Agent-manager): Different repo, independent
|
||
|
||
---
|
||
|
||
## Code structure ready
|
||
|
||
```
|
||
crates/
|
||
mem-core/
|
||
├── gate_parser.rs ✅ (M1)
|
||
├── gated_loop.rs ✅ (M1)
|
||
└── derived_filter.rs ⏳ (M4.2, to create)
|
||
|
||
mem-llm/
|
||
├── chat.rs ✅ (M1)
|
||
├── embeddings.rs ✅ (M2)
|
||
└── rerank.rs ✅ (M3.2, tests pass)
|
||
|
||
mem-cli/
|
||
├── main.rs ⏳ (add Query, Skill(Draft))
|
||
├── ingest_worker.rs ✅ (M1-M2)
|
||
├── query_worker.rs 🟡 (stub exists, needs completion)
|
||
├── skill_draft.rs ⏳ (to create)
|
||
└── http_server.rs ✅ (M3.5)
|
||
|
||
mem-store/
|
||
└── vector_store.rs ✅ (HNSW, edge walking)
|
||
|
||
tests/
|
||
├── it_rerank.rs ✅ (4/5 tests pass, 1 ignored)
|
||
├── it_query.rs ⏳ (to create, 8 tests)
|
||
├── it_skill_draft.rs ⏳ (to create, 7 tests)
|
||
└── it_derived_filter.rs ⏳ (to create)
|
||
```
|
||
|
||
---
|
||
|
||
## Gate progression
|
||
|
||
```
|
||
M3.4 gate ✅
|
||
├─ M3.1 ✅ (L2 synthesis)
|
||
├─ M3.2 ✅ (rerank)
|
||
└─ M3.3 ⏳ (query)
|
||
|
||
M4.3 gate ⏳
|
||
├─ M4.1 🟡 (skill draft, 60% done)
|
||
└─ M4.2 ⏳ (cycle guard)
|
||
|
||
M5.6 gate ⏳
|
||
├─ M5.1 ⏳ (labeling)
|
||
├─ M5.2 ⏳ (calibration)
|
||
├─ M5.3 ⏳ (corpus export)
|
||
├─ M5.4 ⏳ (vLLM, can run parallel)
|
||
└─ M5.5 ⏳ (training)
|
||
```
|
||
|
||
---
|
||
|
||
## Next: Start M3.3
|
||
|
||
Ready to implement. Blocking: none (M3.2 ✅, M2.4 ✅, M2.1 ✅)
|
||
|
||
See tasks/M3.3-mem-query.md for full spec.
|