diff --git a/PROGRESS.md b/PROGRESS.md index e300efe..38c84b7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,12 +1,12 @@ # Poimen Memory System - Implementation Progress **Date:** 2024-08-20 -**Status:** 4 of 51 tasks complete (8%) -**Tests Passing:** 19 integration tests across 4 test suites +**Status:** 8 of 51 tasks complete (16%) — **Phase M0 Complete** ✅ +**Tests Passing:** 35 integration tests (34 passing, 1 ignored) across 8 test suites ## Completed Phases -### ✅ M0 — Read-only spine (Phase 1 of 8) +### ✅ M0 — Read-only spine (Phase 1 of 8) — COMPLETE #### M0.1: Cargo workspace + crate skeletons - **Status:** Complete @@ -45,20 +45,56 @@ - **Status:** Complete - **Tests:** 3 integration tests + 1 ignored (a1-a4) - **Deliverables:** - - Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash guard + - Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash verification - `QwenTokenCounter` implementing tokenizers crate integration - Hash verification: load fails if file modified - `mem tokens ` CLI subcommand - Token counts verified against hand-recorded strings - Budget holds across multiple-record chunks -## Remaining Tasks (47 of 51) +#### M0.5: pi session adapter +- **Status:** Complete +- **Tests:** 5 integration tests (a1-a5) +- **Deliverables:** + - `PiSessionSource` implementing `RecordSource` trait + - Project key extraction from session `cwd` field + - Content flattening for string/block-array/structured shapes + - Role mapping with correct counts (user, assistant, toolResult, system) + - Graceful handling of malformed JSON lines + - Compaction events emitted as system messages -### M0 — Read-only spine (4 tasks remaining) -- [ ] M0.5: pi session adapter (8 task) -- [ ] M0.6: Claude transcript adapter (S task) -- [ ] M0.7: ingest --dry-run (S task) -- [ ] M0.8: M0 composition gate (M task) +#### M0.6: Claude transcript adapter +- **Status:** Complete +- **Tests:** 4 integration tests (a1-a4) +- **Deliverables:** + - `ClaudeTranscriptSource` implementing `RecordSource` trait + - Project key extraction from `cwd` field (different from pi encoding) + - Shared content flattening logic extracted and reused + - Cross-source project key agreement (pi and claude resolve identically) + - System api_error messages preserved + - Irrelevant record types skipped (attachment, queue-operation, etc.) + +#### M0.7: ingest --dry-run +- **Status:** Complete +- **Tests:** 2 integration tests (a1, a5) +- **Deliverables:** + - `mem ingest --project --dry-run` command + - Zero network calls guarantee (no outbound TCP) + - Placeholder for stats accumulation (to be enhanced in M1) + - JSON and text output formats + +#### M0.8: M0 composition gate +- **Status:** Complete +- **Tests:** 5 integration tests (gate assertions) +- **Deliverables:** + - Both `RecordSource` implementations compose identically through chunker + - Sources are swappable via trait boundary + - All four role types properly emitted + - Chunk boundaries respected: no record splitting + - Turn indices (t) guaranteed 1-based and contiguous + - Over-budget records flagged correctly + +## Remaining Tasks (43 of 51) ### M1 — Gated loop at L1 (8 tasks) - [ ] M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate @@ -86,34 +122,58 @@ 1. **Identity is content hash, not sequential ID** — ensures `mem rebuild` is idempotent 2. **JSONL log is authoritative** — vault and vector index are projections that can be rebuilt byte-identically 3. **Streams from day one** — RecordSource designed as async stream to support both batch and live sources -4. **Update gate must discriminate** — reject chunks with low evidence to prevent memory explosion -5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced at test time +4. **Sources are swappable** — both pi and claude implement RecordSource identically; same chunking logic works +5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced +6. **Shared helpers not copies** — content-flattening logic extracted so divergence between sources is impossible ## Build & Test Status ``` cargo build --workspace ✅ Compiles cleanly cargo clippy --workspace ... ✅ Zero warnings with -D warnings -cargo test ✅ 19 tests passing - - it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore) - - it_identity: 6 tests (hash stability, wire format, newtypes) - - it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized records, empty source) - - it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget holds) +cargo test ✅ 35 tests passing (1 ignored) + - it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore) + - it_identity: 6 tests (hash stability, wire format, newtypes) + - it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized, empty) + - it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget) + - it_pi_source: 5 tests (cwd parsing, role counts, content shapes, truncation, provenance) + - it_claude_source: 4 tests (cwd field, cross-source agreement, role mapping, flattener) + - it_dry_run: 2 tests (no network, empty project) + - it_m0_gate: 5 tests (composition, swappability, roles, boundaries, t values) ``` -## Next Steps +## Phase 1 Summary — M0: Read-only spine ✅ -**M0.5 (pi session adapter)** requires: -- Async streaming parser for pi session JSONL files -- Project key extraction from session `cwd` field -- Content flattening for string/block-array/structured content -- Role mapping and provenance construction -- Graceful handling of truncated lines +All 8 M0 tasks complete. The read-only spine foundation is solid: -The architecture is proven solid through M0.1-M0.4. Remaining work is implementation detail per task, with no fundamental design changes expected until the M1 gated loop introduces the LLM gateway. +- ✅ Workspace discipline: dependency direction enforced, mem-core is root +- ✅ Domain types: identity is content-hash, newtypes have no Default +- ✅ Streaming architecture: RecordSource trait proven on two different sources +- ✅ Chunking: budgets respected without splitting records, oversized records handled +- ✅ Tokenization: Qwen2 BPE integrated with hash guard, fallback counter works +- ✅ Source adapters: pi and claude sources fully functional, content flattening shared +- ✅ CLI foundation: ingest --dry-run with no network calls, extensible +- ✅ Composition: all components verified to work together, sources are swappable + +The JSONL log is ready to receive records. Both sources (pi, claude) can parse project sessions. The chunker holds budgets, respects record boundaries, and produces valid turn indices. The tokenizer counts accurately and gate-safely. + +## Next Steps → Phase M1: Gated loop at L1 + +**M1 introduces the update gate** — the LLM that decides which chunks go into memory. Key difference: M0 was read-only (no model calls), M1 will: +- M1.1: LLM chat client (gateway to Claude) +- M1.2: Standing query YAML loader +- M1.3: GRU-Mem prompt template with query fusion +- M1.4: Gate response parser (structured output extraction) +- M1.5: Gated loop — the core: ingest → chunk → query → classify → log +- M1.6: JSONL event log writer (where accepted chunks go) +- M1.7: Full `mem ingest` end-to-end +- M1.8: M1 gate — verify update rate stays under 10% (gate must discriminate) + +Phase M1 is where the memory actually learns and the "43% tool results / 2% accept rate" signature becomes visible. --- **Repository:** `/Users/rockliang/workplace/Poimen/memory` -**Last commit:** feat: complete M0.1-M0.4 phases -**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli) +**Last commit:** feat: complete M0 phase - read-only spine (8/51 tasks) +**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli) +**Test Framework:** tokio async + futures streams, integration-test harness with fixtures