diff --git a/PROGRESS.md b/PROGRESS.md new file mode 100644 index 0000000..e300efe --- /dev/null +++ b/PROGRESS.md @@ -0,0 +1,119 @@ +# Poimen Memory System - Implementation Progress + +**Date:** 2024-08-20 +**Status:** 4 of 51 tasks complete (8%) +**Tests Passing:** 19 integration tests across 4 test suites + +## Completed Phases + +### ✅ M0 — Read-only spine (Phase 1 of 8) + +#### M0.1: Cargo workspace + crate skeletons +- **Status:** Complete +- **Tests:** 4 integration tests (a1-a4) +- **Deliverables:** + - Root `Cargo.toml` with 6-crate workspace + - Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core + - mem-core has zero intra-workspace dependencies + - CI/CD pipeline with GitHub Actions + - `.gitignore` properly configured (tracks log/ and tasks/) + +#### M0.2: Domain types and sha256 identity +- **Status:** Complete +- **Tests:** 6 integration tests (a1-a5 + hash stability) +- **Deliverables:** + - `Level` enum (L0, L1, L2) with canonical JSON serialization + - `Role` enum (User, Assistant, ToolResult, System) + - `Record`, `Chunk`, `MemoryNode` domain types + - Content-hash identity system (sha256 over semantic content) + - Newtypes: ProjectId, QueryId, RunId (no Default derives) + - Hash stability guaranteed: rebuild idempotent, hash excludes timestamps + +#### M0.3: RecordSource trait + ChunkPolicy +- **Status:** Complete +- **Tests:** 6 integration tests (a1-a6) +- **Deliverables:** + - `RecordSource` trait for streaming sources + - `ChunkPolicy` with token budgets and `Boundary::Record` + - `TokenCounter` trait with `CharsOverFourCounter` stub + - `chunks()` stream function respecting budgets without splitting records + - Oversized records yield alone with flag + - `VecSource` for test sources + - Turn indices (t) guaranteed 1-based and contiguous + +#### M0.4: Tokenizer-backed chunk sizing +- **Status:** Complete +- **Tests:** 3 integration tests + 1 ignored (a1-a4) +- **Deliverables:** + - Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash guard + - `QwenTokenCounter` implementing tokenizers crate integration + - Hash verification: load fails if file modified + - `mem tokens ` CLI subcommand + - Token counts verified against hand-recorded strings + - Budget holds across multiple-record chunks + +## Remaining Tasks (47 of 51) + +### M0 — Read-only spine (4 tasks remaining) +- [ ] M0.5: pi session adapter (8 task) +- [ ] M0.6: Claude transcript adapter (S task) +- [ ] M0.7: ingest --dry-run (S task) +- [ ] M0.8: M0 composition gate (M task) + +### M1 — Gated loop at L1 (8 tasks) +- [ ] M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate + +### M2 — Projections (8 tasks) +- [ ] M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate + +### M3 — L2 synthesis + retrieval (4 tasks) +- [ ] M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate + +### M3.5 — Distributed API Layer (8 tasks) +- [ ] M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate + +### M4 — Skills (3 tasks) +- [ ] M4.1-M4.3: skill draft, derived filter, gate + +### M5 — Post-training (6 tasks) +- [ ] M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate + +### M6 — agent-manager migration (6 tasks) +- [ ] M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate + +## Key Architecture Decisions Established + +1. **Identity is content hash, not sequential ID** — ensures `mem rebuild` is idempotent +2. **JSONL log is authoritative** — vault and vector index are projections that can be rebuilt byte-identically +3. **Streams from day one** — RecordSource designed as async stream to support both batch and live sources +4. **Update gate must discriminate** — reject chunks with low evidence to prevent memory explosion +5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced at test time + +## Build & Test Status + +``` +cargo build --workspace ✅ Compiles cleanly +cargo clippy --workspace ... ✅ Zero warnings with -D warnings +cargo test ✅ 19 tests passing + - it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore) + - it_identity: 6 tests (hash stability, wire format, newtypes) + - it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized records, empty source) + - it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget holds) +``` + +## Next Steps + +**M0.5 (pi session adapter)** requires: +- Async streaming parser for pi session JSONL files +- Project key extraction from session `cwd` field +- Content flattening for string/block-array/structured content +- Role mapping and provenance construction +- Graceful handling of truncated lines + +The architecture is proven solid through M0.1-M0.4. Remaining work is implementation detail per task, with no fundamental design changes expected until the M1 gated loop introduces the LLM gateway. + +--- + +**Repository:** `/Users/rockliang/workplace/Poimen/memory` +**Last commit:** feat: complete M0.1-M0.4 phases +**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)