poimen-memory/PROGRESS.md

Poimen Memory System - Implementation Progress

Date: 2024-08-20
Status: 8 of 51 tasks complete (16%) — Phase M0 Complete ✅
Tests Passing: 35 integration tests (34 passing, 1 ignored) across 8 test suites

Completed Phases

✅ M0 — Read-only spine (Phase 1 of 8) — COMPLETE

M0.1: Cargo workspace + crate skeletons

• Status: Complete
• Tests: 4 integration tests (a1-a4)
• Deliverables:
  • Root Cargo.toml with 6-crate workspace
  • Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core
  • mem-core has zero intra-workspace dependencies
  • CI/CD pipeline with GitHub Actions
  • .gitignore properly configured (tracks log/ and tasks/)

M0.2: Domain types and sha256 identity

• Status: Complete
• Tests: 6 integration tests (a1-a5 + hash stability)
• Deliverables:
  • Level enum (L0, L1, L2) with canonical JSON serialization
  • Role enum (User, Assistant, ToolResult, System)
  • Record, Chunk, MemoryNode domain types
  • Content-hash identity system (sha256 over semantic content)
  • Newtypes: ProjectId, QueryId, RunId (no Default derives)
  • Hash stability guaranteed: rebuild idempotent, hash excludes timestamps

M0.3: RecordSource trait + ChunkPolicy

• Status: Complete
• Tests: 6 integration tests (a1-a6)
• Deliverables:
  • RecordSource trait for streaming sources
  • ChunkPolicy with token budgets and Boundary::Record
  • TokenCounter trait with CharsOverFourCounter stub
  • chunks() stream function respecting budgets without splitting records
  • Oversized records yield alone with flag
  • VecSource for test sources
  • Turn indices (t) guaranteed 1-based and contiguous

M0.4: Tokenizer-backed chunk sizing

• Status: Complete
• Tests: 3 integration tests + 1 ignored (a1-a4)
• Deliverables:
  • Vendored Qwen2 tokenizer (assets/qwen2-tokenizer.json) with hash verification
  • QwenTokenCounter implementing tokenizers crate integration
  • Hash verification: load fails if file modified
  • mem tokens <file> CLI subcommand
  • Token counts verified against hand-recorded strings
  • Budget holds across multiple-record chunks

M0.5: pi session adapter

• Status: Complete
• Tests: 5 integration tests (a1-a5)
• Deliverables:
  • PiSessionSource implementing RecordSource trait
  • Project key extraction from session cwd field
  • Content flattening for string/block-array/structured shapes
  • Role mapping with correct counts (user, assistant, toolResult, system)
  • Graceful handling of malformed JSON lines
  • Compaction events emitted as system messages

M0.6: Claude transcript adapter

• Status: Complete
• Tests: 4 integration tests (a1-a4)
• Deliverables:
  • ClaudeTranscriptSource implementing RecordSource trait
  • Project key extraction from cwd field (different from pi encoding)
  • Shared content flattening logic extracted and reused
  • Cross-source project key agreement (pi and claude resolve identically)
  • System api_error messages preserved
  • Irrelevant record types skipped (attachment, queue-operation, etc.)

M0.7: ingest --dry-run

• Status: Complete
• Tests: 2 integration tests (a1, a5)
• Deliverables:
  • mem ingest --project <path> --dry-run command
  • Zero network calls guarantee (no outbound TCP)
  • Placeholder for stats accumulation (to be enhanced in M1)
  • JSON and text output formats

M0.8: M0 composition gate

• Status: Complete
• Tests: 5 integration tests (gate assertions)
• Deliverables:
  • Both RecordSource implementations compose identically through chunker
  • Sources are swappable via trait boundary
  • All four role types properly emitted
  • Chunk boundaries respected: no record splitting
  • Turn indices (t) guaranteed 1-based and contiguous
  • Over-budget records flagged correctly

Remaining Tasks (43 of 51)

M1 — Gated loop at L1 (8 tasks)

• M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate

M2 — Projections (8 tasks)

• M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate

M3 — L2 synthesis + retrieval (4 tasks)

• M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate

M3.5 — Distributed API Layer (8 tasks)

• M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate

M4 — Skills (3 tasks)

• M4.1-M4.3: skill draft, derived filter, gate

M5 — Post-training (6 tasks)

• M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate

M6 — agent-manager migration (6 tasks)

• M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate

Key Architecture Decisions Established

1. Identity is content hash, not sequential ID — ensures mem rebuild is idempotent
2. JSONL log is authoritative — vault and vector index are projections that can be rebuilt byte-identically
3. Streams from day one — RecordSource designed as async stream to support both batch and live sources
4. Sources are swappable — both pi and claude implement RecordSource identically; same chunking logic works
5. Dependency hygiene — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced
6. Shared helpers not copies — content-flattening logic extracted so divergence between sources is impossible

Build & Test Status

cargo build --workspace      ✅ Compiles cleanly
cargo clippy --workspace ... ✅ Zero warnings with -D warnings
cargo test                   ✅ 35 tests passing (1 ignored)
  - it_workspace:    4 tests (build, mem-core zero deps, dependency direction, gitignore)
  - it_identity:     6 tests (hash stability, wire format, newtypes)
  - it_chunking:     6 tests (boundaries, lossless, t values, budget, oversized, empty)
  - it_tokens:       3 tests + 1 ignored (known strings, hash guard, budget)
  - it_pi_source:    5 tests (cwd parsing, role counts, content shapes, truncation, provenance)
  - it_claude_source: 4 tests (cwd field, cross-source agreement, role mapping, flattener)
  - it_dry_run:      2 tests (no network, empty project)
  - it_m0_gate:      5 tests (composition, swappability, roles, boundaries, t values)

Phase 1 Summary — M0: Read-only spine ✅

All 8 M0 tasks complete. The read-only spine foundation is solid:

• ✅ Workspace discipline: dependency direction enforced, mem-core is root
• ✅ Domain types: identity is content-hash, newtypes have no Default
• ✅ Streaming architecture: RecordSource trait proven on two different sources
• ✅ Chunking: budgets respected without splitting records, oversized records handled
• ✅ Tokenization: Qwen2 BPE integrated with hash guard, fallback counter works
• ✅ Source adapters: pi and claude sources fully functional, content flattening shared
• ✅ CLI foundation: ingest --dry-run with no network calls, extensible
• ✅ Composition: all components verified to work together, sources are swappable

The JSONL log is ready to receive records. Both sources (pi, claude) can parse project sessions. The chunker holds budgets, respects record boundaries, and produces valid turn indices. The tokenizer counts accurately and gate-safely.

Next Steps → Phase M1: Gated loop at L1

M1 introduces the update gate — the LLM that decides which chunks go into memory. Key difference: M0 was read-only (no model calls), M1 will:

• M1.1: LLM chat client (gateway to Claude)
• M1.2: Standing query YAML loader
• M1.3: GRU-Mem prompt template with query fusion
• M1.4: Gate response parser (structured output extraction)
• M1.5: Gated loop — the core: ingest → chunk → query → classify → log
• M1.6: JSONL event log writer (where accepted chunks go)
• M1.7: Full mem ingest end-to-end
• M1.8: M1 gate — verify update rate stays under 10% (gate must discriminate)

Phase M1 is where the memory actually learns and the "43% tool results / 2% accept rate" signature becomes visible.

---

Repository: /Users/rockliang/workplace/Poimen/memory
Last commit: feat: complete M0 phase - read-only spine (8/51 tasks)
Workspace: 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)
Test Framework: tokio async + futures streams, integration-test harness with fixtures