poimen-memory/PROGRESS.md

Poimen Memory System - Implementation Progress

Date: 2024-08-20
Status: 4 of 51 tasks complete (8%)
Tests Passing: 19 integration tests across 4 test suites

Completed Phases

✅ M0 — Read-only spine (Phase 1 of 8)

M0.1: Cargo workspace + crate skeletons

• Status: Complete
• Tests: 4 integration tests (a1-a4)
• Deliverables:
  • Root Cargo.toml with 6-crate workspace
  • Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core
  • mem-core has zero intra-workspace dependencies
  • CI/CD pipeline with GitHub Actions
  • .gitignore properly configured (tracks log/ and tasks/)

M0.2: Domain types and sha256 identity

• Status: Complete
• Tests: 6 integration tests (a1-a5 + hash stability)
• Deliverables:
  • Level enum (L0, L1, L2) with canonical JSON serialization
  • Role enum (User, Assistant, ToolResult, System)
  • Record, Chunk, MemoryNode domain types
  • Content-hash identity system (sha256 over semantic content)
  • Newtypes: ProjectId, QueryId, RunId (no Default derives)
  • Hash stability guaranteed: rebuild idempotent, hash excludes timestamps

M0.3: RecordSource trait + ChunkPolicy

• Status: Complete
• Tests: 6 integration tests (a1-a6)
• Deliverables:
  • RecordSource trait for streaming sources
  • ChunkPolicy with token budgets and Boundary::Record
  • TokenCounter trait with CharsOverFourCounter stub
  • chunks() stream function respecting budgets without splitting records
  • Oversized records yield alone with flag
  • VecSource for test sources
  • Turn indices (t) guaranteed 1-based and contiguous

M0.4: Tokenizer-backed chunk sizing

• Status: Complete
• Tests: 3 integration tests + 1 ignored (a1-a4)
• Deliverables:
  • Vendored Qwen2 tokenizer (assets/qwen2-tokenizer.json) with hash guard
  • QwenTokenCounter implementing tokenizers crate integration
  • Hash verification: load fails if file modified
  • mem tokens <file> CLI subcommand
  • Token counts verified against hand-recorded strings
  • Budget holds across multiple-record chunks

Remaining Tasks (47 of 51)

M0 — Read-only spine (4 tasks remaining)

• M0.5: pi session adapter (8 task)
• M0.6: Claude transcript adapter (S task)
• M0.7: ingest --dry-run (S task)
• M0.8: M0 composition gate (M task)

M1 — Gated loop at L1 (8 tasks)

• M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate

M2 — Projections (8 tasks)

• M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate

M3 — L2 synthesis + retrieval (4 tasks)

• M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate

M3.5 — Distributed API Layer (8 tasks)

• M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate

M4 — Skills (3 tasks)

• M4.1-M4.3: skill draft, derived filter, gate

M5 — Post-training (6 tasks)

• M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate

M6 — agent-manager migration (6 tasks)

• M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate

Key Architecture Decisions Established

1. Identity is content hash, not sequential ID — ensures mem rebuild is idempotent
2. JSONL log is authoritative — vault and vector index are projections that can be rebuilt byte-identically
3. Streams from day one — RecordSource designed as async stream to support both batch and live sources
4. Update gate must discriminate — reject chunks with low evidence to prevent memory explosion
5. Dependency hygiene — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced at test time

Build & Test Status

cargo build --workspace      ✅ Compiles cleanly
cargo clippy --workspace ... ✅ Zero warnings with -D warnings
cargo test                   ✅ 19 tests passing
  - it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore)
  - it_identity: 6 tests (hash stability, wire format, newtypes)
  - it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized records, empty source)
  - it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget holds)

Next Steps

M0.5 (pi session adapter) requires:

• Async streaming parser for pi session JSONL files
• Project key extraction from session cwd field
• Content flattening for string/block-array/structured content
• Role mapping and provenance construction
• Graceful handling of truncated lines

The architecture is proven solid through M0.1-M0.4. Remaining work is implementation detail per task, with no fundamental design changes expected until the M1 gated loop introduces the LLM gateway.

---

Repository: /Users/rockliang/workplace/Poimen/memory
Last commit: feat: complete M0.1-M0.4 phases
Workspace: 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)