poimen-memory/PROGRESS.md

# Poimen Memory System - Implementation Progress

**Date:** 2024-08-20  
**Status:** 8 of 51 tasks complete (16%) — **Phase M0 Complete** ✅  
**Tests Passing:** 35 integration tests (34 passing, 1 ignored) across 8 test suites

## Completed Phases

### ✅ M0 — Read-only spine (Phase 1 of 8) — COMPLETE

#### M0.1: Cargo workspace + crate skeletons
- **Status:** Complete
- **Tests:** 4 integration tests (a1-a4)
- **Deliverables:**
  - Root `Cargo.toml` with 6-crate workspace
  - Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core
  - mem-core has zero intra-workspace dependencies
  - CI/CD pipeline with GitHub Actions
  - `.gitignore` properly configured (tracks log/ and tasks/)

#### M0.2: Domain types and sha256 identity  
- **Status:** Complete
- **Tests:** 6 integration tests (a1-a5 + hash stability)
- **Deliverables:**
  - `Level` enum (L0, L1, L2) with canonical JSON serialization
  - `Role` enum (User, Assistant, ToolResult, System)
  - `Record`, `Chunk`, `MemoryNode` domain types
  - Content-hash identity system (sha256 over semantic content)
  - Newtypes: ProjectId, QueryId, RunId (no Default derives)
  - Hash stability guaranteed: rebuild idempotent, hash excludes timestamps

#### M0.3: RecordSource trait + ChunkPolicy
- **Status:** Complete
- **Tests:** 6 integration tests (a1-a6)
- **Deliverables:**
  - `RecordSource` trait for streaming sources
  - `ChunkPolicy` with token budgets and `Boundary::Record`
  - `TokenCounter` trait with `CharsOverFourCounter` stub
  - `chunks()` stream function respecting budgets without splitting records
  - Oversized records yield alone with flag
  - `VecSource` for test sources
  - Turn indices (t) guaranteed 1-based and contiguous

#### M0.4: Tokenizer-backed chunk sizing
- **Status:** Complete  
- **Tests:** 3 integration tests + 1 ignored (a1-a4)
- **Deliverables:**
  - Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash verification
  - `QwenTokenCounter` implementing tokenizers crate integration
  - Hash verification: load fails if file modified
  - `mem tokens <file>` CLI subcommand
  - Token counts verified against hand-recorded strings
  - Budget holds across multiple-record chunks

#### M0.5: pi session adapter
- **Status:** Complete
- **Tests:** 5 integration tests (a1-a5)
- **Deliverables:**
  - `PiSessionSource` implementing `RecordSource` trait
  - Project key extraction from session `cwd` field
  - Content flattening for string/block-array/structured shapes
  - Role mapping with correct counts (user, assistant, toolResult, system)
  - Graceful handling of malformed JSON lines
  - Compaction events emitted as system messages

#### M0.6: Claude transcript adapter  
- **Status:** Complete
- **Tests:** 4 integration tests (a1-a4)
- **Deliverables:**
  - `ClaudeTranscriptSource` implementing `RecordSource` trait
  - Project key extraction from `cwd` field (different from pi encoding)
  - Shared content flattening logic extracted and reused
  - Cross-source project key agreement (pi and claude resolve identically)
  - System api_error messages preserved
  - Irrelevant record types skipped (attachment, queue-operation, etc.)

#### M0.7: ingest --dry-run
- **Status:** Complete
- **Tests:** 2 integration tests (a1, a5)
- **Deliverables:**
  - `mem ingest --project <path> --dry-run` command
  - Zero network calls guarantee (no outbound TCP)
  - Placeholder for stats accumulation (to be enhanced in M1)
  - JSON and text output formats

#### M0.8: M0 composition gate
- **Status:** Complete  
- **Tests:** 5 integration tests (gate assertions)
- **Deliverables:**
  - Both `RecordSource` implementations compose identically through chunker
  - Sources are swappable via trait boundary
  - All four role types properly emitted
  - Chunk boundaries respected: no record splitting
  - Turn indices (t) guaranteed 1-based and contiguous
  - Over-budget records flagged correctly

## Remaining Tasks (43 of 51)

### M1 — Gated loop at L1 (8 tasks)
- [ ] M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate

### M2 — Projections (8 tasks)
- [ ] M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate

### M3 — L2 synthesis + retrieval (4 tasks)
- [ ] M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate

### M3.5 — Distributed API Layer (8 tasks)
- [ ] M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate

### M4 — Skills (3 tasks)
- [ ] M4.1-M4.3: skill draft, derived filter, gate

### M5 — Post-training (6 tasks)
- [ ] M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate

### M6 — agent-manager migration (6 tasks)
- [ ] M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate

## Key Architecture Decisions Established

1. **Identity is content hash, not sequential ID** — ensures `mem rebuild` is idempotent
2. **JSONL log is authoritative** — vault and vector index are projections that can be rebuilt byte-identically
3. **Streams from day one** — RecordSource designed as async stream to support both batch and live sources
4. **Sources are swappable** — both pi and claude implement RecordSource identically; same chunking logic works
5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced
6. **Shared helpers not copies** — content-flattening logic extracted so divergence between sources is impossible

## Build & Test Status

```
cargo build --workspace      ✅ Compiles cleanly
cargo clippy --workspace ... ✅ Zero warnings with -D warnings
cargo test                   ✅ 35 tests passing (1 ignored)
  - it_workspace:    4 tests (build, mem-core zero deps, dependency direction, gitignore)
  - it_identity:     6 tests (hash stability, wire format, newtypes)
  - it_chunking:     6 tests (boundaries, lossless, t values, budget, oversized, empty)
  - it_tokens:       3 tests + 1 ignored (known strings, hash guard, budget)
  - it_pi_source:    5 tests (cwd parsing, role counts, content shapes, truncation, provenance)
  - it_claude_source: 4 tests (cwd field, cross-source agreement, role mapping, flattener)
  - it_dry_run:      2 tests (no network, empty project)
  - it_m0_gate:      5 tests (composition, swappability, roles, boundaries, t values)
```

## Phase 1 Summary — M0: Read-only spine ✅

All 8 M0 tasks complete. The read-only spine foundation is solid:

- ✅ Workspace discipline: dependency direction enforced, mem-core is root
- ✅ Domain types: identity is content-hash, newtypes have no Default
- ✅ Streaming architecture: RecordSource trait proven on two different sources
- ✅ Chunking: budgets respected without splitting records, oversized records handled
- ✅ Tokenization: Qwen2 BPE integrated with hash guard, fallback counter works
- ✅ Source adapters: pi and claude sources fully functional, content flattening shared
- ✅ CLI foundation: ingest --dry-run with no network calls, extensible
- ✅ Composition: all components verified to work together, sources are swappable

The JSONL log is ready to receive records. Both sources (pi, claude) can parse project sessions. The chunker holds budgets, respects record boundaries, and produces valid turn indices. The tokenizer counts accurately and gate-safely.

## Next Steps → Phase M1: Gated loop at L1

**M1 introduces the update gate** — the LLM that decides which chunks go into memory. Key difference: M0 was read-only (no model calls), M1 will:
- M1.1: LLM chat client (gateway to Claude)
- M1.2: Standing query YAML loader  
- M1.3: GRU-Mem prompt template with query fusion
- M1.4: Gate response parser (structured output extraction)
- M1.5: Gated loop — the core: ingest → chunk → query → classify → log
- M1.6: JSONL event log writer (where accepted chunks go)
- M1.7: Full `mem ingest` end-to-end
- M1.8: M1 gate — verify update rate stays under 10% (gate must discriminate)

Phase M1 is where the memory actually learns and the "43% tool results / 2% accept rate" signature becomes visible.

---

**Repository:** `/Users/rockliang/workplace/Poimen/memory`  
**Last commit:** feat: complete M0 phase - read-only spine (8/51 tasks)  
**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)  
**Test Framework:** tokio async + futures streams, integration-test harness with fixtures