poimen-memory/PROGRESS.md

# Poimen Memory System - Implementation Progress

**Date:** 2024-08-20  
**Status:** 4 of 51 tasks complete (8%)  
**Tests Passing:** 19 integration tests across 4 test suites

## Completed Phases

### ✅ M0 — Read-only spine (Phase 1 of 8)

#### M0.1: Cargo workspace + crate skeletons
- **Status:** Complete
- **Tests:** 4 integration tests (a1-a4)
- **Deliverables:**
  - Root `Cargo.toml` with 6-crate workspace
  - Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core
  - mem-core has zero intra-workspace dependencies
  - CI/CD pipeline with GitHub Actions
  - `.gitignore` properly configured (tracks log/ and tasks/)

#### M0.2: Domain types and sha256 identity  
- **Status:** Complete
- **Tests:** 6 integration tests (a1-a5 + hash stability)
- **Deliverables:**
  - `Level` enum (L0, L1, L2) with canonical JSON serialization
  - `Role` enum (User, Assistant, ToolResult, System)
  - `Record`, `Chunk`, `MemoryNode` domain types
  - Content-hash identity system (sha256 over semantic content)
  - Newtypes: ProjectId, QueryId, RunId (no Default derives)
  - Hash stability guaranteed: rebuild idempotent, hash excludes timestamps

#### M0.3: RecordSource trait + ChunkPolicy
- **Status:** Complete
- **Tests:** 6 integration tests (a1-a6)
- **Deliverables:**
  - `RecordSource` trait for streaming sources
  - `ChunkPolicy` with token budgets and `Boundary::Record`
  - `TokenCounter` trait with `CharsOverFourCounter` stub
  - `chunks()` stream function respecting budgets without splitting records
  - Oversized records yield alone with flag
  - `VecSource` for test sources
  - Turn indices (t) guaranteed 1-based and contiguous

#### M0.4: Tokenizer-backed chunk sizing
- **Status:** Complete  
- **Tests:** 3 integration tests + 1 ignored (a1-a4)
- **Deliverables:**
  - Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash guard
  - `QwenTokenCounter` implementing tokenizers crate integration
  - Hash verification: load fails if file modified
  - `mem tokens <file>` CLI subcommand
  - Token counts verified against hand-recorded strings
  - Budget holds across multiple-record chunks

## Remaining Tasks (47 of 51)

### M0 — Read-only spine (4 tasks remaining)
- [ ] M0.5: pi session adapter (8 task)
- [ ] M0.6: Claude transcript adapter (S task)
- [ ] M0.7: ingest --dry-run (S task)
- [ ] M0.8: M0 composition gate (M task)

### M1 — Gated loop at L1 (8 tasks)
- [ ] M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate

### M2 — Projections (8 tasks)
- [ ] M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate

### M3 — L2 synthesis + retrieval (4 tasks)
- [ ] M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate

### M3.5 — Distributed API Layer (8 tasks)
- [ ] M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate

### M4 — Skills (3 tasks)
- [ ] M4.1-M4.3: skill draft, derived filter, gate

### M5 — Post-training (6 tasks)
- [ ] M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate

### M6 — agent-manager migration (6 tasks)
- [ ] M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate

## Key Architecture Decisions Established

1. **Identity is content hash, not sequential ID** — ensures `mem rebuild` is idempotent
2. **JSONL log is authoritative** — vault and vector index are projections that can be rebuilt byte-identically
3. **Streams from day one** — RecordSource designed as async stream to support both batch and live sources
4. **Update gate must discriminate** — reject chunks with low evidence to prevent memory explosion
5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced at test time

## Build & Test Status

```
cargo build --workspace      ✅ Compiles cleanly
cargo clippy --workspace ... ✅ Zero warnings with -D warnings
cargo test                   ✅ 19 tests passing
  - it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore)
  - it_identity: 6 tests (hash stability, wire format, newtypes)
  - it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized records, empty source)
  - it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget holds)
```

## Next Steps

**M0.5 (pi session adapter)** requires:
- Async streaming parser for pi session JSONL files
- Project key extraction from session `cwd` field
- Content flattening for string/block-array/structured content
- Role mapping and provenance construction
- Graceful handling of truncated lines

The architecture is proven solid through M0.1-M0.4. Remaining work is implementation detail per task, with no fundamental design changes expected until the M1 gated loop introduces the LLM gateway.

---

**Repository:** `/Users/rockliang/workplace/Poimen/memory`  
**Last commit:** feat: complete M0.1-M0.4 phases  
**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)