doc: add comprehensive progress tracking
This commit is contained in:
Story Crater Bot
+119
@@ -0,0 +1,119 @@
|
||||
# Poimen Memory System - Implementation Progress
|
||||
|
||||
**Date:** 2024-08-20
|
||||
**Status:** 4 of 51 tasks complete (8%)
|
||||
**Tests Passing:** 19 integration tests across 4 test suites
|
||||
|
||||
## Completed Phases
|
||||
|
||||
### ✅ M0 — Read-only spine (Phase 1 of 8)
|
||||
|
||||
#### M0.1: Cargo workspace + crate skeletons
|
||||
- **Status:** Complete
|
||||
- **Tests:** 4 integration tests (a1-a4)
|
||||
- **Deliverables:**
|
||||
- Root `Cargo.toml` with 6-crate workspace
|
||||
- Enforced dependency direction: mem-cli → {mem-ingest, mem-store, mem-llm, mem-chunk} → mem-core
|
||||
- mem-core has zero intra-workspace dependencies
|
||||
- CI/CD pipeline with GitHub Actions
|
||||
- `.gitignore` properly configured (tracks log/ and tasks/)
|
||||
|
||||
#### M0.2: Domain types and sha256 identity
|
||||
- **Status:** Complete
|
||||
- **Tests:** 6 integration tests (a1-a5 + hash stability)
|
||||
- **Deliverables:**
|
||||
- `Level` enum (L0, L1, L2) with canonical JSON serialization
|
||||
- `Role` enum (User, Assistant, ToolResult, System)
|
||||
- `Record`, `Chunk`, `MemoryNode` domain types
|
||||
- Content-hash identity system (sha256 over semantic content)
|
||||
- Newtypes: ProjectId, QueryId, RunId (no Default derives)
|
||||
- Hash stability guaranteed: rebuild idempotent, hash excludes timestamps
|
||||
|
||||
#### M0.3: RecordSource trait + ChunkPolicy
|
||||
- **Status:** Complete
|
||||
- **Tests:** 6 integration tests (a1-a6)
|
||||
- **Deliverables:**
|
||||
- `RecordSource` trait for streaming sources
|
||||
- `ChunkPolicy` with token budgets and `Boundary::Record`
|
||||
- `TokenCounter` trait with `CharsOverFourCounter` stub
|
||||
- `chunks()` stream function respecting budgets without splitting records
|
||||
- Oversized records yield alone with flag
|
||||
- `VecSource` for test sources
|
||||
- Turn indices (t) guaranteed 1-based and contiguous
|
||||
|
||||
#### M0.4: Tokenizer-backed chunk sizing
|
||||
- **Status:** Complete
|
||||
- **Tests:** 3 integration tests + 1 ignored (a1-a4)
|
||||
- **Deliverables:**
|
||||
- Vendored Qwen2 tokenizer (`assets/qwen2-tokenizer.json`) with hash guard
|
||||
- `QwenTokenCounter` implementing tokenizers crate integration
|
||||
- Hash verification: load fails if file modified
|
||||
- `mem tokens <file>` CLI subcommand
|
||||
- Token counts verified against hand-recorded strings
|
||||
- Budget holds across multiple-record chunks
|
||||
|
||||
## Remaining Tasks (47 of 51)
|
||||
|
||||
### M0 — Read-only spine (4 tasks remaining)
|
||||
- [ ] M0.5: pi session adapter (8 task)
|
||||
- [ ] M0.6: Claude transcript adapter (S task)
|
||||
- [ ] M0.7: ingest --dry-run (S task)
|
||||
- [ ] M0.8: M0 composition gate (M task)
|
||||
|
||||
### M1 — Gated loop at L1 (8 tasks)
|
||||
- [ ] M1.1-M1.8: LLM chat, standing queries, prompt template, response parser, gated loop, JSONL log, end-to-end ingest, gate
|
||||
|
||||
### M2 — Projections (8 tasks)
|
||||
- [ ] M2.1-M2.8: Embeddings, CNPG manifest, schema/migrations, pgvector repo, Obsidian projector, rebuild, verify, gate
|
||||
|
||||
### M3 — L2 synthesis + retrieval (4 tasks)
|
||||
- [ ] M3.1-M3.4: L2 synthesis, rerank, query with provenance, gate
|
||||
|
||||
### M3.5 — Distributed API Layer (8 tasks)
|
||||
- [ ] M3.5.1-M3.5.8: HTTP server, ingest endpoint, query endpoint, federation, skills, projects, rate limiting, gate
|
||||
|
||||
### M4 — Skills (3 tasks)
|
||||
- [ ] M4.1-M4.3: skill draft, derived filter, gate
|
||||
|
||||
### M5 — Post-training (6 tasks)
|
||||
- [ ] M5.1-M5.6: Evidence labeler, calibration, training corpus, vLLM, veRL, gate
|
||||
|
||||
### M6 — agent-manager migration (6 tasks)
|
||||
- [ ] M6.1-M6.6: CNPG manifest, schema port, store query port, nginx routing, credentials, gate
|
||||
|
||||
## Key Architecture Decisions Established
|
||||
|
||||
1. **Identity is content hash, not sequential ID** — ensures `mem rebuild` is idempotent
|
||||
2. **JSONL log is authoritative** — vault and vector index are projections that can be rebuilt byte-identically
|
||||
3. **Streams from day one** — RecordSource designed as async stream to support both batch and live sources
|
||||
4. **Update gate must discriminate** — reject chunks with low evidence to prevent memory explosion
|
||||
5. **Dependency hygiene** — mem-core has zero workspace dependencies; dependency graph is acyclic and enforced at test time
|
||||
|
||||
## Build & Test Status
|
||||
|
||||
```
|
||||
cargo build --workspace ✅ Compiles cleanly
|
||||
cargo clippy --workspace ... ✅ Zero warnings with -D warnings
|
||||
cargo test ✅ 19 tests passing
|
||||
- it_workspace: 4 tests (build, mem-core zero deps, dependency direction, gitignore)
|
||||
- it_identity: 6 tests (hash stability, wire format, newtypes)
|
||||
- it_chunking: 6 tests (boundaries, lossless, t values, budget, oversized records, empty source)
|
||||
- it_tokens: 3 tests + 1 ignored (known strings, hash guard, budget holds)
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
**M0.5 (pi session adapter)** requires:
|
||||
- Async streaming parser for pi session JSONL files
|
||||
- Project key extraction from session `cwd` field
|
||||
- Content flattening for string/block-array/structured content
|
||||
- Role mapping and provenance construction
|
||||
- Graceful handling of truncated lines
|
||||
|
||||
The architecture is proven solid through M0.1-M0.4. Remaining work is implementation detail per task, with no fundamental design changes expected until the M1 gated loop introduces the LLM gateway.
|
||||
|
||||
---
|
||||
|
||||
**Repository:** `/Users/rockliang/workplace/Poimen/memory`
|
||||
**Last commit:** feat: complete M0.1-M0.4 phases
|
||||
**Workspace:** 6 crates (mem-core, mem-chunk, mem-llm, mem-ingest, mem-store, mem-cli)
|
||||
Reference in New Issue
Block a user