feat: complete M0 phase - read-only spine (8/51 tasks)

M0.1 - Cargo workspace + crate skeletons (4 tests)
   6-crate workspace with enforced dependency direction
   GitHub Actions CI pipeline

M0.2 - Domain types and sha256 identity (6 tests)
   Level, Role, Record, Chunk, MemoryNode types
   Content-hash identity (sha256) ensuring rebuild idempotence
   Newtypes (ProjectId, QueryId, RunId) without Default

M0.3 - RecordSource trait + ChunkPolicy (6 tests)
   RecordSource streaming trait
   Chunk policy with token budgets and record boundaries
   Chunking stream that respects budgets without splitting records

M0.4 - Tokenizer-backed chunk sizing (3 tests + 1 ignored)
   Vendored Qwen2 tokenizer with hash verification
   QwenTokenCounter for accurate token counting
   mem tokens CLI subcommand

M0.5 - pi session adapter (5 tests)
   PiSessionSource implementing RecordSource
   Project key extraction from cwd field
   Content flattening for various shapes
   Shared flatten_content helper module

M0.6 - Claude transcript adapter (4 tests)
   ClaudeTranscriptSource implementing RecordSource
   Identical content flattening as pi source
   Cross-source project key agreement

M0.7 - ingest --dry-run (2 tests)
   mem ingest --project --dry-run command
   Zero network calls guarantee

M0.8 - M0 composition gate (5 tests)
   Both sources compose through chunker identically
   Sources are swappable via RecordSource trait
   All role types properly emitted
   Chunk boundaries respected, t values contiguous

Summary:
- 35 integration tests (34 passing, 1 ignored)
- Zero clippy warnings with -D warnings
- All phases compose and verify correctly
- Read-only spine foundation proves extensibility
This commit is contained in:
Story Crater Bot
2026-08-22 23:13:42 -07:00
parent f81006add2
commit 6e6de869be
12 changed files with 885 additions and 21 deletions
+2 -2
View File
@@ -15,7 +15,7 @@ pub struct CharsOverFourCounter;
impl TokenCounter for CharsOverFourCounter {
fn count(&self, record: &Record) -> usize {
// Rough heuristic: 4 characters per token
(record.text.len() + 3) / 4
record.text.len().div_ceil(4)
}
}
@@ -73,7 +73,7 @@ impl TokenCounter for QwenTokenCounter {
Ok(encoding) => encoding.get_tokens().len(),
Err(_) => {
// Fallback to character-based estimate if tokenization fails
(record.text.len() + 3) / 4
record.text.len().div_ceil(4)
}
}
}