feat: complete M0.1-M0.4 phases

M0.1 - Cargo workspace + crate skeletons
  - 6-crate workspace with correct dependency direction
  - CI/CD pipeline with GitHub Actions
  - Integration tests verifying build and dependency structure

M0.2 - Domain types and sha256 identity
  - Level (L0, L1, L2) enum with proper serde formatting
  - Role enum (User, Assistant, ToolResult, System)
  - Record, Chunk, and MemoryNode domain types
  - Content-hash identity system ensuring rebuild idempotence
  - Newtypes (ProjectId, QueryId, RunId) with validation
  - Round-trip serde tests for all types

M0.3 - RecordSource trait + ChunkPolicy
  - RecordSource trait for streaming record sources
  - Chunk policy with token budgets and boundary modes
  - TokenCounter trait with CharsOverFourCounter stub
  - Chunking stream that respects budgets without splitting records
  - VecSource for testing
  - Integration tests verifying lossless chunking and budget adherence

M0.4 - Tokenizer-backed chunk sizing
  - Vendored Qwen2 tokenizer with hash verification
  - QwenTokenCounter implementing proper token counting
  - Hash guard that fails on modified tokenizer
  - mem tokens CLI subcommand for token counting
  - Integration tests with known string counts, hash guards, and budget verification

Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
This commit is contained in:
Story Crater Bot
2026-08-22 23:13:42 -07:00
parent 144fa33574
commit 631cbfa3e9
36 changed files with 3379 additions and 5 deletions
+49
View File
@@ -0,0 +1,49 @@
/// Boundary mode - where chunks can be split.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Boundary {
/// Never split inside a Record
Record,
}
/// Trigger for flushing a chunk.
#[derive(Clone, Debug)]
pub enum FlushTrigger {
/// Flush when this many tokens is reached
Tokens(usize),
// OrIdle(Duration) will land with the first streaming source.
// Carrying the enum now means that change is one variant, not a signature change
// threaded through the loop.
}
/// Chunking policy.
#[derive(Clone, Debug)]
pub struct ChunkPolicy {
/// Maximum tokens per chunk (default 5000 - GRU-Mem paper default)
pub max_tokens: usize,
/// Boundary mode - never split inside a Record
pub split_on: Boundary,
/// Flush trigger
pub flush: FlushTrigger,
}
impl Default for ChunkPolicy {
fn default() -> Self {
ChunkPolicy {
max_tokens: 5000,
split_on: Boundary::Record,
flush: FlushTrigger::Tokens(5000),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_chunk_policy() {
let policy = ChunkPolicy::default();
assert_eq!(policy.max_tokens, 5000);
assert_eq!(policy.split_on, Boundary::Record);
}
}