feat: complete M0.1-M0.4 phases
M0.1 - Cargo workspace + crate skeletons - 6-crate workspace with correct dependency direction - CI/CD pipeline with GitHub Actions - Integration tests verifying build and dependency structure M0.2 - Domain types and sha256 identity - Level (L0, L1, L2) enum with proper serde formatting - Role enum (User, Assistant, ToolResult, System) - Record, Chunk, and MemoryNode domain types - Content-hash identity system ensuring rebuild idempotence - Newtypes (ProjectId, QueryId, RunId) with validation - Round-trip serde tests for all types M0.3 - RecordSource trait + ChunkPolicy - RecordSource trait for streaming record sources - Chunk policy with token budgets and boundary modes - TokenCounter trait with CharsOverFourCounter stub - Chunking stream that respects budgets without splitting records - VecSource for testing - Integration tests verifying lossless chunking and budget adherence M0.4 - Tokenizer-backed chunk sizing - Vendored Qwen2 tokenizer with hash verification - QwenTokenCounter implementing proper token counting - Hash guard that fails on modified tokenizer - mem tokens CLI subcommand for token counting - Integration tests with known string counts, hash guards, and budget verification Total: 19 integration tests passing, all phases verified to compose correctly Workspace builds cleanly with no clippy warnings
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
/// Boundary mode - where chunks can be split.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Boundary {
|
||||
/// Never split inside a Record
|
||||
Record,
|
||||
}
|
||||
|
||||
/// Trigger for flushing a chunk.
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum FlushTrigger {
|
||||
/// Flush when this many tokens is reached
|
||||
Tokens(usize),
|
||||
// OrIdle(Duration) will land with the first streaming source.
|
||||
// Carrying the enum now means that change is one variant, not a signature change
|
||||
// threaded through the loop.
|
||||
}
|
||||
|
||||
/// Chunking policy.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ChunkPolicy {
|
||||
/// Maximum tokens per chunk (default 5000 - GRU-Mem paper default)
|
||||
pub max_tokens: usize,
|
||||
/// Boundary mode - never split inside a Record
|
||||
pub split_on: Boundary,
|
||||
/// Flush trigger
|
||||
pub flush: FlushTrigger,
|
||||
}
|
||||
|
||||
impl Default for ChunkPolicy {
|
||||
fn default() -> Self {
|
||||
ChunkPolicy {
|
||||
max_tokens: 5000,
|
||||
split_on: Boundary::Record,
|
||||
flush: FlushTrigger::Tokens(5000),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_default_chunk_policy() {
|
||||
let policy = ChunkPolicy::default();
|
||||
assert_eq!(policy.max_tokens, 5000);
|
||||
assert_eq!(policy.split_on, Boundary::Record);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user