 Story Crater Bot
|
b18932b10c
|
feat: M3.8.1 phase 1 — content router + log compressor
ContentRouter uses Google Magika ML for content detection (<1ms) with regex
fallback. Detects JSON, code, logs, diffs, config, text.
LogCompressor reuses M3.7.7 patterns (markers, cascade, strip_ansi) to
shrink build logs by keeping errors/stacks and dropping noise.
17 unit tests passing:
- router: json, code, diff, log, text detection
- log: error lines, stack traces, ansi stripping, compression
- optimizer: token estimation, passthrough mode
Magika + ort ONNX runtime added to Cargo.toml.
|
2026-08-28 09:29:56 -07:00 |
|
|
Story Crater Bot
|
33b7150f56
|
feat: complete M0.1-M0.4 phases
M0.1 - Cargo workspace + crate skeletons
- 6-crate workspace with correct dependency direction
- CI/CD pipeline with GitHub Actions
- Integration tests verifying build and dependency structure
M0.2 - Domain types and sha256 identity
- Level (L0, L1, L2) enum with proper serde formatting
- Role enum (User, Assistant, ToolResult, System)
- Record, Chunk, and MemoryNode domain types
- Content-hash identity system ensuring rebuild idempotence
- Newtypes (ProjectId, QueryId, RunId) with validation
- Round-trip serde tests for all types
M0.3 - RecordSource trait + ChunkPolicy
- RecordSource trait for streaming record sources
- Chunk policy with token budgets and boundary modes
- TokenCounter trait with CharsOverFourCounter stub
- Chunking stream that respects budgets without splitting records
- VecSource for testing
- Integration tests verifying lossless chunking and budget adherence
M0.4 - Tokenizer-backed chunk sizing
- Vendored Qwen2 tokenizer with hash verification
- QwenTokenCounter implementing proper token counting
- Hash guard that fails on modified tokenizer
- mem tokens CLI subcommand for token counting
- Integration tests with known string counts, hash guards, and budget verification
Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
|
2026-08-22 23:13:42 -07:00 |
|