feat: complete M0.1-M0.4 phases

M0.1 - Cargo workspace + crate skeletons
  - 6-crate workspace with correct dependency direction
  - CI/CD pipeline with GitHub Actions
  - Integration tests verifying build and dependency structure

M0.2 - Domain types and sha256 identity
  - Level (L0, L1, L2) enum with proper serde formatting
  - Role enum (User, Assistant, ToolResult, System)
  - Record, Chunk, and MemoryNode domain types
  - Content-hash identity system ensuring rebuild idempotence
  - Newtypes (ProjectId, QueryId, RunId) with validation
  - Round-trip serde tests for all types

M0.3 - RecordSource trait + ChunkPolicy
  - RecordSource trait for streaming record sources
  - Chunk policy with token budgets and boundary modes
  - TokenCounter trait with CharsOverFourCounter stub
  - Chunking stream that respects budgets without splitting records
  - VecSource for testing
  - Integration tests verifying lossless chunking and budget adherence

M0.4 - Tokenizer-backed chunk sizing
  - Vendored Qwen2 tokenizer with hash verification
  - QwenTokenCounter implementing proper token counting
  - Hash guard that fails on modified tokenizer
  - mem tokens CLI subcommand for token counting
  - Integration tests with known string counts, hash guards, and budget verification

Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
This commit is contained in:
Story Crater Bot
2026-08-22 23:13:42 -07:00
parent 144fa33574
commit 631cbfa3e9
36 changed files with 3379 additions and 5 deletions
+28
View File
@@ -0,0 +1,28 @@
name: CI
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
env:
CARGO_TERM_COLOR: always
jobs:
build:
name: Build and Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Build workspace
run: cargo build --workspace
- name: Run clippy
run: cargo clippy --workspace -- -D warnings
- name: Run tests
run: cargo test --test it_workspace