 Story Crater Bot
|
6a873088e6
|
feat(M5.1-M5.2): Add evidence labeler and calibration infrastructure
M5.1 — Evidence Labeler (distant supervision):
- EvidenceLabel struct: chunk_sha, t, label, why, model, ts
- LabelerConfig: configurable model_id, max_tokens, max_context
- make_label_prompt(): question + chunk in 16K context budget
- parse_label_response(): extract yes/no + 1-sentence justification
- fits_context_budget(): verify prompt fits reasoning model limits
- Unit tests: 8/8 passing
M5.2 — Labeler Calibration (Cohen's kappa):
- CalibrationResults: tp/tn/fp/fn, accuracy, kappa, precision, recall, f1
- Cohen's kappa formula (corrects for class imbalance, unlike accuracy)
- CalibrationSample: blind worksheet (hides labeler answers from human)
- stratified_sample(): 50/50 positive/negative (not corpus-proportional)
- passes_gate(): kappa >= 0.6 threshold
- Unit tests: 6/6 passing
Integration tests:
tests/it_labeler.rs: 11 tests, all passing
- a1: One label per chunk
- a2: Keyed by sha (survives re-chunking)
- a3: Context budget respected
- a4: Justifications preserved
- a5: Label structure correct
- a6: No tools in prompt (reasoning model requirement)
- a7: Parse variations (YES/no/Yes/No)
- a8-a11: Serialization, rate reporting, edge cases
tests/it_calibration.rs: 12 tests, all passing
- a1: Worksheet blind (labeler answers hidden)
- a2: Stratified sampling (attempts 50/50)
- a3: Kappa perfect agreement = 1.0
- a4: Kappa vs accuracy (high accuracy ≠ good kappa)
- a5: Confusion matrix (all 4 cells tracked)
- a6: Precision/recall separated
- a7: Gate threshold kappa >= 0.6
- a8: F1 score computed
- a9-a12: Roundtrips, disagreement analysis, formula validation
Files created:
crates/mem-llm/src/labeler.rs (250 LOC)
crates/mem-llm/src/calibration.rs (280 LOC)
tests/it_labeler.rs (200 LOC)
tests/it_calibration.rs (300 LOC)
Architecture:
M5.1: Question + Chunk → Reasoning Model → Label + Why
M5.2: Labeler Labels + Human Labels → Kappa + Confusion Matrix → Gate
Blocks: M5.3 (corpus export)
Depends: M4.3 ✓
|
2026-08-25 12:44:23 -07:00 |
|
|
Story Crater Bot
|
631cbfa3e9
|
feat: complete M0.1-M0.4 phases
M0.1 - Cargo workspace + crate skeletons
- 6-crate workspace with correct dependency direction
- CI/CD pipeline with GitHub Actions
- Integration tests verifying build and dependency structure
M0.2 - Domain types and sha256 identity
- Level (L0, L1, L2) enum with proper serde formatting
- Role enum (User, Assistant, ToolResult, System)
- Record, Chunk, and MemoryNode domain types
- Content-hash identity system ensuring rebuild idempotence
- Newtypes (ProjectId, QueryId, RunId) with validation
- Round-trip serde tests for all types
M0.3 - RecordSource trait + ChunkPolicy
- RecordSource trait for streaming record sources
- Chunk policy with token budgets and boundary modes
- TokenCounter trait with CharsOverFourCounter stub
- Chunking stream that respects budgets without splitting records
- VecSource for testing
- Integration tests verifying lossless chunking and budget adherence
M0.4 - Tokenizer-backed chunk sizing
- Vendored Qwen2 tokenizer with hash verification
- QwenTokenCounter implementing proper token counting
- Hash guard that fails on modified tokenizer
- mem tokens CLI subcommand for token counting
- Integration tests with known string counts, hash guards, and budget verification
Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
|
2026-08-22 23:13:42 -07:00 |
|
rock