Story Crater Bot
ecd8f510f3
M3.8.3 ✅ COMPLETE (7 tests) - MetricsCollector: per-project aggregation - Structured logging (tracing) - Prometheus export format M3.8.4 ✅ IMPLICIT (no work needed) - Query path already clean (no compression) - Only cache_metrics() uses optimizer (for observability) M3.8.5 ⏳ ACTIVE (16 tests spec'd) - Compression ratio benchmarks (5 tests: log/json/text/diff/mixed) - Search quality validation (8 tests: pgvector/opensearch/fusion) - Performance baseline (3 tests: latency/throughput/memory) M3.8.6 ⏳ PENDING (13 gate assertions) - Safety (6): no data loss, deterministic, structure preservation - Performance (4): latency p99 <3ms, throughput 1000+/sec, memory <100MB - Quality (3): compression targets, search improvement, cache accuracy Project progress: 64/78 complete (82%), 8/13 gates green Total M3.8 tests: 103 (62+5+7+0+16+13)
172 lines
3.9 KiB
Markdown
172 lines
3.9 KiB
Markdown
# M3.8.6 — M3.8 Composition Gate
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | M3.8 — Context optimization |
|
|
| Size | M — 1 day |
|
|
| Status | ⬜ Not started |
|
|
| Depends | M3.8.5 (benchmarks) |
|
|
| Blocks | Production deployment |
|
|
|
|
## Goal
|
|
|
|
Verify M3.8 implementation meets all safety, performance, and quality constraints
|
|
before production rollout.
|
|
|
|
## Gate Assertions
|
|
|
|
### Safety (6 assertions)
|
|
|
|
1. **No data loss** — Optimized chunks preserve all semantic content
|
|
```rust
|
|
assert!(semantic_similarity(original, optimized) > 0.95);
|
|
```
|
|
|
|
2. **Deterministic output** — Same input always produces same output
|
|
```rust
|
|
assert_eq!(optimize(text), optimize(text));
|
|
```
|
|
|
|
3. **Structure preservation** — JSON/logs remain parseable
|
|
```rust
|
|
assert!(parse_json(&optimized).is_ok());
|
|
assert!(grep_logs(&optimized).count() > 0);
|
|
```
|
|
|
|
4. **Metadata preserved** — Breadcrumb, role, provenance untouched
|
|
```rust
|
|
assert_eq!(original.provenance, optimized.provenance);
|
|
assert_eq!(original.breadcrumb, optimized.breadcrumb);
|
|
```
|
|
|
|
5. **Error handling** — Graceful fallback on optimization failure
|
|
```rust
|
|
assert!(optimize_with_fallback(bad_input).is_ok());
|
|
```
|
|
|
|
6. **Thread safety** — Concurrent optimization doesn't corrupt state
|
|
```rust
|
|
assert!(concurrent_optimize(1000).all_ok());
|
|
```
|
|
|
|
### Performance (4 assertions)
|
|
|
|
1. **Latency** — Per-record optimization <3ms p99
|
|
```rust
|
|
assert!(latency_p99() < Duration::from_millis(3));
|
|
```
|
|
|
|
2. **Throughput** — Sustained 1000+ records/sec
|
|
```rust
|
|
assert!(throughput_records_per_sec() >= 1000);
|
|
```
|
|
|
|
3. **Memory** — Cache stays <100MB (max 1000 entries)
|
|
```rust
|
|
assert!(cache_size_mb() < 100);
|
|
```
|
|
|
|
4. **No regressions** — Existing tests still pass
|
|
```rust
|
|
assert!(all_prompt_tests_pass());
|
|
assert!(all_ingest_tests_pass());
|
|
```
|
|
|
|
### Quality (3 assertions)
|
|
|
|
1. **Compression targets met** — All content types
|
|
```rust
|
|
assert!(log_ratio >= 85.0 && log_ratio <= 95.0);
|
|
assert!(json_ratio >= 70.0 && json_ratio <= 90.0);
|
|
assert!(text_ratio >= 30.0 && text_ratio <= 50.0);
|
|
```
|
|
|
|
2. **Search quality improves** — pgvector + OpenSearch
|
|
```rust
|
|
assert!(embedding_similarity > 0.95);
|
|
assert!(opensearch_mrr_improvement > 10);
|
|
```
|
|
|
|
3. **No false positives** — Cache eligibility accurate
|
|
```rust
|
|
assert!(drift_metric_accurate < 0.05); // <5% error
|
|
```
|
|
|
|
## Test Implementation
|
|
|
|
File: `tests/it_m3_8_gate.rs` (400 LOC)
|
|
|
|
```rust
|
|
#[test]
|
|
fn m3_8_gate_no_data_loss() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_deterministic() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_structure_preservation() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_metadata_preservation() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_error_handling() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_thread_safety() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_latency_p99() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_throughput_sustained() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_memory_bounded() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_no_regressions() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_compression_targets() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_search_quality() { ... }
|
|
|
|
#[test]
|
|
fn m3_8_gate_cache_eligibility() { ... }
|
|
```
|
|
|
|
Total: **13 gate assertions**
|
|
|
|
## Acceptance Criteria
|
|
|
|
✅ All 13 assertions passing
|
|
✅ All 62 M3.8.1 optimizer tests passing
|
|
✅ All 5 M3.8.2 ingest tests passing
|
|
✅ All 7 M3.8.3 metrics tests passing
|
|
✅ All 16 M3.8.5 benchmark tests passing
|
|
✅ All 11 existing prompt tests passing
|
|
✅ No regressions in other modules
|
|
✅ Documentation complete
|
|
|
|
## Success Criteria
|
|
|
|
- **Safety**: 6/6 assertions ✅
|
|
- **Performance**: 4/4 assertions ✅
|
|
- **Quality**: 3/3 assertions ✅
|
|
- **Coverage**: 100% of compressors tested
|
|
- **Documentation**: BENCHMARKS.md + GATE.md
|
|
|
|
## Timeline
|
|
|
|
- M3.8.1: ✅ Done (62 tests)
|
|
- M3.8.2: ✅ Done (5 tests)
|
|
- M3.8.3: ✅ Done (7 tests)
|
|
- M3.8.4: ✅ Done (implicit, 0 tests)
|
|
- M3.8.5: ⏳ In progress (16 tests)
|
|
- M3.8.6: ⏳ Next (13 tests)
|
|
|
|
**Total M3.8**: 103 tests
|
|
**Expected gate pass rate**: 100%
|