Story Crater Bot
ecd8f510f3
M3.8.3 ✅ COMPLETE (7 tests) - MetricsCollector: per-project aggregation - Structured logging (tracing) - Prometheus export format M3.8.4 ✅ IMPLICIT (no work needed) - Query path already clean (no compression) - Only cache_metrics() uses optimizer (for observability) M3.8.5 ⏳ ACTIVE (16 tests spec'd) - Compression ratio benchmarks (5 tests: log/json/text/diff/mixed) - Search quality validation (8 tests: pgvector/opensearch/fusion) - Performance baseline (3 tests: latency/throughput/memory) M3.8.6 ⏳ PENDING (13 gate assertions) - Safety (6): no data loss, deterministic, structure preservation - Performance (4): latency p99 <3ms, throughput 1000+/sec, memory <100MB - Quality (3): compression targets, search improvement, cache accuracy Project progress: 64/78 complete (82%), 8/13 gates green Total M3.8 tests: 103 (62+5+7+0+16+13)
74 lines
2.2 KiB
Markdown
74 lines
2.2 KiB
Markdown
# M3.8.3 — Metrics & Monitoring (UPDATED)
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | M3.8 — Context optimization |
|
|
| Size | M — 1 day |
|
|
| Status | ✅ COMPLETE |
|
|
| Depends | M3.8.1, M3.8.2 |
|
|
| Blocks | M3.8.4 |
|
|
|
|
## Deliverables Completed
|
|
|
|
✅ **MetricsCollector** (7 tests)
|
|
- Per-project aggregation of OptimizationMetrics
|
|
- Merge metrics from multiple optimization runs
|
|
- Structured logging via tracing
|
|
- Prometheus-compatible export format
|
|
|
|
✅ **Test Coverage** (7 tests)
|
|
- `test_collector_merge_single_project` — Store and retrieve per-project metrics
|
|
- `test_collector_merge_multiple_projects` — Aggregate across projects
|
|
- `test_collector_merge_aggregates` — Multiple runs per project
|
|
- `test_collector_nonexistent_project` — Handle missing project gracefully
|
|
- `test_collector_per_compressor_stats` — Track per-compressor breakdowns
|
|
- `test_prometheus_export_format` — Valid Prometheus text format
|
|
- `test_prometheus_compression_ratio` — Correct ratio calculations
|
|
|
|
✅ **Prometheus Export**
|
|
- Counter: `m3_8_optimization_records_total`
|
|
- Gauge: `m3_8_optimization_input_bytes_total`
|
|
- Gauge: `m3_8_optimization_output_bytes_total`
|
|
- Gauge: `m3_8_optimization_compression_ratio`
|
|
- Per-compressor stats with labels
|
|
|
|
## Integration Pattern
|
|
|
|
```rust
|
|
use mem_ingest::{MetricsCollector, optimize_record_with_metrics, OptimizationMetrics};
|
|
use std::sync::{Arc, Mutex};
|
|
|
|
let optimizer = ContextOptimizer::from_env()?;
|
|
let collector = MetricsCollector::new();
|
|
|
|
for project_id in projects {
|
|
let metrics = Arc::new(Mutex::new(OptimizationMetrics::default()));
|
|
|
|
for record in source.records() {
|
|
let optimized = optimize_record_with_metrics(
|
|
record,
|
|
&optimizer,
|
|
&metrics,
|
|
)?;
|
|
embed_and_index(&optimized)?;
|
|
}
|
|
|
|
let final_metrics = metrics.lock().unwrap().clone();
|
|
collector.merge_project(project_id, final_metrics);
|
|
}
|
|
|
|
// Log summary
|
|
collector.log_all_projects();
|
|
|
|
// Export for Prometheus
|
|
let prometheus_text = collector.prometheus_export();
|
|
http_server.register_metrics_endpoint("/metrics", prometheus_text);
|
|
```
|
|
|
|
## Status
|
|
|
|
✅ **Complete and Production-Ready**
|
|
- All 7 tests passing
|
|
- Prometheus format validated
|
|
- Ready to integrate into rebuild pipeline
|