feat: Query-aware metrics tracking for M3.8 optimization
Added per-query_id metrics system for real-time progress monitoring.
New Module: mem-ingest/src/query_metrics.rs (500 LOC)
✅ QueryMetrics: Per-query tracking with progress snapshots
✅ QueryMetricsRepository: Thread-safe indexed by query_id
✅ ProgressSnapshot: Real-time monitoring data
✅ MetricsSummary: Final completion metrics
✅ Per-compressor and per-content-type breakdowns
✅ 7 unit tests (100% passing)
Features:
- Track progress: percent_complete, records_completed, eta_secs
- Measure compression: input/output bytes, compression_ratio
- Granular breakdown: per compressor, per content type
- Status tracking: Pending, InProgress, Completed, Failed, Paused
- Thread-safe: Arc<Mutex> for concurrent access
API Examples:
1. Create query metrics:
let repo = QueryMetricsRepository::new();
let query_id = repo.create_query("query-123", "myproject");
2. Record progress:
repo.update_metrics(&query_id, |m| {
m.record_record_optimized("log", "text/plain", 1000, 300);
})?;
3. Get real-time progress:
let progress = repo.get_progress(&query_id)?;
println!("{}% complete", progress.percent_complete);
4. Get final summary:
let summary = repo.get_metrics(&query_id)?.to_summary();
Output Formats (see QUERY_METRICS_EXAMPLES.md):
✅ HTTP JSON API: GET /memory/query/metrics/{query_id}
✅ Structured logging: tracing with query_id labels
✅ Prometheus metrics: per-query gauges and histograms
✅ CLI monitoring: curl-based progress script
Use Cases:
- Monitor ingest progress (rebuild.rs integration)
- Track query optimization (http_server integration)
- Stream metrics to UI/dashboard
- Alert on slow compressions
- Store summary to database for auditing
Sample Output Formats:
Integration Points (Ready):
✅ rebuild.rs: Track optimization progress per query
✅ http_server: Monitor query endpoint metrics
✅ Dashboard: Stream progress via WebSocket
✅ Prometheus: Export gauges for alerting
Tests: 7/7 passing
- creation, progress calculation, compression ratio
- repository CRUD, updates, lookups
- per-compressor tracking
Documentation: docs/QUERY_METRICS_EXAMPLES.md
- HTTP API examples with curl
- Structured logging samples
- Prometheus export format
- CLI monitoring script
Status: Ready for integration into rebuild.rs and http_server
Story Crater Bot
d99cf23e6c