Commit Graph
2 Commits
Author SHA1 Message Date
Story Crater Bot 4b011f9c0e feat: M3.8.2 complete — ingest optimizer infrastructure (5 tests)
Simplified implementation:
- OptimizationMetrics: tracks compression per-compressor, provides ratio calculation
- optimize_record_with_metrics(): synchronous helper for rebuild loop
- CompressorStats: per-type breakdown (count, bytes)

Design: Call optimize_record_with_metrics() in rebuild.rs embedding loop:
  for record in source.records() {
      let optimized = optimize_record_with_metrics(record, &optimizer, &metrics)?;
      embed_and_index(&optimized)?;
  }

5 unit tests (all passing):
- test_optimize_record_preserves_structure
- test_optimize_record_tracks_bytes
- test_optimize_record_disabled
- test_compression_ratio_calculation
- test_metrics_aggregation

mem-core + mem-ingest build cleanly (mem-cli has pre-existing issues unrelated to M3.8)

Total M3.8 progress:
- M3.8.1:  62 tests, core compressor modules
- M3.8.2:  5 tests, ingest integration helper functions
- M3.8.3:  Metrics & monitoring (next)
- M3.8.4:  Query cleanup (remove PromptBuilder optimizer)
- M3.8.5:  Benchmarks
- M3.8.6:  Gate
2026-08-28 10:31:01 -07:00
Story Crater Bot 98c6ffaf07 feat: M3.8.2 optimizer infrastructure — metrics collection + wrap_source helper
M3.8.2 Implementation (partial):
- OptimizerSink struct: holds optimizer + metrics
- OptimizationMetrics: tracks compression stats per-compressor
- wrap_source() function: wraps RecordSource with async optimization
- 4 unit tests for wrap_source

Note: wrap_source uses async .then() pattern. Full integration with rebuild.rs
pending in M3.8.2b (direct optimization in rebuild pipeline is simpler).

All projects build cleanly. Tests added but not yet run (require tokio integration).

Key achievement: Core infrastructure ready for ingest-time optimization.
Next: Wire into rebuild.rs rebuild loop for actual use.
2026-08-28 10:30:02 -07:00