feat: M3.8 query optimizer (7 tests, ready to wire)
QueryOptimizer implements query-time optimization:
- Async optimize_chunk(chunk) before LLM processing
- Batch optimize_chunks() for multiple results
- Graceful fallback: original on optimization failure
- Metrics tracking for cache alignment analysis
Features:
✓ Content-type inference (JSON/logs/diffs/text)
✓ Environment-driven configuration
✓ Optional service integration
✓ Batch processing support
✓ Metrics calculation
Tests (7 passing):
- Disabled optimizer behavior
- Environment variable handling
- Async chunk optimization
- Content-type inference (JSON, logs, diffs, text)
- Metrics calculation
Build: ✅ mem-core (137 tests total, 7 new)
Ready to wire:
1. Ingest path: optimize_record_with_metrics() in rebuild.rs
2. Query path: QueryOptimizer.optimize_chunks() before LLM context
Architecture:
Ingest: Content → M3.8 compress → clean → embed + index
Query: Search → M3.8 optimize → clean → LLM context
Next: Wire into rebuild.rs and query_executor.rs
This commit is contained in:
@@ -13,6 +13,7 @@ pub mod cache_align;
|
||||
pub mod ccr;
|
||||
pub mod plugin;
|
||||
pub mod builtin;
|
||||
pub mod query_optimizer;
|
||||
|
||||
use anyhow::Result;
|
||||
use serde::{Deserialize, Serialize};
|
||||
@@ -31,6 +32,7 @@ pub use plugin::{
|
||||
pub use builtin::{
|
||||
BuiltinOptimizer, JsonFormatter, JsonlFormatter, RawFormatter, CsvFormatter, YamlFormatter,
|
||||
};
|
||||
pub use query_optimizer::{QueryOptimizer, QueryOptimizationMetrics};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct OptimizedChunk {
|
||||
|
||||
Reference in New Issue
Block a user