M3.8.2 — Ingest Pipeline Integration (OptimizerSink)

Field	Value
Phase	M3.8 — Context optimization
Size	M — 1–2 days
Status	✅ COMPLETE
Depends	M3.8.1 (core modules)
Blocks	M3.8.3

Goal

Wire M3.8 compressors into the ingest pipeline so that chunks are optimized BEFORE embedding + indexing, resulting in:

Better embeddings (clean text)
Better search ranking (signal-rich documents)
Cleaner results for all queries

Deliverables

1. OptimizerSink Wrapper (100 LOC)

New module: crates/mem-ingest/src/optimizer_sink.rs

pub struct OptimizerSink {
    inner: Box<dyn RecordSource>,
    optimizer: ContextOptimizer,
    config: OptimizerConfig,
    metrics: MetricsCollector,
}

impl RecordSource for OptimizerSink {
    fn next_record(&mut self) -> Option<Record> {
        let record = self.inner.next_record()?;
        let optimized = self.optimizer.optimize(&record.content)?;
        
        // Track metrics
        self.metrics.record(OptimizationMetrics {
            input_bytes: record.content.len(),
            output_bytes: optimized.compressed.len(),
            compressor: optimized.compressor_used,
            ..
        });
        
        // Emit optimized chunk
        Some(Record {
            content: optimized.compressed,
            ..record
        })
    }
}

pub fn optimize_source(
    source: Box<dyn RecordSource>,
    project: &str,
) -> Result<OptimizerSink>

Tests (3):

test_optimizer_sink_preserves_structure
test_optimizer_sink_reduces_bytes
test_optimizer_sink_handles_errors

2. Rebuild Integration (30 LOC)

Modify: crates/mem-store/src/rebuild.rs

let source = DocCorpusSource::new(vault_path)?;
let optimized = optimize_source(Box::new(source), &project)?;  // ← NEW

for record in optimized {
    let embedding = embed(&record.content)?;  // clean text
    insert_pgvector(embedding, &record)?;
    insert_opensearch(&record)?;
}

Tests (4):

test_rebuild_with_optimizer_enabled
test_rebuild_with_optimizer_disabled
test_rebuild_compression_ratio
test_rebuild_pgvector_quality_improves

3. Source Integration Tests (150 LOC)

New module: tests/it_ingest_optimizer.rs

Test each ingest source with optimizer:

test_pi_session_source_optimized (Claude transcripts)
test_doc_corpus_source_optimized (markdown files)
test_claude_transcript_source_optimized (agent logs)
test_optimizer_preserves_breadcrumb (M3.6.1 paths)
test_optimizer_respects_level (L0/L1/L2)
test_optimizer_disabled_via_env (MEM_CONTEXT_OPTIMIZER=off)

Tests (6):

Per-source integration tests

4. Metrics Collection (NEW)

Modified: crates/mem-ingest/src/lib.rs

Export MetricsCollector from OptimizerSink:

pub struct OptimizerMetrics {
    pub input_bytes: usize,
    pub output_bytes: usize,
    pub compression_ratio: f32,
    pub compressor_used: String,
    pub timestamp: i64,
    pub project: String,
}

No new tests (M3.8.3 handles metrics comprehensively)

Acceptance

✅ All 13 new tests passing ✅ OptimizerSink integrated with rebuild.rs ✅ All ingest sources work with optimizer ✅ Metrics collected (no performance regression <1ms per chunk) ✅ Backward compatible (optimizer disableable via env) ✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)

3.3 KiB Raw Blame History Unescape Escape