poimen-memory/tasks/M3.8.2-cache-aligner-headers.md

# M3.8.2 — Ingest Pipeline Integration (OptimizerSink)

| Field | Value |
|---|---|
| Phase | M3.8 — Context optimization |
| Size | M — 1–2 days |
| Status | ✅ COMPLETE |
| Depends | M3.8.1 (core modules) |
| Blocks | M3.8.3 |

## Goal

Wire M3.8 compressors into the ingest pipeline so that chunks are optimized
BEFORE embedding + indexing, resulting in:
- Better embeddings (clean text)
- Better search ranking (signal-rich documents)
- Cleaner results for all queries

## Deliverables

### 1. OptimizerSink Wrapper (100 LOC)

New module: `crates/mem-ingest/src/optimizer_sink.rs`

```rust
pub struct OptimizerSink {
    inner: Box<dyn RecordSource>,
    optimizer: ContextOptimizer,
    config: OptimizerConfig,
    metrics: MetricsCollector,
}

impl RecordSource for OptimizerSink {
    fn next_record(&mut self) -> Option<Record> {
        let record = self.inner.next_record()?;
        let optimized = self.optimizer.optimize(&record.content)?;
        
        // Track metrics
        self.metrics.record(OptimizationMetrics {
            input_bytes: record.content.len(),
            output_bytes: optimized.compressed.len(),
            compressor: optimized.compressor_used,
            ..
        });
        
        // Emit optimized chunk
        Some(Record {
            content: optimized.compressed,
            ..record
        })
    }
}

pub fn optimize_source(
    source: Box<dyn RecordSource>,
    project: &str,
) -> Result<OptimizerSink>
```

Tests (3):
- `test_optimizer_sink_preserves_structure`
- `test_optimizer_sink_reduces_bytes`
- `test_optimizer_sink_handles_errors`

### 2. Rebuild Integration (30 LOC)

Modify: `crates/mem-store/src/rebuild.rs`

```rust
let source = DocCorpusSource::new(vault_path)?;
let optimized = optimize_source(Box::new(source), &project)?;  // ← NEW

for record in optimized {
    let embedding = embed(&record.content)?;  // clean text
    insert_pgvector(embedding, &record)?;
    insert_opensearch(&record)?;
}
```

Tests (4):
- `test_rebuild_with_optimizer_enabled`
- `test_rebuild_with_optimizer_disabled`
- `test_rebuild_compression_ratio`
- `test_rebuild_pgvector_quality_improves`

### 3. Source Integration Tests (150 LOC)

New module: `tests/it_ingest_optimizer.rs`

Test each ingest source with optimizer:
- `test_pi_session_source_optimized` (Claude transcripts)
- `test_doc_corpus_source_optimized` (markdown files)
- `test_claude_transcript_source_optimized` (agent logs)
- `test_optimizer_preserves_breadcrumb` (M3.6.1 paths)
- `test_optimizer_respects_level` (L0/L1/L2)
- `test_optimizer_disabled_via_env` (MEM_CONTEXT_OPTIMIZER=off)

Tests (6):
- Per-source integration tests

### 4. Metrics Collection (NEW)

Modified: `crates/mem-ingest/src/lib.rs`

Export MetricsCollector from OptimizerSink:
```rust
pub struct OptimizerMetrics {
    pub input_bytes: usize,
    pub output_bytes: usize,
    pub compression_ratio: f32,
    pub compressor_used: String,
    pub timestamp: i64,
    pub project: String,
}
```

No new tests (M3.8.3 handles metrics comprehensively)

## Acceptance

✅ All 13 new tests passing
✅ OptimizerSink integrated with rebuild.rs
✅ All ingest sources work with optimizer
✅ Metrics collected (no performance regression <1ms per chunk)
✅ Backward compatible (optimizer disableable via env)
✅ Compression ratios match targets (log 85-95%, json 70-90%, etc.)