poimen-memory/tasks/M3.8.1-context-optimizer.md

# M3.8.1 — Context Optimizer Core Modules

| Field | Value |
|---|---|
| Phase | M3.8 — Context optimization |
| Size | L — 3–5 days |
| Status | 🟡 PARTIAL (modules done, needs ingest wiring) |
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
| Blocks | M3.8.2 (ingest integration) |
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |

## Status: PARTIAL ⚠️

✅ **Core Compressor Modules Complete**: 1,100 LOC, 62 tests
- ContentRouter (Magika ML detection)
- LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
- CacheAligner (drift detection)
- CcrStore (reversible compression)
- ContextOptimizer orchestrator

❌ **Integration in Wrong Place**:
- Currently: PromptBuilder.build_cache_aligned() (query path)
- Should be: rebuild.rs ingest pipeline (ingest path)
- Result: Improves only LLM input, not search quality

## What Was Done Right

✅ **Content Detection** (Magika ML + regex)
- <1ms classification
- Detects JSON, code, logs, diffs, config, text
- Thread-safe, ONNX local

✅ **5 Compressor Implementations**
- LogCompressor: 85-95% ratio (keep errors + stack traces)
- JsonCrusher: 70-90% ratio (field variance)
- DiffCompressor: 60-80% ratio (change lines only)
- TextCompressor: 30-50% ratio (token importance)
- ConfigCompressor: passthrough (already compact)

✅ **Cache Alignment**
- Detects dynamic patterns (timestamps, UUIDs, session IDs)
- Drift metric (0.0-1.0)
- Separates stable prefix from dynamic tail

✅ **Reversible Compression** (CCR Store)
- LRU cache with SHA256
- TTL-based expiry
- Model can retrieve originals via hint injection

## What Needs Fixing

### Root Issue: Architecture Misunderstanding

**Documented** (❌ Wrong):
```
Ingest → pgvector + OpenSearch (full noise)
  ↓
Query → M3.8 compression → LLM
```

**Should Be** (✅ Correct):
```
Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
  ↓
Query → retrieve clean results → LLM
```

**Why the correct way is better:**
1. Cleaner text → better embeddings (pgvector)
2. Signal-rich text → better BM25 ranking (OpenSearch)
3. One-time processing at ingest, not per-query
4. All users benefit from cleaner search results
5. LLM already gets optimized chunks

### Next Steps

**M3.8.2**: Ingest Pipeline Integration (1 day)
- Create OptimizerSink wrapper around ingest sources
- Wire into rebuild.rs
- Test with all source types
- Collect metrics

**M3.8.3**: Metrics & Monitoring (1 day)
- Track compression ratio per chunk
- Aggregate per project/source/type
- Emit to tracing/Prometheus
- Dashboard visualization

**M3.8.4**: Query Path Cleanup (0.5 days)
- Remove PromptBuilder.build_cache_aligned() optimizer call
- Keep cache_metrics() for observability (drift tracking)
- Simplify PromptBuilder

## Test Summary

✅ **62 Unit Tests** (all passing)
- Phase 1: 17 (router, log)
- Phase 2: 15 (json, diff)
- Phase 3: 18 (cache align, CCR)
- Phase 4: 12 (text, config)

⏳ **20 New Tests Pending** (M3.8.2-3)
- Ingest source optimization
- Metrics collection
- End-to-end pipeline

## Files

**Implemented** (1,100 LOC):
- `crates/mem-core/src/optimizer/mod.rs`
- `crates/mem-core/src/optimizer/router.rs`
- `crates/mem-core/src/optimizer/log.rs`
- `crates/mem-core/src/optimizer/json.rs`
- `crates/mem-core/src/optimizer/diff.rs`
- `crates/mem-core/src/optimizer/text.rs`
- `crates/mem-core/src/optimizer/cache_align.rs`
- `crates/mem-core/src/optimizer/ccr.rs`

**Pending** (180 LOC):
- `crates/mem-ingest/src/optimizer_sink.rs` (M3.8.2)
- `crates/mem-core/src/optimizer/metrics.rs` (M3.8.3)

## Commits

1. `bf13e3a` — Phase 1: ContentRouter + LogCompressor
2. `a903a3f` — Phase 2: JsonCrusher + DiffCompressor
3. `edcc231` — Phase 3: CacheAligner + CcrStore
4. `8d8addc` — Phase 4: TextCompressor + env config

## Lessons Learned

1. **Ingest-time optimization > query-time**: Better for entire pipeline
2. **Compression ratios vary widely**: Log 85-95% vs text 30-50%
3. **Reversibility matters**: Model needs originals for detailed analysis
4. **Metrics > assumption**: Need to measure actual improvement in search quality

## Remediation

See **`tasks/M3.8-CORRECTED-architecture.md`** for complete re-architecture plan.