poimen-memory/tasks/M3.8.1-context-optimizer.md

M3.8.1 — Context Optimizer Core Modules

Field	Value
Phase	M3.8 — Context optimization
Size	L — 3–5 days
Status	✅ COMPLETE
Spec	docs/CONTEXT_OPTIMIZER.md
Blocks	M3.8.2 (ingest integration)
Depends	M3.7.7 (lesson.rs patterns), M3.7.8 (stop words)

Status: PARTIAL ⚠️

✅ Core Compressor Modules Complete: 1,100 LOC, 62 tests

• ContentRouter (Magika ML detection)
• LogCompressor, JsonCrusher, DiffCompressor, TextCompressor
• CacheAligner (drift detection)
• CcrStore (reversible compression)
• ContextOptimizer orchestrator

❌ Integration in Wrong Place:

• Currently: PromptBuilder.build_cache_aligned() (query path)
• Should be: rebuild.rs ingest pipeline (ingest path)
• Result: Improves only LLM input, not search quality

What Was Done Right

✅ Content Detection (Magika ML + regex)

• <1ms classification
• Detects JSON, code, logs, diffs, config, text
• Thread-safe, ONNX local

✅ 5 Compressor Implementations

• LogCompressor: 85-95% ratio (keep errors + stack traces)
• JsonCrusher: 70-90% ratio (field variance)
• DiffCompressor: 60-80% ratio (change lines only)
• TextCompressor: 30-50% ratio (token importance)
• ConfigCompressor: passthrough (already compact)

✅ Cache Alignment

• Detects dynamic patterns (timestamps, UUIDs, session IDs)
• Drift metric (0.0-1.0)
• Separates stable prefix from dynamic tail

✅ Reversible Compression (CCR Store)

• LRU cache with SHA256
• TTL-based expiry
• Model can retrieve originals via hint injection

What Needs Fixing

Root Issue: Architecture Misunderstanding

Documented (❌ Wrong):

Ingest → pgvector + OpenSearch (full noise)
  ↓
Query → M3.8 compression → LLM

Should Be (✅ Correct):

Ingest → M3.8 optimization → pgvector + OpenSearch (clean)
  ↓
Query → retrieve clean results → LLM

Why the correct way is better:

1. Cleaner text → better embeddings (pgvector)
2. Signal-rich text → better BM25 ranking (OpenSearch)
3. One-time processing at ingest, not per-query
4. All users benefit from cleaner search results
5. LLM already gets optimized chunks

Next Steps

M3.8.2: Ingest Pipeline Integration (1 day)

• Create OptimizerSink wrapper around ingest sources
• Wire into rebuild.rs
• Test with all source types
• Collect metrics

M3.8.3: Metrics & Monitoring (1 day)

• Track compression ratio per chunk
• Aggregate per project/source/type
• Emit to tracing/Prometheus
• Dashboard visualization

M3.8.4: Query Path Cleanup (0.5 days)

• Remove PromptBuilder.build_cache_aligned() optimizer call
• Keep cache_metrics() for observability (drift tracking)
• Simplify PromptBuilder

Test Summary

✅ 62 Unit Tests (all passing)

• Phase 1: 17 (router, log)
• Phase 2: 15 (json, diff)
• Phase 3: 18 (cache align, CCR)
• Phase 4: 12 (text, config)

⏳ 20 New Tests Pending (M3.8.2-3)

• Ingest source optimization
• Metrics collection
• End-to-end pipeline

Files

Implemented (1,100 LOC):

• crates/mem-core/src/optimizer/mod.rs
• crates/mem-core/src/optimizer/router.rs
• crates/mem-core/src/optimizer/log.rs
• crates/mem-core/src/optimizer/json.rs
• crates/mem-core/src/optimizer/diff.rs
• crates/mem-core/src/optimizer/text.rs
• crates/mem-core/src/optimizer/cache_align.rs
• crates/mem-core/src/optimizer/ccr.rs

Pending (180 LOC):

• crates/mem-ingest/src/optimizer_sink.rs (M3.8.2)
• crates/mem-core/src/optimizer/metrics.rs (M3.8.3)

Commits

1. bf13e3a — Phase 1: ContentRouter + LogCompressor
2. a903a3f — Phase 2: JsonCrusher + DiffCompressor
3. edcc231 — Phase 3: CacheAligner + CcrStore
4. 8d8addc — Phase 4: TextCompressor + env config

Lessons Learned

1. Ingest-time optimization > query-time: Better for entire pipeline
2. Compression ratios vary widely: Log 85-95% vs text 30-50%
3. Reversibility matters: Model needs originals for detailed analysis
4. Metrics > assumption: Need to measure actual improvement in search quality

Remediation

See tasks/M3.8-CORRECTED-architecture.md for complete re-architecture plan.