poimen-memory/tasks/M3.8.1-context-optimizer.md

M3.8.1 — Context Optimizer: pre-LLM compression pipeline

Field	Value
Phase	M3.8 — Context optimization
Size	L — 3–5 days
Status	🟡 Phase 1 complete
Flags	—
Spec	docs/CONTEXT_OPTIMIZER.md
Blocks	M3.8.4
Depends	M3.7.7 (lesson.rs patterns), M3.7.8 (stop words), magika crate

Goal

Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks before they enter the GRU-Mem prompt. Search indexes stay untouched.

Deliverables

Phase 1: ContentRouter + LogCompressor (day 1) ✅

Files:

• crates/mem-core/src/optimizer/mod.rs — orchestrator
• crates/mem-core/src/optimizer/router.rs — content type detection
• crates/mem-core/src/optimizer/log.rs — log compression

ContentRouter uses Google Magika (ML) + regex fallback:

// Primary: Magika ML classifier (ONNX, <1ms per classification)
let magika = magika::Session::new()?;
let result = magika.identify_content_sync(content.as_bytes())?;
let label = result.info().label; // "json", "python", "shell", "yaml", etc.

// Map Magika labels → our compressor types
// Fallback to regex heuristics if Magika confidence < threshold

Type	Magika Labels	Regex Fallback
Json	json, jsonl	starts with { or [, valid parse
Log	txt + log heuristics	timestamp patterns, error:, npm ERR!
Diff	diff	---/+++/@@ markers
Code	python, javascript, rust, go, typescript, shell	import/use/fn/def
Config	yaml, toml, ini, xml	key-value patterns
Text	fallback	default

LogCompressor reuses M3.7.7 lesson.rs:

• markers() for error line detection
• is_cascade() for noise suppression
• strip_ansi() for cleanup
• Keep: error lines, stack traces, exit codes
• Drop: INFO/DEBUG noise, passing tests, repeated patterns

Tests (17, all passing ✅):

Unit tests (7):

• optimizer::tests::test_token_estimate
• optimizer::tests::test_optimizer_passthrough_when_disabled
• optimizer::router::tests::test_json_detection
• optimizer::router::tests::test_json_array_detection
• optimizer::router::tests::test_invalid_json_rejected
• optimizer::router::tests::test_code_detection
• optimizer::router::tests::test_diff_detection

Content Router + Fallback (4):

• optimizer::router::tests::test_log_detection_with_error_level
• optimizer::router::tests::test_log_detection_with_npm_err
• optimizer::router::tests::test_text_detection_fallback

LogCompressor (6):

• optimizer::log::tests::test_error_line_detection
• optimizer::log::tests::test_stack_line_detection
• optimizer::log::tests::test_strip_ansi
• optimizer::log::tests::test_cascade_removal
• optimizer::log::tests::test_compress_npm_error_log
• optimizer::log::tests::test_compress_rust_error_log
• optimizer::log::tests::test_compression_ratio

Integration tests (written, compiled):

• 10 scenarios in tests/it_context_optimizer.rs
  • JSON detection, Log detection, Code detection, Diff detection
  • NPM log compression, JSON array handling
  • Passthrough when disabled, compression disabled mode
  • Token counting, Real-world Cargo error scenario

Phase 2: JsonCrusher + DiffCompressor (day 2)

Files:

• crates/mem-core/src/optimizer/json.rs
• crates/mem-core/src/optimizer/diff.rs

JsonCrusher:

• Parse JSON array → analyse field variance per key
• Allocation: 30% start (schema), 15% end (recency), 55% importance
• Keep: all keys, structure, error/null/boolean fields, boundary items
• Drop: mid-array homogeneous elements, long string values

DiffCompressor:

• Keep: +/- lines (actual changes), hunk headers (@@)
• Drop: unchanged context lines, file mode changes
• Preserve enough context for the model to understand the change

Tests (10):

• json_keeps_keys, json_drops_mid_array, json_keeps_boundaries
• json_preserves_errors, json_ratio_70_90pct
• diff_keeps_changes, diff_drops_context, diff_keeps_headers
• diff_preserves_additions, diff_ratio_60_80pct

Phase 3: CacheAligner + CCR Store (day 3)

Files:

• crates/mem-core/src/optimizer/cache_align.rs
• crates/mem-core/src/optimizer/ccr.rs

CacheAligner:

• Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs)
• Reuse lesson.rs normalise() regex patterns
• Move dynamic content to tail, keep static prefix stable
• Report drift metrics

CCR Store:

• LRU cache (bounded, default 1000 entries)
• store(content) → hash / retrieve(hash) → content
• TTL-based expiry (default 1hr, matches gate run duration)
• Inject retrieval hint: <!-- CCR:hash -->

Tests (10):

• align_moves_timestamps_to_tail, align_moves_uuids_to_tail
• align_preserves_static_prefix, align_stable_across_calls
• align_drift_metrics_reported
• ccr_store_and_retrieve, ccr_lru_eviction, ccr_ttl_expiry
• ccr_hash_deterministic, ccr_inject_hint

Phase 4: TextCompressor + Integration (day 4)

Files:

• crates/mem-core/src/optimizer/text.rs
• Update crates/mem-core/src/prompt.rs — wire optimizer into build_cache_aligned

TextCompressor:

• Reuse M3.7.8 stop words for low-value token detection
• Keep: high-entropy tokens (IDs, hashes, error codes, numbers)
• Drop: filler words, repeated phrases, low-information prose

Integration:

• ContextOptimizer::optimize(chunk, config) → OptimizedChunk
• Called inside PromptBuilder::build_cache_aligned() before assembly
• Config: enable/disable per-compressor, token budget, CCR toggle
• Env var: MEM_CONTEXT_OPTIMIZER=on|off (default: on)

Tests (10):

• text_keeps_high_entropy, text_drops_filler, text_ratio_30_50pct
• optimizer_end_to_end_json, optimizer_end_to_end_log
• optimizer_skips_under_budget, optimizer_respects_budget
• optimizer_off_passthrough, optimizer_with_ccr
• prompt_builder_uses_optimizer

Verify

Integration test — tests/it_context_optimizer.rs:

1. a1_log_compression — 50-line npm error log compressed to <10 lines, all error lines preserved, ratio >80%.
2. a2_json_compression — 100-element JSON array compressed to <20 items, keys preserved, boundary items present, ratio >70%.
3. a3_diff_compression — 200-line unified diff compressed to changes only, context lines dropped, ratio >60%.
4. a4_cache_prefix_stable — Same query across 20 chunks produces identical cache prefix (system + query) after alignment.
5. a5_ccr_roundtrip — Compressed chunk has CCR hash, retrieve returns original byte-identical content.
6. a6_search_untouched — Verify optimizer is NOT called during indexing, only during prompt assembly.
7. a7_under_budget_passthrough — Chunk already under token budget passes through unmodified (zero overhead).
8. a8_no_false_negatives — Error lines, stack traces, and exit codes survive compression across all content types.

Command: cargo test -p mem-core optimizer

False pass:

• Compression ratio measured on already-small input. Fixtures must be realistic size (50+ lines for logs, 100+ elements for JSON arrays).
• Testing CCR retrieve without first compressing. The store must be populated by the compression step, not manually seeded.

Acceptance

• Log compression: >80% ratio, zero error lines lost
• JSON compression: >70% ratio, all keys preserved
• Diff compression: >60% ratio, all change lines preserved
• Cache prefix stable across chunks within a run
• CCR retrieval returns byte-identical originals
• Search indexes never see compressed content
• MEM_CONTEXT_OPTIMIZER=off disables entirely (passthrough)