poimen-memory/tasks/M3.8.1-context-optimizer.md

M3.8.1 — Context Optimizer: pre-LLM compression pipeline

Field	Value
Phase	M3.8 — Context optimization
Size	L — 3–5 days
Status	⬜ Not started
Flags	—
Spec	docs/CONTEXT_OPTIMIZER.md
Blocks	M3.8.4
Depends	M3.7.7 (lesson.rs patterns), M3.7.8 (stop words)

Goal

Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks before they enter the GRU-Mem prompt. Search indexes stay untouched.

Deliverables

Phase 1: ContentRouter + LogCompressor (day 1)

Files:

• crates/mem-core/src/optimizer/mod.rs — orchestrator
• crates/mem-core/src/optimizer/router.rs — content type detection
• crates/mem-core/src/optimizer/log.rs — log compression

ContentRouter detects content type via heuristics:

Type	Signal
Json	starts with { or [, valid JSON parse
Log	timestamp patterns, log levels, error:, npm ERR!
Diff	---/+++/@@ markers
Code	import/use/fn/def/class + indentation
Text	fallback

LogCompressor reuses M3.7.7 lesson.rs:

• markers() for error line detection
• is_cascade() for noise suppression
• strip_ansi() for cleanup
• Keep: error lines, stack traces, exit codes
• Drop: INFO/DEBUG noise, passing tests, repeated patterns

Tests (10):

• detect_json, detect_log, detect_diff, detect_code, detect_text
• log_keeps_errors, log_drops_info_noise, log_keeps_stack_traces
• log_compression_ratio_above_80pct, log_strips_ansi

Phase 2: JsonCrusher + DiffCompressor (day 2)

Files:

• crates/mem-core/src/optimizer/json.rs
• crates/mem-core/src/optimizer/diff.rs

JsonCrusher:

• Parse JSON array → analyse field variance per key
• Allocation: 30% start (schema), 15% end (recency), 55% importance
• Keep: all keys, structure, error/null/boolean fields, boundary items
• Drop: mid-array homogeneous elements, long string values

DiffCompressor:

• Keep: +/- lines (actual changes), hunk headers (@@)
• Drop: unchanged context lines, file mode changes
• Preserve enough context for the model to understand the change

Tests (10):

• json_keeps_keys, json_drops_mid_array, json_keeps_boundaries
• json_preserves_errors, json_ratio_70_90pct
• diff_keeps_changes, diff_drops_context, diff_keeps_headers
• diff_preserves_additions, diff_ratio_60_80pct

Phase 3: CacheAligner + CCR Store (day 3)

Files:

• crates/mem-core/src/optimizer/cache_align.rs
• crates/mem-core/src/optimizer/ccr.rs

CacheAligner:

• Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs)
• Reuse lesson.rs normalise() regex patterns
• Move dynamic content to tail, keep static prefix stable
• Report drift metrics

CCR Store:

• LRU cache (bounded, default 1000 entries)
• store(content) → hash / retrieve(hash) → content
• TTL-based expiry (default 1hr, matches gate run duration)
• Inject retrieval hint: <!-- CCR:hash -->

Tests (10):

• align_moves_timestamps_to_tail, align_moves_uuids_to_tail
• align_preserves_static_prefix, align_stable_across_calls
• align_drift_metrics_reported
• ccr_store_and_retrieve, ccr_lru_eviction, ccr_ttl_expiry
• ccr_hash_deterministic, ccr_inject_hint

Phase 4: TextCompressor + Integration (day 4)

Files:

• crates/mem-core/src/optimizer/text.rs
• Update crates/mem-core/src/prompt.rs — wire optimizer into build_cache_aligned

TextCompressor:

• Reuse M3.7.8 stop words for low-value token detection
• Keep: high-entropy tokens (IDs, hashes, error codes, numbers)
• Drop: filler words, repeated phrases, low-information prose

Integration:

• ContextOptimizer::optimize(chunk, config) → OptimizedChunk
• Called inside PromptBuilder::build_cache_aligned() before assembly
• Config: enable/disable per-compressor, token budget, CCR toggle
• Env var: MEM_CONTEXT_OPTIMIZER=on|off (default: on)

Tests (10):

• text_keeps_high_entropy, text_drops_filler, text_ratio_30_50pct
• optimizer_end_to_end_json, optimizer_end_to_end_log
• optimizer_skips_under_budget, optimizer_respects_budget
• optimizer_off_passthrough, optimizer_with_ccr
• prompt_builder_uses_optimizer

Verify

Integration test — tests/it_context_optimizer.rs:

1. a1_log_compression — 50-line npm error log compressed to <10 lines, all error lines preserved, ratio >80%.
2. a2_json_compression — 100-element JSON array compressed to <20 items, keys preserved, boundary items present, ratio >70%.
3. a3_diff_compression — 200-line unified diff compressed to changes only, context lines dropped, ratio >60%.
4. a4_cache_prefix_stable — Same query across 20 chunks produces identical cache prefix (system + query) after alignment.
5. a5_ccr_roundtrip — Compressed chunk has CCR hash, retrieve returns original byte-identical content.
6. a6_search_untouched — Verify optimizer is NOT called during indexing, only during prompt assembly.
7. a7_under_budget_passthrough — Chunk already under token budget passes through unmodified (zero overhead).
8. a8_no_false_negatives — Error lines, stack traces, and exit codes survive compression across all content types.

Command: cargo test -p mem-core optimizer

False pass:

• Compression ratio measured on already-small input. Fixtures must be realistic size (50+ lines for logs, 100+ elements for JSON arrays).
• Testing CCR retrieve without first compressing. The store must be populated by the compression step, not manually seeded.

Acceptance

• Log compression: >80% ratio, zero error lines lost
• JSON compression: >70% ratio, all keys preserved
• Diff compression: >60% ratio, all change lines preserved
• Cache prefix stable across chunks within a run
• CCR retrieval returns byte-identical originals
• Search indexes never see compressed content
• MEM_CONTEXT_OPTIMIZER=off disables entirely (passthrough)