poimen-memory/tasks/M3.8.1-context-optimizer.md

# M3.8.1 — Context Optimizer: pre-LLM compression pipeline

| Field | Value |
|---|---|
| Phase | M3.8 — Context optimization |
| Size | L — 3–5 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
| Blocks | M3.8.4 |
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |

## Goal

Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks
before they enter the GRU-Mem prompt. Search indexes stay untouched.

## Deliverables

### Phase 1: ContentRouter + LogCompressor (day 1)

**Files:**
- `crates/mem-core/src/optimizer/mod.rs` — orchestrator
- `crates/mem-core/src/optimizer/router.rs` — content type detection
- `crates/mem-core/src/optimizer/log.rs` — log compression

**ContentRouter** detects content type via heuristics:

| Type | Signal |
|---|---|
| Json | starts with `{` or `[`, valid JSON parse |
| Log | timestamp patterns, log levels, `error:`, `npm ERR!` |
| Diff | `---`/`+++`/`@@` markers |
| Code | `import`/`use`/`fn`/`def`/`class` + indentation |
| Text | fallback |

**LogCompressor** reuses M3.7.7 `lesson.rs`:
- `markers()` for error line detection
- `is_cascade()` for noise suppression
- `strip_ansi()` for cleanup
- Keep: error lines, stack traces, exit codes
- Drop: INFO/DEBUG noise, passing tests, repeated patterns

**Tests (10):**
- `detect_json`, `detect_log`, `detect_diff`, `detect_code`, `detect_text`
- `log_keeps_errors`, `log_drops_info_noise`, `log_keeps_stack_traces`
- `log_compression_ratio_above_80pct`, `log_strips_ansi`

### Phase 2: JsonCrusher + DiffCompressor (day 2)

**Files:**
- `crates/mem-core/src/optimizer/json.rs`
- `crates/mem-core/src/optimizer/diff.rs`

**JsonCrusher:**
- Parse JSON array → analyse field variance per key
- Allocation: 30% start (schema), 15% end (recency), 55% importance
- Keep: all keys, structure, error/null/boolean fields, boundary items
- Drop: mid-array homogeneous elements, long string values

**DiffCompressor:**
- Keep: `+`/`-` lines (actual changes), hunk headers (`@@`)
- Drop: unchanged context lines, file mode changes
- Preserve enough context for the model to understand the change

**Tests (10):**
- `json_keeps_keys`, `json_drops_mid_array`, `json_keeps_boundaries`
- `json_preserves_errors`, `json_ratio_70_90pct`
- `diff_keeps_changes`, `diff_drops_context`, `diff_keeps_headers`
- `diff_preserves_additions`, `diff_ratio_60_80pct`

### Phase 3: CacheAligner + CCR Store (day 3)

**Files:**
- `crates/mem-core/src/optimizer/cache_align.rs`
- `crates/mem-core/src/optimizer/ccr.rs`

**CacheAligner:**
- Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs)
- Reuse `lesson.rs` normalise() regex patterns
- Move dynamic content to tail, keep static prefix stable
- Report drift metrics

**CCR Store:**
- LRU cache (bounded, default 1000 entries)
- `store(content) → hash` / `retrieve(hash) → content`
- TTL-based expiry (default 1hr, matches gate run duration)
- Inject retrieval hint: `<!-- CCR:hash -->`

**Tests (10):**
- `align_moves_timestamps_to_tail`, `align_moves_uuids_to_tail`
- `align_preserves_static_prefix`, `align_stable_across_calls`
- `align_drift_metrics_reported`
- `ccr_store_and_retrieve`, `ccr_lru_eviction`, `ccr_ttl_expiry`
- `ccr_hash_deterministic`, `ccr_inject_hint`

### Phase 4: TextCompressor + Integration (day 4)

**Files:**
- `crates/mem-core/src/optimizer/text.rs`
- Update `crates/mem-core/src/prompt.rs` — wire optimizer into `build_cache_aligned`

**TextCompressor:**
- Reuse M3.7.8 stop words for low-value token detection
- Keep: high-entropy tokens (IDs, hashes, error codes, numbers)
- Drop: filler words, repeated phrases, low-information prose

**Integration:**
- `ContextOptimizer::optimize(chunk, config) → OptimizedChunk`
- Called inside `PromptBuilder::build_cache_aligned()` before assembly
- Config: enable/disable per-compressor, token budget, CCR toggle
- Env var: `MEM_CONTEXT_OPTIMIZER=on|off` (default: on)

**Tests (10):**
- `text_keeps_high_entropy`, `text_drops_filler`, `text_ratio_30_50pct`
- `optimizer_end_to_end_json`, `optimizer_end_to_end_log`
- `optimizer_skips_under_budget`, `optimizer_respects_budget`
- `optimizer_off_passthrough`, `optimizer_with_ccr`
- `prompt_builder_uses_optimizer`

## Verify

**Integration test** — `tests/it_context_optimizer.rs`:

1. `a1_log_compression` — 50-line npm error log compressed to <10 lines,
   all error lines preserved, ratio >80%.
2. `a2_json_compression` — 100-element JSON array compressed to <20 items,
   keys preserved, boundary items present, ratio >70%.
3. `a3_diff_compression` — 200-line unified diff compressed to changes only,
   context lines dropped, ratio >60%.
4. `a4_cache_prefix_stable` — Same query across 20 chunks produces identical
   cache prefix (system + query) after alignment.
5. `a5_ccr_roundtrip` — Compressed chunk has CCR hash, retrieve returns
   original byte-identical content.
6. `a6_search_untouched` — Verify optimizer is NOT called during indexing,
   only during prompt assembly.
7. `a7_under_budget_passthrough` — Chunk already under token budget passes
   through unmodified (zero overhead).
8. `a8_no_false_negatives` — Error lines, stack traces, and exit codes
   survive compression across all content types.

**Command:** `cargo test -p mem-core optimizer`

**False pass:**
- Compression ratio measured on already-small input. Fixtures must be
  realistic size (50+ lines for logs, 100+ elements for JSON arrays).
- Testing CCR retrieve without first compressing. The store must be
  populated by the compression step, not manually seeded.

## Acceptance

- Log compression: >80% ratio, zero error lines lost
- JSON compression: >70% ratio, all keys preserved
- Diff compression: >60% ratio, all change lines preserved
- Cache prefix stable across chunks within a run
- CCR retrieval returns byte-identical originals
- Search indexes never see compressed content
- `MEM_CONTEXT_OPTIMIZER=off` disables entirely (passthrough)