# M3.8.1 — Context Optimizer: pre-LLM compression pipeline

| Field | Value |
|---|---|
| Phase | M3.8 — Context optimization |
| Size | L — 3–5 days |
| Status | 🟡 Phase 1 complete |
| Flags | — |
| Spec | `docs/CONTEXT_OPTIMIZER.md` |
| Blocks | M3.8.4 |
| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words), magika crate |

## Goal

Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks
before they enter the GRU-Mem prompt. Search indexes stay untouched.

## Deliverables

### Phase 1: ContentRouter + LogCompressor (day 1) ✅

**Files:**
- `crates/mem-core/src/optimizer/mod.rs` — orchestrator
- `crates/mem-core/src/optimizer/router.rs` — content type detection
- `crates/mem-core/src/optimizer/log.rs` — log compression

**ContentRouter** uses Google Magika (ML) + regex fallback:

```rust
// Primary: Magika ML classifier (ONNX, <1ms per classification)
let magika = magika::Session::new()?;
let result = magika.identify_content_sync(content.as_bytes())?;
let label = result.info().label; // "json", "python", "shell", "yaml", etc.

// Map Magika labels → our compressor types
// Fallback to regex heuristics if Magika confidence < threshold
```

| Type | Magika Labels | Regex Fallback |
|---|---|---|
| Json | `json`, `jsonl` | starts with `{` or `[`, valid parse |
| Log | `txt` + log heuristics | timestamp patterns, `error:`, `npm ERR!` |
| Diff | `diff` | `---`/`+++`/`@@` markers |
| Code | `python`, `javascript`, `rust`, `go`, `typescript`, `shell` | `import`/`use`/`fn`/`def` |
| Config | `yaml`, `toml`, `ini`, `xml` | key-value patterns |
| Text | fallback | default |

**LogCompressor** reuses M3.7.7 `lesson.rs`:
- `markers()` for error line detection
- `is_cascade()` for noise suppression
- `strip_ansi()` for cleanup
- Keep: error lines, stack traces, exit codes
- Drop: INFO/DEBUG noise, passing tests, repeated patterns

**Tests (17, all passing ✅):**

*Unit tests (7):*
- `optimizer::tests::test_token_estimate`
- `optimizer::tests::test_optimizer_passthrough_when_disabled`
- `optimizer::router::tests::test_json_detection`
- `optimizer::router::tests::test_json_array_detection`
- `optimizer::router::tests::test_invalid_json_rejected`
- `optimizer::router::tests::test_code_detection`
- `optimizer::router::tests::test_diff_detection`

*Content Router + Fallback (4):*
- `optimizer::router::tests::test_log_detection_with_error_level`
- `optimizer::router::tests::test_log_detection_with_npm_err`
- `optimizer::router::tests::test_text_detection_fallback`

*LogCompressor (6):*
- `optimizer::log::tests::test_error_line_detection`
- `optimizer::log::tests::test_stack_line_detection`
- `optimizer::log::tests::test_strip_ansi`
- `optimizer::log::tests::test_cascade_removal`
- `optimizer::log::tests::test_compress_npm_error_log`
- `optimizer::log::tests::test_compress_rust_error_log`
- `optimizer::log::tests::test_compression_ratio`

*Integration tests (written, compiled):*
- 10 scenarios in `tests/it_context_optimizer.rs`
  - JSON detection, Log detection, Code detection, Diff detection
  - NPM log compression, JSON array handling
  - Passthrough when disabled, compression disabled mode
  - Token counting, Real-world Cargo error scenario

### Phase 2: JsonCrusher + DiffCompressor (day 2)

**Files:**
- `crates/mem-core/src/optimizer/json.rs`
- `crates/mem-core/src/optimizer/diff.rs`

**JsonCrusher:**
- Parse JSON array → analyse field variance per key
- Allocation: 30% start (schema), 15% end (recency), 55% importance
- Keep: all keys, structure, error/null/boolean fields, boundary items
- Drop: mid-array homogeneous elements, long string values

**DiffCompressor:**
- Keep: `+`/`-` lines (actual changes), hunk headers (`@@`)
- Drop: unchanged context lines, file mode changes
- Preserve enough context for the model to understand the change

**Tests (10):**
- `json_keeps_keys`, `json_drops_mid_array`, `json_keeps_boundaries`
- `json_preserves_errors`, `json_ratio_70_90pct`
- `diff_keeps_changes`, `diff_drops_context`, `diff_keeps_headers`
- `diff_preserves_additions`, `diff_ratio_60_80pct`

### Phase 3: CacheAligner + CCR Store (day 3)

**Files:**
- `crates/mem-core/src/optimizer/cache_align.rs`
- `crates/mem-core/src/optimizer/ccr.rs`

**CacheAligner:**
- Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs)
- Reuse `lesson.rs` normalise() regex patterns
- Move dynamic content to tail, keep static prefix stable
- Report drift metrics

**CCR Store:**
- LRU cache (bounded, default 1000 entries)
- `store(content) → hash` / `retrieve(hash) → content`
- TTL-based expiry (default 1hr, matches gate run duration)
- Inject retrieval hint: `<!-- CCR:hash -->`

**Tests (10):**
- `align_moves_timestamps_to_tail`, `align_moves_uuids_to_tail`
- `align_preserves_static_prefix`, `align_stable_across_calls`
- `align_drift_metrics_reported`
- `ccr_store_and_retrieve`, `ccr_lru_eviction`, `ccr_ttl_expiry`
- `ccr_hash_deterministic`, `ccr_inject_hint`

### Phase 4: TextCompressor + Integration (day 4)

**Files:**
- `crates/mem-core/src/optimizer/text.rs`
- Update `crates/mem-core/src/prompt.rs` — wire optimizer into `build_cache_aligned`

**TextCompressor:**
- Reuse M3.7.8 stop words for low-value token detection
- Keep: high-entropy tokens (IDs, hashes, error codes, numbers)
- Drop: filler words, repeated phrases, low-information prose

**Integration:**
- `ContextOptimizer::optimize(chunk, config) → OptimizedChunk`
- Called inside `PromptBuilder::build_cache_aligned()` before assembly
- Config: enable/disable per-compressor, token budget, CCR toggle
- Env var: `MEM_CONTEXT_OPTIMIZER=on|off` (default: on)

**Tests (10):**
- `text_keeps_high_entropy`, `text_drops_filler`, `text_ratio_30_50pct`
- `optimizer_end_to_end_json`, `optimizer_end_to_end_log`
- `optimizer_skips_under_budget`, `optimizer_respects_budget`
- `optimizer_off_passthrough`, `optimizer_with_ccr`
- `prompt_builder_uses_optimizer`

## Verify

**Integration test** — `tests/it_context_optimizer.rs`:

1. `a1_log_compression` — 50-line npm error log compressed to <10 lines,
   all error lines preserved, ratio >80%.
2. `a2_json_compression` — 100-element JSON array compressed to <20 items,
   keys preserved, boundary items present, ratio >70%.
3. `a3_diff_compression` — 200-line unified diff compressed to changes only,
   context lines dropped, ratio >60%.
4. `a4_cache_prefix_stable` — Same query across 20 chunks produces identical
   cache prefix (system + query) after alignment.
5. `a5_ccr_roundtrip` — Compressed chunk has CCR hash, retrieve returns
   original byte-identical content.
6. `a6_search_untouched` — Verify optimizer is NOT called during indexing,
   only during prompt assembly.
7. `a7_under_budget_passthrough` — Chunk already under token budget passes
   through unmodified (zero overhead).
8. `a8_no_false_negatives` — Error lines, stack traces, and exit codes
   survive compression across all content types.

**Command:** `cargo test -p mem-core optimizer`

**False pass:**
- Compression ratio measured on already-small input. Fixtures must be
  realistic size (50+ lines for logs, 100+ elements for JSON arrays).
- Testing CCR retrieve without first compressing. The store must be
  populated by the compression step, not manually seeded.

## Acceptance

- Log compression: >80% ratio, zero error lines lost
- JSON compression: >70% ratio, all keys preserved
- Diff compression: >60% ratio, all change lines preserved
- Cache prefix stable across chunks within a run
- CCR retrieval returns byte-identical originals
- Search indexes never see compressed content
- `MEM_CONTEXT_OPTIMIZER=off` disables entirely (passthrough)