plan: M3.8 context optimizer (4 tasks, Headroom-inspired)

tasks/INDEX.md

@@ -66,15 +66,17 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
 | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
 | 5.5 | Reference corpora | M3.6.x | 5 | 1 | 0 | 4 | ⬜ M3.6.6 |
 | 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | ⬜ M3.7.6 |
+| 5.7 | Context optimization | M3.8.x | 4 | 0 | 0 | 4 | ⬜ M3.8.4 |
 | 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
 | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
 | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
 | 9 | Hybrid search | M8.x | 9 | 1 | 1 | 7 | ⬜ M8.9 |
-| | **Total** | | **70** | **60** | **0** | **10** | 6/11 green |
+| | **Total** | | **74** | **60** | **0** | **14** | 6/12 green |
 
 **Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
 
 **Current work:**
+- M3.8.1: Context optimizer (Headroom-inspired pre-LLM compression)
 - M8.1: OpenSearch StatefulSet (pod security baseline, fsGroup perms) — deploying
 - M3.7.4: Context endpoint glue (uses tier logic + hybrid search)
 - Obsidian service: ✅ Deployed (ppatlabs/obsidian:latest, REST API on 27124)
@@ -194,6 +196,19 @@ Ids are `M3.7.x` and frozen. `M3.7.1`, `M3.7.2`, `M3.7.3`, `M3.7.5` retired.
 | M3.7.7 | Failure signature extraction + normalisation | M | — | ✅ ARCHIVED |
 | M3.7.8 | Symptom projection at ingest | M | — | ✅ ARCHIVED |
 
+## 5.7 — Context optimization · M3.8.x
+
+Headroom-inspired pre-LLM compression. Sits between hybrid search retrieval
+and the LLM gateway. Search indexes (pgvector + OpenSearch) stay at full
+fidelity; only evidence chunks entering the prompt get optimized.
+
+| Task | Title | Size | Flags | Status |
+|---|---|---|---|---|
+| [M3.8.1](M3.8.1-context-optimizer.md) | Context optimizer pipeline (router + compressors + CCR) | L | — | ⬜ |
+| M3.8.2 | CacheAligner integration with prompt builder | M | — | ⬜ |
+| M3.8.3 | Compression benchmarks + tuning | M | — | ⬜ |
+| M3.8.4 | **M3.8 composition gate** | M | gate | ⬜ |
+
 ## 6 — Post-training · M5.x
 
 Python, separate from the Rust workspace. The boundary is the JSONL log.

tasks/M3.8.1-context-optimizer.md

@@ -0,0 +1,158 @@
+# M3.8.1 — Context Optimizer: pre-LLM compression pipeline
+
+| Field | Value |
+|---|---|
+| Phase | M3.8 — Context optimization |
+| Size | L — 3–5 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | `docs/CONTEXT_OPTIMIZER.md` |
+| Blocks | M3.8.4 |
+| Depends | M3.7.7 (lesson.rs patterns), M3.7.8 (stop words) |
+
+## Goal
+
+Build a 4-stage pre-LLM pipeline that compresses retrieved evidence chunks
+before they enter the GRU-Mem prompt. Search indexes stay untouched.
+
+## Deliverables
+
+### Phase 1: ContentRouter + LogCompressor (day 1)
+
+**Files:**
+- `crates/mem-core/src/optimizer/mod.rs` — orchestrator
+- `crates/mem-core/src/optimizer/router.rs` — content type detection
+- `crates/mem-core/src/optimizer/log.rs` — log compression
+
+**ContentRouter** detects content type via heuristics:
+
+| Type | Signal |
+|---|---|
+| Json | starts with `{` or `[`, valid JSON parse |
+| Log | timestamp patterns, log levels, `error:`, `npm ERR!` |
+| Diff | `---`/`+++`/`@@` markers |
+| Code | `import`/`use`/`fn`/`def`/`class` + indentation |
+| Text | fallback |
+
+**LogCompressor** reuses M3.7.7 `lesson.rs`:
+- `markers()` for error line detection
+- `is_cascade()` for noise suppression
+- `strip_ansi()` for cleanup
+- Keep: error lines, stack traces, exit codes
+- Drop: INFO/DEBUG noise, passing tests, repeated patterns
+
+**Tests (10):**
+- `detect_json`, `detect_log`, `detect_diff`, `detect_code`, `detect_text`
+- `log_keeps_errors`, `log_drops_info_noise`, `log_keeps_stack_traces`
+- `log_compression_ratio_above_80pct`, `log_strips_ansi`
+
+### Phase 2: JsonCrusher + DiffCompressor (day 2)
+
+**Files:**
+- `crates/mem-core/src/optimizer/json.rs`
+- `crates/mem-core/src/optimizer/diff.rs`
+
+**JsonCrusher:**
+- Parse JSON array → analyse field variance per key
+- Allocation: 30% start (schema), 15% end (recency), 55% importance
+- Keep: all keys, structure, error/null/boolean fields, boundary items
+- Drop: mid-array homogeneous elements, long string values
+
+**DiffCompressor:**
+- Keep: `+`/`-` lines (actual changes), hunk headers (`@@`)
+- Drop: unchanged context lines, file mode changes
+- Preserve enough context for the model to understand the change
+
+**Tests (10):**
+- `json_keeps_keys`, `json_drops_mid_array`, `json_keeps_boundaries`
+- `json_preserves_errors`, `json_ratio_70_90pct`
+- `diff_keeps_changes`, `diff_drops_context`, `diff_keeps_headers`
+- `diff_preserves_additions`, `diff_ratio_60_80pct`
+
+### Phase 3: CacheAligner + CCR Store (day 3)
+
+**Files:**
+- `crates/mem-core/src/optimizer/cache_align.rs`
+- `crates/mem-core/src/optimizer/ccr.rs`
+
+**CacheAligner:**
+- Detect dynamic patterns in prompt prefix (timestamps, UUIDs, temp paths, SHAs)
+- Reuse `lesson.rs` normalise() regex patterns
+- Move dynamic content to tail, keep static prefix stable
+- Report drift metrics
+
+**CCR Store:**
+- LRU cache (bounded, default 1000 entries)
+- `store(content) → hash` / `retrieve(hash) → content`
+- TTL-based expiry (default 1hr, matches gate run duration)
+- Inject retrieval hint: `<!-- CCR:hash -->`
+
+**Tests (10):**
+- `align_moves_timestamps_to_tail`, `align_moves_uuids_to_tail`
+- `align_preserves_static_prefix`, `align_stable_across_calls`
+- `align_drift_metrics_reported`
+- `ccr_store_and_retrieve`, `ccr_lru_eviction`, `ccr_ttl_expiry`
+- `ccr_hash_deterministic`, `ccr_inject_hint`
+
+### Phase 4: TextCompressor + Integration (day 4)
+
+**Files:**
+- `crates/mem-core/src/optimizer/text.rs`
+- Update `crates/mem-core/src/prompt.rs` — wire optimizer into `build_cache_aligned`
+
+**TextCompressor:**
+- Reuse M3.7.8 stop words for low-value token detection
+- Keep: high-entropy tokens (IDs, hashes, error codes, numbers)
+- Drop: filler words, repeated phrases, low-information prose
+
+**Integration:**
+- `ContextOptimizer::optimize(chunk, config) → OptimizedChunk`
+- Called inside `PromptBuilder::build_cache_aligned()` before assembly
+- Config: enable/disable per-compressor, token budget, CCR toggle
+- Env var: `MEM_CONTEXT_OPTIMIZER=on|off` (default: on)
+
+**Tests (10):**
+- `text_keeps_high_entropy`, `text_drops_filler`, `text_ratio_30_50pct`
+- `optimizer_end_to_end_json`, `optimizer_end_to_end_log`
+- `optimizer_skips_under_budget`, `optimizer_respects_budget`
+- `optimizer_off_passthrough`, `optimizer_with_ccr`
+- `prompt_builder_uses_optimizer`
+
+## Verify
+
+**Integration test** — `tests/it_context_optimizer.rs`:
+
+1. `a1_log_compression` — 50-line npm error log compressed to <10 lines,
+   all error lines preserved, ratio >80%.
+2. `a2_json_compression` — 100-element JSON array compressed to <20 items,
+   keys preserved, boundary items present, ratio >70%.
+3. `a3_diff_compression` — 200-line unified diff compressed to changes only,
+   context lines dropped, ratio >60%.
+4. `a4_cache_prefix_stable` — Same query across 20 chunks produces identical
+   cache prefix (system + query) after alignment.
+5. `a5_ccr_roundtrip` — Compressed chunk has CCR hash, retrieve returns
+   original byte-identical content.
+6. `a6_search_untouched` — Verify optimizer is NOT called during indexing,
+   only during prompt assembly.
+7. `a7_under_budget_passthrough` — Chunk already under token budget passes
+   through unmodified (zero overhead).
+8. `a8_no_false_negatives` — Error lines, stack traces, and exit codes
+   survive compression across all content types.
+
+**Command:** `cargo test -p mem-core optimizer`
+
+**False pass:**
+- Compression ratio measured on already-small input. Fixtures must be
+  realistic size (50+ lines for logs, 100+ elements for JSON arrays).
+- Testing CCR retrieve without first compressing. The store must be
+  populated by the compression step, not manually seeded.
+
+## Acceptance
+
+- Log compression: >80% ratio, zero error lines lost
+- JSON compression: >70% ratio, all keys preserved
+- Diff compression: >60% ratio, all change lines preserved
+- Cache prefix stable across chunks within a run
+- CCR retrieval returns byte-identical originals
+- Search indexes never see compressed content
+- `MEM_CONTEXT_OPTIMIZER=off` disables entirely (passthrough)