docs: add M3.8 context optimizer to memory-flow.md

memory-flow.md

@@ -7,7 +7,9 @@
 4. [Agent Context Flow](#agent-context-flow)
 5. [System Architecture](#system-architecture)
 6. [OpenSearch + JWT Authentication](#opensearch--jwt-authentication)
-7. [Pod Infrastructure](#pod-infrastructure)
+7. [M3.7.7 → M3.7.8: Failure Diagnosis Pipeline](#m377--m378-failure-diagnosis-pipeline)
+8. [M3.8: Context Optimizer](#m38-context-optimizer)
+9. [Pod Infrastructure](#pod-infrastructure)
 ---
 
 ## Read Flow
@@ -911,6 +913,187 @@ mem sig explain --tool=npm --query="unable to resolve dependency"
 
 ---
 
+## M3.8: Context Optimizer
+
+Headroom-inspired pre-LLM compression layer. Sits between hybrid search
+retrieval and the LLM gateway. Search indexes stay at full fidelity.
+
+### Why
+
+Agent transcripts are ~43% tool results. Raw evidence chunks contain timestamps,
+temp paths, ANSI codes, verbose JSON arrays, and passing test output. Feeding
+this noise to the GRU-Mem gate wastes tokens, risks hallucination on irrelevant
+details, and breaks LLM provider KV cache (dynamic content in prefix).
+
+### Where It Sits
+
+```
+Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
+            │
+            │  full-fidelity chunks (untouched)
+            ▼
+    ┌───────────────────────┐
+    │   CONTEXT OPTIMIZER   │
+    │                       │
+    │  1. Magika Detect     │  ML content type classification (<1ms)
+    │  2. CacheAligner      │  Move timestamps/UUIDs to tail
+    │  3. Compressor         │  Per-type compression
+    │  4. CCR Store          │  Cache originals for retrieval
+    │                       │
+    └───────────┬───────────┘
+                │
+                │  optimized chunks (30–90% smaller)
+                ▼
+    ┌───────────────────────┐
+    │  Cache-Aligned Prompt │  system | query | turn
+    └───────────┬───────────┘
+                │
+                ▼
+           LLM Gateway
+```
+
+**Critical invariant:** Search indexes (pgvector + OpenSearch) NEVER see
+compressed content. Compression only happens in the prompt assembly path.
+
+### Stage 1: Content Detection (Magika ML)
+
+Google’s Magika ONNX model classifies content type in <1ms. No LLM calls,
+no network — embedded model runs locally.
+
+```rust
+use magika::Session;
+
+let magika = magika::Session::new()?;
+let result = magika.identify_content_sync(content.as_bytes())?;
+let label = result.info().label; // "json", "python", "diff", etc.
+```
+
+Falls back to regex heuristics when Magika confidence < 0.7.
+
+| Magika Label | Our Type | Compressor |
+|---|---|---|
+| `json`, `jsonl` | Json | JsonCrusher (70–90% savings) |
+| `python`, `rust`, `go`, `typescript` | Code | CodeCompressor (40–70%) |
+| `diff` | Diff | DiffCompressor (60–80%) |
+| `yaml`, `toml`, `ini` | Config | passthrough |
+| `txt` + log patterns | Log | LogCompressor (85–95%) |
+| fallback | Text | TextCompressor (30–50%) |
+
+### Stage 2: CacheAligner
+
+LLM providers cache based on exact prefix match. A single changing timestamp
+early in the prompt invalidates the entire KV cache.
+
+CacheAligner detects dynamic patterns and moves them to the context tail:
+
+```
+BEFORE (cache miss every call):
+  "At 2026-08-28T09:15:00Z, run abc123 failed with..."
+  ↑ timestamp + run ID break prefix match
+
+AFTER (cache hit on prefix):
+  "Run failed with..."            ← stable prefix (cached)
+  "[ctx: t=2026-08-28T09:15:00Z, run=abc123]"  ← dynamic tail
+```
+
+Reuses normalisation patterns from M3.7.7 `lesson.rs` (timestamp, SHA,
+path, line:col, duration, temp path regexes).
+
+### Stage 3: Per-Type Compression
+
+**JsonCrusher** — Statistical field analysis on JSON arrays:
+- Measures per-field variance, uniqueness, distribution boundaries
+- Allocation: 30% start (schema), 15% end (recency), 55% importance
+- Keeps: all keys, structure, error/null/boolean fields, boundary items
+- Drops: homogeneous mid-array elements, long string values
+
+**LogCompressor** — Reuses M3.7.7 signature extraction:
+- `markers()` for error line detection (npm, cargo, kubectl, docker)
+- `is_cascade()` for noise suppression
+- `strip_ansi()` for cleanup
+- Keeps: error lines, stack traces, exit codes, FAIL markers
+- Drops: INFO/DEBUG noise, passing tests, repeated patterns
+
+**CodeCompressor** — Signature preservation (opt-in):
+- Keeps: imports, function/method signatures, type annotations
+- Drops: function bodies, inline comments, blank lines
+- Simple brace-counting heuristics (not full AST parser)
+
+**DiffCompressor** — Change-only extraction:
+- Keeps: `+`/`-` lines (actual changes), hunk headers (`@@`)
+- Drops: unchanged context lines
+
+**TextCompressor** — Token importance scoring:
+- Reuses M3.7.8 stop words for low-value token detection
+- Keeps: high-entropy tokens (IDs, hashes, error codes)
+- Drops: filler words, repeated phrases
+
+### Stage 4: CCR Store (Compress-Cache-Retrieve)
+
+Compression is aggressive but reversible. Full originals cached with SHA256
+hash. Retrieval hint injected into compressed output:
+
+```
+[compressed evidence...]
+<!-- CCR:7f3a8bc... — full content available -->
+```
+
+If the model needs more detail, it can request the original via hash lookup.
+LRU cache with TTL (default 1hr, matches gate run duration).
+
+### Compression Targets
+
+| Content Type | Ratio | Speed | Preserved |
+|---|---|---|---|
+| JSON arrays | 70–90% | ~1ms | All keys, structure, boundaries |
+| Build logs | 85–95% | ~1ms | Errors, stack traces, exit codes |
+| Source code | 40–70% | ~2ms | Signatures, imports, types |
+| Unified diffs | 60–80% | ~1ms | Change lines, hunk headers |
+| Plain text | 30–50% | ~2ms | High-entropy tokens |
+
+### Code Reuse
+
+| Existing Module | Reused For |
+|---|---|
+| `lesson.rs` normalise() | CacheAligner pattern detection |
+| `lesson.rs` markers() | LogCompressor error detection |
+| `lesson.rs` is_cascade() | LogCompressor noise suppression |
+| `lesson.rs` strip_ansi() | Pre-processing cleanup |
+| `symptom_projection.rs` STOP_WORDS | TextCompressor low-value tokens |
+
+### Configuration
+
+```bash
+# Enable/disable (default: on)
+MEM_CONTEXT_OPTIMIZER=on
+
+# Per-compressor toggle
+MEM_COMPRESS_JSON=on
+MEM_COMPRESS_LOGS=on
+MEM_COMPRESS_CODE=off    # opt-in
+MEM_COMPRESS_DIFF=on
+MEM_COMPRESS_TEXT=on
+
+# CCR store
+MEM_CCR_ENABLED=on
+MEM_CCR_MAX_ENTRIES=1000
+MEM_CCR_TTL_SECS=3600
+```
+
+### Task Breakdown (M3.8.x)
+
+| Task | What | Status |
+|---|---|---|
+| M3.8.1 | ContentRouter (Magika) + all compressors + CCR | ⬜ |
+| M3.8.2 | CacheAligner integration with PromptBuilder | ⬜ |
+| M3.8.3 | Compression benchmarks + ratio tuning | ⬜ |
+| M3.8.4 | Composition gate | ⬜ |
+
+See `docs/CONTEXT_OPTIMIZER.md` for full design.
+Inspired by [Headroom](https://docs.headroomlabs.ai/docs/how-compression-works).
+
+---
+
 ## Pod Infrastructure
 
 Complete pod inventory deployed in `poimen` namespace.