poimen-memory/docs/CONTEXT_OPTIMIZER.md

# Context Optimizer — Pre-LLM Compression Layer

## Motivation

Agent transcripts and tool outputs are noisy. A 50-chunk ingestion run might
feed the GRU-Mem gate evidence that's 43% tool results, full of timestamps,
temp paths, ANSI codes, and verbose JSON. The model wastes tokens parsing noise,
risks hallucinating on irrelevant details, and we pay full price for bloated
input. LLM provider KV caches miss because dynamic content (timestamps, session
IDs) pollutes the prefix.

**This layer sits between retrieval and the LLM call.** Search indexes
(pgvector + OpenSearch) stay untouched at full fidelity. Only the evidence
chunks entering the prompt get optimized.

## Architecture

```
Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
            │
            │  full-fidelity chunks (untouched)
            ▼
    ┌───────────────────────┐
    │   CONTEXT OPTIMIZER   │  ← THIS MODULE
    │                       │
    │  1. CacheAligner      │  Move dynamic content (timestamps, UUIDs)
    │                       │  to end of context. Keep static prefix
    │                       │  stable for KV cache hits.
    │                       │
    │  2. ContentRouter     │  Auto-detect content type per chunk:
    │                       │  JSON, code, logs, diffs, plain text.
    │                       │  Route each to best compressor.
    │                       │
    │  3. Compressors       │  Per-type compression:
    │     - JsonCrusher     │    Statistical field analysis, keep keys
    │     - LogCompressor   │    Keep errors/stack traces, drop noise
    │     - CodeCompressor  │    AST-aware: keep signatures, drop bodies
    │     - TextCompressor  │    Token importance scoring
    │     - DiffCompressor  │    Keep change hunks, drop context
    │                       │
    │  4. CCR Store         │  Cache full originals with hash reference.
    │                       │  Inject retrieval hint so model can fetch
    │                       │  full content if needed.
    │                       │
    └───────────┬───────────┘
                │
                │  optimized chunks (smaller, cleaner)
                ▼
    ┌───────────────────────┐
    │  Cache-Aligned Prompt │  (already built)
    │  system | query | turn│
    └───────────┬───────────┘
                │
                ▼
           LLM Gateway
```

## What Each Stage Does

### Stage 1: CacheAligner

**Goal:** Maximize LLM provider KV cache hits by stabilizing the prompt prefix.

LLM providers (Anthropic, OpenAI) cache based on exact prefix match. A single
changing timestamp or session ID early in the prompt invalidates the entire
cache. CacheAligner:

1. Scans for dynamic patterns in the prompt prefix:
   - ISO timestamps (`2026-08-28T...`)
   - UUIDs (`550e8400-e29b-...`)
   - Session tokens, run IDs
   - Temp paths (`/tmp/abc123`)

2. Moves detected dynamic content to the end of the context (after static
   instructions and query), preserving the stable prefix for cache hits.

3. Reports drift metrics: how much of the prefix changed vs. previous call.

**Implementation:** Regex-based pattern detection + reordering. No ML needed.
Reuses existing normalisation patterns from `lesson.rs` (M3.7.7).

```rust
pub struct CacheAligner;

impl CacheAligner {
    /// Stabilize prompt prefix by moving dynamic content to tail.
    /// Returns (stable_prefix, dynamic_tail).
    pub fn align(content: &str) -> AlignedContent {
        // Detect and extract dynamic patterns
        // Reorder so static content comes first
    }
}
```

### Stage 2: ContentRouter (Magika ML + regex fallback)

**Goal:** Auto-detect content type and route to the best compressor.

**Primary classifier:** Google Magika (`magika` crate v1.1.0) — fast encoder-only
ONNX model that classifies content into 100+ types. <1ms per classification.
No LLM calls, no network — runs locally with embedded ONNX model.

**Fallback:** Regex heuristics for content types Magika doesn't distinguish
well (e.g., build logs vs. plain text) or when confidence is below threshold.

```rust
use magika::Session;

pub struct ContentRouter {
    magika: Session,
    confidence_threshold: f32, // default 0.7
}

pub enum ContentType {
    Json,
    Code { language: String },
    Log,
    Diff,
    Config,
    Text,
}

impl ContentRouter {
    pub fn detect(&self, content: &str) -> ContentType {
        // 1. Try Magika ML classification
        if let Ok(result) = self.magika.identify_content_sync(content.as_bytes()) {
            let label = result.info().label;
            let score = result.score();
            
            if score >= self.confidence_threshold {
                return match label {
                    "json" | "jsonl" => ContentType::Json,
                    "python" | "javascript" | "typescript" | "rust" | "go" | "shell"
                        => ContentType::Code { language: label.to_string() },
                    "diff" => ContentType::Diff,
                    "yaml" | "toml" | "ini" | "xml" => ContentType::Config,
                    _ => self.regex_fallback(content),
                };
            }
        }
        
        // 2. Fallback to regex heuristics
        self.regex_fallback(content)
    }
    
    fn regex_fallback(&self, content: &str) -> ContentType {
        if is_json(content) { return ContentType::Json; }
        if is_log(content)  { return ContentType::Log; }
        if is_diff(content) { return ContentType::Diff; }
        if is_code(content) { return ContentType::Code { language: "unknown".into() }; }
        ContentType::Text
    }
}
```

**Magika label → compressor mapping:**

| Magika Label | ContentType | Compressor |
|---|---|---|
| `json`, `jsonl` | Json | JsonCrusher |
| `python`, `javascript`, `rust`, `go`, `typescript`, `shell` | Code | CodeCompressor |
| `diff` | Diff | DiffCompressor |
| `yaml`, `toml`, `ini`, `xml` | Config | (passthrough, already compact) |
| `txt` + log heuristics | Log | LogCompressor |
| everything else | Text | TextCompressor |

### Stage 3: Compressors

**Goal:** Reduce token count per content type while preserving signal.

#### JsonCrusher (70-90% savings)
- Analyse field-level variance across JSON array elements
- Keep: keys, structure, error markers, boundary items (first/last)
- Drop: redundant mid-array elements, long string values, whitespace
- Allocation: 30% start (schema), 15% end (recency), 55% importance

#### LogCompressor (85-95% savings)
- Keep: error lines, stack traces, exit codes, FAIL markers
- Drop: passing test output, INFO-level noise, repeated patterns
- Reuses M3.7.7 signature extraction patterns (markers, cascade detection)

#### CodeCompressor (40-70% savings, opt-in)
- Keep: imports, function/method signatures, type annotations
- Drop: function bodies, inline comments, blank lines
- Uses simple AST heuristics (brace counting), not full parser

#### DiffCompressor (60-80% savings)
- Keep: change hunks (`+`/`-` lines), hunk headers
- Drop: unchanged context lines (the `@@` surrounding context)

#### TextCompressor (30-50% savings)
- Token importance scoring: keep high-entropy tokens (IDs, hashes, error codes)
- Drop: low-information prose, repeated phrases, filler words

```rust
pub trait Compressor {
    fn compress(&self, content: &str, budget: usize) -> CompressResult;
}

pub struct CompressResult {
    pub compressed: String,
    pub original_tokens: usize,
    pub compressed_tokens: usize,
    pub content_type: ContentType,
    /// Hash of original content for CCR retrieval
    pub ccr_hash: Option<String>,
}
```

### Stage 4: CCR Store (Compress-Cache-Retrieve)

**Goal:** Lossless compression — model can retrieve full originals if needed.

When a chunk is compressed, the full original is cached with a SHA256 hash.
A retrieval hint is injected into the compressed output:

```
[compressed content...]
<!-- CCR:abc123 — full content available via retrieval -->
```

If the model determines it needs more detail, it can request the full content.
This makes compression aggressive but reversible.

```rust
pub struct CcrStore {
    cache: HashMap<String, String>,  // hash → original content
    max_entries: usize,
    ttl: Duration,
}

impl CcrStore {
    pub fn store(&mut self, content: &str) -> String; // returns hash
    pub fn retrieve(&self, hash: &str) -> Option<&str>;
}
```

## Integration with Existing Code

### Where It Plugs In

The context optimizer sits in `mem-core` as a new module, called by
`PromptBuilder::build_cache_aligned()` before assembling the final prompt:

```rust
// In PromptBuilder::build_cache_aligned()
let chunk_text = Self::render_chunk(chunk)?;

// NEW: Optimize before prompt assembly
let optimized = ContextOptimizer::optimize(&chunk_text, &OptimizeConfig {
    cache_align: true,
    compress: true,
    ccr_enabled: true,
    token_budget: BUDGET_CHUNK_MAX,
})?;

let turn_msg = CACHE_TURN
    .replace("{memory}", memory_text)
    .replace("{chunk}", &optimized.compressed);
```

### What Already Exists (Reuse)

| Existing Code | Reuse For |
|---|---|
| `lesson.rs` normalise() | CacheAligner pattern detection (timestamps, paths, SHAs) |
| `lesson.rs` markers() | LogCompressor error line detection |
| `lesson.rs` is_cascade() | LogCompressor cascade suppression |
| `lesson.rs` strip_ansi() | Pre-processing for all compressors |
| `symptom_projection.rs` stop words | TextCompressor low-value token detection |

### What's New

| New Code | Location | Est. LOC |
|---|---|---|
| `context_optimizer.rs` | `crates/mem-core/src/` | 150 |
| `content_router.rs` | `crates/mem-core/src/` | 150 |
| `compressors/json.rs` | `crates/mem-core/src/` | 200 |
| `compressors/log.rs` | `crates/mem-core/src/` | 150 |
| `compressors/code.rs` | `crates/mem-core/src/` | 150 |
| `compressors/diff.rs` | `crates/mem-core/src/` | 100 |
| `compressors/text.rs` | `crates/mem-core/src/` | 100 |
| `ccr_store.rs` | `crates/mem-core/src/` | 80 |
| **Total** | | **~1030** |

## Performance Targets

| Metric | Target |
|---|---|
| Detection + compression | < 10ms per chunk |
| JSON compression ratio | 70-90% |
| Log compression ratio | 85-95% |
| Code compression ratio | 40-70% |
| Cache hit rate improvement | > 50% on repeated queries |
| Zero false negatives | Model should never miss real errors |

## Phasing

### Phase 1: Foundation (1-2 days)
- `ContextOptimizer` orchestrator
- `ContentRouter` with detection heuristics
- `LogCompressor` (reuses M3.7.7 lesson.rs patterns)
- 10 unit tests

### Phase 2: JSON + Diff (1-2 days)
- `JsonCrusher` with field variance analysis
- `DiffCompressor` with hunk preservation
- 10 unit tests

### Phase 3: CCR + CacheAligner (1 day)
- `CcrStore` with LRU cache
- `CacheAligner` with dynamic pattern extraction
- Integration with `PromptBuilder::build_cache_aligned()`
- 10 unit tests

### Phase 4: Code + Text (1 day)
- `CodeCompressor` (opt-in, brace-counting heuristics)
- `TextCompressor` (token importance scoring)
- 10 unit tests

## Key Design Decisions

1. **Magika ML for detection, rules for compression.** Content type detection
   uses Google's Magika ONNX model (<1ms, local, no network). Compression
   itself uses deterministic algorithms (no LLM calls in hot path).

2. **Search indexes untouched.** Compression happens AFTER retrieval. pgvector
   and OpenSearch see full-fidelity text. Only the LLM prompt is optimized.

3. **Reversible via CCR.** Every compression is reversible. The model can
   request full originals via hash lookup. Aggressive compression is safe
   because nothing is permanently lost.

4. **Reuse existing code.** M3.7.7's normalisation patterns, marker detection,
   and cascade suppression are directly reusable for log compression. M3.7.8's
   stop words help text compression.

5. **Budget-aware.** Each compressor respects a token budget. If the chunk is
   already under budget, no compression is applied (zero overhead).

---

Inspired by [Headroom](https://docs.headroomlabs.ai/docs/how-compression-works).
Adapted for Rust, integrated with poimen-memory's hybrid search architecture.