poimen-memory/docs/CONTEXT_OPTIMIZER.md

Context Optimizer — Pre-LLM Compression Layer

Motivation

Agent transcripts and tool outputs are noisy. A 50-chunk ingestion run might feed the GRU-Mem gate evidence that's 43% tool results, full of timestamps, temp paths, ANSI codes, and verbose JSON. The model wastes tokens parsing noise, risks hallucinating on irrelevant details, and we pay full price for bloated input. LLM provider KV caches miss because dynamic content (timestamps, session IDs) pollutes the prefix.

This layer sits between retrieval and the LLM call. Search indexes (pgvector + OpenSearch) stay untouched at full fidelity. Only the evidence chunks entering the prompt get optimized.

Architecture

Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
            │
            │  full-fidelity chunks (untouched)
            ▼
    ┌───────────────────────┐
    │   CONTEXT OPTIMIZER   │  ← THIS MODULE
    │                       │
    │  1. CacheAligner      │  Move dynamic content (timestamps, UUIDs)
    │                       │  to end of context. Keep static prefix
    │                       │  stable for KV cache hits.
    │                       │
    │  2. ContentRouter     │  Auto-detect content type per chunk:
    │                       │  JSON, code, logs, diffs, plain text.
    │                       │  Route each to best compressor.
    │                       │
    │  3. Compressors       │  Per-type compression:
    │     - JsonCrusher     │    Statistical field analysis, keep keys
    │     - LogCompressor   │    Keep errors/stack traces, drop noise
    │     - CodeCompressor  │    AST-aware: keep signatures, drop bodies
    │     - TextCompressor  │    Token importance scoring
    │     - DiffCompressor  │    Keep change hunks, drop context
    │                       │
    │  4. CCR Store         │  Cache full originals with hash reference.
    │                       │  Inject retrieval hint so model can fetch
    │                       │  full content if needed.
    │                       │
    └───────────┬───────────┘
                │
                │  optimized chunks (smaller, cleaner)
                ▼
    ┌───────────────────────┐
    │  Cache-Aligned Prompt │  (already built)
    │  system | query | turn│
    └───────────┬───────────┘
                │
                ▼
           LLM Gateway

What Each Stage Does

Stage 1: CacheAligner

Goal: Maximize LLM provider KV cache hits by stabilizing the prompt prefix.

LLM providers (Anthropic, OpenAI) cache based on exact prefix match. A single changing timestamp or session ID early in the prompt invalidates the entire cache. CacheAligner:

1. Scans for dynamic patterns in the prompt prefix:

   • ISO timestamps (2026-08-28T...)
   • UUIDs (550e8400-e29b-...)
   • Session tokens, run IDs
   • Temp paths (/tmp/abc123)

2. Moves detected dynamic content to the end of the context (after static instructions and query), preserving the stable prefix for cache hits.

3. Reports drift metrics: how much of the prefix changed vs. previous call.

Implementation: Regex-based pattern detection + reordering. No ML needed. Reuses existing normalisation patterns from lesson.rs (M3.7.7).

pub struct CacheAligner;

impl CacheAligner {
    /// Stabilize prompt prefix by moving dynamic content to tail.
    /// Returns (stable_prefix, dynamic_tail).
    pub fn align(content: &str) -> AlignedContent {
        // Detect and extract dynamic patterns
        // Reorder so static content comes first
    }
}

Stage 2: ContentRouter

Goal: Auto-detect content type and route to the best compressor.

Each evidence chunk might be JSON, source code, build logs, a diff, or plain text. The router classifies using structural heuristics:

Content Type	Detection Signal	Compressor
JSON	Valid JSON, { or [ start, key-value pairs	JsonCrusher
Source code	Import/use statements, function defs, indentation	CodeCompressor
Build/test logs	Timestamps, log levels, error:, FAIL	LogCompressor
Unified diffs	---, +++, @@ markers	DiffCompressor
Plain text	Default fallback	TextCompressor

Implementation: Pattern matching + simple heuristics. No ML classifier needed initially (can add Magika later).

pub enum ContentType {
    Json,
    Code,
    Log,
    Diff,
    Text,
}

pub fn detect(content: &str) -> ContentType {
    if is_json(content) { return ContentType::Json; }
    if is_code(content) { return ContentType::Code; }
    if is_log(content)  { return ContentType::Log; }
    if is_diff(content) { return ContentType::Diff; }
    ContentType::Text
}

Stage 3: Compressors

Goal: Reduce token count per content type while preserving signal.

JsonCrusher (70-90% savings)

• Analyse field-level variance across JSON array elements
• Keep: keys, structure, error markers, boundary items (first/last)
• Drop: redundant mid-array elements, long string values, whitespace
• Allocation: 30% start (schema), 15% end (recency), 55% importance

LogCompressor (85-95% savings)

• Keep: error lines, stack traces, exit codes, FAIL markers
• Drop: passing test output, INFO-level noise, repeated patterns
• Reuses M3.7.7 signature extraction patterns (markers, cascade detection)

CodeCompressor (40-70% savings, opt-in)

• Keep: imports, function/method signatures, type annotations
• Drop: function bodies, inline comments, blank lines
• Uses simple AST heuristics (brace counting), not full parser

DiffCompressor (60-80% savings)

• Keep: change hunks (+/- lines), hunk headers
• Drop: unchanged context lines (the @@ surrounding context)

TextCompressor (30-50% savings)

• Token importance scoring: keep high-entropy tokens (IDs, hashes, error codes)
• Drop: low-information prose, repeated phrases, filler words

pub trait Compressor {
    fn compress(&self, content: &str, budget: usize) -> CompressResult;
}

pub struct CompressResult {
    pub compressed: String,
    pub original_tokens: usize,
    pub compressed_tokens: usize,
    pub content_type: ContentType,
    /// Hash of original content for CCR retrieval
    pub ccr_hash: Option<String>,
}

Stage 4: CCR Store (Compress-Cache-Retrieve)

Goal: Lossless compression — model can retrieve full originals if needed.

When a chunk is compressed, the full original is cached with a SHA256 hash. A retrieval hint is injected into the compressed output:

[compressed content...]
<!-- CCR:abc123 — full content available via retrieval -->

If the model determines it needs more detail, it can request the full content. This makes compression aggressive but reversible.

pub struct CcrStore {
    cache: HashMap<String, String>,  // hash → original content
    max_entries: usize,
    ttl: Duration,
}

impl CcrStore {
    pub fn store(&mut self, content: &str) -> String; // returns hash
    pub fn retrieve(&self, hash: &str) -> Option<&str>;
}

Integration with Existing Code

Where It Plugs In

The context optimizer sits in mem-core as a new module, called by PromptBuilder::build_cache_aligned() before assembling the final prompt:

// In PromptBuilder::build_cache_aligned()
let chunk_text = Self::render_chunk(chunk)?;

// NEW: Optimize before prompt assembly
let optimized = ContextOptimizer::optimize(&chunk_text, &OptimizeConfig {
    cache_align: true,
    compress: true,
    ccr_enabled: true,
    token_budget: BUDGET_CHUNK_MAX,
})?;

let turn_msg = CACHE_TURN
    .replace("{memory}", memory_text)
    .replace("{chunk}", &optimized.compressed);

What Already Exists (Reuse)

Existing Code	Reuse For
lesson.rs normalise()	CacheAligner pattern detection (timestamps, paths, SHAs)
lesson.rs markers()	LogCompressor error line detection
lesson.rs is_cascade()	LogCompressor cascade suppression
lesson.rs strip_ansi()	Pre-processing for all compressors
symptom_projection.rs stop words	TextCompressor low-value token detection

What's New

New Code	Location	Est. LOC
context_optimizer.rs	crates/mem-core/src/	150
content_router.rs	crates/mem-core/src/	100
compressors/json.rs	crates/mem-core/src/	200
compressors/log.rs	crates/mem-core/src/	150
compressors/code.rs	crates/mem-core/src/	150
compressors/diff.rs	crates/mem-core/src/	100
compressors/text.rs	crates/mem-core/src/	100
ccr_store.rs	crates/mem-core/src/	80
Total		~1030

Performance Targets

Metric	Target
Detection + compression	< 10ms per chunk
JSON compression ratio	70-90%
Log compression ratio	85-95%
Code compression ratio	40-70%
Cache hit rate improvement	> 50% on repeated queries
Zero false negatives	Model should never miss real errors

Phasing

Phase 1: Foundation (1-2 days)

• ContextOptimizer orchestrator
• ContentRouter with detection heuristics
• LogCompressor (reuses M3.7.7 lesson.rs patterns)
• 10 unit tests

Phase 2: JSON + Diff (1-2 days)

• JsonCrusher with field variance analysis
• DiffCompressor with hunk preservation
• 10 unit tests

Phase 3: CCR + CacheAligner (1 day)

• CcrStore with LRU cache
• CacheAligner with dynamic pattern extraction
• Integration with PromptBuilder::build_cache_aligned()
• 10 unit tests

Phase 4: Code + Text (1 day)

• CodeCompressor (opt-in, brace-counting heuristics)
• TextCompressor (token importance scoring)
• 10 unit tests

Key Design Decisions

1. No ML in hot path. All detection and compression uses rules, regex, and statistics. ML classifiers (Magika) can be added later as an optional enhancement.

2. Search indexes untouched. Compression happens AFTER retrieval. pgvector and OpenSearch see full-fidelity text. Only the LLM prompt is optimized.

3. Reversible via CCR. Every compression is reversible. The model can request full originals via hash lookup. Aggressive compression is safe because nothing is permanently lost.

4. Reuse existing code. M3.7.7's normalisation patterns, marker detection, and cascade suppression are directly reusable for log compression. M3.7.8's stop words help text compression.

5. Budget-aware. Each compressor respects a token budget. If the chunk is already under budget, no compression is applied (zero overhead).

---

Inspired by Headroom. Adapted for Rust, integrated with poimen-memory's hybrid search architecture.