diff --git a/docs/CONTEXT_OPTIMIZER.md b/docs/CONTEXT_OPTIMIZER.md index 32ea9d3..a6eb0fc 100644 --- a/docs/CONTEXT_OPTIMIZER.md +++ b/docs/CONTEXT_OPTIMIZER.md @@ -92,42 +92,78 @@ impl CacheAligner { } ``` -### Stage 2: ContentRouter +### Stage 2: ContentRouter (Magika ML + regex fallback) **Goal:** Auto-detect content type and route to the best compressor. -Each evidence chunk might be JSON, source code, build logs, a diff, or plain -text. The router classifies using structural heuristics: +**Primary classifier:** Google Magika (`magika` crate v1.1.0) — fast encoder-only +ONNX model that classifies content into 100+ types. <1ms per classification. +No LLM calls, no network — runs locally with embedded ONNX model. -| Content Type | Detection Signal | Compressor | -|---|---|---| -| JSON | Valid JSON, `{` or `[` start, key-value pairs | JsonCrusher | -| Source code | Import/use statements, function defs, indentation | CodeCompressor | -| Build/test logs | Timestamps, log levels, `error:`, `FAIL` | LogCompressor | -| Unified diffs | `---`, `+++`, `@@` markers | DiffCompressor | -| Plain text | Default fallback | TextCompressor | - -**Implementation:** Pattern matching + simple heuristics. No ML classifier -needed initially (can add Magika later). +**Fallback:** Regex heuristics for content types Magika doesn't distinguish +well (e.g., build logs vs. plain text) or when confidence is below threshold. ```rust +use magika::Session; + +pub struct ContentRouter { + magika: Session, + confidence_threshold: f32, // default 0.7 +} + pub enum ContentType { Json, - Code, + Code { language: String }, Log, Diff, + Config, Text, } -pub fn detect(content: &str) -> ContentType { - if is_json(content) { return ContentType::Json; } - if is_code(content) { return ContentType::Code; } - if is_log(content) { return ContentType::Log; } - if is_diff(content) { return ContentType::Diff; } - ContentType::Text +impl ContentRouter { + pub fn detect(&self, content: &str) -> ContentType { + // 1. Try Magika ML classification + if let Ok(result) = self.magika.identify_content_sync(content.as_bytes()) { + let label = result.info().label; + let score = result.score(); + + if score >= self.confidence_threshold { + return match label { + "json" | "jsonl" => ContentType::Json, + "python" | "javascript" | "typescript" | "rust" | "go" | "shell" + => ContentType::Code { language: label.to_string() }, + "diff" => ContentType::Diff, + "yaml" | "toml" | "ini" | "xml" => ContentType::Config, + _ => self.regex_fallback(content), + }; + } + } + + // 2. Fallback to regex heuristics + self.regex_fallback(content) + } + + fn regex_fallback(&self, content: &str) -> ContentType { + if is_json(content) { return ContentType::Json; } + if is_log(content) { return ContentType::Log; } + if is_diff(content) { return ContentType::Diff; } + if is_code(content) { return ContentType::Code { language: "unknown".into() }; } + ContentType::Text + } } ``` +**Magika label → compressor mapping:** + +| Magika Label | ContentType | Compressor | +|---|---|---| +| `json`, `jsonl` | Json | JsonCrusher | +| `python`, `javascript`, `rust`, `go`, `typescript`, `shell` | Code | CodeCompressor | +| `diff` | Diff | DiffCompressor | +| `yaml`, `toml`, `ini`, `xml` | Config | (passthrough, already compact) | +| `txt` + log heuristics | Log | LogCompressor | +| everything else | Text | TextCompressor | + ### Stage 3: Compressors **Goal:** Reduce token count per content type while preserving signal. @@ -238,7 +274,7 @@ let turn_msg = CACHE_TURN | New Code | Location | Est. LOC | |---|---|---| | `context_optimizer.rs` | `crates/mem-core/src/` | 150 | -| `content_router.rs` | `crates/mem-core/src/` | 100 | +| `content_router.rs` | `crates/mem-core/src/` | 150 | | `compressors/json.rs` | `crates/mem-core/src/` | 200 | | `compressors/log.rs` | `crates/mem-core/src/` | 150 | | `compressors/code.rs` | `crates/mem-core/src/` | 150 | @@ -284,9 +320,9 @@ let turn_msg = CACHE_TURN ## Key Design Decisions -1. **No ML in hot path.** All detection and compression uses rules, regex, and - statistics. ML classifiers (Magika) can be added later as an optional - enhancement. +1. **Magika ML for detection, rules for compression.** Content type detection + uses Google's Magika ONNX model (<1ms, local, no network). Compression + itself uses deterministic algorithms (no LLM calls in hot path). 2. **Search indexes untouched.** Compression happens AFTER retrieval. pgvector and OpenSearch see full-fidelity text. Only the LLM prompt is optimized. diff --git a/k8s/infra/databases/opensearch.yaml b/k8s/infra/databases/opensearch.yaml index 26dd819..bd35eba 100644 --- a/k8s/infra/databases/opensearch.yaml +++ b/k8s/infra/databases/opensearch.yaml @@ -114,6 +114,16 @@ spec: serviceAccountName: opensearch hostNetwork: false + initContainers: + - name: sysctl + image: busybox:1.28 + command: + - sysctl + - -w + - vm.max_map_count=262144 + securityContext: + privileged: true + containers: - name: opensearch image: opensearchproject/opensearch:2.11.0