plan: add Magika ML classifier to content router

docs/CONTEXT_OPTIMIZER.md

@@ -92,42 +92,78 @@ impl CacheAligner {
 }
 ```
 
-### Stage 2: ContentRouter
+### Stage 2: ContentRouter (Magika ML + regex fallback)
 
 **Goal:** Auto-detect content type and route to the best compressor.
 
-Each evidence chunk might be JSON, source code, build logs, a diff, or plain
-text. The router classifies using structural heuristics:
+**Primary classifier:** Google Magika (`magika` crate v1.1.0) — fast encoder-only
+ONNX model that classifies content into 100+ types. <1ms per classification.
+No LLM calls, no network — runs locally with embedded ONNX model.
 
-| Content Type | Detection Signal | Compressor |
-|---|---|---|
-| JSON | Valid JSON, `{` or `[` start, key-value pairs | JsonCrusher |
-| Source code | Import/use statements, function defs, indentation | CodeCompressor |
-| Build/test logs | Timestamps, log levels, `error:`, `FAIL` | LogCompressor |
-| Unified diffs | `---`, `+++`, `@@` markers | DiffCompressor |
-| Plain text | Default fallback | TextCompressor |
-
-**Implementation:** Pattern matching + simple heuristics. No ML classifier
-needed initially (can add Magika later).
+**Fallback:** Regex heuristics for content types Magika doesn't distinguish
+well (e.g., build logs vs. plain text) or when confidence is below threshold.
 
 ```rust
+use magika::Session;
+
+pub struct ContentRouter {
+    magika: Session,
+    confidence_threshold: f32, // default 0.7
+}
+
 pub enum ContentType {
     Json,
-    Code,
+    Code { language: String },
     Log,
     Diff,
+    Config,
     Text,
 }
 
-pub fn detect(content: &str) -> ContentType {
-    if is_json(content) { return ContentType::Json; }
-    if is_code(content) { return ContentType::Code; }
-    if is_log(content)  { return ContentType::Log; }
-    if is_diff(content) { return ContentType::Diff; }
-    ContentType::Text
+impl ContentRouter {
+    pub fn detect(&self, content: &str) -> ContentType {
+        // 1. Try Magika ML classification
+        if let Ok(result) = self.magika.identify_content_sync(content.as_bytes()) {
+            let label = result.info().label;
+            let score = result.score();
+            
+            if score >= self.confidence_threshold {
+                return match label {
+                    "json" | "jsonl" => ContentType::Json,
+                    "python" | "javascript" | "typescript" | "rust" | "go" | "shell"
+                        => ContentType::Code { language: label.to_string() },
+                    "diff" => ContentType::Diff,
+                    "yaml" | "toml" | "ini" | "xml" => ContentType::Config,
+                    _ => self.regex_fallback(content),
+                };
+            }
+        }
+        
+        // 2. Fallback to regex heuristics
+        self.regex_fallback(content)
+    }
+    
+    fn regex_fallback(&self, content: &str) -> ContentType {
+        if is_json(content) { return ContentType::Json; }
+        if is_log(content)  { return ContentType::Log; }
+        if is_diff(content) { return ContentType::Diff; }
+        if is_code(content) { return ContentType::Code { language: "unknown".into() }; }
+        ContentType::Text
+    }
 }
 ```
 
+**Magika label → compressor mapping:**
+
+| Magika Label | ContentType | Compressor |
+|---|---|---|
+| `json`, `jsonl` | Json | JsonCrusher |
+| `python`, `javascript`, `rust`, `go`, `typescript`, `shell` | Code | CodeCompressor |
+| `diff` | Diff | DiffCompressor |
+| `yaml`, `toml`, `ini`, `xml` | Config | (passthrough, already compact) |
+| `txt` + log heuristics | Log | LogCompressor |
+| everything else | Text | TextCompressor |
+
 ### Stage 3: Compressors
 
 **Goal:** Reduce token count per content type while preserving signal.
@@ -238,7 +274,7 @@ let turn_msg = CACHE_TURN
 | New Code | Location | Est. LOC |
 |---|---|---|
 | `context_optimizer.rs` | `crates/mem-core/src/` | 150 |
-| `content_router.rs` | `crates/mem-core/src/` | 100 |
+| `content_router.rs` | `crates/mem-core/src/` | 150 |
 | `compressors/json.rs` | `crates/mem-core/src/` | 200 |
 | `compressors/log.rs` | `crates/mem-core/src/` | 150 |
 | `compressors/code.rs` | `crates/mem-core/src/` | 150 |
@@ -284,9 +320,9 @@ let turn_msg = CACHE_TURN
 
 ## Key Design Decisions
 
-1. **No ML in hot path.** All detection and compression uses rules, regex, and
-   statistics. ML classifiers (Magika) can be added later as an optional
-   enhancement.
+1. **Magika ML for detection, rules for compression.** Content type detection
+   uses Google's Magika ONNX model (<1ms, local, no network). Compression
+   itself uses deterministic algorithms (no LLM calls in hot path).
 
 2. **Search indexes untouched.** Compression happens AFTER retrieval. pgvector
    and OpenSearch see full-fidelity text. Only the LLM prompt is optimized.

k8s/infra/databases/opensearch.yaml

@@ -114,6 +114,16 @@ spec:
       serviceAccountName: opensearch
       hostNetwork: false
       
+      initContainers:
+        - name: sysctl
+          image: busybox:1.28
+          command:
+            - sysctl
+            - -w
+            - vm.max_map_count=262144
+          securityContext:
+            privileged: true
+      
       containers:
         - name: opensearch
           image: opensearchproject/opensearch:2.11.0