From 2c37d7b6f2a88a52eafa1c9804b495c14779679f Mon Sep 17 00:00:00 2001
From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com>
Date: Fri, 28 Aug 2026 09:19:35 -0700
Subject: [PATCH] docs: clarify optimizer sits in query path only, full
 lifecycle diagram

---
 memory-flow.md | 70 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 59 insertions(+), 11 deletions(-)

diff --git a/memory-flow.md b/memory-flow.md
index ebd2245..fdf1dc8 100644
--- a/memory-flow.md
+++ b/memory-flow.md
@@ -925,20 +925,51 @@ temp paths, ANSI codes, verbose JSON arrays, and passing test output. Feeding
 this noise to the GRU-Mem gate wastes tokens, risks hallucination on irrelevant
 details, and breaks LLM provider KV cache (dynamic content in prefix).
 
-### Where It Sits
+### Full Lifecycle: Where Optimization Happens
+
+Two paths through the system. Optimization touches **only the query path**,
+never the ingest path.
 
 ```
-Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
+═══ INGEST PATH (full fidelity, NO optimization) ═══════════════════
+
+  Agent transcript / tool output / log
             │
-            │  full-fidelity chunks (untouched)
             ▼
     ┌───────────────────────┐
-    │   CONTEXT OPTIMIZER   │
+    │   Chunker (M3.6.1)    │  heading-boundary split
+    └───────────┬───────────┘
+                │
+                ▼
+    ┌───────────────────────┐
+    │   Embed (nomic 768d)  │  full text → vector
+    └───────────┬───────────┘
+                │
+         ┌─────┴─────┐
+         ▼           ▼
+    pgvector      OpenSearch       ← FULL TEXT stored here
+    (semantic)    (BM25 lexical)      never compressed
+
+
+═══ QUERY PATH (optimized before LLM) ══════════════════════════
+
+  User query
+            │
+            ▼
+    ┌───────────────────────┐
+    │   Hybrid Search       │  pgvector 60% + OpenSearch 40%
+    │   (full-text match)   │  searches FULL text, not compressed
+    └───────────┬───────────┘
+                │
+                │  retrieved chunks (full fidelity)
+                ▼
+    ┌───────────────────────┐
+    │   CONTEXT OPTIMIZER   │  ← COMPRESSION HAPPENS HERE
     │                       │
-    │  1. Magika Detect     │  ML content type classification (<1ms)
-    │  2. CacheAligner      │  Move timestamps/UUIDs to tail
-    │  3. Compressor         │  Per-type compression
-    │  4. CCR Store          │  Cache originals for retrieval
+    │  1. Magika Detect     │  classify content type (<1ms)
+    │  2. CacheAligner      │  stabilize prefix for KV cache
+    │  3. Compressor        │  shrink per content type
+    │  4. CCR Store         │  cache originals (reversible)
     │                       │
     └───────────┬───────────┘
                 │
@@ -949,11 +980,28 @@ Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
     └───────────┬───────────┘
                 │
                 ▼
-           LLM Gateway
+    ┌───────────────────────┐
+    │   LLM Gate (GRU-Mem)  │  evaluate evidence, update memory
+    └───────────┬───────────┘
+                │
+                │  <check>yes/no</check>
+                │  <update>memory</update>
+                ▼
+    ┌───────────────────────┐
+    │   JSONL Event Log     │  append memory update event
+    └───────────┬───────────┘
+                │
+                │  event stored (full text, not compressed)
+                ▼
+         ┌─────┴─────┐
+         ▼           ▼
+    pgvector      OpenSearch       ← memory update indexed
+    (re-embed)    (re-index)          at full fidelity
 ```
 
-**Critical invariant:** Search indexes (pgvector + OpenSearch) NEVER see
-compressed content. Compression only happens in the prompt assembly path.
+**Key rule:** Compression is ephemeral. It exists only in the prompt for one
+LLM call. The event log, search indexes, and stored memories never see
+compressed content. If you rebuild from log, you get full-fidelity text.
 
 ### Stage 1: Content Detection (Magika ML)