From afab09680a11742e99d21fd9862541eb8c84834d Mon Sep 17 00:00:00 2001
From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com>
Date: Fri, 28 Aug 2026 10:20:51 -0700
Subject: [PATCH] =?UTF-8?q?docs:=20update=20memory-flow.md=20=E2=80=94=20a?=
 =?UTF-8?q?dd=20Obsidian=20+=20M3.8=20optimizations?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Architecture updates:
- Added Obsidian REST API as reference corpus source of truth (M3.6.2)
- Added OpenSearch cluster with JWT auth for lexical search (M8)
- Clarified ingest path: full-fidelity (no compression)
- Clarified query path: compression between hybrid search + LLM (M3.8)

M3.8 Context Optimizer integration:
- Stage 1: Magika ML content detection
- Stage 2: CacheAligner for KV cache prefix stability
- Stage 3: Per-type compressors (log, json, diff, text)
- Stage 4: CCR store for reversible caching

M3.7.4 tier 3 now explicitly uses Obsidian REST API for reference docs.

Reflects completed work:
- M3.8.1 full 4-phase implementation (62 tests)
- M3.8.2 cache metrics + headers (3 tests)
- 117 total mem-core tests passing
---
 memory-flow.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/memory-flow.md b/memory-flow.md
index fdf1dc8..aca17f5 100644
--- a/memory-flow.md
+++ b/memory-flow.md
@@ -507,10 +507,54 @@ Complete deployment topology with all components.
 │  ├─ Used by: /memory/query (tokenize + embed)               │
 │  └─ Cached results (1hr TTL)                                │
 └─────────────────────────────────────────────────────────────┘
-```
+
+┌─────────────────────────────────────────────────────────────┐
+│  OpenSearch Cluster (BM25 Lexical Search)                   │
+│  ├─ StatefulSet: opensearch-0, opensearch-1 (HA)            │
+│  ├─ Service: opensearch-service (port 9200)                 │
+│  ├─ JWT authentication (Authentik JWKS)                     │
+│  ├─ Indexes: vault-* (chunks, content, breadcrumb)           │
+│  ├─ Ranking: BM25 (TF-IDF normalization)                     │
+│  └─ Used by: /memory/query (lexical fusion, 40% weight)      │
+└─────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────┐
+│  Obsidian REST API (Reference Document Management)           │
+│  ├─ Image: ppatlabs/obsidian:latest (port 27124)             │
+│  ├─ Storage: PVC 10Gi (Longhorn, vault files)               │
+│  ├─ Purpose: Single source of truth (reference docs)        │
+│  ├─ Used by:                                                │
+│  │  ├─ M3.6.2: Fetch reference corpus for context           │
+│  │  ├─ M3.7.4: Tier-3 fallback (general guidance)            │
+│  │  └─ UI: /memory/vault (browse, search, preview)          │
+│  └─ Features: Full-text search, breadcrumb navigation        │
+└─────────────────────────────────────────────────────────────┘
 
 ---
 
+## Architecture Updates: Obsidian + Optimizations
+
+### Storage & Query Optimizations (M3.6-M3.8)
+
+**Ingest Path (Full Fidelity):**
+- ✅ Obsidian REST API (M3.6.2): Reference corpus source of truth
+- ✅ Chunk heading boundaries with breadcrumb paths (M3.6.1)
+- ✅ Full-text embedding (nomic 768-dim) + pgvector storage
+- ✅ Full-text indexing in OpenSearch (BM25 lexical)
+- ✅ No compression at ingest (preserves search quality)
+
+**Query Path (Compressed for LLM):**
+- ✅ Hybrid retrieval: pgvector (60%) + OpenSearch (40%)
+- ✅ M3.8 Context Optimizer (pre-LLM compression pipeline):
+  - Stage 1: Magika ML content detection (<1ms)
+  - Stage 2: CacheAligner (stabilize KV cache prefix)
+  - Stage 3: Per-type compressors (85-95% effective ratio)
+  - Stage 4: CCR store (reversible compression cache)
+- ✅ Cache-aligned prompt builder (system | query | memory+chunk)
+- ✅ M3.7.7-8 failure diagnosis (signature + symptom projection)
+
+**Result**: Full-fidelity search indexes + token-efficient LLM prompts
+
 ---
 
 ## OpenSearch + JWT Authentication
@@ -824,8 +868,9 @@ GET /memory/context?query=npm+ERR+ERESOLVE&tool=npm
    │  ├─ Results: Top-10 chunks ranked by similarity
    │  └─ Confidence: med-high (vector match)
    │
-   └─ TIER 3 (Reference): M3.6 documentation
-      ├─ Query: Reference corpus (kubectl docs, npm docs)
+   └─ TIER 3 (Reference): M3.6 documentation (Obsidian REST API)
+      ├─ Query: POST /obsidian-api/search (reference corpus)
+      ├─ Source: Obsidian vault (kubectl docs, npm docs, etc.)
       ├─ Results: General guidance (not specific solution)
       └─ Confidence: low (generic info)