docs: update memory-flow.md — add Obsidian + M3.8 optimizations

Architecture updates:
- Added Obsidian REST API as reference corpus source of truth (M3.6.2)
- Added OpenSearch cluster with JWT auth for lexical search (M8)
- Clarified ingest path: full-fidelity (no compression)
- Clarified query path: compression between hybrid search + LLM (M3.8)

M3.8 Context Optimizer integration:
- Stage 1: Magika ML content detection
- Stage 2: CacheAligner for KV cache prefix stability
- Stage 3: Per-type compressors (log, json, diff, text)
- Stage 4: CCR store for reversible caching

M3.7.4 tier 3 now explicitly uses Obsidian REST API for reference docs.

Reflects completed work:
- M3.8.1 full 4-phase implementation (62 tests)
- M3.8.2 cache metrics + headers (3 tests)
- 117 total mem-core tests passing
This commit is contained in:
Story Crater Bot
2026-08-28 10:20:51 -07:00
parent bcb4e30ec2
commit 846298b68d
+48 -3
View File
@@ -507,10 +507,54 @@ Complete deployment topology with all components.
│ ├─ Used by: /memory/query (tokenize + embed) │
│ └─ Cached results (1hr TTL) │
└─────────────────────────────────────────────────────────────┘
```
┌─────────────────────────────────────────────────────────────┐
│ OpenSearch Cluster (BM25 Lexical Search) │
│ ├─ StatefulSet: opensearch-0, opensearch-1 (HA) │
│ ├─ Service: opensearch-service (port 9200) │
│ ├─ JWT authentication (Authentik JWKS) │
│ ├─ Indexes: vault-* (chunks, content, breadcrumb) │
│ ├─ Ranking: BM25 (TF-IDF normalization) │
│ └─ Used by: /memory/query (lexical fusion, 40% weight) │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Obsidian REST API (Reference Document Management) │
│ ├─ Image: ppatlabs/obsidian:latest (port 27124) │
│ ├─ Storage: PVC 10Gi (Longhorn, vault files) │
│ ├─ Purpose: Single source of truth (reference docs) │
│ ├─ Used by: │
│ │ ├─ M3.6.2: Fetch reference corpus for context │
│ │ ├─ M3.7.4: Tier-3 fallback (general guidance) │
│ │ └─ UI: /memory/vault (browse, search, preview) │
│ └─ Features: Full-text search, breadcrumb navigation │
└─────────────────────────────────────────────────────────────┘
---
## Architecture Updates: Obsidian + Optimizations
### Storage & Query Optimizations (M3.6-M3.8)
**Ingest Path (Full Fidelity):**
- ✅ Obsidian REST API (M3.6.2): Reference corpus source of truth
- ✅ Chunk heading boundaries with breadcrumb paths (M3.6.1)
- ✅ Full-text embedding (nomic 768-dim) + pgvector storage
- ✅ Full-text indexing in OpenSearch (BM25 lexical)
- ✅ No compression at ingest (preserves search quality)
**Query Path (Compressed for LLM):**
- ✅ Hybrid retrieval: pgvector (60%) + OpenSearch (40%)
- ✅ M3.8 Context Optimizer (pre-LLM compression pipeline):
- Stage 1: Magika ML content detection (<1ms)
- Stage 2: CacheAligner (stabilize KV cache prefix)
- Stage 3: Per-type compressors (85-95% effective ratio)
- Stage 4: CCR store (reversible compression cache)
- ✅ Cache-aligned prompt builder (system | query | memory+chunk)
- ✅ M3.7.7-8 failure diagnosis (signature + symptom projection)
**Result**: Full-fidelity search indexes + token-efficient LLM prompts
---
## OpenSearch + JWT Authentication
@@ -824,8 +868,9 @@ GET /memory/context?query=npm+ERR+ERESOLVE&tool=npm
│ ├─ Results: Top-10 chunks ranked by similarity
│ └─ Confidence: med-high (vector match)
└─ TIER 3 (Reference): M3.6 documentation
├─ Query: Reference corpus (kubectl docs, npm docs)
└─ TIER 3 (Reference): M3.6 documentation (Obsidian REST API)
├─ Query: POST /obsidian-api/search (reference corpus)
├─ Source: Obsidian vault (kubectl docs, npm docs, etc.)
├─ Results: General guidance (not specific solution)
└─ Confidence: low (generic info)