docs: update memory-flow.md — add Obsidian + M3.8 optimizations
Architecture updates: - Added Obsidian REST API as reference corpus source of truth (M3.6.2) - Added OpenSearch cluster with JWT auth for lexical search (M8) - Clarified ingest path: full-fidelity (no compression) - Clarified query path: compression between hybrid search + LLM (M3.8) M3.8 Context Optimizer integration: - Stage 1: Magika ML content detection - Stage 2: CacheAligner for KV cache prefix stability - Stage 3: Per-type compressors (log, json, diff, text) - Stage 4: CCR store for reversible caching M3.7.4 tier 3 now explicitly uses Obsidian REST API for reference docs. Reflects completed work: - M3.8.1 full 4-phase implementation (62 tests) - M3.8.2 cache metrics + headers (3 tests) - 117 total mem-core tests passing
This commit is contained in:
+48
-3
@@ -507,10 +507,54 @@ Complete deployment topology with all components.
|
||||
│ ├─ Used by: /memory/query (tokenize + embed) │
|
||||
│ └─ Cached results (1hr TTL) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ OpenSearch Cluster (BM25 Lexical Search) │
|
||||
│ ├─ StatefulSet: opensearch-0, opensearch-1 (HA) │
|
||||
│ ├─ Service: opensearch-service (port 9200) │
|
||||
│ ├─ JWT authentication (Authentik JWKS) │
|
||||
│ ├─ Indexes: vault-* (chunks, content, breadcrumb) │
|
||||
│ ├─ Ranking: BM25 (TF-IDF normalization) │
|
||||
│ └─ Used by: /memory/query (lexical fusion, 40% weight) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Obsidian REST API (Reference Document Management) │
|
||||
│ ├─ Image: ppatlabs/obsidian:latest (port 27124) │
|
||||
│ ├─ Storage: PVC 10Gi (Longhorn, vault files) │
|
||||
│ ├─ Purpose: Single source of truth (reference docs) │
|
||||
│ ├─ Used by: │
|
||||
│ │ ├─ M3.6.2: Fetch reference corpus for context │
|
||||
│ │ ├─ M3.7.4: Tier-3 fallback (general guidance) │
|
||||
│ │ └─ UI: /memory/vault (browse, search, preview) │
|
||||
│ └─ Features: Full-text search, breadcrumb navigation │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
---
|
||||
|
||||
## Architecture Updates: Obsidian + Optimizations
|
||||
|
||||
### Storage & Query Optimizations (M3.6-M3.8)
|
||||
|
||||
**Ingest Path (Full Fidelity):**
|
||||
- ✅ Obsidian REST API (M3.6.2): Reference corpus source of truth
|
||||
- ✅ Chunk heading boundaries with breadcrumb paths (M3.6.1)
|
||||
- ✅ Full-text embedding (nomic 768-dim) + pgvector storage
|
||||
- ✅ Full-text indexing in OpenSearch (BM25 lexical)
|
||||
- ✅ No compression at ingest (preserves search quality)
|
||||
|
||||
**Query Path (Compressed for LLM):**
|
||||
- ✅ Hybrid retrieval: pgvector (60%) + OpenSearch (40%)
|
||||
- ✅ M3.8 Context Optimizer (pre-LLM compression pipeline):
|
||||
- Stage 1: Magika ML content detection (<1ms)
|
||||
- Stage 2: CacheAligner (stabilize KV cache prefix)
|
||||
- Stage 3: Per-type compressors (85-95% effective ratio)
|
||||
- Stage 4: CCR store (reversible compression cache)
|
||||
- ✅ Cache-aligned prompt builder (system | query | memory+chunk)
|
||||
- ✅ M3.7.7-8 failure diagnosis (signature + symptom projection)
|
||||
|
||||
**Result**: Full-fidelity search indexes + token-efficient LLM prompts
|
||||
|
||||
---
|
||||
|
||||
## OpenSearch + JWT Authentication
|
||||
@@ -824,8 +868,9 @@ GET /memory/context?query=npm+ERR+ERESOLVE&tool=npm
|
||||
│ ├─ Results: Top-10 chunks ranked by similarity
|
||||
│ └─ Confidence: med-high (vector match)
|
||||
│
|
||||
└─ TIER 3 (Reference): M3.6 documentation
|
||||
├─ Query: Reference corpus (kubectl docs, npm docs)
|
||||
└─ TIER 3 (Reference): M3.6 documentation (Obsidian REST API)
|
||||
├─ Query: POST /obsidian-api/search (reference corpus)
|
||||
├─ Source: Obsidian vault (kubectl docs, npm docs, etc.)
|
||||
├─ Results: General guidance (not specific solution)
|
||||
└─ Confidence: low (generic info)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user