diff --git a/memory-flow.md b/memory-flow.md index fdf1dc8..aca17f5 100644 --- a/memory-flow.md +++ b/memory-flow.md @@ -507,10 +507,54 @@ Complete deployment topology with all components. │ ├─ Used by: /memory/query (tokenize + embed) │ │ └─ Cached results (1hr TTL) │ └─────────────────────────────────────────────────────────────┘ -``` + +┌─────────────────────────────────────────────────────────────┐ +│ OpenSearch Cluster (BM25 Lexical Search) │ +│ ├─ StatefulSet: opensearch-0, opensearch-1 (HA) │ +│ ├─ Service: opensearch-service (port 9200) │ +│ ├─ JWT authentication (Authentik JWKS) │ +│ ├─ Indexes: vault-* (chunks, content, breadcrumb) │ +│ ├─ Ranking: BM25 (TF-IDF normalization) │ +│ └─ Used by: /memory/query (lexical fusion, 40% weight) │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ Obsidian REST API (Reference Document Management) │ +│ ├─ Image: ppatlabs/obsidian:latest (port 27124) │ +│ ├─ Storage: PVC 10Gi (Longhorn, vault files) │ +│ ├─ Purpose: Single source of truth (reference docs) │ +│ ├─ Used by: │ +│ │ ├─ M3.6.2: Fetch reference corpus for context │ +│ │ ├─ M3.7.4: Tier-3 fallback (general guidance) │ +│ │ └─ UI: /memory/vault (browse, search, preview) │ +│ └─ Features: Full-text search, breadcrumb navigation │ +└─────────────────────────────────────────────────────────────┘ --- +## Architecture Updates: Obsidian + Optimizations + +### Storage & Query Optimizations (M3.6-M3.8) + +**Ingest Path (Full Fidelity):** +- ✅ Obsidian REST API (M3.6.2): Reference corpus source of truth +- ✅ Chunk heading boundaries with breadcrumb paths (M3.6.1) +- ✅ Full-text embedding (nomic 768-dim) + pgvector storage +- ✅ Full-text indexing in OpenSearch (BM25 lexical) +- ✅ No compression at ingest (preserves search quality) + +**Query Path (Compressed for LLM):** +- ✅ Hybrid retrieval: pgvector (60%) + OpenSearch (40%) +- ✅ M3.8 Context Optimizer (pre-LLM compression pipeline): + - Stage 1: Magika ML content detection (<1ms) + - Stage 2: CacheAligner (stabilize KV cache prefix) + - Stage 3: Per-type compressors (85-95% effective ratio) + - Stage 4: CCR store (reversible compression cache) +- ✅ Cache-aligned prompt builder (system | query | memory+chunk) +- ✅ M3.7.7-8 failure diagnosis (signature + symptom projection) + +**Result**: Full-fidelity search indexes + token-efficient LLM prompts + --- ## OpenSearch + JWT Authentication @@ -824,8 +868,9 @@ GET /memory/context?query=npm+ERR+ERESOLVE&tool=npm │ ├─ Results: Top-10 chunks ranked by similarity │ └─ Confidence: med-high (vector match) │ - └─ TIER 3 (Reference): M3.6 documentation - ├─ Query: Reference corpus (kubectl docs, npm docs) + └─ TIER 3 (Reference): M3.6 documentation (Obsidian REST API) + ├─ Query: POST /obsidian-api/search (reference corpus) + ├─ Source: Obsidian vault (kubectl docs, npm docs, etc.) ├─ Results: General guidance (not specific solution) └─ Confidence: low (generic info)