chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
Story Crater Bot
+833
@@ -0,0 +1,833 @@
|
||||
# Memory UI Flow - Complete Workflow
|
||||
|
||||
## Table of Contents
|
||||
1. [Read Flow](#read-flow)
|
||||
2. [Search Flow (Semantic + Lexical Hybrid)](#search-flow-semantic--lexical-hybrid)
|
||||
3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow)
|
||||
4. [Agent Context Flow](#agent-context-flow)
|
||||
5. [System Architecture](#system-architecture)
|
||||
6. [OpenSearch + JWT Authentication](#opensearch--jwt-authentication)
|
||||
7. [Pod Infrastructure](#pod-infrastructure)
|
||||
---
|
||||
|
||||
## Read Flow
|
||||
|
||||
Browse vault documents from the web UI.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ User: memory.riotpiao.com │
|
||||
│ (Browser, JWT token in localStorage) │
|
||||
└────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
│ GET /memory/vault?project=poimen
|
||||
│ Authorization: Bearer <JWT>
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Memory Service Pod │
|
||||
│ ├─ Load vault files from PVC │
|
||||
│ ├─ Build file tree (directory structure) │
|
||||
│ └─ Return JSON response │
|
||||
└────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
│ {files: [{path, title, updated_at}...]}
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ UI: Render Vault Browser │
|
||||
│ ├─ Project selector (dropdown) │
|
||||
│ ├─ File tree (collapsible folders) │
|
||||
│ ├─ Breadcrumb navigation │
|
||||
│ └─ Preview panel (markdown rendering) │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Search Flow (Semantic + Lexical Hybrid)
|
||||
|
||||
Hybrid retrieval pipeline combining:
|
||||
- **Semantic path**: pgvector embeddings (query understanding)
|
||||
- **Lexical path**: OpenSearch BM25 (exact term matching)
|
||||
- **Fusion**: Weighted linear combination (60% semantic, 40% lexical)
|
||||
- **Result limit**: Top 50 from each engine, merge to top 10 final
|
||||
|
||||
### Complete Retrieval Pipeline
|
||||
|
||||
```
|
||||
User Query: "fix kubernetes port 8080 conflict"
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Stage 1: Query Normalization │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ ├─ Tokenize: ["fix", "kubernetes", ...] │
|
||||
│ ├─ Extract entities: {port: "8080"} │
|
||||
│ ├─ Generate embedding (LLM) │
|
||||
│ └─ Create QueryContext │
|
||||
└────────────────────────┬───────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Stage 2: Parallel Retrieval (Both Engines) │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─ SEMANTIC (pgvector) │
|
||||
│ │ ├─ Query embedding <-> vector │
|
||||
│ │ ├─ Filter: project_id, level, date │
|
||||
│ │ ├─ ORDER BY cosine_similarity DESC │
|
||||
│ │ └─ Return: Top 50 with scores │
|
||||
│ │ │
|
||||
│ └─ LEXICAL (OpenSearch + JWT) │
|
||||
│ ├─ multi_match on [content^2, breadcrumb] │
|
||||
│ ├─ BM25 ranking with fuzziness │
|
||||
│ ├─ Filter: project_id, level, date │
|
||||
│ └─ Return: Top 50 with raw scores │
|
||||
│ │
|
||||
└────────────────┬──────────────────────────┬────────────┘
|
||||
│ │
|
||||
sem_results: [(doc1, 0.92), ...] lex_results: [(doc1, 8.5), ...]
|
||||
│ │
|
||||
└──────────┬───────────────┘
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Stage 3: Score Normalization │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ ├─ Normalize semantic: [0.0 ... 1.0] │
|
||||
│ │ (already 0-1 from cosine) │
|
||||
│ │ │
|
||||
│ ├─ Normalize lexical: [0.0 ... 1.0] │
|
||||
│ │ (min-max: (score-min)/(max-min)) │
|
||||
│ │ │
|
||||
│ └─ Result: Both in [0.0, 1.0] range │
|
||||
│ │
|
||||
└────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
sem_norm: [(doc1, 1.0), ...] lex_norm: [(doc1, 0.98), ...]
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Stage 4: Fusion (Weighted Linear) │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ ├─ Merge all doc IDs from both results │
|
||||
│ ├─ For each doc: score = 0.6*sem + 0.4*lex │
|
||||
│ │ │
|
||||
│ │ doc1: 0.6*1.0 + 0.4*0.98 = 0.992 │
|
||||
│ │ doc2: 0.6*0.96 + 0.4*0.0 = 0.576 │
|
||||
│ │ doc3: 0.6*0.0 + 0.4*0.88 = 0.352 │
|
||||
│ │ │
|
||||
│ ├─ Sort descending: [doc1, doc2, doc3] │
|
||||
│ └─ Take top-10 │
|
||||
│ │
|
||||
└────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Stage 5: Score Breakdown (Transparency) │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ doc1: { │
|
||||
│ "score": 0.992, │
|
||||
│ "sem_component": 1.0, │
|
||||
│ "lex_component": 0.98, │
|
||||
│ "sem_weight": 0.6, │
|
||||
│ "lex_weight": 0.4, │
|
||||
│ "reason": "Exact semantic match + strong lexical" │
|
||||
│ } │
|
||||
│ │
|
||||
└────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ Final Results (Top-10) │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ 1. port-forward.md (0.992) │
|
||||
│ - Semantic: 1.0 | Lexical: 0.98 │
|
||||
│ │
|
||||
│ 2. troubleshooting.md (0.576) │
|
||||
│ - Semantic: 0.96 | Lexical: 0.0 │
|
||||
│ │
|
||||
│ 3. k8s-basics.md (0.352) │
|
||||
│ - Semantic: 0.0 | Lexical: 0.88 │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Query Routing Decision Tree
|
||||
|
||||
```
|
||||
Query Received
|
||||
│
|
||||
├─ Token count < 3?
|
||||
│ ├─ YES → Use LEXICAL_ONLY
|
||||
│ │ (short queries: "fix port" → better BM25)
|
||||
│ │
|
||||
│ └─ NO → Continue...
|
||||
│
|
||||
├─ Contains special syntax (#tag, @mention)?
|
||||
│ ├─ YES → Use LEXICAL_WITH_FILTERS
|
||||
│ │ (preserve exact matches)
|
||||
│ │
|
||||
│ └─ NO → Continue...
|
||||
│
|
||||
├─ Can we embed the query?
|
||||
│ ├─ YES → Use HYBRID
|
||||
│ │ (both engines)
|
||||
│ │
|
||||
│ └─ NO → Use LEXICAL_ONLY
|
||||
│ (LLM unavailable, fallback)
|
||||
│
|
||||
└─ Execute chosen strategy
|
||||
```
|
||||
|
||||
### Index Optimization
|
||||
|
||||
**PostgreSQL (pgvector) - Semantic Path:**
|
||||
- Index: `ivfflat (embedding vector_cosine_ops) WITH (lists=100)`
|
||||
- Filter: `project_id, level IN ('L0','L1','L2'), created_at > now-1y`
|
||||
- Retrieve: Top 50 chunks, then merge with lexical
|
||||
- Lookup: O(log n) pre-filter + O(1) embedding distance
|
||||
|
||||
**OpenSearch (BM25) - Lexical Path:**
|
||||
- Analyzer: `standard` (lowercase, stop words)
|
||||
- Fields: `content^2` (2x boost) + `breadcrumb` + `source`
|
||||
- Tokenizer: Standard + n-gram for typo tolerance
|
||||
- Retrieve: Top 50 results via BM25, then merge with semantic
|
||||
- Lookup: O(n) inverted index scan + TF-IDF ranking
|
||||
|
||||
**Merging Strategy:**
|
||||
1. Normalize both score ranges to [0.0, 1.0]
|
||||
2. Weighted sum: `0.6 * semantic + 0.4 * lexical`
|
||||
3. Sort by final score
|
||||
4. Return top-10 to user
|
||||
|
||||
**Accuracy Metrics (A/B Testing):**
|
||||
- MRR (Mean Reciprocal Rank): Position of first correct result
|
||||
- NDCG@10 (Normalized Discounted Cumulative Gain): Quality of top-10 ranking
|
||||
- Precision@K: Relevant results in top-K
|
||||
- Recall@K: Coverage of all relevant results in top-K
|
||||
|
||||
------
|
||||
|
||||
## Edit Flow (GRM Workflow)
|
||||
|
||||
Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ User: Clicks "Edit" on document │
|
||||
│ Example: runbook-deploy.md │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ UI: Switch to Edit Mode │
|
||||
│ ├─ Load document content from Memory Service │
|
||||
│ ├─ Show markdown editor (CodeMirror) │
|
||||
│ ├─ Disable Save button (drafts only) │
|
||||
│ └─ Show "Submit for Review" button │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
├─ User makes edits (e.g., update deploy steps)
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ User: Click "Submit for Review" │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
│
|
||||
╔═══════╩═══════════════════════════════════════════╗
|
||||
║ STEP 1: CREATE BRANCH ║
|
||||
╚═══════╤═══════════════════════════════════════════╝
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Frontend: POST /memory/grc/draft │
|
||||
│ { │
|
||||
│ "document_path": "vault/runbooks/deploy.md", │
|
||||
│ "content": "<new markdown content>", │
|
||||
│ "message": "Update deploy steps", │
|
||||
│ "user": "[email protected]" │
|
||||
│ } │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Memory Service Pod: GRC Handler │
|
||||
│ ├─ Generate branch name: edit/rock/deploy-<ts> │
|
||||
│ ├─ Call Forgejo API (create branch) │
|
||||
│ ├─ Commit changes to branch │
|
||||
│ └─ Return PR URL + branch name │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Forgejo Git Service │
|
||||
│ ├─ Create branch: edit/rock/deploy-<ts> │
|
||||
│ ├─ From: main │
|
||||
│ ├─ Commit: "Update deploy steps" │
|
||||
│ └─ Trigger CI checks (markdown lint) │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
╔═══════╩═══════════════════════════════════════════╗
|
||||
║ STEP 2: AUTO-CREATE MERGE REQUEST ║
|
||||
╚═══════╤═══════════════════════════════════════════╝
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ UI Feedback │
|
||||
│ ✅ "Draft saved - Merge Request created" │
|
||||
│ └─ Show clickable MR link │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
│
|
||||
╔═══════╩═══════════════════════════════════════════╗
|
||||
║ STEP 3: HUMAN REVIEW (in Forgejo) ║
|
||||
╚═══════╤═══════════════════════════════════════════╝
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Reviewer (e.g., lead engineer) │
|
||||
│ ├─ Open MR in Forgejo web UI │
|
||||
│ ├─ Review diff (before/after) │
|
||||
│ ├─ Comment/suggest edits │
|
||||
│ ├─ Approve or request changes │
|
||||
│ └─ Click "Merge to main" │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
╔═══════╩═══════════════════════════════════════════╗
|
||||
║ STEP 4: AUTO-SYNC TO VAULT ║
|
||||
╚═══════╤═══════════════════════════════════════════╝
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Forgejo: Merge Complete │
|
||||
│ ├─ Branch merged to main │
|
||||
│ ├─ Trigger webhook: pull_request_merged │
|
||||
│ └─ Payload: {pr_id, merged_at, branch} │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
│ Webhook trigger
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ ArgoCD Application │
|
||||
│ ├─ Webhook receiver │
|
||||
│ ├─ Trigger sync of poimen-memory-app │
|
||||
│ └─ Pull latest from git (main) │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Git-Sync Sidecar Pod (poimen namespace) │
|
||||
│ ├─ Receive ArgoCD sync signal │
|
||||
│ ├─ `git pull origin main` in vault/ │
|
||||
│ ├─ File appears in PVC │
|
||||
│ └─ Update complete │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Memory Service Pod: Indexing Job │
|
||||
│ ├─ Detect vault file change │
|
||||
│ ├─ Tokenize + embed new content │
|
||||
│ ├─ Insert into pgvector index │
|
||||
│ └─ Document now searchable │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ UI Notification │
|
||||
│ ✅ "Document published!" │
|
||||
│ ├─ Document now visible to all │
|
||||
│ ├─ Embeddings indexed │
|
||||
│ └─ Available in search │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agent Context Flow
|
||||
|
||||
Real-time agent execution with memory retrieval tracking.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ User: Navigate to "Agent Workspace" tab │
|
||||
│ (Shows live agent execution) │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
│ Establish connection
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Frontend: WebSocket /memory/agents/stream │
|
||||
│ (Fallback: HTTP polling) │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Memory Service Pod │
|
||||
│ ├─ Tail agent execution log │
|
||||
│ ├─ Emit events: │
|
||||
│ │ - agent_started │
|
||||
│ │ - memory_retrieved {query, chunks, scores} │
|
||||
│ │ - tool_invoked {tool_name, args} │
|
||||
│ │ - tool_result {result} │
|
||||
│ │ - agent_decision {reasoning} │
|
||||
│ │ - agent_complete │
|
||||
│ └─ Stream as JSON events │
|
||||
└────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ UI: Real-time Dashboard │
|
||||
│ ├─ Timeline of agent actions (bottom-up) │
|
||||
│ ├─ Memory chunks used (with similarity scores) │
|
||||
│ ├─ Tool calls + outputs (expandable) │
|
||||
│ ├─ Decision tree (branching logic) │
|
||||
│ └─ Knowledge graph overlay (related docs) │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
Complete deployment topology with all components.
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │
|
||||
└────────────┬───────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ HTTPS
|
||||
│
|
||||
↓
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ K8s Ingress Controller (nginx-ingress) │
|
||||
│ ├─ TLS termination (memory.riotpiao.com) │
|
||||
│ ├─ Route to frontend Service (port 80) │
|
||||
│ └─ Route to memory Service (port 8080) │
|
||||
└────┬───────────────────────────────────┬───────────────────────────────┘
|
||||
│ │
|
||||
↓ (frontend) ↓ (API)
|
||||
┌──────────────────────────┐ ┌────────────────────────────────────┐
|
||||
│ Frontend Service │ │ Memory Service (8080) │
|
||||
│ (port 80) │ │ ├─ LoadBalancer type │
|
||||
└────┬─────────────────────┘ └────┬───────────────────────────────┘
|
||||
│ │
|
||||
↓ ↓
|
||||
┌──────────────────────────┐ ┌────────────────────────────────────┐
|
||||
│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │
|
||||
│ ├─ React app │ │ ├─ HTTP server (actix-web) │
|
||||
│ ├─ Vite build │ │ ├─ JWT validation │
|
||||
│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │
|
||||
│ └─ API client │ │ ├─ Vault browser │
|
||||
└──────────────────────────┘ │ ├─ Query (embedding) handler │
|
||||
│ ├─ Skills handler │
|
||||
┌──────────────────────────┼─ Projects handler │
|
||||
│ │ └─ Volume: /data/vault (PVC) │
|
||||
│ │ │
|
||||
│ └────┬───────────────────────────────┘
|
||||
│ │
|
||||
│ Memory Pod 2 (HA replica) │
|
||||
│ (identical to Pod 1) │
|
||||
│ │
|
||||
├───────────────────────────────┤
|
||||
│ │
|
||||
↓ ↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │
|
||||
│ ├─ Primary: memory-db-0 (PVC: 20Gi) │
|
||||
│ ├─ Replica: memory-db-1 (PVC: 20Gi) │
|
||||
│ ├─ Service: memory-db (headless) │
|
||||
│ ├─ Tables: │
|
||||
│ │ ├─ chunks (id, text, project_id, embedding, source) │
|
||||
│ │ ├─ skills (id, name, metadata) │
|
||||
│ │ ├─ projects (id, name) │
|
||||
│ │ └─ agent_logs (id, agent_id, action, timestamp) │
|
||||
│ └─ Extension: pgvector (vector similarity) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ INDEX: embedding <-> vector[]
|
||||
│
|
||||
└─ Used by: /memory/query (similarity search)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │
|
||||
│ ├─ Mount path: /data/vault │
|
||||
│ ├─ Content: │
|
||||
│ │ ├─ vault/skills/ │
|
||||
│ │ ├─ vault/runbooks/ │
|
||||
│ │ ├─ vault/evidence/ │
|
||||
│ │ └─ .git/ (full git history) │
|
||||
│ └─ Sync: git-sync sidecar (on file changes) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ ArgoCD monitors + syncs
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Git-Sync Sidecar (runs in Memory Pod) │
|
||||
│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │
|
||||
│ ├─ Branch: main │
|
||||
│ ├─ Sync interval: 30s │
|
||||
│ ├─ On merge: pulls to /data/vault │
|
||||
│ └─ Triggers indexing │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ Webhooks
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ External: Forgejo + ArgoCD │
|
||||
│ ├─ Forgejo webhook: pr_merged → ArgoCD │
|
||||
│ ├─ ArgoCD watches: poimen-memory-app (in git) │
|
||||
│ ├─ Auto-sync enabled (prune + selfHeal) │
|
||||
│ └─ Revision tracking │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ External: Authentik (OIDC) │
|
||||
│ ├─ Issuer: https://authentik.riotpiao.com/.../ │
|
||||
│ ├─ JWKS: .../jwks/ │
|
||||
│ ├─ OAuth2 App: poimen-memory │
|
||||
│ └─ Used by: JWT validation in Memory Service │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ External: LLM Service (Embeddings) │
|
||||
│ ├─ Provider: Vertex AI / Hugging Face / etc │
|
||||
│ ├─ Used by: /memory/query (tokenize + embed) │
|
||||
│ └─ Cached results (1hr TTL) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## OpenSearch + JWT Authentication
|
||||
|
||||
### JWT Flow with OpenSearch
|
||||
|
||||
```
|
||||
Frontend
|
||||
│ Authorization: Bearer <JWT>
|
||||
│ (Authentik-signed token)
|
||||
│
|
||||
↓
|
||||
Memory Service
|
||||
├─ Extract JWT from header
|
||||
├─ Validate signature (Authentik JWKS)
|
||||
├─ Verify expiry + issuer + audience
|
||||
└─ Extract claims (sub, roles, permissions)
|
||||
│
|
||||
├─ Can query pgvector (no auth needed)
|
||||
│
|
||||
└─ Forward JWT to OpenSearch
|
||||
│ Authorization: Bearer <same-JWT>
|
||||
│
|
||||
↓
|
||||
OpenSearch
|
||||
├─ Receive JWT in Authorization header
|
||||
├─ JWT realm validates signature
|
||||
├─ Extract roles from JWT claims
|
||||
├─ Map to internal roles (read_vault, write_vault)
|
||||
└─ Check index permissions
|
||||
│
|
||||
├─ Query allowed → return results
|
||||
└─ Write denied → 403 Forbidden
|
||||
```
|
||||
|
||||
### Hybrid Search: Semantic + Lexical
|
||||
|
||||
**Memory Service executes parallel searches:**
|
||||
|
||||
```
|
||||
POST /memory/query + JWT
|
||||
│
|
||||
├─ Path 1 (Semantic): pgvector
|
||||
│ ├─ LLM embedding
|
||||
│ ├─ Cosine similarity
|
||||
│ └─ Score: 0.88 (understanding)
|
||||
│
|
||||
├─ Path 2 (Lexical): OpenSearch + JWT
|
||||
│ ├─ Tokenize query
|
||||
│ ├─ BM25 ranking
|
||||
│ └─ Score: 0.96 (exact terms)
|
||||
│
|
||||
└─ Rerank (Weighted: 60% semantic + 40% lexical)
|
||||
└─ Final score: 0.92
|
||||
|
||||
Returns: Combined results sorted by hybrid score
|
||||
```
|
||||
|
||||
### OpenSearch JWT Realm Configuration
|
||||
|
||||
```yaml
|
||||
opensearch_security:
|
||||
authc:
|
||||
realms:
|
||||
jwt_realm:
|
||||
type: jwt
|
||||
order: 1
|
||||
|
||||
# Token extraction
|
||||
token_name: Authorization
|
||||
token_extractor: "Bearer " # Strip "Bearer " prefix
|
||||
|
||||
# JWKS from Authentik (auto-refresh hourly)
|
||||
jwks_uri: "https://authentik.riotpiao.com/application/o/poimen-memory/jwks/"
|
||||
jwks_refresh_interval_ms: 3600000
|
||||
|
||||
# Issuer validation
|
||||
issuer: "https://authentik.riotpiao.com/application/o/poimen-memory/"
|
||||
|
||||
# Extract claims
|
||||
roles_key: "roles" # From JWT claim
|
||||
subject_key: "sub" # User identifier
|
||||
```
|
||||
|
||||
### Roles Mapping (JWT → OpenSearch)
|
||||
|
||||
**JWT claims example:**
|
||||
```json
|
||||
{
|
||||
"iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
|
||||
"sub": "[email protected]",
|
||||
"roles": ["read_vault", "write_vault"],
|
||||
"permissions": ["memory:read", "memory:write"]
|
||||
}
|
||||
```
|
||||
|
||||
**OpenSearch role definitions:**
|
||||
```yaml
|
||||
read_vault:
|
||||
cluster_permissions: ["cluster:monitor/health"]
|
||||
index_permissions:
|
||||
- index_patterns: ["vault-*"]
|
||||
allowed_actions: ["indices:data/read/search"]
|
||||
|
||||
write_vault:
|
||||
cluster_permissions: ["cluster:monitor/health"]
|
||||
index_permissions:
|
||||
- index_patterns: ["vault-*"]
|
||||
allowed_actions: ["indices:data/write/index", "indices:data/read/search"]
|
||||
```
|
||||
|
||||
### Network Security (K8s NetworkPolicy)
|
||||
|
||||
```yaml
|
||||
# Only Memory Service can access OpenSearch
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: opensearch-access
|
||||
namespace: poimen
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: opensearch
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: poimen-memory
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9200
|
||||
```
|
||||
|
||||
|
||||
## Pod Infrastructure
|
||||
|
||||
Complete pod inventory deployed in `poimen` namespace.
|
||||
|
||||
### Production Pods
|
||||
|
||||
| Pod Name | Role | Replicas | PVC | Purpose |
|
||||
|----------|------|----------|-----|---------|
|
||||
| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, hybrid search |
|
||||
| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | pgvector semantic search index |
|
||||
| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |
|
||||
| **opensearch-0** | OpenSearch Primary | 1 | 30Gi | Lexical (BM25) search, JWT realm |
|
||||
| **opensearch-1** | OpenSearch Replica | 1 | 30Gi | HA cluster node, JWT validation |
|
||||
| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |
|
||||
|
||||
### Supporting Infrastructure (External)
|
||||
|
||||
| Component | Role | Location |
|
||||
|-----------|------|----------|
|
||||
| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod |
|
||||
| **ArgoCD Application** | CD orchestration | argocd namespace |
|
||||
| **Ingress Controller** | Reverse proxy | ingress-nginx namespace |
|
||||
| **Longhorn** | Storage provider | Storage layer |
|
||||
|
||||
### Total Pod Count: **8 Production Pods**
|
||||
|
||||
```
|
||||
Namespace: poimen
|
||||
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
|
||||
├─ memory-db-0 (StatefulSet) ............................... (1)
|
||||
├─ memory-db-1 (StatefulSet) ............................... (1)
|
||||
├─ opensearch-0 (StatefulSet) .............................. (1) [NEW]
|
||||
├─ opensearch-1 (StatefulSet) .............................. (1) [NEW]
|
||||
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)
|
||||
|
||||
Namespace: argocd
|
||||
└─ argocd-server, argocd-repo-server, etc .................. (5+)
|
||||
|
||||
Namespace: ingress-nginx
|
||||
└─ nginx-ingress-controller ................................ (1)
|
||||
|
||||
TOTAL: 14-16 pods (8 core + 6-8 supporting)
|
||||
```
|
||||
|
||||
### Pod Responsibilities
|
||||
|
||||
#### Memory Service Pod (×2, HA)
|
||||
- **Listen**: 0.0.0.0:8080
|
||||
- **Endpoints**:
|
||||
- `GET /memory/vault` — Read vault files
|
||||
- `POST /memory/query` — Hybrid search (semantic + lexical)
|
||||
- `GET /memory/skills` — List skills
|
||||
- `POST /memory/grc/draft` — Create branch + MR
|
||||
- `GET /memory/grc/status` — Check MR status
|
||||
- `GET /memory/agents/logs` — Stream agent events
|
||||
- **Auth**: JWT (Authentik)
|
||||
- **Hybrid Search Logic**:
|
||||
- Validates JWT (Authentik JWKS)
|
||||
- Queries pgvector (semantic in parallel)
|
||||
- Queries OpenSearch with JWT (lexical in parallel)
|
||||
- Reranks results (weighted: 60% semantic, 40% lexical)
|
||||
- Returns combined results
|
||||
- **Connections**:
|
||||
- PostgreSQL (pgvector semantic search)
|
||||
- OpenSearch (lexical search with JWT)
|
||||
- Forgejo API (GRC)
|
||||
- LLM service (embeddings)
|
||||
- PVC (vault files)
|
||||
|
||||
#### PostgreSQL Pod (×2, Primary + Replica)
|
||||
- **Listen**: 5432
|
||||
- **Service**: memory-db (headless for StatefulSet)
|
||||
- **Storage**: 20Gi per pod (PVC)
|
||||
- **Replication**: Streaming replication (primary → replica)
|
||||
- **Extensions**: pgvector
|
||||
- **Data**:
|
||||
- chunks table (with vector index for semantic search)
|
||||
- skills table
|
||||
- projects table
|
||||
- agent_logs table
|
||||
- **Role**: Semantic search engine (embeddings)
|
||||
|
||||
#### OpenSearch Pod (×2, Primary + Replica)
|
||||
- **Listen**: 9200 (HTTP), 9300 (cluster communication)
|
||||
- **Service**: opensearch (headless for cluster), opensearch-internal (for queries)
|
||||
- **Storage**: 30Gi per pod (PVC)
|
||||
- **Cluster**: poimen-memory (2-node minimum)
|
||||
- **Security**:
|
||||
- JWT realm enabled (validates Authentik tokens)
|
||||
- JWKS endpoint: https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
|
||||
- Role mapping: Extract roles from JWT claims
|
||||
- Index permissions: read_vault, write_vault roles
|
||||
- **Indices**:
|
||||
- vault-* (BM25 text search with TF-IDF scoring)
|
||||
- **Role**: Lexical search engine (exact terms + TF-IDF)
|
||||
|
||||
#### Frontend Pod (×1-2)
|
||||
- **Listen**: 80
|
||||
- **Serve**: React SPA static files
|
||||
- **Endpoints**:
|
||||
- `/` — App shell
|
||||
- `/api/*` — Proxy to Memory Service (8080)
|
||||
- **Auth**: JWT (localStorage)
|
||||
- **Build**: Vite (production bundle)
|
||||
|
||||
#### Git-Sync Sidecar (embedded in Memory Pod)
|
||||
- **Runs**: As a init container + background process
|
||||
- **Watch**: Forgejo main branch
|
||||
- **Sync interval**: 30 seconds
|
||||
- **Action on merge**: `git pull` → trigger re-index
|
||||
|
||||
---
|
||||
|
||||
## Traffic Flow Diagram
|
||||
|
||||
```
|
||||
Internet (Users)
|
||||
│
|
||||
│ HTTPS
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────┐
|
||||
│ Ingress Controller │
|
||||
│ (nginx) │
|
||||
└──┬────────────┬─────────┘
|
||||
│ │
|
||||
│ port 80 │ port 8080
|
||||
│ │
|
||||
↓ ↓
|
||||
┌──────────┐ ┌──────────────────┐
|
||||
│ Frontend │ │ Memory Service │
|
||||
│ (React) │ │ (Rust + Actix) │
|
||||
└────┬─────┘ └──┬───────────┬────┘
|
||||
│ │ │
|
||||
│ ┌──┘ └──┐
|
||||
│ │ │
|
||||
↓ ↓ ↓
|
||||
┌───────────────────┐ ┌──────────────┐
|
||||
│ PostgreSQL │ │ Vault PVC │
|
||||
│ (pgvector index) │ │ (git files) │
|
||||
└───────────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
### Core Services
|
||||
- [x] Memory Service (2 pods) deployed in `poimen` namespace
|
||||
- [x] PostgreSQL StatefulSet (2 pods) deployed with pgvector
|
||||
- [x] OpenSearch StatefulSet (2 pods) deployed with JWT realm
|
||||
- [x] PVC: poimen-memory-vault (10Gi) attached
|
||||
- [x] PVC: opensearch data (30Gi per pod) attached
|
||||
|
||||
### Security & Auth
|
||||
- [x] JWT auth enabled (Authentik integration at Memory Service)
|
||||
- [x] OpenSearch JWT realm configured (validates Authentik tokens)
|
||||
- [x] K8s NetworkPolicy (only Memory Service → OpenSearch)
|
||||
- [x] Role mapping (JWT claims → OpenSearch roles)
|
||||
|
||||
### Integration
|
||||
- [x] Git-sync sidecar configured (auto-pull on merge)
|
||||
- [x] Ingress configured (memory.riotpiao.com)
|
||||
- [x] Hybrid search endpoints (/memory/query?search_method=hybrid)
|
||||
|
||||
### Design Documentation
|
||||
- [x] Retrieval pipeline architecture (4-stage: normalize → parallel → fusion → ranking)
|
||||
- [x] Index optimization (pgvector ivfflat + OpenSearch BM25 tuning)
|
||||
- [x] Score fusion strategy (weighted linear + RRF alternative)
|
||||
- [x] Query routing decision tree (short queries → lexical, normal → hybrid)
|
||||
- [x] Accuracy metrics (MRR, NDCG@10, Precision@K, Recall@K)
|
||||
- [x] Weight tuning strategy (A/B testing framework)
|
||||
- [ ] **Reference**: See `docs/HYBRID_SEARCH_DESIGN.md` (19KB, comprehensive design)
|
||||
|
||||
### Testing & Deployment
|
||||
- [ ] Test fixture setup (query + expected results dataset)
|
||||
- [ ] NDCG/MRR baseline measurements (semantic vs lexical)
|
||||
- [ ] Weight tuning experiments (0.5/0.5, 0.6/0.4, 0.7/0.3, 0.4/0.6)
|
||||
- [ ] Performance benchmarks (latency: parallel vs serial)
|
||||
- [ ] Frontend pod deployment
|
||||
- [ ] End-to-end hybrid search tests
|
||||
- [ ] Gradual rollout (Phase 1: lexical-only → Phase 2: hybrid 10% → Phase 3: 100%)
|
||||
- [ ] GRC endpoints tested
|
||||
- [ ] Agent logging endpoints tested
|
||||
|
||||
Reference in New Issue
Block a user