rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3f096e8f9cfdd9f59e3b7e02468e35ab6b1dd9b9
poimen-memory/memory-flow.md
T
Story Crater Bot 959c596b1d chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1) 
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
2026-08-27 20:25:05 -07:00

834 lines
44 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Memory UI Flow - Complete Workflow
## Table of Contents
1. [Read Flow](#read-flow)
2. [Search Flow (Semantic + Lexical Hybrid)](#search-flow-semantic--lexical-hybrid)
3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow)
4. [Agent Context Flow](#agent-context-flow)
5. [System Architecture](#system-architecture)
6. [OpenSearch + JWT Authentication](#opensearch--jwt-authentication)
7. [Pod Infrastructure](#pod-infrastructure)
---
## Read Flow
Browse vault documents from the web UI.
```
┌─────────────────────────────────────────────────────┐
│ User: memory.riotpiao.com │
│ (Browser, JWT token in localStorage) │
└────────────┬────────────────────────────────────────┘
│
│ GET /memory/vault?project=poimen
│ Authorization: Bearer <JWT>
│
↓
┌─────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Load vault files from PVC │
│ ├─ Build file tree (directory structure) │
│ └─ Return JSON response │
└────────────┬────────────────────────────────────────┘
│
│ {files: [{path, title, updated_at}...]}
│
↓
┌─────────────────────────────────────────────────────┐
│ UI: Render Vault Browser │
│ ├─ Project selector (dropdown) │
│ ├─ File tree (collapsible folders) │
│ ├─ Breadcrumb navigation │
│ └─ Preview panel (markdown rendering) │
└─────────────────────────────────────────────────────┘
```
---
## Search Flow (Semantic + Lexical Hybrid)
Hybrid retrieval pipeline combining:
- **Semantic path**: pgvector embeddings (query understanding)
- **Lexical path**: OpenSearch BM25 (exact term matching)
- **Fusion**: Weighted linear combination (60% semantic, 40% lexical)
- **Result limit**: Top 50 from each engine, merge to top 10 final
### Complete Retrieval Pipeline
```
User Query: "fix kubernetes port 8080 conflict"
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 1: Query Normalization │
├────────────────────────────────────────────────────────┤
│ ├─ Tokenize: ["fix", "kubernetes", ...] │
│ ├─ Extract entities: {port: "8080"} │
│ ├─ Generate embedding (LLM) │
│ └─ Create QueryContext │
└────────────────────────┬───────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 2: Parallel Retrieval (Both Engines) │
├────────────────────────────────────────────────────────┤
│ │
│ ┌─ SEMANTIC (pgvector) │
│ │ ├─ Query embedding <-> vector │
│ │ ├─ Filter: project_id, level, date │
│ │ ├─ ORDER BY cosine_similarity DESC │
│ │ └─ Return: Top 50 with scores │
│ │ │
│ └─ LEXICAL (OpenSearch + JWT) │
│ ├─ multi_match on [content^2, breadcrumb] │
│ ├─ BM25 ranking with fuzziness │
│ ├─ Filter: project_id, level, date │
│ └─ Return: Top 50 with raw scores │
│ │
└────────────────┬──────────────────────────┬────────────┘
│ │
sem_results: [(doc1, 0.92), ...] lex_results: [(doc1, 8.5), ...]
│ │
└──────────┬───────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 3: Score Normalization │
├────────────────────────────────────────────────────────┤
│ ├─ Normalize semantic: [0.0 ... 1.0] │
│ │ (already 0-1 from cosine) │
│ │ │
│ ├─ Normalize lexical: [0.0 ... 1.0] │
│ │ (min-max: (score-min)/(max-min)) │
│ │ │
│ └─ Result: Both in [0.0, 1.0] range │
│ │
└────────────────┬──────────────────────────────────────┘
│
sem_norm: [(doc1, 1.0), ...] lex_norm: [(doc1, 0.98), ...]
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 4: Fusion (Weighted Linear) │
├────────────────────────────────────────────────────────┤
│ ├─ Merge all doc IDs from both results │
│ ├─ For each doc: score = 0.6*sem + 0.4*lex │
│ │ │
│ │ doc1: 0.6*1.0 + 0.4*0.98 = 0.992 │
│ │ doc2: 0.6*0.96 + 0.4*0.0 = 0.576 │
│ │ doc3: 0.6*0.0 + 0.4*0.88 = 0.352 │
│ │ │
│ ├─ Sort descending: [doc1, doc2, doc3] │
│ └─ Take top-10 │
│ │
└────────────────┬──────────────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 5: Score Breakdown (Transparency) │
├────────────────────────────────────────────────────────┤
│ doc1: { │
│ "score": 0.992, │
│ "sem_component": 1.0, │
│ "lex_component": 0.98, │
│ "sem_weight": 0.6, │
│ "lex_weight": 0.4, │
│ "reason": "Exact semantic match + strong lexical" │
│ } │
│ │
└────────────────┬──────────────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Final Results (Top-10) │
├────────────────────────────────────────────────────────┤
│ 1. port-forward.md (0.992) │
│ - Semantic: 1.0 | Lexical: 0.98 │
│ │
│ 2. troubleshooting.md (0.576) │
│ - Semantic: 0.96 | Lexical: 0.0 │
│ │
│ 3. k8s-basics.md (0.352) │
│ - Semantic: 0.0 | Lexical: 0.88 │
│ │
└────────────────────────────────────────────────────────┘
```
### Query Routing Decision Tree
```
Query Received
│
├─ Token count < 3?
│ ├─ YES → Use LEXICAL_ONLY
│ │ (short queries: "fix port" → better BM25)
│ │
│ └─ NO → Continue...
│
├─ Contains special syntax (#tag, @mention)?
│ ├─ YES → Use LEXICAL_WITH_FILTERS
│ │ (preserve exact matches)
│ │
│ └─ NO → Continue...
│
├─ Can we embed the query?
│ ├─ YES → Use HYBRID
│ │ (both engines)
│ │
│ └─ NO → Use LEXICAL_ONLY
│ (LLM unavailable, fallback)
│
└─ Execute chosen strategy
```
### Index Optimization
**PostgreSQL (pgvector) - Semantic Path:**
- Index: `ivfflat (embedding vector_cosine_ops) WITH (lists=100)`
- Filter: `project_id, level IN ('L0','L1','L2'), created_at > now-1y`
- Retrieve: Top 50 chunks, then merge with lexical
- Lookup: O(log n) pre-filter + O(1) embedding distance
**OpenSearch (BM25) - Lexical Path:**
- Analyzer: `standard` (lowercase, stop words)
- Fields: `content^2` (2x boost) + `breadcrumb` + `source`
- Tokenizer: Standard + n-gram for typo tolerance
- Retrieve: Top 50 results via BM25, then merge with semantic
- Lookup: O(n) inverted index scan + TF-IDF ranking
**Merging Strategy:**
1. Normalize both score ranges to [0.0, 1.0]
2. Weighted sum: `0.6 * semantic + 0.4 * lexical`
3. Sort by final score
4. Return top-10 to user
**Accuracy Metrics (A/B Testing):**
- MRR (Mean Reciprocal Rank): Position of first correct result
- NDCG@10 (Normalized Discounted Cumulative Gain): Quality of top-10 ranking
- Precision@K: Relevant results in top-K
- Recall@K: Coverage of all relevant results in top-K
------
## Edit Flow (GRM Workflow)
Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.
```
┌──────────────────────────────────────────────────────┐
│ User: Clicks "Edit" on document │
│ Example: runbook-deploy.md │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Switch to Edit Mode │
│ ├─ Load document content from Memory Service │
│ ├─ Show markdown editor (CodeMirror) │
│ ├─ Disable Save button (drafts only) │
│ └─ Show "Submit for Review" button │
└────────────┬─────────────────────────────────────────┘
│
├─ User makes edits (e.g., update deploy steps)
│
↓
┌──────────────────────────────────────────────────────┐
│ User: Click "Submit for Review" │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 1: CREATE BRANCH ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: POST /memory/grc/draft │
│ { │
│ "document_path": "vault/runbooks/deploy.md", │
│ "content": "<new markdown content>", │
│ "message": "Update deploy steps", │
│ "user": "[email protected]" │
│ } │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: GRC Handler │
│ ├─ Generate branch name: edit/rock/deploy-<ts> │
│ ├─ Call Forgejo API (create branch) │
│ ├─ Commit changes to branch │
│ └─ Return PR URL + branch name │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo Git Service │
│ ├─ Create branch: edit/rock/deploy-<ts> │
│ ├─ From: main │
│ ├─ Commit: "Update deploy steps" │
│ └─ Trigger CI checks (markdown lint) │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 2: AUTO-CREATE MERGE REQUEST ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Feedback │
│ ✅ "Draft saved - Merge Request created" │
│ └─ Show clickable MR link │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 3: HUMAN REVIEW (in Forgejo) ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Reviewer (e.g., lead engineer) │
│ ├─ Open MR in Forgejo web UI │
│ ├─ Review diff (before/after) │
│ ├─ Comment/suggest edits │
│ ├─ Approve or request changes │
│ └─ Click "Merge to main" │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 4: AUTO-SYNC TO VAULT ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo: Merge Complete │
│ ├─ Branch merged to main │
│ ├─ Trigger webhook: pull_request_merged │
│ └─ Payload: {pr_id, merged_at, branch} │
└────────────┬─────────────────────────────────────────┘
│
│ Webhook trigger
│
↓
┌──────────────────────────────────────────────────────┐
│ ArgoCD Application │
│ ├─ Webhook receiver │
│ ├─ Trigger sync of poimen-memory-app │
│ └─ Pull latest from git (main) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Git-Sync Sidecar Pod (poimen namespace) │
│ ├─ Receive ArgoCD sync signal │
│ ├─ `git pull origin main` in vault/ │
│ ├─ File appears in PVC │
│ └─ Update complete │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: Indexing Job │
│ ├─ Detect vault file change │
│ ├─ Tokenize + embed new content │
│ ├─ Insert into pgvector index │
│ └─ Document now searchable │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Notification │
│ ✅ "Document published!" │
│ ├─ Document now visible to all │
│ ├─ Embeddings indexed │
│ └─ Available in search │
└──────────────────────────────────────────────────────┘
```
---
## Agent Context Flow
Real-time agent execution with memory retrieval tracking.
```
┌──────────────────────────────────────────────────────┐
│ User: Navigate to "Agent Workspace" tab │
│ (Shows live agent execution) │
└────────────┬─────────────────────────────────────────┘
│
│ Establish connection
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: WebSocket /memory/agents/stream │
│ (Fallback: HTTP polling) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Tail agent execution log │
│ ├─ Emit events: │
│ │ - agent_started │
│ │ - memory_retrieved {query, chunks, scores} │
│ │ - tool_invoked {tool_name, args} │
│ │ - tool_result {result} │
│ │ - agent_decision {reasoning} │
│ │ - agent_complete │
│ └─ Stream as JSON events │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Real-time Dashboard │
│ ├─ Timeline of agent actions (bottom-up) │
│ ├─ Memory chunks used (with similarity scores) │
│ ├─ Tool calls + outputs (expandable) │
│ ├─ Decision tree (branching logic) │
│ └─ Knowledge graph overlay (related docs) │
└──────────────────────────────────────────────────────┘
```
---
## System Architecture
Complete deployment topology with all components.
```
┌────────────────────────────────────────────────────────────────────────┐
│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │
└────────────┬───────────────────────────────────────────────────────────┘
│
│ HTTPS
│
↓
┌────────────────────────────────────────────────────────────────────────┐
│ K8s Ingress Controller (nginx-ingress) │
│ ├─ TLS termination (memory.riotpiao.com) │
│ ├─ Route to frontend Service (port 80) │
│ └─ Route to memory Service (port 8080) │
└────┬───────────────────────────────────┬───────────────────────────────┘
│ │
↓ (frontend) ↓ (API)
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Service │ │ Memory Service (8080) │
│ (port 80) │ │ ├─ LoadBalancer type │
└────┬─────────────────────┘ └────┬───────────────────────────────┘
│ │
↓ ↓
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │
│ ├─ React app │ │ ├─ HTTP server (actix-web) │
│ ├─ Vite build │ │ ├─ JWT validation │
│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │
│ └─ API client │ │ ├─ Vault browser │
└──────────────────────────┘ │ ├─ Query (embedding) handler │
│ ├─ Skills handler │
┌──────────────────────────┼─ Projects handler │
│ │ └─ Volume: /data/vault (PVC) │
│ │ │
│ └────┬───────────────────────────────┘
│ │
│ Memory Pod 2 (HA replica) │
│ (identical to Pod 1) │
│ │
├───────────────────────────────┤
│ │
↓ ↓
┌─────────────────────────────────────────────────────────────┐
│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │
│ ├─ Primary: memory-db-0 (PVC: 20Gi) │
│ ├─ Replica: memory-db-1 (PVC: 20Gi) │
│ ├─ Service: memory-db (headless) │
│ ├─ Tables: │
│ │ ├─ chunks (id, text, project_id, embedding, source) │
│ │ ├─ skills (id, name, metadata) │
│ │ ├─ projects (id, name) │
│ │ └─ agent_logs (id, agent_id, action, timestamp) │
│ └─ Extension: pgvector (vector similarity) │
└─────────────────────────────────────────────────────────────┘
│
│ INDEX: embedding <-> vector[]
│
└─ Used by: /memory/query (similarity search)
┌─────────────────────────────────────────────────────────────┐
│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │
│ ├─ Mount path: /data/vault │
│ ├─ Content: │
│ │ ├─ vault/skills/ │
│ │ ├─ vault/runbooks/ │
│ │ ├─ vault/evidence/ │
│ │ └─ .git/ (full git history) │
│ └─ Sync: git-sync sidecar (on file changes) │
└─────────────────────────────────────────────────────────────┘
│
│ ArgoCD monitors + syncs
│
↓
┌─────────────────────────────────────────────────────────────┐
│ Git-Sync Sidecar (runs in Memory Pod) │
│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │
│ ├─ Branch: main │
│ ├─ Sync interval: 30s │
│ ├─ On merge: pulls to /data/vault │
│ └─ Triggers indexing │
└─────────────────────────────────────────────────────────────┘
│
│ Webhooks
│
↓
┌─────────────────────────────────────────────────────────────┐
│ External: Forgejo + ArgoCD │
│ ├─ Forgejo webhook: pr_merged → ArgoCD │
│ ├─ ArgoCD watches: poimen-memory-app (in git) │
│ ├─ Auto-sync enabled (prune + selfHeal) │
│ └─ Revision tracking │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: Authentik (OIDC) │
│ ├─ Issuer: https://authentik.riotpiao.com/.../ │
│ ├─ JWKS: .../jwks/ │
│ ├─ OAuth2 App: poimen-memory │
│ └─ Used by: JWT validation in Memory Service │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: LLM Service (Embeddings) │
│ ├─ Provider: Vertex AI / Hugging Face / etc │
│ ├─ Used by: /memory/query (tokenize + embed) │
│ └─ Cached results (1hr TTL) │
└─────────────────────────────────────────────────────────────┘
```
---
---
## OpenSearch + JWT Authentication
### JWT Flow with OpenSearch
```
Frontend
│ Authorization: Bearer <JWT>
│ (Authentik-signed token)
│
↓
Memory Service
├─ Extract JWT from header
├─ Validate signature (Authentik JWKS)
├─ Verify expiry + issuer + audience
└─ Extract claims (sub, roles, permissions)
│
├─ Can query pgvector (no auth needed)
│
└─ Forward JWT to OpenSearch
│ Authorization: Bearer <same-JWT>
│
↓
OpenSearch
├─ Receive JWT in Authorization header
├─ JWT realm validates signature
├─ Extract roles from JWT claims
├─ Map to internal roles (read_vault, write_vault)
└─ Check index permissions
│
├─ Query allowed → return results
└─ Write denied → 403 Forbidden
```
### Hybrid Search: Semantic + Lexical
**Memory Service executes parallel searches:**
```
POST /memory/query + JWT
│
├─ Path 1 (Semantic): pgvector
│ ├─ LLM embedding
│ ├─ Cosine similarity
│ └─ Score: 0.88 (understanding)
│
├─ Path 2 (Lexical): OpenSearch + JWT
│ ├─ Tokenize query
│ ├─ BM25 ranking
│ └─ Score: 0.96 (exact terms)
│
└─ Rerank (Weighted: 60% semantic + 40% lexical)
└─ Final score: 0.92
Returns: Combined results sorted by hybrid score
```
### OpenSearch JWT Realm Configuration
```yaml
opensearch_security:
authc:
realms:
jwt_realm:
type: jwt
order: 1
# Token extraction
token_name: Authorization
token_extractor: "Bearer " # Strip "Bearer " prefix
# JWKS from Authentik (auto-refresh hourly)
jwks_uri: "https://authentik.riotpiao.com/application/o/poimen-memory/jwks/"
jwks_refresh_interval_ms: 3600000
# Issuer validation
issuer: "https://authentik.riotpiao.com/application/o/poimen-memory/"
# Extract claims
roles_key: "roles" # From JWT claim
subject_key: "sub" # User identifier
```
### Roles Mapping (JWT → OpenSearch)
**JWT claims example:**
```json
{
"iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
"sub": "[email protected]",
"roles": ["read_vault", "write_vault"],
"permissions": ["memory:read", "memory:write"]
}
```
**OpenSearch role definitions:**
```yaml
read_vault:
cluster_permissions: ["cluster:monitor/health"]
index_permissions:
- index_patterns: ["vault-*"]
allowed_actions: ["indices:data/read/search"]
write_vault:
cluster_permissions: ["cluster:monitor/health"]
index_permissions:
- index_patterns: ["vault-*"]
allowed_actions: ["indices:data/write/index", "indices:data/read/search"]
```
### Network Security (K8s NetworkPolicy)
```yaml
# Only Memory Service can access OpenSearch
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: opensearch-access
namespace: poimen
spec:
podSelector:
matchLabels:
app: opensearch
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: poimen-memory
ports:
- protocol: TCP
port: 9200
```
## Pod Infrastructure
Complete pod inventory deployed in `poimen` namespace.
### Production Pods
| Pod Name | Role | Replicas | PVC | Purpose |
|----------|------|----------|-----|---------|
| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, hybrid search |
| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | pgvector semantic search index |
| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |
| **opensearch-0** | OpenSearch Primary | 1 | 30Gi | Lexical (BM25) search, JWT realm |
| **opensearch-1** | OpenSearch Replica | 1 | 30Gi | HA cluster node, JWT validation |
| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |
### Supporting Infrastructure (External)
| Component | Role | Location |
|-----------|------|----------|
| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod |
| **ArgoCD Application** | CD orchestration | argocd namespace |
| **Ingress Controller** | Reverse proxy | ingress-nginx namespace |
| **Longhorn** | Storage provider | Storage layer |
### Total Pod Count: **8 Production Pods**
```
Namespace: poimen
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
├─ memory-db-0 (StatefulSet) ............................... (1)
├─ memory-db-1 (StatefulSet) ............................... (1)
├─ opensearch-0 (StatefulSet) .............................. (1) [NEW]
├─ opensearch-1 (StatefulSet) .............................. (1) [NEW]
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)
Namespace: argocd
└─ argocd-server, argocd-repo-server, etc .................. (5+)
Namespace: ingress-nginx
└─ nginx-ingress-controller ................................ (1)
TOTAL: 14-16 pods (8 core + 6-8 supporting)
```
### Pod Responsibilities
#### Memory Service Pod (×2, HA)
- **Listen**: 0.0.0.0:8080
- **Endpoints**:
- `GET /memory/vault` — Read vault files
- `POST /memory/query` — Hybrid search (semantic + lexical)
- `GET /memory/skills` — List skills
- `POST /memory/grc/draft` — Create branch + MR
- `GET /memory/grc/status` — Check MR status
- `GET /memory/agents/logs` — Stream agent events
- **Auth**: JWT (Authentik)
- **Hybrid Search Logic**:
- Validates JWT (Authentik JWKS)
- Queries pgvector (semantic in parallel)
- Queries OpenSearch with JWT (lexical in parallel)
- Reranks results (weighted: 60% semantic, 40% lexical)
- Returns combined results
- **Connections**:
- PostgreSQL (pgvector semantic search)
- OpenSearch (lexical search with JWT)
- Forgejo API (GRC)
- LLM service (embeddings)
- PVC (vault files)
#### PostgreSQL Pod (×2, Primary + Replica)
- **Listen**: 5432
- **Service**: memory-db (headless for StatefulSet)
- **Storage**: 20Gi per pod (PVC)
- **Replication**: Streaming replication (primary → replica)
- **Extensions**: pgvector
- **Data**:
- chunks table (with vector index for semantic search)
- skills table
- projects table
- agent_logs table
- **Role**: Semantic search engine (embeddings)
#### OpenSearch Pod (×2, Primary + Replica)
- **Listen**: 9200 (HTTP), 9300 (cluster communication)
- **Service**: opensearch (headless for cluster), opensearch-internal (for queries)
- **Storage**: 30Gi per pod (PVC)
- **Cluster**: poimen-memory (2-node minimum)
- **Security**:
- JWT realm enabled (validates Authentik tokens)
- JWKS endpoint: https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
- Role mapping: Extract roles from JWT claims
- Index permissions: read_vault, write_vault roles
- **Indices**:
- vault-* (BM25 text search with TF-IDF scoring)
- **Role**: Lexical search engine (exact terms + TF-IDF)
#### Frontend Pod (×1-2)
- **Listen**: 80
- **Serve**: React SPA static files
- **Endpoints**:
- `/` — App shell
- `/api/*` — Proxy to Memory Service (8080)
- **Auth**: JWT (localStorage)
- **Build**: Vite (production bundle)
#### Git-Sync Sidecar (embedded in Memory Pod)
- **Runs**: As a init container + background process
- **Watch**: Forgejo main branch
- **Sync interval**: 30 seconds
- **Action on merge**: `git pull` → trigger re-index
---
## Traffic Flow Diagram
```
Internet (Users)
│
│ HTTPS
│
↓
┌─────────────────────────┐
│ Ingress Controller │
│ (nginx) │
└──┬────────────┬─────────┘
│ │
│ port 80 │ port 8080
│ │
↓ ↓
┌──────────┐ ┌──────────────────┐
│ Frontend │ │ Memory Service │
│ (React) │ │ (Rust + Actix) │
└────┬─────┘ └──┬───────────┬────┘
│ │ │
│ ┌──┘ └──┐
│ │ │
↓ ↓ ↓
┌───────────────────┐ ┌──────────────┐
│ PostgreSQL │ │ Vault PVC │
│ (pgvector index) │ │ (git files) │
└───────────────────┘ └──────────────┘
```
---
## Deployment Checklist
### Core Services
- [x] Memory Service (2 pods) deployed in `poimen` namespace
- [x] PostgreSQL StatefulSet (2 pods) deployed with pgvector
- [x] OpenSearch StatefulSet (2 pods) deployed with JWT realm
- [x] PVC: poimen-memory-vault (10Gi) attached
- [x] PVC: opensearch data (30Gi per pod) attached
### Security & Auth
- [x] JWT auth enabled (Authentik integration at Memory Service)
- [x] OpenSearch JWT realm configured (validates Authentik tokens)
- [x] K8s NetworkPolicy (only Memory Service → OpenSearch)
- [x] Role mapping (JWT claims → OpenSearch roles)
### Integration
- [x] Git-sync sidecar configured (auto-pull on merge)
- [x] Ingress configured (memory.riotpiao.com)
- [x] Hybrid search endpoints (/memory/query?search_method=hybrid)
### Design Documentation
- [x] Retrieval pipeline architecture (4-stage: normalize → parallel → fusion → ranking)
- [x] Index optimization (pgvector ivfflat + OpenSearch BM25 tuning)
- [x] Score fusion strategy (weighted linear + RRF alternative)
- [x] Query routing decision tree (short queries → lexical, normal → hybrid)
- [x] Accuracy metrics (MRR, NDCG@10, Precision@K, Recall@K)
- [x] Weight tuning strategy (A/B testing framework)
- [ ] **Reference**: See `docs/HYBRID_SEARCH_DESIGN.md` (19KB, comprehensive design)
### Testing & Deployment
- [ ] Test fixture setup (query + expected results dataset)
- [ ] NDCG/MRR baseline measurements (semantic vs lexical)
- [ ] Weight tuning experiments (0.5/0.5, 0.6/0.4, 0.7/0.3, 0.4/0.6)
- [ ] Performance benchmarks (latency: parallel vs serial)
- [ ] Frontend pod deployment
- [ ] End-to-end hybrid search tests
- [ ] Gradual rollout (Phase 1: lexical-only → Phase 2: hybrid 10% → Phase 3: 100%)
- [ ] GRC endpoints tested
- [ ] Agent logging endpoints tested
Reference in New Issue View Git Blame Copy Permalink