rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
463958be1485e86ec6a939e85d0d76075470917b
poimen-memory/memory-flow.md
T

834 lines
44 KiB
Markdown
Raw Normal View History

chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)
2026-08-27 20:25:05 -07:00
# Memory UI Flow - Complete Workflow
## Table of Contents
1. [Read Flow](#read-flow)
2. [Search Flow (Semantic + Lexical Hybrid)](#search-flow-semantic--lexical-hybrid)
3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow)
4. [Agent Context Flow](#agent-context-flow)
5. [System Architecture](#system-architecture)
6. [OpenSearch + JWT Authentication](#opensearch--jwt-authentication)
7. [Pod Infrastructure](#pod-infrastructure)
---
## Read Flow
Browse vault documents from the web UI.
```
┌─────────────────────────────────────────────────────┐
│ User: memory.riotpiao.com │
│ (Browser, JWT token in localStorage) │
└────────────┬────────────────────────────────────────┘
│
│ GET /memory/vault?project=poimen
│ Authorization: Bearer <JWT>
│
↓
┌─────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Load vault files from PVC │
│ ├─ Build file tree (directory structure) │
│ └─ Return JSON response │
└────────────┬────────────────────────────────────────┘
│
│ {files: [{path, title, updated_at}...]}
│
↓
┌─────────────────────────────────────────────────────┐
│ UI: Render Vault Browser │
│ ├─ Project selector (dropdown) │
│ ├─ File tree (collapsible folders) │
│ ├─ Breadcrumb navigation │
│ └─ Preview panel (markdown rendering) │
└─────────────────────────────────────────────────────┘
```
---
## Search Flow (Semantic + Lexical Hybrid)
Hybrid retrieval pipeline combining:
- **Semantic path**: pgvector embeddings (query understanding)
- **Lexical path**: OpenSearch BM25 (exact term matching)
- **Fusion**: Weighted linear combination (60% semantic, 40% lexical)
- **Result limit**: Top 50 from each engine, merge to top 10 final
### Complete Retrieval Pipeline
```
User Query: "fix kubernetes port 8080 conflict"
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 1: Query Normalization │
├────────────────────────────────────────────────────────┤
│ ├─ Tokenize: ["fix", "kubernetes", ...] │
│ ├─ Extract entities: {port: "8080"} │
│ ├─ Generate embedding (LLM) │
│ └─ Create QueryContext │
└────────────────────────┬───────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 2: Parallel Retrieval (Both Engines) │
├────────────────────────────────────────────────────────┤
│ │
│ ┌─ SEMANTIC (pgvector) │
│ │ ├─ Query embedding <-> vector │
│ │ ├─ Filter: project_id, level, date │
│ │ ├─ ORDER BY cosine_similarity DESC │
│ │ └─ Return: Top 50 with scores │
│ │ │
│ └─ LEXICAL (OpenSearch + JWT) │
│ ├─ multi_match on [content^2, breadcrumb] │
│ ├─ BM25 ranking with fuzziness │
│ ├─ Filter: project_id, level, date │
│ └─ Return: Top 50 with raw scores │
│ │
└────────────────┬──────────────────────────┬────────────┘
│ │
sem_results: [(doc1, 0.92), ...] lex_results: [(doc1, 8.5), ...]
│ │
└──────────┬───────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 3: Score Normalization │
├────────────────────────────────────────────────────────┤
│ ├─ Normalize semantic: [0.0 ... 1.0] │
│ │ (already 0-1 from cosine) │
│ │ │
│ ├─ Normalize lexical: [0.0 ... 1.0] │
│ │ (min-max: (score-min)/(max-min)) │
│ │ │
│ └─ Result: Both in [0.0, 1.0] range │
│ │
└────────────────┬──────────────────────────────────────┘
│
sem_norm: [(doc1, 1.0), ...] lex_norm: [(doc1, 0.98), ...]
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 4: Fusion (Weighted Linear) │
├────────────────────────────────────────────────────────┤
│ ├─ Merge all doc IDs from both results │
│ ├─ For each doc: score = 0.6*sem + 0.4*lex │
│ │ │
│ │ doc1: 0.6*1.0 + 0.4*0.98 = 0.992 │
│ │ doc2: 0.6*0.96 + 0.4*0.0 = 0.576 │
│ │ doc3: 0.6*0.0 + 0.4*0.88 = 0.352 │
│ │ │
│ ├─ Sort descending: [doc1, doc2, doc3] │
│ └─ Take top-10 │
│ │
└────────────────┬──────────────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Stage 5: Score Breakdown (Transparency) │
├────────────────────────────────────────────────────────┤
│ doc1: { │
│ "score": 0.992, │
│ "sem_component": 1.0, │
│ "lex_component": 0.98, │
│ "sem_weight": 0.6, │
│ "lex_weight": 0.4, │
│ "reason": "Exact semantic match + strong lexical" │
│ } │
│ │
└────────────────┬──────────────────────────────────────┘
│
↓
┌────────────────────────────────────────────────────────┐
│ Final Results (Top-10) │
├────────────────────────────────────────────────────────┤
│ 1. port-forward.md (0.992) │
│ - Semantic: 1.0 | Lexical: 0.98 │
│ │
│ 2. troubleshooting.md (0.576) │
│ - Semantic: 0.96 | Lexical: 0.0 │
│ │
│ 3. k8s-basics.md (0.352) │
│ - Semantic: 0.0 | Lexical: 0.88 │
│ │
└────────────────────────────────────────────────────────┘
```
### Query Routing Decision Tree
```
Query Received
│
├─ Token count < 3?
│ ├─ YES → Use LEXICAL_ONLY
│ │ (short queries: "fix port" → better BM25)
│ │
│ └─ NO → Continue...
│
├─ Contains special syntax (#tag, @mention)?
│ ├─ YES → Use LEXICAL_WITH_FILTERS
│ │ (preserve exact matches)
│ │
│ └─ NO → Continue...
│
├─ Can we embed the query?
│ ├─ YES → Use HYBRID
│ │ (both engines)
│ │
│ └─ NO → Use LEXICAL_ONLY
│ (LLM unavailable, fallback)
│
└─ Execute chosen strategy
```
### Index Optimization
**PostgreSQL (pgvector) - Semantic Path:**
- Index: `ivfflat (embedding vector_cosine_ops) WITH (lists=100)`
- Filter: `project_id, level IN ('L0','L1','L2'), created_at > now-1y`
- Retrieve: Top 50 chunks, then merge with lexical
- Lookup: O(log n) pre-filter + O(1) embedding distance
**OpenSearch (BM25) - Lexical Path:**
- Analyzer: `standard` (lowercase, stop words)
- Fields: `content^2` (2x boost) + `breadcrumb` + `source`
- Tokenizer: Standard + n-gram for typo tolerance
- Retrieve: Top 50 results via BM25, then merge with semantic
- Lookup: O(n) inverted index scan + TF-IDF ranking
**Merging Strategy:**
1. Normalize both score ranges to [0.0, 1.0]
2. Weighted sum: `0.6 * semantic + 0.4 * lexical`
3. Sort by final score
4. Return top-10 to user
**Accuracy Metrics (A/B Testing):**
- MRR (Mean Reciprocal Rank): Position of first correct result
- NDCG@10 (Normalized Discounted Cumulative Gain): Quality of top-10 ranking
- Precision@K: Relevant results in top-K
- Recall@K: Coverage of all relevant results in top-K
------
## Edit Flow (GRM Workflow)
Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.
```
┌──────────────────────────────────────────────────────┐
│ User: Clicks "Edit" on document │
│ Example: runbook-deploy.md │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Switch to Edit Mode │
│ ├─ Load document content from Memory Service │
│ ├─ Show markdown editor (CodeMirror) │
│ ├─ Disable Save button (drafts only) │
│ └─ Show "Submit for Review" button │
└────────────┬─────────────────────────────────────────┘
│
├─ User makes edits (e.g., update deploy steps)
│
↓
┌──────────────────────────────────────────────────────┐
│ User: Click "Submit for Review" │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 1: CREATE BRANCH ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: POST /memory/grc/draft │
│ { │
│ "document_path": "vault/runbooks/deploy.md", │
│ "content": "<new markdown content>", │
│ "message": "Update deploy steps", │
│ "user": "[email protected]" │
│ } │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: GRC Handler │
│ ├─ Generate branch name: edit/rock/deploy-<ts> │
│ ├─ Call Forgejo API (create branch) │
│ ├─ Commit changes to branch │
│ └─ Return PR URL + branch name │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo Git Service │
│ ├─ Create branch: edit/rock/deploy-<ts> │
│ ├─ From: main │
│ ├─ Commit: "Update deploy steps" │
│ └─ Trigger CI checks (markdown lint) │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 2: AUTO-CREATE MERGE REQUEST ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Feedback │
│ ✅ "Draft saved - Merge Request created" │
│ └─ Show clickable MR link │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 3: HUMAN REVIEW (in Forgejo) ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Reviewer (e.g., lead engineer) │
│ ├─ Open MR in Forgejo web UI │
│ ├─ Review diff (before/after) │
│ ├─ Comment/suggest edits │
│ ├─ Approve or request changes │
│ └─ Click "Merge to main" │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 4: AUTO-SYNC TO VAULT ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo: Merge Complete │
│ ├─ Branch merged to main │
│ ├─ Trigger webhook: pull_request_merged │
│ └─ Payload: {pr_id, merged_at, branch} │
└────────────┬─────────────────────────────────────────┘
│
│ Webhook trigger
│
↓
┌──────────────────────────────────────────────────────┐
│ ArgoCD Application │
│ ├─ Webhook receiver │
│ ├─ Trigger sync of poimen-memory-app │
│ └─ Pull latest from git (main) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Git-Sync Sidecar Pod (poimen namespace) │
│ ├─ Receive ArgoCD sync signal │
│ ├─ `git pull origin main` in vault/ │
│ ├─ File appears in PVC │
│ └─ Update complete │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: Indexing Job │
│ ├─ Detect vault file change │
│ ├─ Tokenize + embed new content │
│ ├─ Insert into pgvector index │
│ └─ Document now searchable │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Notification │
│ ✅ "Document published!" │
│ ├─ Document now visible to all │
│ ├─ Embeddings indexed │
│ └─ Available in search │
└──────────────────────────────────────────────────────┘
```
---
## Agent Context Flow
Real-time agent execution with memory retrieval tracking.
```
┌──────────────────────────────────────────────────────┐
│ User: Navigate to "Agent Workspace" tab │
│ (Shows live agent execution) │
└────────────┬─────────────────────────────────────────┘
│
│ Establish connection
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: WebSocket /memory/agents/stream │
│ (Fallback: HTTP polling) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Tail agent execution log │
│ ├─ Emit events: │
│ │ - agent_started │
│ │ - memory_retrieved {query, chunks, scores} │
│ │ - tool_invoked {tool_name, args} │
│ │ - tool_result {result} │
│ │ - agent_decision {reasoning} │
│ │ - agent_complete │
│ └─ Stream as JSON events │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Real-time Dashboard │
│ ├─ Timeline of agent actions (bottom-up) │
│ ├─ Memory chunks used (with similarity scores) │
│ ├─ Tool calls + outputs (expandable) │
│ ├─ Decision tree (branching logic) │
│ └─ Knowledge graph overlay (related docs) │
└──────────────────────────────────────────────────────┘
```
---
## System Architecture
Complete deployment topology with all components.
```
┌────────────────────────────────────────────────────────────────────────┐
│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │
└────────────┬───────────────────────────────────────────────────────────┘
│
│ HTTPS
│
↓
┌────────────────────────────────────────────────────────────────────────┐
│ K8s Ingress Controller (nginx-ingress) │
│ ├─ TLS termination (memory.riotpiao.com) │
│ ├─ Route to frontend Service (port 80) │
│ └─ Route to memory Service (port 8080) │
└────┬───────────────────────────────────┬───────────────────────────────┘
│ │
↓ (frontend) ↓ (API)
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Service │ │ Memory Service (8080) │
│ (port 80) │ │ ├─ LoadBalancer type │
└────┬─────────────────────┘ └────┬───────────────────────────────┘
│ │
↓ ↓
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │
│ ├─ React app │ │ ├─ HTTP server (actix-web) │
│ ├─ Vite build │ │ ├─ JWT validation │
│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │
│ └─ API client │ │ ├─ Vault browser │
└──────────────────────────┘ │ ├─ Query (embedding) handler │
│ ├─ Skills handler │
┌──────────────────────────┼─ Projects handler │
│ │ └─ Volume: /data/vault (PVC) │
│ │ │
│ └────┬───────────────────────────────┘
│ │
│ Memory Pod 2 (HA replica) │
│ (identical to Pod 1) │
│ │
├───────────────────────────────┤
│ │
↓ ↓
┌─────────────────────────────────────────────────────────────┐
│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │
│ ├─ Primary: memory-db-0 (PVC: 20Gi) │
│ ├─ Replica: memory-db-1 (PVC: 20Gi) │
│ ├─ Service: memory-db (headless) │
│ ├─ Tables: │
│ │ ├─ chunks (id, text, project_id, embedding, source) │
│ │ ├─ skills (id, name, metadata) │
│ │ ├─ projects (id, name) │
│ │ └─ agent_logs (id, agent_id, action, timestamp) │
│ └─ Extension: pgvector (vector similarity) │
└─────────────────────────────────────────────────────────────┘
│
│ INDEX: embedding <-> vector[]
│
└─ Used by: /memory/query (similarity search)
┌─────────────────────────────────────────────────────────────┐
│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │
│ ├─ Mount path: /data/vault │
│ ├─ Content: │
│ │ ├─ vault/skills/ │
│ │ ├─ vault/runbooks/ │
│ │ ├─ vault/evidence/ │
│ │ └─ .git/ (full git history) │
│ └─ Sync: git-sync sidecar (on file changes) │
└─────────────────────────────────────────────────────────────┘
│
│ ArgoCD monitors + syncs
│
↓
┌─────────────────────────────────────────────────────────────┐
│ Git-Sync Sidecar (runs in Memory Pod) │
│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │
│ ├─ Branch: main │
│ ├─ Sync interval: 30s │
│ ├─ On merge: pulls to /data/vault │
│ └─ Triggers indexing │
└─────────────────────────────────────────────────────────────┘
│
│ Webhooks
│
↓
┌─────────────────────────────────────────────────────────────┐
│ External: Forgejo + ArgoCD │
│ ├─ Forgejo webhook: pr_merged → ArgoCD │
│ ├─ ArgoCD watches: poimen-memory-app (in git) │
│ ├─ Auto-sync enabled (prune + selfHeal) │
│ └─ Revision tracking │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: Authentik (OIDC) │
│ ├─ Issuer: https://authentik.riotpiao.com/.../ │
│ ├─ JWKS: .../jwks/ │
│ ├─ OAuth2 App: poimen-memory │
│ └─ Used by: JWT validation in Memory Service │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: LLM Service (Embeddings) │
│ ├─ Provider: Vertex AI / Hugging Face / etc │
│ ├─ Used by: /memory/query (tokenize + embed) │
│ └─ Cached results (1hr TTL) │
└─────────────────────────────────────────────────────────────┘
```
---
---
## OpenSearch + JWT Authentication
### JWT Flow with OpenSearch
```
Frontend
│ Authorization: Bearer <JWT>
│ (Authentik-signed token)
│
↓
Memory Service
├─ Extract JWT from header
├─ Validate signature (Authentik JWKS)
├─ Verify expiry + issuer + audience
└─ Extract claims (sub, roles, permissions)
│
├─ Can query pgvector (no auth needed)
│
└─ Forward JWT to OpenSearch
│ Authorization: Bearer <same-JWT>
│
↓
OpenSearch
├─ Receive JWT in Authorization header
├─ JWT realm validates signature
├─ Extract roles from JWT claims
├─ Map to internal roles (read_vault, write_vault)
└─ Check index permissions
│
├─ Query allowed → return results
└─ Write denied → 403 Forbidden
```
### Hybrid Search: Semantic + Lexical
**Memory Service executes parallel searches:**
```
POST /memory/query + JWT
│
├─ Path 1 (Semantic): pgvector
│ ├─ LLM embedding
│ ├─ Cosine similarity
│ └─ Score: 0.88 (understanding)
│
├─ Path 2 (Lexical): OpenSearch + JWT
│ ├─ Tokenize query
│ ├─ BM25 ranking
│ └─ Score: 0.96 (exact terms)
│
└─ Rerank (Weighted: 60% semantic + 40% lexical)
└─ Final score: 0.92
Returns: Combined results sorted by hybrid score
```
### OpenSearch JWT Realm Configuration
```yaml
opensearch_security:
authc:
realms:
jwt_realm:
type: jwt
order: 1
# Token extraction
token_name: Authorization
token_extractor: "Bearer " # Strip "Bearer " prefix
# JWKS from Authentik (auto-refresh hourly)
jwks_uri: "https://authentik.riotpiao.com/application/o/poimen-memory/jwks/"
jwks_refresh_interval_ms: 3600000
# Issuer validation
issuer: "https://authentik.riotpiao.com/application/o/poimen-memory/"
# Extract claims
roles_key: "roles" # From JWT claim
subject_key: "sub" # User identifier
```
### Roles Mapping (JWT → OpenSearch)
**JWT claims example:**
```json
{
"iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
"sub": "[email protected]",
"roles": ["read_vault", "write_vault"],
"permissions": ["memory:read", "memory:write"]
}
```
**OpenSearch role definitions:**
```yaml
read_vault:
cluster_permissions: ["cluster:monitor/health"]
index_permissions:
- index_patterns: ["vault-*"]
allowed_actions: ["indices:data/read/search"]
write_vault:
cluster_permissions: ["cluster:monitor/health"]
index_permissions:
- index_patterns: ["vault-*"]
allowed_actions: ["indices:data/write/index", "indices:data/read/search"]
```
### Network Security (K8s NetworkPolicy)
```yaml
# Only Memory Service can access OpenSearch
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: opensearch-access
namespace: poimen
spec:
podSelector:
matchLabels:
app: opensearch
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: poimen-memory
ports:
- protocol: TCP
port: 9200
```
## Pod Infrastructure
Complete pod inventory deployed in `poimen` namespace.
### Production Pods
| Pod Name | Role | Replicas | PVC | Purpose |
|----------|------|----------|-----|---------|
| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, hybrid search |
| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | pgvector semantic search index |
| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |
| **opensearch-0** | OpenSearch Primary | 1 | 30Gi | Lexical (BM25) search, JWT realm |
| **opensearch-1** | OpenSearch Replica | 1 | 30Gi | HA cluster node, JWT validation |
| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |
### Supporting Infrastructure (External)
| Component | Role | Location |
|-----------|------|----------|
| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod |
| **ArgoCD Application** | CD orchestration | argocd namespace |
| **Ingress Controller** | Reverse proxy | ingress-nginx namespace |
| **Longhorn** | Storage provider | Storage layer |
### Total Pod Count: **8 Production Pods**
```
Namespace: poimen
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
├─ memory-db-0 (StatefulSet) ............................... (1)
├─ memory-db-1 (StatefulSet) ............................... (1)
├─ opensearch-0 (StatefulSet) .............................. (1) [NEW]
├─ opensearch-1 (StatefulSet) .............................. (1) [NEW]
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)
Namespace: argocd
└─ argocd-server, argocd-repo-server, etc .................. (5+)
Namespace: ingress-nginx
└─ nginx-ingress-controller ................................ (1)
TOTAL: 14-16 pods (8 core + 6-8 supporting)
```
### Pod Responsibilities
#### Memory Service Pod (×2, HA)
- **Listen**: 0.0.0.0:8080
- **Endpoints**:
- `GET /memory/vault` — Read vault files
- `POST /memory/query` — Hybrid search (semantic + lexical)
- `GET /memory/skills` — List skills
- `POST /memory/grc/draft` — Create branch + MR
- `GET /memory/grc/status` — Check MR status
- `GET /memory/agents/logs` — Stream agent events
- **Auth**: JWT (Authentik)
- **Hybrid Search Logic**:
- Validates JWT (Authentik JWKS)
- Queries pgvector (semantic in parallel)
- Queries OpenSearch with JWT (lexical in parallel)
- Reranks results (weighted: 60% semantic, 40% lexical)
- Returns combined results
- **Connections**:
- PostgreSQL (pgvector semantic search)
- OpenSearch (lexical search with JWT)
- Forgejo API (GRC)
- LLM service (embeddings)
- PVC (vault files)
#### PostgreSQL Pod (×2, Primary + Replica)
- **Listen**: 5432
- **Service**: memory-db (headless for StatefulSet)
- **Storage**: 20Gi per pod (PVC)
- **Replication**: Streaming replication (primary → replica)
- **Extensions**: pgvector
- **Data**:
- chunks table (with vector index for semantic search)
- skills table
- projects table
- agent_logs table
- **Role**: Semantic search engine (embeddings)
#### OpenSearch Pod (×2, Primary + Replica)
- **Listen**: 9200 (HTTP), 9300 (cluster communication)
- **Service**: opensearch (headless for cluster), opensearch-internal (for queries)
- **Storage**: 30Gi per pod (PVC)
- **Cluster**: poimen-memory (2-node minimum)
- **Security**:
- JWT realm enabled (validates Authentik tokens)
- JWKS endpoint: https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
- Role mapping: Extract roles from JWT claims
- Index permissions: read_vault, write_vault roles
- **Indices**:
- vault-* (BM25 text search with TF-IDF scoring)
- **Role**: Lexical search engine (exact terms + TF-IDF)
#### Frontend Pod (×1-2)
- **Listen**: 80
- **Serve**: React SPA static files
- **Endpoints**:
- `/` — App shell
- `/api/*` — Proxy to Memory Service (8080)
- **Auth**: JWT (localStorage)
- **Build**: Vite (production bundle)
#### Git-Sync Sidecar (embedded in Memory Pod)
- **Runs**: As a init container + background process
- **Watch**: Forgejo main branch
- **Sync interval**: 30 seconds
- **Action on merge**: `git pull` → trigger re-index
---
## Traffic Flow Diagram
```
Internet (Users)
│
│ HTTPS
│
↓
┌─────────────────────────┐
│ Ingress Controller │
│ (nginx) │
└──┬────────────┬─────────┘
│ │
│ port 80 │ port 8080
│ │
↓ ↓
┌──────────┐ ┌──────────────────┐
│ Frontend │ │ Memory Service │
│ (React) │ │ (Rust + Actix) │
└────┬─────┘ └──┬───────────┬────┘
│ │ │
│ ┌──┘ └──┐
│ │ │
↓ ↓ ↓
┌───────────────────┐ ┌──────────────┐
│ PostgreSQL │ │ Vault PVC │
│ (pgvector index) │ │ (git files) │
└───────────────────┘ └──────────────┘
```
---
## Deployment Checklist
### Core Services
- [x] Memory Service (2 pods) deployed in `poimen` namespace
- [x] PostgreSQL StatefulSet (2 pods) deployed with pgvector
- [x] OpenSearch StatefulSet (2 pods) deployed with JWT realm
- [x] PVC: poimen-memory-vault (10Gi) attached
- [x] PVC: opensearch data (30Gi per pod) attached
### Security & Auth
- [x] JWT auth enabled (Authentik integration at Memory Service)
- [x] OpenSearch JWT realm configured (validates Authentik tokens)
- [x] K8s NetworkPolicy (only Memory Service → OpenSearch)
- [x] Role mapping (JWT claims → OpenSearch roles)
### Integration
- [x] Git-sync sidecar configured (auto-pull on merge)
- [x] Ingress configured (memory.riotpiao.com)
- [x] Hybrid search endpoints (/memory/query?search_method=hybrid)
### Design Documentation
- [x] Retrieval pipeline architecture (4-stage: normalize → parallel → fusion → ranking)
- [x] Index optimization (pgvector ivfflat + OpenSearch BM25 tuning)
- [x] Score fusion strategy (weighted linear + RRF alternative)
- [x] Query routing decision tree (short queries → lexical, normal → hybrid)
- [x] Accuracy metrics (MRR, NDCG@10, Precision@K, Recall@K)
- [x] Weight tuning strategy (A/B testing framework)
- [ ] **Reference**: See `docs/HYBRID_SEARCH_DESIGN.md` (19KB, comprehensive design)
### Testing & Deployment
- [ ] Test fixture setup (query + expected results dataset)
- [ ] NDCG/MRR baseline measurements (semantic vs lexical)
- [ ] Weight tuning experiments (0.5/0.5, 0.6/0.4, 0.7/0.3, 0.4/0.6)
- [ ] Performance benchmarks (latency: parallel vs serial)
- [ ] Frontend pod deployment
- [ ] End-to-end hybrid search tests
- [ ] Gradual rollout (Phase 1: lexical-only → Phase 2: hybrid 10% → Phase 3: 100%)
- [ ] GRC endpoints tested
- [ ] Agent logging endpoints tested
Reference in New Issue Copy Permalink