Story Crater Bot
1018c04c56
Complete documentation of: - Read flow (vault browser) - Search flow (embeddings + pgvector) - Edit flow (4-step GRM workflow with branch → MR → review → merge → auto-sync) - Agent context flow (real-time execution logs) - System architecture (K8s topology) - Pod infrastructure (5-6 core + 6-8 supporting pods) - Traffic flow diagram - Deployment checklist Total pods documented: - 2× poimen-memory (API server, HA) - 2× memory-db (PostgreSQL primary + replica, 20Gi each) - 1-2× frontend (React SPA, vault UI) - 5+ ArgoCD pods (CD orchestration) - 1× nginx-ingress (reverse proxy)
32 KiB
32 KiB
Memory UI Flow - Complete Workflow
Table of Contents
- Read Flow
- Search Flow
- Edit Flow (GRM Workflow)
- Agent Context Flow
- System Architecture
- Pod Infrastructure
Read Flow
Browse vault documents from the web UI.
┌─────────────────────────────────────────────────────┐
│ User: memory.riotpiao.com │
│ (Browser, JWT token in localStorage) │
└────────────┬────────────────────────────────────────┘
│
│ GET /memory/vault?project=poimen
│ Authorization: Bearer <JWT>
│
↓
┌─────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Load vault files from PVC │
│ ├─ Build file tree (directory structure) │
│ └─ Return JSON response │
└────────────┬────────────────────────────────────────┘
│
│ {files: [{path, title, updated_at}...]}
│
↓
┌─────────────────────────────────────────────────────┐
│ UI: Render Vault Browser │
│ ├─ Project selector (dropdown) │
│ ├─ File tree (collapsible folders) │
│ ├─ Breadcrumb navigation │
│ └─ Preview panel (markdown rendering) │
└─────────────────────────────────────────────────────┘
Search Flow
Semantic search using embeddings and pgvector similarity.
┌──────────────────────────────────────────────────────┐
│ User: Type query "deployment issues" │
│ Select filters (project=poimen, level=L1) │
└────────────┬─────────────────────────────────────────┘
│
│ POST /memory/query
│ {
│ "query": "deployment issues",
│ "project": "poimen",
│ "limit": 10
│ }
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Tokenize query │
│ ├─ Call LLM API for embedding │
│ └─ pgvector similarity search │
└────────────┬─────────────────────────────────────────┘
│
│ Query: similarity vector
│
↓
┌──────────────────────────────────────────────────────┐
│ PostgreSQL Pod + pgvector │
│ ├─ Index: embedding <-> <dimension> │
│ ├─ Query: │
│ │ SELECT chunks, similarity_score │
│ │ ORDER BY embedding <-> query_vec │
│ │ LIMIT 10 │
│ └─ Return ranked results │
└────────────┬─────────────────────────────────────────┘
│
│ [{chunk, score, source, level, breadcrumb}...]
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Render Results │
│ ├─ Result cards with similarity badges (0.92) │
│ ├─ Syntax highlighted code blocks │
│ ├─ Link to source document │
│ └─ "Related" section (similarity graph) │
└──────────────────────────────────────────────────────┘
Edit Flow (GRM Workflow)
Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.
┌──────────────────────────────────────────────────────┐
│ User: Clicks "Edit" on document │
│ Example: runbook-deploy.md │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Switch to Edit Mode │
│ ├─ Load document content from Memory Service │
│ ├─ Show markdown editor (CodeMirror) │
│ ├─ Disable Save button (drafts only) │
│ └─ Show "Submit for Review" button │
└────────────┬─────────────────────────────────────────┘
│
├─ User makes edits (e.g., update deploy steps)
│
↓
┌──────────────────────────────────────────────────────┐
│ User: Click "Submit for Review" │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 1: CREATE BRANCH ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: POST /memory/grc/draft │
│ { │
│ "document_path": "vault/runbooks/deploy.md", │
│ "content": "<new markdown content>", │
│ "message": "Update deploy steps", │
│ "user": "[email protected]" │
│ } │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: GRC Handler │
│ ├─ Generate branch name: edit/rock/deploy-<ts> │
│ ├─ Call Forgejo API (create branch) │
│ ├─ Commit changes to branch │
│ └─ Return PR URL + branch name │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo Git Service │
│ ├─ Create branch: edit/rock/deploy-<ts> │
│ ├─ From: main │
│ ├─ Commit: "Update deploy steps" │
│ └─ Trigger CI checks (markdown lint) │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 2: AUTO-CREATE MERGE REQUEST ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Feedback │
│ ✅ "Draft saved - Merge Request created" │
│ └─ Show clickable MR link │
└────────────┬─────────────────────────────────────────┘
│
│
╔═══════╩═══════════════════════════════════════════╗
║ STEP 3: HUMAN REVIEW (in Forgejo) ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Reviewer (e.g., lead engineer) │
│ ├─ Open MR in Forgejo web UI │
│ ├─ Review diff (before/after) │
│ ├─ Comment/suggest edits │
│ ├─ Approve or request changes │
│ └─ Click "Merge to main" │
└────────────┬─────────────────────────────────────────┘
│
↓
╔═══════╩═══════════════════════════════════════════╗
║ STEP 4: AUTO-SYNC TO VAULT ║
╚═══════╤═══════════════════════════════════════════╝
│
↓
┌──────────────────────────────────────────────────────┐
│ Forgejo: Merge Complete │
│ ├─ Branch merged to main │
│ ├─ Trigger webhook: pull_request_merged │
│ └─ Payload: {pr_id, merged_at, branch} │
└────────────┬─────────────────────────────────────────┘
│
│ Webhook trigger
│
↓
┌──────────────────────────────────────────────────────┐
│ ArgoCD Application │
│ ├─ Webhook receiver │
│ ├─ Trigger sync of poimen-memory-app │
│ └─ Pull latest from git (main) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Git-Sync Sidecar Pod (poimen namespace) │
│ ├─ Receive ArgoCD sync signal │
│ ├─ `git pull origin main` in vault/ │
│ ├─ File appears in PVC │
│ └─ Update complete │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: Indexing Job │
│ ├─ Detect vault file change │
│ ├─ Tokenize + embed new content │
│ ├─ Insert into pgvector index │
│ └─ Document now searchable │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI Notification │
│ ✅ "Document published!" │
│ ├─ Document now visible to all │
│ ├─ Embeddings indexed │
│ └─ Available in search │
└──────────────────────────────────────────────────────┘
Agent Context Flow
Real-time agent execution with memory retrieval tracking.
┌──────────────────────────────────────────────────────┐
│ User: Navigate to "Agent Workspace" tab │
│ (Shows live agent execution) │
└────────────┬─────────────────────────────────────────┘
│
│ Establish connection
│
↓
┌──────────────────────────────────────────────────────┐
│ Frontend: WebSocket /memory/agents/stream │
│ (Fallback: HTTP polling) │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod │
│ ├─ Tail agent execution log │
│ ├─ Emit events: │
│ │ - agent_started │
│ │ - memory_retrieved {query, chunks, scores} │
│ │ - tool_invoked {tool_name, args} │
│ │ - tool_result {result} │
│ │ - agent_decision {reasoning} │
│ │ - agent_complete │
│ └─ Stream as JSON events │
└────────────┬─────────────────────────────────────────┘
│
↓
┌──────────────────────────────────────────────────────┐
│ UI: Real-time Dashboard │
│ ├─ Timeline of agent actions (bottom-up) │
│ ├─ Memory chunks used (with similarity scores) │
│ ├─ Tool calls + outputs (expandable) │
│ ├─ Decision tree (branching logic) │
│ └─ Knowledge graph overlay (related docs) │
└──────────────────────────────────────────────────────┘
System Architecture
Complete deployment topology with all components.
┌────────────────────────────────────────────────────────────────────────┐
│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │
└────────────┬───────────────────────────────────────────────────────────┘
│
│ HTTPS
│
↓
┌────────────────────────────────────────────────────────────────────────┐
│ K8s Ingress Controller (nginx-ingress) │
│ ├─ TLS termination (memory.riotpiao.com) │
│ ├─ Route to frontend Service (port 80) │
│ └─ Route to memory Service (port 8080) │
└────┬───────────────────────────────────┬───────────────────────────────┘
│ │
↓ (frontend) ↓ (API)
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Service │ │ Memory Service (8080) │
│ (port 80) │ │ ├─ LoadBalancer type │
└────┬──────────────────────┘ └────┬───────────────────────────────┘
│ │
↓ ↓
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │
│ ├─ React app │ │ ├─ HTTP server (actix-web) │
│ ├─ Vite build │ │ ├─ JWT validation │
│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │
│ └─ API client │ │ ├─ Vault browser │
└──────────────────────────┘ │ ├─ Query (embedding) handler │
│ ├─ Skills handler │
┌──────────────────────────┼─ Projects handler │
│ │ └─ Volume: /data/vault (PVC) │
│ │ │
│ └────┬───────────────────────────────┘
│ │
│ Memory Pod 2 (HA replica) │
│ (identical to Pod 1) │
│ │
├───────────────────────────────┤
│ │
↓ ↓
┌─────────────────────────────────────────────────────────────┐
│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │
│ ├─ Primary: memory-db-0 (PVC: 20Gi) │
│ ├─ Replica: memory-db-1 (PVC: 20Gi) │
│ ├─ Service: memory-db (headless) │
│ ├─ Tables: │
│ │ ├─ chunks (id, text, project_id, embedding, source) │
│ │ ├─ skills (id, name, metadata) │
│ │ ├─ projects (id, name) │
│ │ └─ agent_logs (id, agent_id, action, timestamp) │
│ └─ Extension: pgvector (vector similarity) │
└─────────────────────────────────────────────────────────────┘
│
│ INDEX: embedding <-> vector[]
│
└─ Used by: /memory/query (similarity search)
┌─────────────────────────────────────────────────────────────┐
│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │
│ ├─ Mount path: /data/vault │
│ ├─ Content: │
│ │ ├─ vault/skills/ │
│ │ ├─ vault/runbooks/ │
│ │ ├─ vault/evidence/ │
│ │ └─ .git/ (full git history) │
│ └─ Sync: git-sync sidecar (on file changes) │
└─────────────────────────────────────────────────────────────┘
│
│ ArgoCD monitors + syncs
│
↓
┌─────────────────────────────────────────────────────────────┐
│ Git-Sync Sidecar (runs in Memory Pod) │
│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │
│ ├─ Branch: main │
│ ├─ Sync interval: 30s │
│ ├─ On merge: pulls to /data/vault │
│ └─ Triggers indexing │
└─────────────────────────────────────────────────────────────┘
│
│ Webhooks
│
↓
┌─────────────────────────────────────────────────────────────┐
│ External: Forgejo + ArgoCD │
│ ├─ Forgejo webhook: pr_merged → ArgoCD │
│ ├─ ArgoCD watches: poimen-memory-app (in git) │
│ ├─ Auto-sync enabled (prune + selfHeal) │
│ └─ Revision tracking │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: Authentik (OIDC) │
│ ├─ Issuer: https://authentik.riotpiao.com/.../ │
│ ├─ JWKS: .../jwks/ │
│ ├─ OAuth2 App: poimen-memory │
│ └─ Used by: JWT validation in Memory Service │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ External: LLM Service (Embeddings) │
│ ├─ Provider: Vertex AI / Hugging Face / etc │
│ ├─ Used by: /memory/query (tokenize + embed) │
│ └─ Cached results (1hr TTL) │
└─────────────────────────────────────────────────────────────┘
Pod Infrastructure
Complete pod inventory deployed in poimen namespace.
Production Pods
| Pod Name | Role | Replicas | PVC | Purpose |
|---|---|---|---|---|
| poimen-memory-* | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, Query handler |
| memory-db-0 | PostgreSQL Primary | 1 | 20Gi | Main database, chunks + metadata |
| memory-db-1 | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |
| frontend-* | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |
Supporting Infrastructure (External)
| Component | Role | Location |
|---|---|---|
| Git-Sync Sidecar | Auto-pull vault | Embedded in memory pod |
| ArgoCD Application | CD orchestration | argocd namespace |
| Ingress Controller | Reverse proxy | ingress-nginx namespace |
| Longhorn | Storage provider | Storage layer |
Total Pod Count: 6 Production Pods
Namespace: poimen
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
├─ memory-db-0 (StatefulSet) ............................... (1)
├─ memory-db-1 (StatefulSet) ............................... (1)
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)
Namespace: argocd
└─ argocd-server, argocd-repo-server, etc .................. (5+)
Namespace: ingress-nginx
└─ nginx-ingress-controller ................................ (1)
TOTAL: 12-14 pods (6 core + 6-8 supporting)
Pod Responsibilities
Memory Service Pod (×2, HA)
- Listen: 0.0.0.0:8080
- Endpoints:
GET /memory/vault— Read vault filesPOST /memory/query— Semantic search (embeddings)GET /memory/skills— List skillsPOST /memory/grc/draft— Create branch + MRGET /memory/grc/status— Check MR statusGET /memory/agents/logs— Stream agent events
- Auth: JWT (Authentik)
- Connections:
- PostgreSQL (query + index)
- Forgejo API (GRC)
- LLM service (embeddings)
- PVC (vault files)
PostgreSQL Pod (×2, Primary + Replica)
- Listen: 5432
- Service: memory-db (headless for StatefulSet)
- Storage: 20Gi per pod (PVC)
- Replication: Streaming replication (primary → replica)
- Extensions: pgvector
- Data:
- chunks table (with vector index)
- skills table
- projects table
- agent_logs table
Frontend Pod (×1-2)
- Listen: 80
- Serve: React SPA static files
- Endpoints:
/— App shell/api/*— Proxy to Memory Service (8080)
- Auth: JWT (localStorage)
- Build: Vite (production bundle)
Git-Sync Sidecar (embedded in Memory Pod)
- Runs: As a init container + background process
- Watch: Forgejo main branch
- Sync interval: 30 seconds
- Action on merge:
git pull→ trigger re-index
Traffic Flow Diagram
Internet (Users)
│
│ HTTPS
│
↓
┌─────────────────────────┐
│ Ingress Controller │
│ (nginx) │
└──┬────────────┬─────────┘
│ │
│ port 80 │ port 8080
│ │
↓ ↓
┌──────────┐ ┌──────────────────┐
│ Frontend │ │ Memory Service │
│ (React) │ │ (Rust + Actix) │
└────┬─────┘ └──┬───────────┬────┘
│ │ │
│ ┌──┘ └──┐
│ │ │
↓ ↓ ↓
┌───────────────────┐ ┌──────────────┐
│ PostgreSQL │ │ Vault PVC │
│ (pgvector index) │ │ (git files) │
└───────────────────┘ └──────────────┘
Deployment Checklist
- Memory Service (2 pods) deployed in
poimennamespace - PostgreSQL StatefulSet (2 pods) deployed
- PVC: poimen-memory-vault (10Gi) attached
- Git-sync sidecar configured (auto-pull on merge)
- JWT auth enabled (Authentik integration)
- Ingress configured (memory.riotpiao.com)
- Frontend pod deployment (TODO)
- GRC endpoints tested (TODO)
- Agent logging endpoints tested (TODO)