Story Crater Bot
1018c04c56
Complete documentation of: - Read flow (vault browser) - Search flow (embeddings + pgvector) - Edit flow (4-step GRM workflow with branch → MR → review → merge → auto-sync) - Agent context flow (real-time execution logs) - System architecture (K8s topology) - Pod infrastructure (5-6 core + 6-8 supporting pods) - Traffic flow diagram - Deployment checklist Total pods documented: - 2× poimen-memory (API server, HA) - 2× memory-db (PostgreSQL primary + replica, 20Gi each) - 1-2× frontend (React SPA, vault UI) - 5+ ArgoCD pods (CD orchestration) - 1× nginx-ingress (reverse proxy)
530 lines
32 KiB
Markdown
530 lines
32 KiB
Markdown
# Memory UI Flow - Complete Workflow
|
||
|
||
## Table of Contents
|
||
1. [Read Flow](#read-flow)
|
||
2. [Search Flow](#search-flow)
|
||
3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow)
|
||
4. [Agent Context Flow](#agent-context-flow)
|
||
5. [System Architecture](#system-architecture)
|
||
6. [Pod Infrastructure](#pod-infrastructure)
|
||
|
||
---
|
||
|
||
## Read Flow
|
||
|
||
Browse vault documents from the web UI.
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ User: memory.riotpiao.com │
|
||
│ (Browser, JWT token in localStorage) │
|
||
└────────────┬────────────────────────────────────────┘
|
||
│
|
||
│ GET /memory/vault?project=poimen
|
||
│ Authorization: Bearer <JWT>
|
||
│
|
||
↓
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ Memory Service Pod │
|
||
│ ├─ Load vault files from PVC │
|
||
│ ├─ Build file tree (directory structure) │
|
||
│ └─ Return JSON response │
|
||
└────────────┬────────────────────────────────────────┘
|
||
│
|
||
│ {files: [{path, title, updated_at}...]}
|
||
│
|
||
↓
|
||
┌─────────────────────────────────────────────────────┐
|
||
│ UI: Render Vault Browser │
|
||
│ ├─ Project selector (dropdown) │
|
||
│ ├─ File tree (collapsible folders) │
|
||
│ ├─ Breadcrumb navigation │
|
||
│ └─ Preview panel (markdown rendering) │
|
||
└─────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Search Flow
|
||
|
||
Semantic search using embeddings and pgvector similarity.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ User: Type query "deployment issues" │
|
||
│ Select filters (project=poimen, level=L1) │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│ POST /memory/query
|
||
│ {
|
||
│ "query": "deployment issues",
|
||
│ "project": "poimen",
|
||
│ "limit": 10
|
||
│ }
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Memory Service Pod │
|
||
│ ├─ Tokenize query │
|
||
│ ├─ Call LLM API for embedding │
|
||
│ └─ pgvector similarity search │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│ Query: similarity vector
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ PostgreSQL Pod + pgvector │
|
||
│ ├─ Index: embedding <-> <dimension> │
|
||
│ ├─ Query: │
|
||
│ │ SELECT chunks, similarity_score │
|
||
│ │ ORDER BY embedding <-> query_vec │
|
||
│ │ LIMIT 10 │
|
||
│ └─ Return ranked results │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│ [{chunk, score, source, level, breadcrumb}...]
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ UI: Render Results │
|
||
│ ├─ Result cards with similarity badges (0.92) │
|
||
│ ├─ Syntax highlighted code blocks │
|
||
│ ├─ Link to source document │
|
||
│ └─ "Related" section (similarity graph) │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Edit Flow (GRM Workflow)
|
||
|
||
Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ User: Clicks "Edit" on document │
|
||
│ Example: runbook-deploy.md │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ UI: Switch to Edit Mode │
|
||
│ ├─ Load document content from Memory Service │
|
||
│ ├─ Show markdown editor (CodeMirror) │
|
||
│ ├─ Disable Save button (drafts only) │
|
||
│ └─ Show "Submit for Review" button │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
├─ User makes edits (e.g., update deploy steps)
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ User: Click "Submit for Review" │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│
|
||
╔═══════╩═══════════════════════════════════════════╗
|
||
║ STEP 1: CREATE BRANCH ║
|
||
╚═══════╤═══════════════════════════════════════════╝
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Frontend: POST /memory/grc/draft │
|
||
│ { │
|
||
│ "document_path": "vault/runbooks/deploy.md", │
|
||
│ "content": "<new markdown content>", │
|
||
│ "message": "Update deploy steps", │
|
||
│ "user": "[email protected]" │
|
||
│ } │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Memory Service Pod: GRC Handler │
|
||
│ ├─ Generate branch name: edit/rock/deploy-<ts> │
|
||
│ ├─ Call Forgejo API (create branch) │
|
||
│ ├─ Commit changes to branch │
|
||
│ └─ Return PR URL + branch name │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Forgejo Git Service │
|
||
│ ├─ Create branch: edit/rock/deploy-<ts> │
|
||
│ ├─ From: main │
|
||
│ ├─ Commit: "Update deploy steps" │
|
||
│ └─ Trigger CI checks (markdown lint) │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
╔═══════╩═══════════════════════════════════════════╗
|
||
║ STEP 2: AUTO-CREATE MERGE REQUEST ║
|
||
╚═══════╤═══════════════════════════════════════════╝
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ UI Feedback │
|
||
│ ✅ "Draft saved - Merge Request created" │
|
||
│ └─ Show clickable MR link │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│
|
||
╔═══════╩═══════════════════════════════════════════╗
|
||
║ STEP 3: HUMAN REVIEW (in Forgejo) ║
|
||
╚═══════╤═══════════════════════════════════════════╝
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Reviewer (e.g., lead engineer) │
|
||
│ ├─ Open MR in Forgejo web UI │
|
||
│ ├─ Review diff (before/after) │
|
||
│ ├─ Comment/suggest edits │
|
||
│ ├─ Approve or request changes │
|
||
│ └─ Click "Merge to main" │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
╔═══════╩═══════════════════════════════════════════╗
|
||
║ STEP 4: AUTO-SYNC TO VAULT ║
|
||
╚═══════╤═══════════════════════════════════════════╝
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Forgejo: Merge Complete │
|
||
│ ├─ Branch merged to main │
|
||
│ ├─ Trigger webhook: pull_request_merged │
|
||
│ └─ Payload: {pr_id, merged_at, branch} │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│ Webhook trigger
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ ArgoCD Application │
|
||
│ ├─ Webhook receiver │
|
||
│ ├─ Trigger sync of poimen-memory-app │
|
||
│ └─ Pull latest from git (main) │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Git-Sync Sidecar Pod (poimen namespace) │
|
||
│ ├─ Receive ArgoCD sync signal │
|
||
│ ├─ `git pull origin main` in vault/ │
|
||
│ ├─ File appears in PVC │
|
||
│ └─ Update complete │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Memory Service Pod: Indexing Job │
|
||
│ ├─ Detect vault file change │
|
||
│ ├─ Tokenize + embed new content │
|
||
│ ├─ Insert into pgvector index │
|
||
│ └─ Document now searchable │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ UI Notification │
|
||
│ ✅ "Document published!" │
|
||
│ ├─ Document now visible to all │
|
||
│ ├─ Embeddings indexed │
|
||
│ └─ Available in search │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Agent Context Flow
|
||
|
||
Real-time agent execution with memory retrieval tracking.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ User: Navigate to "Agent Workspace" tab │
|
||
│ (Shows live agent execution) │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
│ Establish connection
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Frontend: WebSocket /memory/agents/stream │
|
||
│ (Fallback: HTTP polling) │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ Memory Service Pod │
|
||
│ ├─ Tail agent execution log │
|
||
│ ├─ Emit events: │
|
||
│ │ - agent_started │
|
||
│ │ - memory_retrieved {query, chunks, scores} │
|
||
│ │ - tool_invoked {tool_name, args} │
|
||
│ │ - tool_result {result} │
|
||
│ │ - agent_decision {reasoning} │
|
||
│ │ - agent_complete │
|
||
│ └─ Stream as JSON events │
|
||
└────────────┬─────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ UI: Real-time Dashboard │
|
||
│ ├─ Timeline of agent actions (bottom-up) │
|
||
│ ├─ Memory chunks used (with similarity scores) │
|
||
│ ├─ Tool calls + outputs (expandable) │
|
||
│ ├─ Decision tree (branching logic) │
|
||
│ └─ Knowledge graph overlay (related docs) │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## System Architecture
|
||
|
||
Complete deployment topology with all components.
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────────┐
|
||
│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │
|
||
└────────────┬───────────────────────────────────────────────────────────┘
|
||
│
|
||
│ HTTPS
|
||
│
|
||
↓
|
||
┌────────────────────────────────────────────────────────────────────────┐
|
||
│ K8s Ingress Controller (nginx-ingress) │
|
||
│ ├─ TLS termination (memory.riotpiao.com) │
|
||
│ ├─ Route to frontend Service (port 80) │
|
||
│ └─ Route to memory Service (port 8080) │
|
||
└────┬───────────────────────────────────┬───────────────────────────────┘
|
||
│ │
|
||
↓ (frontend) ↓ (API)
|
||
┌──────────────────────────┐ ┌────────────────────────────────────┐
|
||
│ Frontend Service │ │ Memory Service (8080) │
|
||
│ (port 80) │ │ ├─ LoadBalancer type │
|
||
└────┬──────────────────────┘ └────┬───────────────────────────────┘
|
||
│ │
|
||
↓ ↓
|
||
┌──────────────────────────┐ ┌────────────────────────────────────┐
|
||
│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │
|
||
│ ├─ React app │ │ ├─ HTTP server (actix-web) │
|
||
│ ├─ Vite build │ │ ├─ JWT validation │
|
||
│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │
|
||
│ └─ API client │ │ ├─ Vault browser │
|
||
└──────────────────────────┘ │ ├─ Query (embedding) handler │
|
||
│ ├─ Skills handler │
|
||
┌──────────────────────────┼─ Projects handler │
|
||
│ │ └─ Volume: /data/vault (PVC) │
|
||
│ │ │
|
||
│ └────┬───────────────────────────────┘
|
||
│ │
|
||
│ Memory Pod 2 (HA replica) │
|
||
│ (identical to Pod 1) │
|
||
│ │
|
||
├───────────────────────────────┤
|
||
│ │
|
||
↓ ↓
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │
|
||
│ ├─ Primary: memory-db-0 (PVC: 20Gi) │
|
||
│ ├─ Replica: memory-db-1 (PVC: 20Gi) │
|
||
│ ├─ Service: memory-db (headless) │
|
||
│ ├─ Tables: │
|
||
│ │ ├─ chunks (id, text, project_id, embedding, source) │
|
||
│ │ ├─ skills (id, name, metadata) │
|
||
│ │ ├─ projects (id, name) │
|
||
│ │ └─ agent_logs (id, agent_id, action, timestamp) │
|
||
│ └─ Extension: pgvector (vector similarity) │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
│ INDEX: embedding <-> vector[]
|
||
│
|
||
└─ Used by: /memory/query (similarity search)
|
||
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │
|
||
│ ├─ Mount path: /data/vault │
|
||
│ ├─ Content: │
|
||
│ │ ├─ vault/skills/ │
|
||
│ │ ├─ vault/runbooks/ │
|
||
│ │ ├─ vault/evidence/ │
|
||
│ │ └─ .git/ (full git history) │
|
||
│ └─ Sync: git-sync sidecar (on file changes) │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
│ ArgoCD monitors + syncs
|
||
│
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Git-Sync Sidecar (runs in Memory Pod) │
|
||
│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │
|
||
│ ├─ Branch: main │
|
||
│ ├─ Sync interval: 30s │
|
||
│ ├─ On merge: pulls to /data/vault │
|
||
│ └─ Triggers indexing │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
│ Webhooks
|
||
│
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ External: Forgejo + ArgoCD │
|
||
│ ├─ Forgejo webhook: pr_merged → ArgoCD │
|
||
│ ├─ ArgoCD watches: poimen-memory-app (in git) │
|
||
│ ├─ Auto-sync enabled (prune + selfHeal) │
|
||
│ └─ Revision tracking │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ External: Authentik (OIDC) │
|
||
│ ├─ Issuer: https://authentik.riotpiao.com/.../ │
|
||
│ ├─ JWKS: .../jwks/ │
|
||
│ ├─ OAuth2 App: poimen-memory │
|
||
│ └─ Used by: JWT validation in Memory Service │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ External: LLM Service (Embeddings) │
|
||
│ ├─ Provider: Vertex AI / Hugging Face / etc │
|
||
│ ├─ Used by: /memory/query (tokenize + embed) │
|
||
│ └─ Cached results (1hr TTL) │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Pod Infrastructure
|
||
|
||
Complete pod inventory deployed in `poimen` namespace.
|
||
|
||
### Production Pods
|
||
|
||
| Pod Name | Role | Replicas | PVC | Purpose |
|
||
|----------|------|----------|-----|---------|
|
||
| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, Query handler |
|
||
| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | Main database, chunks + metadata |
|
||
| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |
|
||
| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |
|
||
|
||
### Supporting Infrastructure (External)
|
||
|
||
| Component | Role | Location |
|
||
|-----------|------|----------|
|
||
| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod |
|
||
| **ArgoCD Application** | CD orchestration | argocd namespace |
|
||
| **Ingress Controller** | Reverse proxy | ingress-nginx namespace |
|
||
| **Longhorn** | Storage provider | Storage layer |
|
||
|
||
### Total Pod Count: **6 Production Pods**
|
||
|
||
```
|
||
Namespace: poimen
|
||
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
|
||
├─ memory-db-0 (StatefulSet) ............................... (1)
|
||
├─ memory-db-1 (StatefulSet) ............................... (1)
|
||
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)
|
||
|
||
Namespace: argocd
|
||
└─ argocd-server, argocd-repo-server, etc .................. (5+)
|
||
|
||
Namespace: ingress-nginx
|
||
└─ nginx-ingress-controller ................................ (1)
|
||
|
||
TOTAL: 12-14 pods (6 core + 6-8 supporting)
|
||
```
|
||
|
||
### Pod Responsibilities
|
||
|
||
#### Memory Service Pod (×2, HA)
|
||
- **Listen**: 0.0.0.0:8080
|
||
- **Endpoints**:
|
||
- `GET /memory/vault` — Read vault files
|
||
- `POST /memory/query` — Semantic search (embeddings)
|
||
- `GET /memory/skills` — List skills
|
||
- `POST /memory/grc/draft` — Create branch + MR
|
||
- `GET /memory/grc/status` — Check MR status
|
||
- `GET /memory/agents/logs` — Stream agent events
|
||
- **Auth**: JWT (Authentik)
|
||
- **Connections**:
|
||
- PostgreSQL (query + index)
|
||
- Forgejo API (GRC)
|
||
- LLM service (embeddings)
|
||
- PVC (vault files)
|
||
|
||
#### PostgreSQL Pod (×2, Primary + Replica)
|
||
- **Listen**: 5432
|
||
- **Service**: memory-db (headless for StatefulSet)
|
||
- **Storage**: 20Gi per pod (PVC)
|
||
- **Replication**: Streaming replication (primary → replica)
|
||
- **Extensions**: pgvector
|
||
- **Data**:
|
||
- chunks table (with vector index)
|
||
- skills table
|
||
- projects table
|
||
- agent_logs table
|
||
|
||
#### Frontend Pod (×1-2)
|
||
- **Listen**: 80
|
||
- **Serve**: React SPA static files
|
||
- **Endpoints**:
|
||
- `/` — App shell
|
||
- `/api/*` — Proxy to Memory Service (8080)
|
||
- **Auth**: JWT (localStorage)
|
||
- **Build**: Vite (production bundle)
|
||
|
||
#### Git-Sync Sidecar (embedded in Memory Pod)
|
||
- **Runs**: As a init container + background process
|
||
- **Watch**: Forgejo main branch
|
||
- **Sync interval**: 30 seconds
|
||
- **Action on merge**: `git pull` → trigger re-index
|
||
|
||
---
|
||
|
||
## Traffic Flow Diagram
|
||
|
||
```
|
||
Internet (Users)
|
||
│
|
||
│ HTTPS
|
||
│
|
||
↓
|
||
┌─────────────────────────┐
|
||
│ Ingress Controller │
|
||
│ (nginx) │
|
||
└──┬────────────┬─────────┘
|
||
│ │
|
||
│ port 80 │ port 8080
|
||
│ │
|
||
↓ ↓
|
||
┌──────────┐ ┌──────────────────┐
|
||
│ Frontend │ │ Memory Service │
|
||
│ (React) │ │ (Rust + Actix) │
|
||
└────┬─────┘ └──┬───────────┬────┘
|
||
│ │ │
|
||
│ ┌──┘ └──┐
|
||
│ │ │
|
||
↓ ↓ ↓
|
||
┌───────────────────┐ ┌──────────────┐
|
||
│ PostgreSQL │ │ Vault PVC │
|
||
│ (pgvector index) │ │ (git files) │
|
||
└───────────────────┘ └──────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Deployment Checklist
|
||
|
||
- [x] Memory Service (2 pods) deployed in `poimen` namespace
|
||
- [x] PostgreSQL StatefulSet (2 pods) deployed
|
||
- [x] PVC: poimen-memory-vault (10Gi) attached
|
||
- [x] Git-sync sidecar configured (auto-pull on merge)
|
||
- [x] JWT auth enabled (Authentik integration)
|
||
- [x] Ingress configured (memory.riotpiao.com)
|
||
- [ ] Frontend pod deployment (TODO)
|
||
- [ ] GRC endpoints tested (TODO)
|
||
- [ ] Agent logging endpoints tested (TODO)
|
||
|