From 1018c04c56536c2cf1aa225fe94a8270f72a3767 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Thu, 27 Aug 2026 15:47:43 -0700 Subject: [PATCH] docs: Add memory-flow.md - UI flows, architecture, and pod inventory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete documentation of: - Read flow (vault browser) - Search flow (embeddings + pgvector) - Edit flow (4-step GRM workflow with branch → MR → review → merge → auto-sync) - Agent context flow (real-time execution logs) - System architecture (K8s topology) - Pod infrastructure (5-6 core + 6-8 supporting pods) - Traffic flow diagram - Deployment checklist Total pods documented: - 2× poimen-memory (API server, HA) - 2× memory-db (PostgreSQL primary + replica, 20Gi each) - 1-2× frontend (React SPA, vault UI) - 5+ ArgoCD pods (CD orchestration) - 1× nginx-ingress (reverse proxy) --- memory-flow.md | 529 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 529 insertions(+) create mode 100644 memory-flow.md diff --git a/memory-flow.md b/memory-flow.md new file mode 100644 index 0000000..053908a --- /dev/null +++ b/memory-flow.md @@ -0,0 +1,529 @@ +# Memory UI Flow - Complete Workflow + +## Table of Contents +1. [Read Flow](#read-flow) +2. [Search Flow](#search-flow) +3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow) +4. [Agent Context Flow](#agent-context-flow) +5. [System Architecture](#system-architecture) +6. [Pod Infrastructure](#pod-infrastructure) + +--- + +## Read Flow + +Browse vault documents from the web UI. + +``` +┌─────────────────────────────────────────────────────┐ +│ User: memory.riotpiao.com │ +│ (Browser, JWT token in localStorage) │ +└────────────┬────────────────────────────────────────┘ + │ + │ GET /memory/vault?project=poimen + │ Authorization: Bearer + │ + ↓ +┌─────────────────────────────────────────────────────┐ +│ Memory Service Pod │ +│ ├─ Load vault files from PVC │ +│ ├─ Build file tree (directory structure) │ +│ └─ Return JSON response │ +└────────────┬────────────────────────────────────────┘ + │ + │ {files: [{path, title, updated_at}...]} + │ + ↓ +┌─────────────────────────────────────────────────────┐ +│ UI: Render Vault Browser │ +│ ├─ Project selector (dropdown) │ +│ ├─ File tree (collapsible folders) │ +│ ├─ Breadcrumb navigation │ +│ └─ Preview panel (markdown rendering) │ +└─────────────────────────────────────────────────────┘ +``` + +--- + +## Search Flow + +Semantic search using embeddings and pgvector similarity. + +``` +┌──────────────────────────────────────────────────────┐ +│ User: Type query "deployment issues" │ +│ Select filters (project=poimen, level=L1) │ +└────────────┬─────────────────────────────────────────┘ + │ + │ POST /memory/query + │ { + │ "query": "deployment issues", + │ "project": "poimen", + │ "limit": 10 + │ } + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Memory Service Pod │ +│ ├─ Tokenize query │ +│ ├─ Call LLM API for embedding │ +│ └─ pgvector similarity search │ +└────────────┬─────────────────────────────────────────┘ + │ + │ Query: similarity vector + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ PostgreSQL Pod + pgvector │ +│ ├─ Index: embedding <-> │ +│ ├─ Query: │ +│ │ SELECT chunks, similarity_score │ +│ │ ORDER BY embedding <-> query_vec │ +│ │ LIMIT 10 │ +│ └─ Return ranked results │ +└────────────┬─────────────────────────────────────────┘ + │ + │ [{chunk, score, source, level, breadcrumb}...] + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ UI: Render Results │ +│ ├─ Result cards with similarity badges (0.92) │ +│ ├─ Syntax highlighted code blocks │ +│ ├─ Link to source document │ +│ └─ "Related" section (similarity graph) │ +└──────────────────────────────────────────────────────┘ +``` + +--- + +## Edit Flow (GRM Workflow) + +Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault. + +``` +┌──────────────────────────────────────────────────────┐ +│ User: Clicks "Edit" on document │ +│ Example: runbook-deploy.md │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ UI: Switch to Edit Mode │ +│ ├─ Load document content from Memory Service │ +│ ├─ Show markdown editor (CodeMirror) │ +│ ├─ Disable Save button (drafts only) │ +│ └─ Show "Submit for Review" button │ +└────────────┬─────────────────────────────────────────┘ + │ + ├─ User makes edits (e.g., update deploy steps) + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ User: Click "Submit for Review" │ +└────────────┬─────────────────────────────────────────┘ + │ + │ + ╔═══════╩═══════════════════════════════════════════╗ + ║ STEP 1: CREATE BRANCH ║ + ╚═══════╤═══════════════════════════════════════════╝ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Frontend: POST /memory/grc/draft │ +│ { │ +│ "document_path": "vault/runbooks/deploy.md", │ +│ "content": "", │ +│ "message": "Update deploy steps", │ +│ "user": "rock@riotpiao.com" │ +│ } │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Memory Service Pod: GRC Handler │ +│ ├─ Generate branch name: edit/rock/deploy- │ +│ ├─ Call Forgejo API (create branch) │ +│ ├─ Commit changes to branch │ +│ └─ Return PR URL + branch name │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Forgejo Git Service │ +│ ├─ Create branch: edit/rock/deploy- │ +│ ├─ From: main │ +│ ├─ Commit: "Update deploy steps" │ +│ └─ Trigger CI checks (markdown lint) │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ + ╔═══════╩═══════════════════════════════════════════╗ + ║ STEP 2: AUTO-CREATE MERGE REQUEST ║ + ╚═══════╤═══════════════════════════════════════════╝ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ UI Feedback │ +│ ✅ "Draft saved - Merge Request created" │ +│ └─ Show clickable MR link │ +└────────────┬─────────────────────────────────────────┘ + │ + │ + ╔═══════╩═══════════════════════════════════════════╗ + ║ STEP 3: HUMAN REVIEW (in Forgejo) ║ + ╚═══════╤═══════════════════════════════════════════╝ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Reviewer (e.g., lead engineer) │ +│ ├─ Open MR in Forgejo web UI │ +│ ├─ Review diff (before/after) │ +│ ├─ Comment/suggest edits │ +│ ├─ Approve or request changes │ +│ └─ Click "Merge to main" │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ + ╔═══════╩═══════════════════════════════════════════╗ + ║ STEP 4: AUTO-SYNC TO VAULT ║ + ╚═══════╤═══════════════════════════════════════════╝ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Forgejo: Merge Complete │ +│ ├─ Branch merged to main │ +│ ├─ Trigger webhook: pull_request_merged │ +│ └─ Payload: {pr_id, merged_at, branch} │ +└────────────┬─────────────────────────────────────────┘ + │ + │ Webhook trigger + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ ArgoCD Application │ +│ ├─ Webhook receiver │ +│ ├─ Trigger sync of poimen-memory-app │ +│ └─ Pull latest from git (main) │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Git-Sync Sidecar Pod (poimen namespace) │ +│ ├─ Receive ArgoCD sync signal │ +│ ├─ `git pull origin main` in vault/ │ +│ ├─ File appears in PVC │ +│ └─ Update complete │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Memory Service Pod: Indexing Job │ +│ ├─ Detect vault file change │ +│ ├─ Tokenize + embed new content │ +│ ├─ Insert into pgvector index │ +│ └─ Document now searchable │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ UI Notification │ +│ ✅ "Document published!" │ +│ ├─ Document now visible to all │ +│ ├─ Embeddings indexed │ +│ └─ Available in search │ +└──────────────────────────────────────────────────────┘ +``` + +--- + +## Agent Context Flow + +Real-time agent execution with memory retrieval tracking. + +``` +┌──────────────────────────────────────────────────────┐ +│ User: Navigate to "Agent Workspace" tab │ +│ (Shows live agent execution) │ +└────────────┬─────────────────────────────────────────┘ + │ + │ Establish connection + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Frontend: WebSocket /memory/agents/stream │ +│ (Fallback: HTTP polling) │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ Memory Service Pod │ +│ ├─ Tail agent execution log │ +│ ├─ Emit events: │ +│ │ - agent_started │ +│ │ - memory_retrieved {query, chunks, scores} │ +│ │ - tool_invoked {tool_name, args} │ +│ │ - tool_result {result} │ +│ │ - agent_decision {reasoning} │ +│ │ - agent_complete │ +│ └─ Stream as JSON events │ +└────────────┬─────────────────────────────────────────┘ + │ + ↓ +┌──────────────────────────────────────────────────────┐ +│ UI: Real-time Dashboard │ +│ ├─ Timeline of agent actions (bottom-up) │ +│ ├─ Memory chunks used (with similarity scores) │ +│ ├─ Tool calls + outputs (expandable) │ +│ ├─ Decision tree (branching logic) │ +│ └─ Knowledge graph overlay (related docs) │ +└──────────────────────────────────────────────────────┘ +``` + +--- + +## System Architecture + +Complete deployment topology with all components. + +``` +┌────────────────────────────────────────────────────────────────────────┐ +│ EXTERNAL: User → memory.riotpiao.com (DNS A record) │ +└────────────┬───────────────────────────────────────────────────────────┘ + │ + │ HTTPS + │ + ↓ +┌────────────────────────────────────────────────────────────────────────┐ +│ K8s Ingress Controller (nginx-ingress) │ +│ ├─ TLS termination (memory.riotpiao.com) │ +│ ├─ Route to frontend Service (port 80) │ +│ └─ Route to memory Service (port 8080) │ +└────┬───────────────────────────────────┬───────────────────────────────┘ + │ │ + ↓ (frontend) ↓ (API) +┌──────────────────────────┐ ┌────────────────────────────────────┐ +│ Frontend Service │ │ Memory Service (8080) │ +│ (port 80) │ │ ├─ LoadBalancer type │ +└────┬──────────────────────┘ └────┬───────────────────────────────┘ + │ │ + ↓ ↓ +┌──────────────────────────┐ ┌────────────────────────────────────┐ +│ Frontend Pod (React SPA)│ │ Memory Pod 1 (poimen-memory-*) │ +│ ├─ React app │ │ ├─ HTTP server (actix-web) │ +│ ├─ Vite build │ │ ├─ JWT validation │ +│ ├─ Static files │ │ ├─ GRC handler (Forgejo API) │ +│ └─ API client │ │ ├─ Vault browser │ +└──────────────────────────┘ │ ├─ Query (embedding) handler │ + │ ├─ Skills handler │ + ┌──────────────────────────┼─ Projects handler │ + │ │ └─ Volume: /data/vault (PVC) │ + │ │ │ + │ └────┬───────────────────────────────┘ + │ │ + │ Memory Pod 2 (HA replica) │ + │ (identical to Pod 1) │ + │ │ + ├───────────────────────────────┤ + │ │ + ↓ ↓ +┌─────────────────────────────────────────────────────────────┐ +│ PostgreSQL StatefulSet (memory-db-0, memory-db-1) │ +│ ├─ Primary: memory-db-0 (PVC: 20Gi) │ +│ ├─ Replica: memory-db-1 (PVC: 20Gi) │ +│ ├─ Service: memory-db (headless) │ +│ ├─ Tables: │ +│ │ ├─ chunks (id, text, project_id, embedding, source) │ +│ │ ├─ skills (id, name, metadata) │ +│ │ ├─ projects (id, name) │ +│ │ └─ agent_logs (id, agent_id, action, timestamp) │ +│ └─ Extension: pgvector (vector similarity) │ +└─────────────────────────────────────────────────────────────┘ + │ + │ INDEX: embedding <-> vector[] + │ + └─ Used by: /memory/query (similarity search) + +┌─────────────────────────────────────────────────────────────┐ +│ Storage: PVC (poimen-memory-vault, 10Gi, Longhorn) │ +│ ├─ Mount path: /data/vault │ +│ ├─ Content: │ +│ │ ├─ vault/skills/ │ +│ │ ├─ vault/runbooks/ │ +│ │ ├─ vault/evidence/ │ +│ │ └─ .git/ (full git history) │ +│ └─ Sync: git-sync sidecar (on file changes) │ +└─────────────────────────────────────────────────────────────┘ + │ + │ ArgoCD monitors + syncs + │ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Git-Sync Sidecar (runs in Memory Pod) │ +│ ├─ Watches: https://forgejo.riotpiao.com/.../memory.git │ +│ ├─ Branch: main │ +│ ├─ Sync interval: 30s │ +│ ├─ On merge: pulls to /data/vault │ +│ └─ Triggers indexing │ +└─────────────────────────────────────────────────────────────┘ + │ + │ Webhooks + │ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ External: Forgejo + ArgoCD │ +│ ├─ Forgejo webhook: pr_merged → ArgoCD │ +│ ├─ ArgoCD watches: poimen-memory-app (in git) │ +│ ├─ Auto-sync enabled (prune + selfHeal) │ +│ └─ Revision tracking │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ External: Authentik (OIDC) │ +│ ├─ Issuer: https://authentik.riotpiao.com/.../ │ +│ ├─ JWKS: .../jwks/ │ +│ ├─ OAuth2 App: poimen-memory │ +│ └─ Used by: JWT validation in Memory Service │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ External: LLM Service (Embeddings) │ +│ ├─ Provider: Vertex AI / Hugging Face / etc │ +│ ├─ Used by: /memory/query (tokenize + embed) │ +│ └─ Cached results (1hr TTL) │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## Pod Infrastructure + +Complete pod inventory deployed in `poimen` namespace. + +### Production Pods + +| Pod Name | Role | Replicas | PVC | Purpose | +|----------|------|----------|-----|---------| +| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, Query handler | +| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | Main database, chunks + metadata | +| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas | +| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) | + +### Supporting Infrastructure (External) + +| Component | Role | Location | +|-----------|------|----------| +| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod | +| **ArgoCD Application** | CD orchestration | argocd namespace | +| **Ingress Controller** | Reverse proxy | ingress-nginx namespace | +| **Longhorn** | Storage provider | Storage layer | + +### Total Pod Count: **6 Production Pods** + +``` +Namespace: poimen +├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2) +├─ memory-db-0 (StatefulSet) ............................... (1) +├─ memory-db-1 (StatefulSet) ............................... (1) +└─ frontend (Deployment) × 1-2 pods ......................... (1-2) + +Namespace: argocd +└─ argocd-server, argocd-repo-server, etc .................. (5+) + +Namespace: ingress-nginx +└─ nginx-ingress-controller ................................ (1) + +TOTAL: 12-14 pods (6 core + 6-8 supporting) +``` + +### Pod Responsibilities + +#### Memory Service Pod (×2, HA) +- **Listen**: 0.0.0.0:8080 +- **Endpoints**: + - `GET /memory/vault` — Read vault files + - `POST /memory/query` — Semantic search (embeddings) + - `GET /memory/skills` — List skills + - `POST /memory/grc/draft` — Create branch + MR + - `GET /memory/grc/status` — Check MR status + - `GET /memory/agents/logs` — Stream agent events +- **Auth**: JWT (Authentik) +- **Connections**: + - PostgreSQL (query + index) + - Forgejo API (GRC) + - LLM service (embeddings) + - PVC (vault files) + +#### PostgreSQL Pod (×2, Primary + Replica) +- **Listen**: 5432 +- **Service**: memory-db (headless for StatefulSet) +- **Storage**: 20Gi per pod (PVC) +- **Replication**: Streaming replication (primary → replica) +- **Extensions**: pgvector +- **Data**: + - chunks table (with vector index) + - skills table + - projects table + - agent_logs table + +#### Frontend Pod (×1-2) +- **Listen**: 80 +- **Serve**: React SPA static files +- **Endpoints**: + - `/` — App shell + - `/api/*` — Proxy to Memory Service (8080) +- **Auth**: JWT (localStorage) +- **Build**: Vite (production bundle) + +#### Git-Sync Sidecar (embedded in Memory Pod) +- **Runs**: As a init container + background process +- **Watch**: Forgejo main branch +- **Sync interval**: 30 seconds +- **Action on merge**: `git pull` → trigger re-index + +--- + +## Traffic Flow Diagram + +``` +Internet (Users) + │ + │ HTTPS + │ + ↓ +┌─────────────────────────┐ +│ Ingress Controller │ +│ (nginx) │ +└──┬────────────┬─────────┘ + │ │ + │ port 80 │ port 8080 + │ │ + ↓ ↓ +┌──────────┐ ┌──────────────────┐ +│ Frontend │ │ Memory Service │ +│ (React) │ │ (Rust + Actix) │ +└────┬─────┘ └──┬───────────┬────┘ + │ │ │ + │ ┌──┘ └──┐ + │ │ │ + ↓ ↓ ↓ + ┌───────────────────┐ ┌──────────────┐ + │ PostgreSQL │ │ Vault PVC │ + │ (pgvector index) │ │ (git files) │ + └───────────────────┘ └──────────────┘ +``` + +--- + +## Deployment Checklist + +- [x] Memory Service (2 pods) deployed in `poimen` namespace +- [x] PostgreSQL StatefulSet (2 pods) deployed +- [x] PVC: poimen-memory-vault (10Gi) attached +- [x] Git-sync sidecar configured (auto-pull on merge) +- [x] JWT auth enabled (Authentik integration) +- [x] Ingress configured (memory.riotpiao.com) +- [ ] Frontend pod deployment (TODO) +- [ ] GRC endpoints tested (TODO) +- [ ] Agent logging endpoints tested (TODO) +