poimen-memory/memory-flow.md

# Memory UI Flow - Complete Workflow

## Table of Contents
1. [Read Flow](#read-flow)
2. [Search Flow (Semantic + Lexical Hybrid)](#search-flow-semantic--lexical-hybrid)
3. [Edit Flow (GRM Workflow)](#edit-flow-grm-workflow)
4. [Agent Context Flow](#agent-context-flow)
5. [System Architecture](#system-architecture)
6. [OpenSearch + JWT Authentication](#opensearch--jwt-authentication)
7. [M3.7.7 → M3.7.8: Failure Diagnosis Pipeline](#m377--m378-failure-diagnosis-pipeline)
8. [M3.8: Context Optimizer](#m38-context-optimizer)
9. [Pod Infrastructure](#pod-infrastructure)
---

## Read Flow

Browse vault documents from the web UI.

```
┌─────────────────────────────────────────────────────┐
│ User: memory.riotpiao.com                           │
│ (Browser, JWT token in localStorage)                │
└────────────┬────────────────────────────────────────┘
             │
             │ GET /memory/vault?project=poimen
             │ Authorization: Bearer <JWT>
             │
             ↓
┌─────────────────────────────────────────────────────┐
│ Memory Service Pod                                  │
│ ├─ Load vault files from PVC                        │
│ ├─ Build file tree (directory structure)            │
│ └─ Return JSON response                             │
└────────────┬────────────────────────────────────────┘
             │
             │ {files: [{path, title, updated_at}...]}
             │
             ↓
┌─────────────────────────────────────────────────────┐
│ UI: Render Vault Browser                            │
│ ├─ Project selector (dropdown)                      │
│ ├─ File tree (collapsible folders)                  │
│ ├─ Breadcrumb navigation                            │
│ └─ Preview panel (markdown rendering)               │
└─────────────────────────────────────────────────────┘
```

---

## Search Flow (Semantic + Lexical Hybrid)

Hybrid retrieval pipeline combining:
- **Semantic path**: pgvector embeddings (query understanding)
- **Lexical path**: OpenSearch BM25 (exact term matching)  
- **Fusion**: Weighted linear combination (60% semantic, 40% lexical)
- **Result limit**: Top 50 from each engine, merge to top 10 final

### Complete Retrieval Pipeline

```
User Query: "fix kubernetes port 8080 conflict"
   │
   ↓
┌────────────────────────────────────────────────────────┐
│ Stage 1: Query Normalization                           │
├────────────────────────────────────────────────────────┤
│ ├─ Tokenize: ["fix", "kubernetes", ...]               │
│ ├─ Extract entities: {port: "8080"}                    │
│ ├─ Generate embedding (LLM)                            │
│ └─ Create QueryContext                                │
└────────────────────────┬───────────────────────────────┘
                         │
                         ↓
┌────────────────────────────────────────────────────────┐
│ Stage 2: Parallel Retrieval (Both Engines)             │
├────────────────────────────────────────────────────────┤
│                                                        │
│  ┌─ SEMANTIC (pgvector)                                │
│  │  ├─ Query embedding <-> vector                      │
│  │  ├─ Filter: project_id, level, date                 │
│  │  ├─ ORDER BY cosine_similarity DESC                 │
│  │  └─ Return: Top 50 with scores                      │
│  │                                                     │
│  └─ LEXICAL (OpenSearch + JWT)                         │
│     ├─ multi_match on [content^2, breadcrumb]          │
│     ├─ BM25 ranking with fuzziness                     │
│     ├─ Filter: project_id, level, date                 │
│     └─ Return: Top 50 with raw scores                  │
│                                                        │
└────────────────┬──────────────────────────┬────────────┘
                 │                          │
   sem_results: [(doc1, 0.92), ...]         lex_results: [(doc1, 8.5), ...]
                 │                          │
                 └──────────┬───────────────┘
                            │
                            ↓
┌────────────────────────────────────────────────────────┐
│ Stage 3: Score Normalization                           │
├────────────────────────────────────────────────────────┤
│ ├─ Normalize semantic: [0.0 ... 1.0]                   │
│ │  (already 0-1 from cosine)                           │
│ │                                                      │
│ ├─ Normalize lexical: [0.0 ... 1.0]                    │
│ │  (min-max: (score-min)/(max-min))                    │
│ │                                                      │
│ └─ Result: Both in [0.0, 1.0] range                    │
│                                                        │
└────────────────┬──────────────────────────────────────┘
                 │
   sem_norm: [(doc1, 1.0), ...]  lex_norm: [(doc1, 0.98), ...]
                 │
                 ↓
┌────────────────────────────────────────────────────────┐
│ Stage 4: Fusion (Weighted Linear)                      │
├────────────────────────────────────────────────────────┤
│ ├─ Merge all doc IDs from both results                 │
│ ├─ For each doc: score = 0.6*sem + 0.4*lex             │
│ │                                                      │
│ │  doc1: 0.6*1.0 + 0.4*0.98 = 0.992                    │
│ │  doc2: 0.6*0.96 + 0.4*0.0 = 0.576                    │
│ │  doc3: 0.6*0.0 + 0.4*0.88 = 0.352                    │
│ │                                                      │
│ ├─ Sort descending: [doc1, doc2, doc3]                 │
│ └─ Take top-10                                         │
│                                                        │
└────────────────┬──────────────────────────────────────┘
                 │
                 ↓
┌────────────────────────────────────────────────────────┐
│ Stage 5: Score Breakdown (Transparency)                │
├────────────────────────────────────────────────────────┤
│ doc1: {                                                │
│   "score": 0.992,                                      │
│   "sem_component": 1.0,                                │
│   "lex_component": 0.98,                               │
│   "sem_weight": 0.6,                                   │
│   "lex_weight": 0.4,                                   │
│   "reason": "Exact semantic match + strong lexical"    │
│ }                                                      │
│                                                        │
└────────────────┬──────────────────────────────────────┘
                 │
                 ↓
┌────────────────────────────────────────────────────────┐
│ Final Results (Top-10)                                 │
├────────────────────────────────────────────────────────┤
│ 1. port-forward.md (0.992)                             │
│    - Semantic: 1.0 | Lexical: 0.98                     │
│                                                        │
│ 2. troubleshooting.md (0.576)                          │
│    - Semantic: 0.96 | Lexical: 0.0                     │
│                                                        │
│ 3. k8s-basics.md (0.352)                               │
│    - Semantic: 0.0 | Lexical: 0.88                     │
│                                                        │
└────────────────────────────────────────────────────────┘
```

### Query Routing Decision Tree

```
Query Received
   │
   ├─ Token count < 3?
   │  ├─ YES → Use LEXICAL_ONLY
   │  │        (short queries: "fix port" → better BM25)
   │  │
   │  └─ NO → Continue...
   │
   ├─ Contains special syntax (#tag, @mention)?
   │  ├─ YES → Use LEXICAL_WITH_FILTERS
   │  │        (preserve exact matches)
   │  │
   │  └─ NO → Continue...
   │
   ├─ Can we embed the query?
   │  ├─ YES → Use HYBRID
   │  │        (both engines)
   │  │
   │  └─ NO → Use LEXICAL_ONLY
   │           (LLM unavailable, fallback)
   │
   └─ Execute chosen strategy
```

### Index Optimization

**PostgreSQL (pgvector) - Semantic Path:**
- Index: `ivfflat (embedding vector_cosine_ops) WITH (lists=100)`
- Filter: `project_id, level IN ('L0','L1','L2'), created_at > now-1y`
- Retrieve: Top 50 chunks, then merge with lexical
- Lookup: O(log n) pre-filter + O(1) embedding distance

**OpenSearch (BM25) - Lexical Path:**
- Analyzer: `standard` (lowercase, stop words)
- Fields: `content^2` (2x boost) + `breadcrumb` + `source`
- Tokenizer: Standard + n-gram for typo tolerance
- Retrieve: Top 50 results via BM25, then merge with semantic
- Lookup: O(n) inverted index scan + TF-IDF ranking

**Merging Strategy:**
1. Normalize both score ranges to [0.0, 1.0]
2. Weighted sum: `0.6 * semantic + 0.4 * lexical`
3. Sort by final score
4. Return top-10 to user

**Accuracy Metrics (A/B Testing):**
- MRR (Mean Reciprocal Rank): Position of first correct result
- NDCG@10 (Normalized Discounted Cumulative Gain): Quality of top-10 ranking
- Precision@K: Relevant results in top-K
- Recall@K: Coverage of all relevant results in top-K

------

## Edit Flow (GRM Workflow)

Full Git Review Merge workflow: create branch → MR → human approval → auto-sync vault.

```
┌──────────────────────────────────────────────────────┐
│ User: Clicks "Edit" on document                      │
│ Example: runbook-deploy.md                           │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ UI: Switch to Edit Mode                              │
│ ├─ Load document content from Memory Service         │
│ ├─ Show markdown editor (CodeMirror)                 │
│ ├─ Disable Save button (drafts only)                 │
│ └─ Show "Submit for Review" button                   │
└────────────┬─────────────────────────────────────────┘
             │
             ├─ User makes edits (e.g., update deploy steps)
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ User: Click "Submit for Review"                      │
└────────────┬─────────────────────────────────────────┘
             │
             │
     ╔═══════╩═══════════════════════════════════════════╗
     ║  STEP 1: CREATE BRANCH                            ║
     ╚═══════╤═══════════════════════════════════════════╝
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Frontend: POST /memory/grc/draft                      │
│ {                                                     │
│   "document_path": "vault/runbooks/deploy.md",       │
│   "content": "<new markdown content>",               │
│   "message": "Update deploy steps",                  │
│   "user": "[email protected]"                        │
│ }                                                     │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: GRC Handler                       │
│ ├─ Generate branch name: edit/rock/deploy-<ts>      │
│ ├─ Call Forgejo API (create branch)                  │
│ ├─ Commit changes to branch                          │
│ └─ Return PR URL + branch name                       │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Forgejo Git Service                                  │
│ ├─ Create branch: edit/rock/deploy-<ts>             │
│ ├─ From: main                                        │
│ ├─ Commit: "Update deploy steps"                     │
│ └─ Trigger CI checks (markdown lint)                 │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
     ╔═══════╩═══════════════════════════════════════════╗
     ║  STEP 2: AUTO-CREATE MERGE REQUEST                ║
     ╚═══════╤═══════════════════════════════════════════╝
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ UI Feedback                                           │
│ ✅ "Draft saved - Merge Request created"             │
│ └─ Show clickable MR link                            │
└────────────┬─────────────────────────────────────────┘
             │
             │
     ╔═══════╩═══════════════════════════════════════════╗
     ║  STEP 3: HUMAN REVIEW (in Forgejo)               ║
     ╚═══════╤═══════════════════════════════════════════╝
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Reviewer (e.g., lead engineer)                       │
│ ├─ Open MR in Forgejo web UI                         │
│ ├─ Review diff (before/after)                        │
│ ├─ Comment/suggest edits                             │
│ ├─ Approve or request changes                        │
│ └─ Click "Merge to main"                             │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
     ╔═══════╩═══════════════════════════════════════════╗
     ║  STEP 4: AUTO-SYNC TO VAULT                       ║
     ╚═══════╤═══════════════════════════════════════════╝
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Forgejo: Merge Complete                              │
│ ├─ Branch merged to main                             │
│ ├─ Trigger webhook: pull_request_merged              │
│ └─ Payload: {pr_id, merged_at, branch}              │
└────────────┬─────────────────────────────────────────┘
             │
             │ Webhook trigger
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ ArgoCD Application                                   │
│ ├─ Webhook receiver                                  │
│ ├─ Trigger sync of poimen-memory-app                 │
│ └─ Pull latest from git (main)                       │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Git-Sync Sidecar Pod (poimen namespace)              │
│ ├─ Receive ArgoCD sync signal                        │
│ ├─ `git pull origin main` in vault/                  │
│ ├─ File appears in PVC                               │
│ └─ Update complete                                   │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod: Indexing Job                      │
│ ├─ Detect vault file change                          │
│ ├─ Tokenize + embed new content                      │
│ ├─ Insert into pgvector index                        │
│ └─ Document now searchable                           │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ UI Notification                                      │
│ ✅ "Document published!"                             │
│ ├─ Document now visible to all                       │
│ ├─ Embeddings indexed                                │
│ └─ Available in search                               │
└──────────────────────────────────────────────────────┘
```

---

## Agent Context Flow

Real-time agent execution with memory retrieval tracking.

```
┌──────────────────────────────────────────────────────┐
│ User: Navigate to "Agent Workspace" tab              │
│ (Shows live agent execution)                         │
└────────────┬─────────────────────────────────────────┘
             │
             │ Establish connection
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Frontend: WebSocket /memory/agents/stream            │
│ (Fallback: HTTP polling)                             │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ Memory Service Pod                                   │
│ ├─ Tail agent execution log                          │
│ ├─ Emit events:                                      │
│ │  - agent_started                                   │
│ │  - memory_retrieved {query, chunks, scores}        │
│ │  - tool_invoked {tool_name, args}                  │
│ │  - tool_result {result}                            │
│ │  - agent_decision {reasoning}                      │
│ │  - agent_complete                                  │
│ └─ Stream as JSON events                             │
└────────────┬─────────────────────────────────────────┘
             │
             ↓
┌──────────────────────────────────────────────────────┐
│ UI: Real-time Dashboard                              │
│ ├─ Timeline of agent actions (bottom-up)             │
│ ├─ Memory chunks used (with similarity scores)       │
│ ├─ Tool calls + outputs (expandable)                 │
│ ├─ Decision tree (branching logic)                   │
│ └─ Knowledge graph overlay (related docs)            │
└──────────────────────────────────────────────────────┘
```

---

## System Architecture

Complete deployment topology with all components.

```
┌────────────────────────────────────────────────────────────────────────┐
│  EXTERNAL: User → memory.riotpiao.com (DNS A record)                  │
└────────────┬───────────────────────────────────────────────────────────┘
             │
             │ HTTPS
             │
             ↓
┌────────────────────────────────────────────────────────────────────────┐
│  K8s Ingress Controller (nginx-ingress)                                │
│  ├─ TLS termination (memory.riotpiao.com)                              │
│  ├─ Route to frontend Service (port 80)                                │
│  └─ Route to memory Service (port 8080)                                │
└────┬───────────────────────────────────┬───────────────────────────────┘
     │                                   │
     ↓ (frontend)                        ↓ (API)
┌──────────────────────────┐   ┌────────────────────────────────────┐
│  Frontend Service        │   │  Memory Service (8080)             │
│  (port 80)               │   │  ├─ LoadBalancer type              │
└────┬─────────────────────┘   └────┬───────────────────────────────┘
     │                               │
     ↓                               ↓
┌──────────────────────────┐    ┌────────────────────────────────────┐
│  Frontend Pod (React SPA)│    │  Memory Pod 1 (poimen-memory-*)    │
│  ├─ React app            │    │  ├─ HTTP server (actix-web)        │
│  ├─ Vite build           │    │  ├─ JWT validation                 │
│  ├─ Static files         │    │  ├─ GRC handler (Forgejo API)      │
│  └─ API client           │    │  ├─ Vault browser                  │
└──────────────────────────┘    │  ├─ Query (embedding) handler      │
                                │  ├─ Skills handler                 │
     ┌──────────────────────────┼─ Projects handler                  │
     │                          │  └─ Volume: /data/vault (PVC)      │
     │                          │                                    │
     │                          └────┬───────────────────────────────┘
     │                               │
     │  Memory Pod 2 (HA replica)    │
     │  (identical to Pod 1)         │
     │                               │
     ├───────────────────────────────┤
     │                               │
     ↓                               ↓
┌─────────────────────────────────────────────────────────────┐
│  PostgreSQL StatefulSet (memory-db-0, memory-db-1)          │
│  ├─ Primary: memory-db-0 (PVC: 20Gi)                        │
│  ├─ Replica: memory-db-1 (PVC: 20Gi)                        │
│  ├─ Service: memory-db (headless)                           │
│  ├─ Tables:                                                 │
│  │  ├─ chunks (id, text, project_id, embedding, source)     │
│  │  ├─ skills (id, name, metadata)                          │
│  │  ├─ projects (id, name)                                  │
│  │  └─ agent_logs (id, agent_id, action, timestamp)         │
│  └─ Extension: pgvector (vector similarity)                 │
└─────────────────────────────────────────────────────────────┘
     │
     │ INDEX: embedding <-> vector[]
     │
     └─ Used by: /memory/query (similarity search)

┌─────────────────────────────────────────────────────────────┐
│  Storage: PVC (poimen-memory-vault, 10Gi, Longhorn)         │
│  ├─ Mount path: /data/vault                                 │
│  ├─ Content:                                                │
│  │  ├─ vault/skills/                                        │
│  │  ├─ vault/runbooks/                                      │
│  │  ├─ vault/evidence/                                      │
│  │  └─ .git/ (full git history)                             │
│  └─ Sync: git-sync sidecar (on file changes)                │
└─────────────────────────────────────────────────────────────┘
     │
     │ ArgoCD monitors + syncs
     │
     ↓
┌─────────────────────────────────────────────────────────────┐
│  Git-Sync Sidecar (runs in Memory Pod)                      │
│  ├─ Watches: https://forgejo.riotpiao.com/.../memory.git    │
│  ├─ Branch: main                                            │
│  ├─ Sync interval: 30s                                      │
│  ├─ On merge: pulls to /data/vault                          │
│  └─ Triggers indexing                                       │
└─────────────────────────────────────────────────────────────┘
     │
     │ Webhooks
     │
     ↓
┌─────────────────────────────────────────────────────────────┐
│  External: Forgejo + ArgoCD                                 │
│  ├─ Forgejo webhook: pr_merged → ArgoCD                     │
│  ├─ ArgoCD watches: poimen-memory-app (in git)              │
│  ├─ Auto-sync enabled (prune + selfHeal)                    │
│  └─ Revision tracking                                       │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│  External: Authentik (OIDC)                                 │
│  ├─ Issuer: https://authentik.riotpiao.com/.../             │
│  ├─ JWKS: .../jwks/                                         │
│  ├─ OAuth2 App: poimen-memory                               │
│  └─ Used by: JWT validation in Memory Service               │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│  External: LLM Service (Embeddings)                         │
│  ├─ Provider: Vertex AI / Hugging Face / etc                │
│  ├─ Used by: /memory/query (tokenize + embed)               │
│  └─ Cached results (1hr TTL)                                │
└─────────────────────────────────────────────────────────────┘
```

---

---

## OpenSearch + JWT Authentication

### JWT Flow with OpenSearch

```
Frontend
  │ Authorization: Bearer <JWT>
  │ (Authentik-signed token)
  │
  ↓
Memory Service
  ├─ Extract JWT from header
  ├─ Validate signature (Authentik JWKS)
  ├─ Verify expiry + issuer + audience
  └─ Extract claims (sub, roles, permissions)
    │
    ├─ Can query pgvector (no auth needed)
    │
    └─ Forward JWT to OpenSearch
       │ Authorization: Bearer <same-JWT>
       │
       ↓
    OpenSearch
    ├─ Receive JWT in Authorization header
    ├─ JWT realm validates signature
    ├─ Extract roles from JWT claims
    ├─ Map to internal roles (read_vault, write_vault)
    └─ Check index permissions
      │
      ├─ Query allowed → return results
      └─ Write denied → 403 Forbidden
```

### Hybrid Search: Semantic + Lexical

**Memory Service executes parallel searches:**

```
POST /memory/query + JWT
  │
  ├─ Path 1 (Semantic): pgvector
  │  ├─ LLM embedding
  │  ├─ Cosine similarity
  │  └─ Score: 0.88 (understanding)
  │
  ├─ Path 2 (Lexical): OpenSearch + JWT
  │  ├─ Tokenize query
  │  ├─ BM25 ranking
  │  └─ Score: 0.96 (exact terms)
  │
  └─ Rerank (Weighted: 60% semantic + 40% lexical)
     └─ Final score: 0.92

Returns: Combined results sorted by hybrid score
```

### OpenSearch JWT Realm Configuration

```yaml
opensearch_security:
  authc:
    realms:
      jwt_realm:
        type: jwt
        order: 1
        
        # Token extraction
        token_name: Authorization
        token_extractor: "Bearer "  # Strip "Bearer " prefix
        
        # JWKS from Authentik (auto-refresh hourly)
        jwks_uri: "https://authentik.riotpiao.com/application/o/poimen-memory/jwks/"
        jwks_refresh_interval_ms: 3600000
        
        # Issuer validation
        issuer: "https://authentik.riotpiao.com/application/o/poimen-memory/"
        
        # Extract claims
        roles_key: "roles"       # From JWT claim
        subject_key: "sub"       # User identifier
```

### Roles Mapping (JWT → OpenSearch)

**JWT claims example:**
```json
{
  "iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
  "sub": "[email protected]",
  "roles": ["read_vault", "write_vault"],
  "permissions": ["memory:read", "memory:write"]
}
```

**OpenSearch role definitions:**
```yaml
read_vault:
  cluster_permissions: ["cluster:monitor/health"]
  index_permissions:
    - index_patterns: ["vault-*"]
      allowed_actions: ["indices:data/read/search"]

write_vault:
  cluster_permissions: ["cluster:monitor/health"]
  index_permissions:
    - index_patterns: ["vault-*"]
      allowed_actions: ["indices:data/write/index", "indices:data/read/search"]
```

### Network Security (K8s NetworkPolicy)

```yaml
# Only Memory Service can access OpenSearch
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: opensearch-access
  namespace: poimen
spec:
  podSelector:
    matchLabels:
      app: opensearch
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: poimen-memory
      ports:
        - protocol: TCP
          port: 9200
```


## M3.7.7 → M3.7.8: Failure Diagnosis Pipeline

### M3.7.7 Complete: Signature Extraction (✅ 18 tests passing)

**Purpose:** Reduce failure logs to canonical signatures that are byte-identical across runs.

**CLI Command: `mem sig --tool=<TOOL> --file=<LOG>`**

Extract and explain a failure signature from any log file:

```bash
# Extract signature from file
mem sig --tool=npm --file=ci-logs/npm-install-failed.txt

# Or from stdin
cat failure.log | mem sig --tool=cargo

# Output:
# {
#   "tool": "npm",
#   "raw": "npm ERR! code ERESOLVE unable to resolve dependency tree",
#   "normalised": "npm error eresolve dependency tree",
#   "sig_sha": "abc123def456789abcdef456789abc123def456",
#   "rule": "npm_error_line",
#   "confidence": 0.95,
#   "matched_line": 42,
#   "context": [
#     "npm ERR! npm ERR! code ERESOLVE",
#     "npm ERR! [... 100+ lines of consequence ...]",
#     "npm ERR! npm ERR! Could not resolve dependency:",
#   ]
# }
```

**Example Usage in CI/CD:**

```bash
# GitHub Actions: capture failure and explain
if [ $? -ne 0 ]; then
  echo "=== FAILURE SIGNATURE ==="
  mem sig --tool=npm --file=$LOG_FILE >> $GITHUB_STEP_SUMMARY
  exit 1
fi

# Kubernetes: explain pod logs
kubectl logs <pod> | mem sig --tool=kubectl

# Local development: diagnose build errors
cargo build 2>&1 | mem sig --tool=cargo
```

**Key Features:**
- ✅ **Deterministic:** Same failure always produces identical sig_sha
- ✅ **Tool-aware:** npm ERR vs cargo error vs kubectl message handling
- ✅ **Noise-resistant:** Strips timestamps, paths, SHAs, addresses
- ✅ **Root-cause detection:** Picks root error, skips consequence lines
- ✅ **Fast:** <50ms on 50KB logs (rule-based, no LLM)

```
Failure Log (50KB, noisy):
  2026-08-21T10:02:11.482Z
  /home/runner/work/Poimen/memory/...
  npm ERR! code ERESOLVE unable to resolve dependency tree
  npm ERR! [... 100 lines of consequence errors ...]
              │
              ↓ [M3.7.7: extract() + normalise()]
              │
Signature:
  tool: "npm"
  raw: "npm ERR! code ERESOLVE unable to resolve dependency tree"
  normalised: "npm error eresolve dependency tree"
  sig_sha: "abc123def456..." ← deterministic hash
  rule: "npm_error_line"
```

**Key invariants (all 18 tests verify):**
- ✅ Same failure from 2 different runs → identical sig_sha
- ✅ Different failures → different sig_sha
- ✅ Removes timestamps, paths, SHAs, line:col, durations, addresses
- ✅ Picks first (root) error, not consequence lines
- ✅ Unknown tools fallback gracefully
- ✅ Tool part of signature identity (npm vs cargo errors differ)

---

### M3.7.8 In Progress: Symptom Projection

**Purpose:** Transform user queries into normalized symptom vectors that can match extracted signatures.

**The Problem:**
```
User reports:  "npm can't find tslib module"
Canonical sig: "npm ERR! code ERESOLVE unable to resolve dependency tree"

These don't hash the same way, so tier 1 (exact signature match) fails.
Symptom projection bridges this gap.
```

**Design (3-stage normalization pipeline):**

#### Stage 1: Extract Symptom Keywords
```rust
Input query: "npm error: unable to resolve dependency tree"
              │
              ├─ Tool detection: "npm" (from query or context)
              ├─ Error pattern extraction: ["unable", "resolve", "dependency"]
              ├─ Stop word removal: [unable, to, the, of, ...]
              └─ Normalize abbreviations: ERESOLVE → "resolve error"
                                         ERR → "error"
                                         RC → "return code"
                                         OOM → "out of memory"
                                         EACCES → "permission denied"
                                         ENOENT → "not found"
              │
              ↓
Keywords: ["npm", "error", "resolve", "dependency", "tree"]
```

#### Stage 2: Normalize to Canonical Form
```rust
Keywords: ["npm", "error", "resolve", "dependency", "tree"]
          │
          ├─ Remove stop words (a, the, is, can, may, etc.)
          ├─ Lowercase & stemming (resolved → resolve)
          ├─ Expand shorthands:
          │   - ERESOLVE → "resolve error"
          │   - ERR → "error"
          │   - EOF → "end of file"
          │   - EACCES → "permission denied"
          │
          └─ Tool mapping (npm, cargo, kubectl, etc.)
             (different tools handle same error differently)
              │
              ↓
Normalized: "npm error resolve dependency tree"
```

#### Stage 3: Generate Symptom Hash
```rust
Normalized: "npm error resolve dependency tree"
            │
            ├─ Sort keywords alphabetically (for consistency)
            │  → "dependency error npm resolve tree"
            │
            ├─ Join with spaces
            ├─ SHA256 hash (same algorithm as M3.7.7)
            │  sym_sha = SHA256("npm" + "\n" + normalized)
            │
            └─ Store: SymptomVector { tool, normalised, sym_sha }
                      │
                      ├─ sym_sha: "xyz789abc..." ← for lookup
                      ├─ normalised: "dependency error npm resolve tree"
                      └─ keywords: ["dependency", "error", "npm", "resolve", "tree"]
```

**Key insight:** If extracted signature and user query normalize to the same `sym_sha`, they match (tier 1 exact hit). If not, fall through to tier 2 (hybrid search).

---

### M3.7.4: Three-Tier Context Endpoint Integration

```
GET /memory/context?query=npm+ERR+ERESOLVE&tool=npm
   │
   │ Normalize with M3.7.8: SymptomVector
   │
   ├─ TIER 1 (Exact): sym_sha lookup
   │  ├─ Query: SELECT * FROM lessons WHERE sym_sha = ?
   │  ├─ Hit: Return past solution (cached in memory)
   │  └─ Miss: Continue to Tier 2
   │
   ├─ TIER 2 (Semantic/Hybrid): M8 search
   │  ├─ Query: POST /memory/query (hybrid: 60% pgvector + 40% OpenSearch)
   │  ├─ Results: Top-10 chunks ranked by similarity
   │  └─ Confidence: med-high (vector match)
   │
   └─ TIER 3 (Reference): M3.6 documentation
      ├─ Query: Reference corpus (kubectl docs, npm docs)
      ├─ Results: General guidance (not specific solution)
      └─ Confidence: low (generic info)

Response:
{
  "tier": 1,
  "confidence": "high",
  "results": [
    {
      "source": "lesson",
      "title": "Fix npm ERESOLVE errors",
      "content": "npm ERR! code ERESOLVE...",
      "solution": "Run npm ci instead of npm install"
    }
  ]
}
```

---

### CLI: `mem sig explain` — Query Signature Database

Once signatures are extracted and stored, query by signature to find past solutions:

```bash
# Find all solutions for npm ERESOLVE errors
mem sig explain --sig-sha=abc123def456789abcdef456789abc123def456

# Or by tool + normalized query
mem sig explain --tool=npm --query="unable to resolve dependency"

# Output:
# {
#   "found": true,
#   "sig_sha": "abc123...",
#   "count": 3,
#   "solutions": [
#     {
#       "source": "GH Action #1234",
#       "timestamp": "2026-08-20T15:30:00Z",
#       "solution": "npm ci instead of npm install",
#       "success_rate": 0.95
#     },
#     ...
#   ]
# }
```

**Workflow Integration:**
1. CI/CD captures failure log
2. `mem sig --tool=npm --file=log.txt` extracts sig_sha
3. `mem sig explain --sig-sha=<sha>` finds past solutions
4. If found → apply solution, skip manual debugging
5. If not found → fall back to M3.7.4 context endpoint (hybrid search)

---

### M3.7.8 Implementation Plan

**Files to create:**
1. `crates/mem-core/src/symptom_projection.rs` (250 LOC)
   - `project_symptom(tool: &str, query: &str) -> SymptomVector`
   - `normalize_query(text: &str) -> String`
   - `abbrev_expand(word: &str) -> String`
   - `stop_words()` — predefined list
   - Abbreviation mappings per tool

2. `tests/it_symptom_projection.rs` (400 LOC, 6 assertions)
   - `a1_same_symptom_same_hash` — query variants → same sym_sha
   - `a2_abbrev_expansion` — ERESOLVE, ERR, OOM normalize correctly
   - `a3_stop_word_removal` — "unable to resolve" → "resolve"
   - `a4_tool_consistency` — tool name included in sym_sha
   - `a5_case_insensitive` — "NPM" = "npm"
   - `a6_keyword_order_irrelevant` — sorted before hashing

**Test fixtures:**
- 6 query examples (npm, cargo, kubectl) with expected normalized form
- Abbreviation expansions per tool

**Integration with M3.7.4 endpoint:**
- M3.7.4 calls `symptom_projection::project_symptom(tool, user_query)`
- Returns `SymptomVector { sym_sha, normalised, keywords }`
- Looks up sym_sha in lesson cache
- If miss: delegates to M8 hybrid search

---

## M3.8: Context Optimizer

Headroom-inspired pre-LLM compression layer. Sits between hybrid search
retrieval and the LLM gateway. Search indexes stay at full fidelity.

### Why

Agent transcripts are ~43% tool results. Raw evidence chunks contain timestamps,
temp paths, ANSI codes, verbose JSON arrays, and passing test output. Feeding
this noise to the GRU-Mem gate wastes tokens, risks hallucination on irrelevant
details, and breaks LLM provider KV cache (dynamic content in prefix).

### Where It Sits

```
Query → Hybrid Search (pgvector 60% + OpenSearch 40%)
            │
            │  full-fidelity chunks (untouched)
            ▼
    ┌───────────────────────┐
    │   CONTEXT OPTIMIZER   │
    │                       │
    │  1. Magika Detect     │  ML content type classification (<1ms)
    │  2. CacheAligner      │  Move timestamps/UUIDs to tail
    │  3. Compressor         │  Per-type compression
    │  4. CCR Store          │  Cache originals for retrieval
    │                       │
    └───────────┬───────────┘
                │
                │  optimized chunks (30–90% smaller)
                ▼
    ┌───────────────────────┐
    │  Cache-Aligned Prompt │  system | query | turn
    └───────────┬───────────┘
                │
                ▼
           LLM Gateway
```

**Critical invariant:** Search indexes (pgvector + OpenSearch) NEVER see
compressed content. Compression only happens in the prompt assembly path.

### Stage 1: Content Detection (Magika ML)

Google’s Magika ONNX model classifies content type in <1ms. No LLM calls,
no network — embedded model runs locally.

```rust
use magika::Session;

let magika = magika::Session::new()?;
let result = magika.identify_content_sync(content.as_bytes())?;
let label = result.info().label; // "json", "python", "diff", etc.
```

Falls back to regex heuristics when Magika confidence < 0.7.

| Magika Label | Our Type | Compressor |
|---|---|---|
| `json`, `jsonl` | Json | JsonCrusher (70–90% savings) |
| `python`, `rust`, `go`, `typescript` | Code | CodeCompressor (40–70%) |
| `diff` | Diff | DiffCompressor (60–80%) |
| `yaml`, `toml`, `ini` | Config | passthrough |
| `txt` + log patterns | Log | LogCompressor (85–95%) |
| fallback | Text | TextCompressor (30–50%) |

### Stage 2: CacheAligner

LLM providers cache based on exact prefix match. A single changing timestamp
early in the prompt invalidates the entire KV cache.

CacheAligner detects dynamic patterns and moves them to the context tail:

```
BEFORE (cache miss every call):
  "At 2026-08-28T09:15:00Z, run abc123 failed with..."
  ↑ timestamp + run ID break prefix match

AFTER (cache hit on prefix):
  "Run failed with..."            ← stable prefix (cached)
  "[ctx: t=2026-08-28T09:15:00Z, run=abc123]"  ← dynamic tail
```

Reuses normalisation patterns from M3.7.7 `lesson.rs` (timestamp, SHA,
path, line:col, duration, temp path regexes).

### Stage 3: Per-Type Compression

**JsonCrusher** — Statistical field analysis on JSON arrays:
- Measures per-field variance, uniqueness, distribution boundaries
- Allocation: 30% start (schema), 15% end (recency), 55% importance
- Keeps: all keys, structure, error/null/boolean fields, boundary items
- Drops: homogeneous mid-array elements, long string values

**LogCompressor** — Reuses M3.7.7 signature extraction:
- `markers()` for error line detection (npm, cargo, kubectl, docker)
- `is_cascade()` for noise suppression
- `strip_ansi()` for cleanup
- Keeps: error lines, stack traces, exit codes, FAIL markers
- Drops: INFO/DEBUG noise, passing tests, repeated patterns

**CodeCompressor** — Signature preservation (opt-in):
- Keeps: imports, function/method signatures, type annotations
- Drops: function bodies, inline comments, blank lines
- Simple brace-counting heuristics (not full AST parser)

**DiffCompressor** — Change-only extraction:
- Keeps: `+`/`-` lines (actual changes), hunk headers (`@@`)
- Drops: unchanged context lines

**TextCompressor** — Token importance scoring:
- Reuses M3.7.8 stop words for low-value token detection
- Keeps: high-entropy tokens (IDs, hashes, error codes)
- Drops: filler words, repeated phrases

### Stage 4: CCR Store (Compress-Cache-Retrieve)

Compression is aggressive but reversible. Full originals cached with SHA256
hash. Retrieval hint injected into compressed output:

```
[compressed evidence...]
<!-- CCR:7f3a8bc... — full content available -->
```

If the model needs more detail, it can request the original via hash lookup.
LRU cache with TTL (default 1hr, matches gate run duration).

### Compression Targets

| Content Type | Ratio | Speed | Preserved |
|---|---|---|---|
| JSON arrays | 70–90% | ~1ms | All keys, structure, boundaries |
| Build logs | 85–95% | ~1ms | Errors, stack traces, exit codes |
| Source code | 40–70% | ~2ms | Signatures, imports, types |
| Unified diffs | 60–80% | ~1ms | Change lines, hunk headers |
| Plain text | 30–50% | ~2ms | High-entropy tokens |

### Code Reuse

| Existing Module | Reused For |
|---|---|
| `lesson.rs` normalise() | CacheAligner pattern detection |
| `lesson.rs` markers() | LogCompressor error detection |
| `lesson.rs` is_cascade() | LogCompressor noise suppression |
| `lesson.rs` strip_ansi() | Pre-processing cleanup |
| `symptom_projection.rs` STOP_WORDS | TextCompressor low-value tokens |

### Configuration

```bash
# Enable/disable (default: on)
MEM_CONTEXT_OPTIMIZER=on

# Per-compressor toggle
MEM_COMPRESS_JSON=on
MEM_COMPRESS_LOGS=on
MEM_COMPRESS_CODE=off    # opt-in
MEM_COMPRESS_DIFF=on
MEM_COMPRESS_TEXT=on

# CCR store
MEM_CCR_ENABLED=on
MEM_CCR_MAX_ENTRIES=1000
MEM_CCR_TTL_SECS=3600
```

### Task Breakdown (M3.8.x)

| Task | What | Status |
|---|---|---|
| M3.8.1 | ContentRouter (Magika) + all compressors + CCR | ⬜ |
| M3.8.2 | CacheAligner integration with PromptBuilder | ⬜ |
| M3.8.3 | Compression benchmarks + ratio tuning | ⬜ |
| M3.8.4 | Composition gate | ⬜ |

See `docs/CONTEXT_OPTIMIZER.md` for full design.
Inspired by [Headroom](https://docs.headroomlabs.ai/docs/how-compression-works).

---

## Pod Infrastructure

Complete pod inventory deployed in `poimen` namespace.

### Production Pods

| Pod Name | Role | Replicas | PVC | Purpose |
|----------|------|----------|-----|---------|
| **poimen-memory-\*** | API Server | 2 | 10Gi vault | HTTP server, JWT auth, GRC, hybrid search |  
| **memory-db-0** | PostgreSQL Primary | 1 | 20Gi | pgvector semantic search index |  
| **memory-db-1** | PostgreSQL Replica | 1 | 20Gi | High availability, read replicas |  
| **opensearch-0** | OpenSearch Primary | 1 | 30Gi | Lexical (BM25) search, JWT realm |  
| **opensearch-1** | OpenSearch Replica | 1 | 30Gi | HA cluster node, JWT validation |  
| **frontend-\*** | React SPA | 1+ | — | Web UI (memory.riotpiao.com) |

### Supporting Infrastructure (External)

| Component | Role | Location |
|-----------|------|----------|
| **Git-Sync Sidecar** | Auto-pull vault | Embedded in memory pod |
| **ArgoCD Application** | CD orchestration | argocd namespace |
| **Ingress Controller** | Reverse proxy | ingress-nginx namespace |
| **Longhorn** | Storage provider | Storage layer |

### Total Pod Count: **8 Production Pods**

```
Namespace: poimen
├─ poimen-memory (ReplicaSet) × 2 pods ...................... (2)
├─ memory-db-0 (StatefulSet) ............................... (1)
├─ memory-db-1 (StatefulSet) ............................... (1)
├─ opensearch-0 (StatefulSet) .............................. (1) [NEW]
├─ opensearch-1 (StatefulSet) .............................. (1) [NEW]
└─ frontend (Deployment) × 1-2 pods ......................... (1-2)

Namespace: argocd
└─ argocd-server, argocd-repo-server, etc .................. (5+)

Namespace: ingress-nginx
└─ nginx-ingress-controller ................................ (1)

TOTAL: 14-16 pods (8 core + 6-8 supporting)
```

### Pod Responsibilities

#### Memory Service Pod (×2, HA)
- **Listen**: 0.0.0.0:8080
- **Endpoints**:
  - `GET /memory/vault` — Read vault files
  - `POST /memory/query` — Hybrid search (semantic + lexical)
  - `GET /memory/skills` — List skills
  - `POST /memory/grc/draft` — Create branch + MR
  - `GET /memory/grc/status` — Check MR status
  - `GET /memory/agents/logs` — Stream agent events
- **Auth**: JWT (Authentik)
- **Hybrid Search Logic**:
  - Validates JWT (Authentik JWKS)
  - Queries pgvector (semantic in parallel)
  - Queries OpenSearch with JWT (lexical in parallel)
  - Reranks results (weighted: 60% semantic, 40% lexical)
  - Returns combined results
- **Connections**:
  - PostgreSQL (pgvector semantic search)
  - OpenSearch (lexical search with JWT)
  - Forgejo API (GRC)
  - LLM service (embeddings)
  - PVC (vault files)

#### PostgreSQL Pod (×2, Primary + Replica)
- **Listen**: 5432
- **Service**: memory-db (headless for StatefulSet)
- **Storage**: 20Gi per pod (PVC)
- **Replication**: Streaming replication (primary → replica)
- **Extensions**: pgvector
- **Data**:
  - chunks table (with vector index for semantic search)
  - skills table
  - projects table
  - agent_logs table
- **Role**: Semantic search engine (embeddings)

#### OpenSearch Pod (×2, Primary + Replica)
- **Listen**: 9200 (HTTP), 9300 (cluster communication)
- **Service**: opensearch (headless for cluster), opensearch-internal (for queries)
- **Storage**: 30Gi per pod (PVC)
- **Cluster**: poimen-memory (2-node minimum)
- **Security**:
  - JWT realm enabled (validates Authentik tokens)
  - JWKS endpoint: https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
  - Role mapping: Extract roles from JWT claims
  - Index permissions: read_vault, write_vault roles
- **Indices**:
  - vault-* (BM25 text search with TF-IDF scoring)
- **Role**: Lexical search engine (exact terms + TF-IDF)

#### Frontend Pod (×1-2)
- **Listen**: 80
- **Serve**: React SPA static files
- **Endpoints**:
  - `/` — App shell
  - `/api/*` — Proxy to Memory Service (8080)
- **Auth**: JWT (localStorage)
- **Build**: Vite (production bundle)

#### Git-Sync Sidecar (embedded in Memory Pod)
- **Runs**: As a init container + background process
- **Watch**: Forgejo main branch
- **Sync interval**: 30 seconds
- **Action on merge**: `git pull` → trigger re-index

---

## Traffic Flow Diagram

```
Internet (Users)
     │
     │ HTTPS
     │
     ↓
┌─────────────────────────┐
│ Ingress Controller      │
│ (nginx)                 │
└──┬────────────┬─────────┘
   │            │
   │ port 80    │ port 8080
   │            │
   ↓            ↓
┌──────────┐  ┌──────────────────┐
│ Frontend │  │ Memory Service   │
│ (React)  │  │ (Rust + Actix)   │
└────┬─────┘  └──┬───────────┬────┘
     │           │           │
     │        ┌──┘           └──┐
     │        │                 │
     ↓        ↓                 ↓
   ┌───────────────────┐  ┌──────────────┐
   │  PostgreSQL       │  │  Vault PVC   │
   │  (pgvector index) │  │  (git files) │
   └───────────────────┘  └──────────────┘
```

---

## Deployment Checklist

### Core Services
- [x] Memory Service (2 pods) deployed in `poimen` namespace
- [x] PostgreSQL StatefulSet (2 pods) deployed with pgvector
- [x] OpenSearch StatefulSet (2 pods) deployed with JWT realm
- [x] PVC: poimen-memory-vault (10Gi) attached
- [x] PVC: opensearch data (30Gi per pod) attached

### Security & Auth
- [x] JWT auth enabled (Authentik integration at Memory Service)
- [x] OpenSearch JWT realm configured (validates Authentik tokens)
- [x] K8s NetworkPolicy (only Memory Service → OpenSearch)
- [x] Role mapping (JWT claims → OpenSearch roles)

### Integration
- [x] Git-sync sidecar configured (auto-pull on merge)
- [x] Ingress configured (memory.riotpiao.com)
- [x] Hybrid search endpoints (/memory/query?search_method=hybrid)

### Design Documentation
- [x] Retrieval pipeline architecture (4-stage: normalize → parallel → fusion → ranking)
- [x] Index optimization (pgvector ivfflat + OpenSearch BM25 tuning)
- [x] Score fusion strategy (weighted linear + RRF alternative)
- [x] Query routing decision tree (short queries → lexical, normal → hybrid)
- [x] Accuracy metrics (MRR, NDCG@10, Precision@K, Recall@K)
- [x] Weight tuning strategy (A/B testing framework)
- [ ] **Reference**: See `docs/HYBRID_SEARCH_DESIGN.md` (19KB, comprehensive design)

### Testing & Deployment
- [ ] Test fixture setup (query + expected results dataset)
- [ ] NDCG/MRR baseline measurements (semantic vs lexical)
- [ ] Weight tuning experiments (0.5/0.5, 0.6/0.4, 0.7/0.3, 0.4/0.6)
- [ ] Performance benchmarks (latency: parallel vs serial)
- [ ] Frontend pod deployment
- [ ] End-to-end hybrid search tests
- [ ] Gradual rollout (Phase 1: lexical-only → Phase 2: hybrid 10% → Phase 3: 100%)
- [ ] GRC endpoints tested
- [ ] Agent logging endpoints tested