poimen-memory/memory-flow.md

Complete Memory API Call Flows & Routes

Project Status: ✅ All 78 tasks complete, 100% feature-ready

Table of Contents

1. API Endpoints Overview
2. Detailed Call Flows by Route
3. Authorization & Authentication
4. Error Handling & Fallbacks
5. Performance Characteristics
6. System Architecture

---

API Endpoints Overview

Endpoint	Method	Auth	Rate Limit	Purpose
/health	GET	—	—	Service health check
/memory/vault	GET	JWT	—	Browse vault files
/memory/query	POST	JWT	1000/hr	Hybrid search (semantic + lexical)
/memory/context	POST	JWT	100/hr	Three-tier context retrieval
/memory/ingest	POST	JWT	100/hr	Ingest new memory records
/memory/rebuild	POST	JWT	—	Rebuild indexes from log
/memory/verify	GET	JWT	—	Composition gate validation
/memory/skills	GET	JWT	—	List available skills
/memory/agents/logs	GET	JWT	—	Stream agent execution logs
/memory/grc/draft	POST	JWT	—	Create GRC branch & MR
/memory/grc/status	GET	JWT	—	Check MR merge status

---

Detailed Call Flows by Route

Route 1: GET /health

Purpose: Service health check (no auth required)

REQUEST:
  GET http://localhost:8080/health
  [No headers required]

CALL FLOW:
  1. http_server.rs::handle_health()
     └─> Return { "status": "ok", "timestamp": "..." }

RESPONSE: 200 OK
  {
    "status": "ok",
    "timestamp": "2025-01-29T10:00:00Z",
    "version": "0.1.0"
  }

ERROR PATHS:
  - 500 Internal Server Error: If database unavailable
    └─> Return { "status": "unhealthy", "reason": "db_connection_failed" }

---

Route 2: GET /memory/vault?project=

Purpose: List all vault files, filtered by project

REQUEST:
  GET http://localhost:8080/memory/vault?project=poimen
  Authorization: Bearer <JWT>
  
  Query Params:
    - project: string (required) — project identifier
    - level_filter: L1,L2,R (optional) — filter by level
    - path_prefix: docs/ (optional) — limit to directory

FULL CALL FLOW:
  1. http_server.rs::handle_vault()
     ├─> Step 1: Validate Authorization
     │   ├─ Extract JWT from Authorization header
     │   ├─ jwt_validator.rs::validate_token()
     │   │  ├─ Check token signature (RS256, Authentik JWKS)
     │   │  ├─ Verify issuer matches config
     │   │  ├─ Check expiry (exp claim)
     │   │  ├─ Validate audience (aud = "poimen-memory")
     │   │  └─ Return: { user, roles, permissions }
     │   │
     │   └─ Check capability: "memory:read" in permissions?
     │      └─ If missing → 403 Forbidden
     │
     ├─> Step 2: List Vault Files
     │   ├─ vault_projector.rs::list_files(project)
     │   │  ├─ event_log.rs::read_event_log()
     │   │  │  └─ Scan JSONL log for all records
     │   │  │
     │   │  ├─ For each record:
     │   │  │  ├─ Parse JSON
     │   │  │  ├─ Check project_id matches
     │   │  │  ├─ Extract: level, path, text, timestamp, source
     │   │  │  └─ Group by file path
     │   │  │
     │   │  ├─ Build FileInfo:
     │   │  │  ├─ path: string
     │   │  │  ├─ title: inferred from path
     │   │  │  ├─ level: L0|L1|L2|R
     │   │  │  ├─ updated_at: max(timestamps)
     │   │  │  ├─ record_count: count of records
     │   │  │  ├─ source: transcript|docs|reference
     │   │  │  └─ breadcrumb: file path > section > subsection
     │   │  │
     │   │  └─ Return: Vec<FileInfo>
     │   │
     │   ├─ Apply filters (if provided):
     │   │  ├─ level_filter: keep only L1, L2, etc.
     │   │  └─ path_prefix: keep only docs/*, etc.
     │   │
     │   └─ Sort by:
     │      ├─ Primary: updated_at DESC (most recent first)
     │      └─ Secondary: path ASC (alphabetical)
     │
     ├─> Step 3: Build Response
     │   ├─ Count total records across all files
     │   ├─ Compute response size
     │   └─ Return: { project, files, total_records }
     │
     └─> Return 200 OK

RESPONSE: 200 OK (example)
  {
    "project": "poimen",
    "files": [
      {
        "path": "kubernetes/debugging.md",
        "title": "Debugging",
        "level": "L1",
        "updated_at": "2025-01-28T10:00:00Z",
        "record_count": 23,
        "breadcrumb": "kubernetes.md > Debugging > Pod Issues",
        "source": "transcript://session-123"
      },
      {
        "path": "reference/docs/kubectl.md",
        "title": "kubectl Reference",
        "level": "R",
        "updated_at": "2025-01-20T15:30:00Z",
        "record_count": 156,
        "breadcrumb": "kubectl.md > Common Commands",
        "source": "obsidian://poimen-vault/kubectl.md"
      }
    ],
    "total_records": 542,
    "search_time_ms": 34
  }

ERROR PATHS:
  - 401 Unauthorized: JWT missing or invalid
    └─> { "error": "unauthorized", "reason": "invalid_token" }
  - 403 Forbidden: Token lacks "memory:read" capability
    └─> { "error": "forbidden", "reason": "missing_capability" }
  - 404 Not Found: Project doesn't exist
    └─> { "error": "not_found", "reason": "project_not_found" }
  - 500 Internal Server Error: Event log read failure
    └─> { "error": "internal_error", "reason": "log_read_failed" }

PERFORMANCE:
  - Typical: 20-50ms (depends on event log size)
  - Worst case: 500ms (large project, slow disk)
  - Cached for: 60 seconds (per project)

---

Route 3: POST /memory/query

Purpose: Hybrid semantic + lexical search with three query routes

REQUEST:
  POST http://localhost:8080/memory/query
  Authorization: Bearer <JWT>
  Content-Type: application/json
  
  {
    "project": "poimen",
    "query": "fix kubernetes port 8080 conflict",
    "level_filter": ["L1", "L2"],    // optional: exclude R
    "floor": 0.6,                     // optional: min relevance
    "limit": 10,                      // optional: default 10, max 100
    "scope": "all"                    // optional: "learned"|"reference"|"all"
  }

FULL CALL FLOW:
  1. http_server.rs::handle_query()
     ├─> Step 1: Validate & Extract JWT
     │   ├─ jwt_validator.rs::validate_token()
     │   ├─ Check "memory:read" capability
     │   └─ If missing → 403 Forbidden
     │
     ├─> Step 2: Rate Limit Check
     │   ├─ rate_limiter.rs::check_limit(apikey, "query")
     │   │  ├─ Get token bucket state (redis-like)
     │   │  ├─ Tokens available? (1000/hour = 1 per 3.6 seconds)
     │   │  └─ If depleted → 429 Too Many Requests
     │   │     └─ Return: Retry-After: 45 (seconds)
     │   │
     │   └─ Decrement bucket
     │
     ├─> Step 3: Query Classification (M8.3)
     │   ├─ query_optimizer.rs::classify_question()
     │   │  ├─ Tokenize query into words
     │   │  ├─ Detect intent:
     │   │  │  ├─ Bug fix keywords: "error", "fail", "bug", "not working"
     │   │  │  │  └─ Route: "hybrid" (both engines critical)
     │   │  │  ├─ How-to keywords: "how", "guide", "setup", "configure"
     │   │  │  │  └─ Route: "semantic" (understanding over exact match)
     │   │  │  ├─ FAQ keywords: exact phrase match patterns
     │   │  │  │  └─ Route: "lexical" (BM25 for phrase retrieval)
     │   │  │  └─ Default:
     │   │  │     └─ Route: "hybrid" (safest default)
     │   │  │
     │   │  └─ Return: { intent, route: "semantic"|"lexical"|"hybrid" }
     │   │
     │   └─ Store route for later
     │
     ├─> Step 4: Hybrid Query Execution
     │   │
     │   ├─ ROUTE 1: Semantic Only (if route == "semantic")
     │   │  ├─ embeddings.rs::embed_query(query)
     │   │  │  ├─ Call LLM (nomic-embed-text-1.5, 768-dim)
     │   │  │  ├─ Get: query_embedding [768 floats]
     │   │  │  └─ Normalize to unit vector (L2 norm)
     │   │  │
     │   │  ├─ pgvector_repo.rs::vector_search()
     │   │  │  ├─ Query: SELECT id, text, breadcrumb, embedding
     │   │  │  │          FROM memory_vector
     │   │  │  │          WHERE project_id = ?
     │   │  │  │          AND level IN (?) [L0, L1, L2]
     │   │  │  │          ORDER BY embedding <=> query_vec DESC
     │   │  │  │          LIMIT 50
     │   │  │  │
     │   │  │  ├─ Returns: [(id, text, sim_score_0_to_1), ...]
     │   │  │  │ where sim_score = cosine_similarity(embedding, query_vec)
     │   │  │  │
     │   │  │  └─ Example scores: [(doc1, 0.92), (doc2, 0.78), ...]
     │   │  │
     │   │  └─ Semantic path complete
     │   │
     │   │
     │   ├─ ROUTE 2: Lexical Only (if route == "lexical")
     │   │  ├─ Tokenize query: ["fix", "kubernetes", "port", ...]
     │   │  │
     │   │  ├─ opensearch_client.rs::search()
     │   │  │  ├─ Build OpenSearch query:
     │   │  │  │  {
     │   │  │  │    "query": {
     │   │  │  │      "multi_match": {
     │   │  │  │        "query": "fix kubernetes port 8080 conflict",
     │   │  │  │        "fields": [
     │   │  │  │          "content^2",      // 2x boost on full text
     │   │  │  │          "breadcrumb",
     │   │  │  │          "source"
     │   │  │  │        ],
     │   │  │  │        "fuzziness": "AUTO",  // typo tolerance
     │   │  │  │        "operator": "or"      // match any term
     │   │  │  │      }
     │   │  │  │    },
     │   │  │  │    "filter": [
     │   │  │  │      { "term": { "project_id": "poimen" } },
     │   │  │  │      { "terms": { "level": ["L0", "L1", "L2"] } },
     │   │  │  │      { "range": { "created_at": { "gte": "now-1y" } } }
     │   │  │  │    ],
     │   │  │  │    "size": 50,
     │   │  │  │    "track_scores": true
     │   │  │  │  }
     │   │  │  │
     │   │  │  ├─ Send JWT in Authorization header to OpenSearch
     │   │  │  ├─ OpenSearch validates token (JWT realm):
     │   │  │  │  ├─ Extract JWT from Authorization header
     │   │  │  │  ├─ Validate signature (JWKS from Authentik)
     │   │  │  │  ├─ Extract roles from claims
     │   │  │  │  └─ Check index permissions (read_vault role)
     │   │  │  │
     │   │  │  └─ Returns: [(id, text, bm25_score_raw), ...]
     │   │  │     Example: [(doc1, 8.5), (doc2, 6.2), ...]
     │   │  │
     │   │  └─ Lexical path complete
     │   │
     │   │
     │   ├─ ROUTE 3: Hybrid (if route == "hybrid")
     │   │  ├─ Execute BOTH paths in parallel:
     │   │  │  ├─ Task 1: embeddings + pgvector_search (semantic path)
     │   │  │  └─ Task 2: opensearch_search + JWT (lexical path)
     │   │  │
     │   │  └─ Wait for both to complete (tokio::join!)
     │   │
     │   │  ├─ Score Normalization:
     │   │  │  ├─ Semantic scores already [0.0, 1.0] (cosine)
     │   │  │  │
     │   │  │  ├─ Lexical scores raw (e.g., 0-50 range):
     │   │  │  │  ├─ Find min & max of returned scores
     │   │  │  │  ├─ min-max normalize: (score - min) / (max - min)
     │   │  │  │  └─ Result: [0.0, 1.0]
     │   │  │  │
     │   │  │  └─ Both now in [0.0, 1.0] range
     │   │  │
     │   │  ├─ RRF Fusion (rrf_fusion.rs::fuse_results):
     │   │  │  ├─ Collect all unique doc IDs from both result sets
     │   │  │  │
     │   │  │  ├─ For each doc:
     │   │  │  │  ├─ Get semantic score (default 0.0 if not in results)
     │   │  │  │  ├─ Get lexical score (default 0.0 if not in results)
     │   │  │  │  │
     │   │  │  │  ├─ Compute fused score:
     │   │  │  │  │  fused = 0.6 * semantic_norm + 0.4 * lexical_norm
     │   │  │  │  │
     │   │  │  │  └─ Example:
     │   │  │  │     doc1: 0.6*1.0 + 0.4*0.98 = 0.992
     │   │  │  │     doc2: 0.6*0.96 + 0.4*0.0 = 0.576
     │   │  │  │     doc3: 0.6*0.0 + 0.4*0.88 = 0.352
     │   │  │  │
     │   │  │  ├─ Sort by fused score (descending)
     │   │  │  └─ Take top 10 (or user's limit)
     │   │  │
     │   │  └─ Hybrid path complete
     │   │
     │   └─ Path complete (semantic, lexical, or hybrid)
     │
     ├─> Step 5: Query Levels Filtering (M3.6.5)
     │   ├─ query_levels.rs::apply_filters(results)
     │   │  ├─ For each result:
     │   │  │  ├─ Check level_filter: is result's level in allowed list?
     │   │  │  │  └─ If not → exclude
     │   │  │  │
     │   │  │  ├─ Check floor threshold: is score >= floor?
     │   │  │  │  └─ If not → exclude
     │   │  │  │
     │   │  │  ├─ Check scope:
     │   │  │  │  ├─ "learned": only L0/L1/L2 (exclude R)
     │   │  │  │  ├─ "reference": only R
     │   │  │  │  └─ "all": no exclusion
     │   │  │  │
     │   │  │  └─ Keep result if all checks pass
     │   │  │
     │   │  └─ Return: filtered_results
     │   │
     │   └─ Filtering complete
     │
     ├─> Step 6: Record Query for Metrics
     │   ├─ accuracy_metrics.rs::record_query(query, results)
     │   │  ├─ Store for NDCG/MRR calculation
     │   │  ├─ Track query intent distribution
     │   │  └─ Used for M8.9 composition gate
     │   │
     │   └─ Metrics recorded
     │
     ├─> Step 7: Build Response
     │   ├─ For each result:
     │   │  ├─ Include: id, level, score, text, breadcrumb, source
     │   │  ├─ If hybrid: include semantic_score, lexical_score breakdown
     │   │  ├─ Truncate text to 500 chars (keep breadcrumb intact)
     │   │  └─ Parse breadcrumb for hierarchy display
     │   │
     │   └─ Return: { query, results, total_hits, search_time_ms }
     │
     └─> Return 200 OK

RESPONSE: 200 OK (example)
  {
    "query": "fix kubernetes port 8080 conflict",
    "intent": "bug_fix",
    "route": "hybrid",
    "results": [
      {
        "id": "chunk-abc123",
        "level": "L1",
        "score": 0.992,
        "semantic_score": 1.0,
        "lexical_score": 0.98,
        "semantic_weight": 0.6,
        "lexical_weight": 0.4,
        "text": "To fix port conflicts, check if port 8080 is already in use...",
        "breadcrumb": "kubernetes.md > Troubleshooting > Port Conflicts",
        "source": "transcript://session-123",
        "matched_fields": ["content", "breadcrumb"]
      },
      {
        "id": "chunk-def456",
        "level": "L2",
        "score": 0.576,
        "semantic_score": 0.96,
        "lexical_score": 0.0,
        "text": "Common Kubernetes debugging patterns include...",
        "breadcrumb": "kubernetes.md > Debugging > Patterns",
        "source": "transcript://session-456"
      },
      // ... 8 more results
    ],
    "total_hits": 127,
    "search_time_ms": 145,
    "returned_count": 10
  }

ERROR PATHS:
  - 401 Unauthorized: JWT missing/invalid
    └─> { "error": "unauthorized", "reason": "invalid_token" }
  - 403 Forbidden: Missing "memory:read" capability
    └─> { "error": "forbidden", "reason": "insufficient_permissions" }
  - 429 Too Many Requests: Rate limit exceeded (1000/hr)
    └─> { "error": "rate_limit_exceeded", "retry_after": 45 }
  - 400 Bad Request: Invalid query format
    └─> { "error": "invalid_request", "reason": "query_too_long" }
  - 503 Service Unavailable: OpenSearch unreachable
    └─> Fallback to semantic-only search
       └─> { "query": "...", "results": [...], "degraded": true, "reason": "lexical_engine_unavailable" }
  - 504 Gateway Timeout: Search > 10 seconds
    └─> Return partial results with timeout flag
       └─> { "query": "...", "results": [...], "timeout": true, "partial": true }

PERFORMANCE:
  - Semantic only: 50-100ms (LLM embedding + pgvector search)
  - Lexical only: 30-80ms (OpenSearch BM25)
  - Hybrid (parallel): 80-150ms (max of both + merge overhead)
  - Rate limit: 1000 queries/hour (1 per 3.6 seconds)
  - Typical query returns 50 semantic + 50 lexical, merged to top-10

WEIGHTS (Tunable):
  - Semantic: 60% (understanding matters more)
  - Lexical: 40% (exact terms provide disambiguation)
  - Tuning via M8.3: adjust based on query intent

---

Route 4: POST /memory/context

Purpose: Three-tier context retrieval for tool execution (M3.7)

REQUEST:
  POST http://localhost:8080/memory/context
  Authorization: Bearer <JWT>
  Content-Type: application/json
  
  {
    "project": "poimen",
    "tool": "kubectl",
    "task": "debug-pod",
    "signature_source": "failure_log",
    "scope": "tool_context",
    "budget": 8192  // Max response bytes
  }

FULL CALL FLOW:
  1. http_server.rs::handle_context()
     ├─> Step 1: JWT Validation
     │   ├─ jwt_validator.rs::validate_token()
     │   ├─ Check "memory:read" capability
     │   └─ Deny if missing (403)
     │
     ├─> Step 2: Rate Limit Check
     │   ├─ rate_limiter.rs::check_limit(apikey, "projects")
     │   ├─ Limit: 100/hour
     │   └─ Deny if exceeded (429)
     │
     ├─> Step 3: Context Lookup (context_endpoint.rs)
     │   │
     │   ├─ TIER 1: Exact Signature Match
     │   │  ├─ signature_lookup.rs::find_by_source()
     │   │  │  ├─ Extract signature from request (failure_log field)
     │   │  │  ├─ M3.7.7 signature extraction:
     │   │  │  │  ├─ Tokenize signature source
     │   │  │  │  ├─ Run tool-specific extractors (npm, cargo, kubectl)
     │   │  │  │  ├─ Normalize (remove timestamps, paths, hashes)
     │   │  │  │  └─ Compute SHA256: sig_sha
     │   │  │  │
     │   │  │  ├─ Query: SELECT lessons FROM memory_lessons
     │   │  │  │          WHERE project_id = ?
     │   │  │  │          AND sig_sha = ?
     │   │  │  │
     │   │  │  └─ Return: [Lesson { tier: 1, score: 1.0, text, ... }]
     │   │  │     (or empty if no match)
     │   │  │
     │   │  └─ Tier-1 complete
     │   │
     │   │
     │   ├─ TIER 2: Vector Search (only if budget permits or Tier-1 miss)
     │   │  ├─ context_query.rs::embed_context(tool, task)
     │   │  │  ├─ Build context string: "{tool} {task}"
     │   │  │  ├─ LLM embed (768-dim)
     │   │  │  └─ Get: context_embedding
     │   │  │
     │   │  ├─ Call simple_hybrid_search.rs::hybrid_search()
     │   │  │  ├─ Parallel paths (same as /memory/query):
     │   │  │  │  ├─ Semantic: pgvector_repo.rs::vector_search()
     │   │  │  │  └─ Lexical: opensearch_client.rs::search() + JWT
     │   │  │  │
     │   │  │  ├─ Score normalization & RRF fusion
     │   │  │  └─ Return: [(doc_id, fused_score), ...]
     │   │  │
     │   │  ├─ Filter for Tier-2 only:
     │   │  │  ├─ Keep only L2 records (high-confidence synthesis)
     │   │  │  ├─ Or high-confidence L1 (score > 0.85)
     │   │  │  └─ Exclude R (reference documents)
     │   │  │
     │   │  ├─ Take top-20 candidates
     │   │  ├─ Return: [Lesson { tier: 2, score, text, ... }]
     │   │  │
     │   │  └─ Tier-2 complete
     │   │
     │   │
     │   ├─ TIER 3: Reference Fallback (if budget allows)
     │   │  ├─ obsidian_ref_source.rs::fetch_reference_sections()
     │   │  │  ├─ Call Obsidian REST API: GET /api/vault/listFiles
     │   │  │  │  (Obsidian pod, port 27124)
     │   │  │  │
     │   │  │  ├─ For each reference file:
     │   │  │  │  ├─ Call: GET /api/vault/readFile?path={path}
     │   │  │  │  ├─ Chunk via M3.6.1 (heading boundaries)
     │   │  │  │  └─ Compute relevance to tool/task
     │   │  │  │
     │   │  │  └─ Return: chunks ordered by relevance
     │   │  │
     │   │  ├─ reference_cycle_guard.rs::detect_derived_reference()
     │   │  │  ├─ Check shingle overlap with ingested content
     │   │  │  ├─ If overlap > 0.5 → mark as "derived"
     │   │  │  └─ Exclude derived refs (don't duplicate evidence)
     │   │  │
     │   │  ├─ Take top-5 non-derived reference chunks
     │   │  └─ Return: [Lesson { tier: 3, score, source: "obsidian://...", ... }]
     │   │
     │   ├─ Tier-3 complete (or skipped if Obsidian unavailable)
     │   │
     │   │
     ├─> Step 4: Budget-Aware Response Assembly
     │   ├─ context_optimizer.rs::assemble_with_budget()
     │   │  ├─ requested_budget = 8192 bytes
     │   │  ├─ used_budget = 0
     │   │  │
     │   │  ├─ Add Tier-1 lessons (never drop):
     │   │  │  └─ used_budget += tier1_lessons.len()
     │   │  │
     │   │  ├─ Try to add Tier-2 lessons:
     │   │  │  ├─ For each tier-2 lesson (highest score first):
     │   │  │  │  ├─ size = lesson.text.len()
     │   │  │  │  ├─ if (used_budget + size) <= requested_budget:
     │   │  │  │  │    add lesson
     │   │  │  │  │  else:
     │   │  │  │  │    break
     │   │  │  │  │
     │   │  │  └─ used_budget += added_lessons.len()
     │   │  │
     │   │  ├─ Try to add Tier-3 lessons (if space):
     │   │  │  ├─ Same logic as Tier-2
     │   │  │  └─ used_budget += added_lessons.len()
     │   │  │
     │   │  ├─ If over budget:
     │   │  │  ├─ Drop Tier-3 (reference) first
     │   │  │  ├─ Then drop Tier-2 (lowest scores first)
     │   │  │  ├─ Record degradation reason
     │   │  │  └─ Keep Tier-1 always
     │   │  │
     │   │  └─ dropped_budget = requested_budget - used_budget
     │   │
     │   └─ Assembly complete
     │
     ├─> Step 5: Skill Linking
     │   ├─ derived_filter.rs::find_linked_skills()
     │   │  ├─ For each Tier-1 hit:
     │   │  │  ├─ Query skill manifest (M4.2)
     │   │  │  ├─ Find skills matching this lesson's sha256
     │   │  │  └─ Add to skills list
     │   │  │
     │   │  ├─ For each Tier-2 hit (confidence > 0.8):
     │   │  │  ├─ Fuzzy match against skill names
     │   │  │  └─ Add if match > 0.9
     │   │  │
     │   │  └─ Deduplicate skills, sort by relevance
     │   │
     │   └─ Skills collected
     │
     ├─> Step 6: Build Response
     │   ├─ Set tier = max(tier_with_results)
     │   │  (1 if Tier-1 hit, 2 if only Tier-2 hits, etc.)
     │   │
     │   ├─ For each lesson:
     │   │  ├─ Include: tier, level, score, text, matched_kind, seen_count
     │   │  ├─ last_seen: timestamp of most recent occurrence
     │   │  └─ parents: breadcrumb hierarchy
     │   │
     │   ├─ Build budget info:
     │   │  ├─ requested: 8192
     │   │  ├─ used: actual bytes used
     │   │  ├─ dropped: bytes dropped (if over budget)
     │   │  └─ degradation: null or reason string
     │   │
     │   └─ Return: { tier, lessons, skills, budget }
     │
     └─> Return 200 OK

RESPONSE: 200 OK (example)
  {
    "tier": 1,
    "confidence": "high",
    "lessons": [
      {
        "tier": 1,
        "level": "L1",
        "score": 1.0,
        "text": "Pod in CrashLoopBackOff: check logs with kubectl logs <pod>",
        "matched_kind": "signature",
        "seen_count": 23,
        "last_seen": "2025-01-28T15:30:00Z",
        "parents": ["kubectl.md", "Troubleshooting", "Pod Issues"]
      },
      {
        "tier": 2,
        "level": "L2",
        "score": 0.87,
        "text": "Common Kubernetes debugging patterns include...",
        "matched_kind": "vector",
        "seen_count": 5,
        "last_seen": "2025-01-25T10:00:00Z"
      },
      {
        "tier": 3,
        "level": "R",
        "score": 0.65,
        "text": "See kubectl troubleshooting guide for general reference",
        "matched_kind": "reference",
        "source": "obsidian://poimen-vault/kubectl.md"
      }
    ],
    "skills": [
      {
        "name": "diagnose-pod-failure",
        "description": "Diagnose Kubernetes pod issues",
        "why": "Tier-1 signature matched"
      }
    ],
    "budget": {
      "requested": 8192,
      "used": 4156,
      "dropped": 0,
      "degradation": null
    }
  }

ERROR PATHS:
  - 401 Unauthorized: JWT missing/invalid
  - 403 Forbidden: Missing "memory:read"
  - 429 Too Many Requests: Rate limit exceeded (100/hr)
  - 400 Bad Request: Missing tool or task
  - 503 Service Unavailable: Obsidian API unreachable
    └─> Return Tier-1 & Tier-2 only (graceful degradation)
  - 504 Gateway Timeout: Obsidian takes > 5 seconds
    └─> Return best-effort Tier-1

PERFORMANCE:
  - Tier-1 (exact match): 5-10ms
  - Tier-2 (hybrid search): 80-150ms
  - Tier-3 (Obsidian fetch): 100-500ms
  - Total: 100-200ms (typical), 500ms (with Obsidian, worst case)
  - Budget assembly: 5ms

---

Route 5: POST /memory/ingest

Purpose: Ingest new memory records (async processing via queue)

REQUEST:
  POST http://localhost:8080/memory/ingest
  Authorization: Bearer <JWT>
  Content-Type: application/json
  
  {
    "project": "poimen",
    "source": "transcript://session-123",
    "kind": "L1",
    "text": "Kubernetes port 8080 conflict resolved by checking netstat...",
    "metadata": {
      "session_id": "sess-123",
      "topic": "troubleshooting",
      "tool": "kubectl"
    }
  }

FULL CALL FLOW:
  1. http_server.rs::handle_ingest()
     ├─> Step 1: JWT Validation
     │   ├─ jwt_validator.rs::validate_token()
     │   ├─ Check "memory:write" capability
     │   └─ Deny if missing (403)
     │
     ├─> Step 2: Rate Limit Check
     │   ├─ rate_limiter.rs::check_limit(apikey, "ingest")
     │   ├─ Limit: 100/hour (ingest-specific)
     │   └─ Deny if exceeded (429)
     │
     ├─> Step 3: Idempotency Check
     │   ├─ idempotency.rs::is_duplicate(idempotency_key)
     │   │  ├─ Generate idempotency_key from {project, source, sha256(text)}
     │   │  ├─ Query: SELECT * FROM idempotency_store
     │   │  │          WHERE key = ? AND created_at > now - 24h
     │   │  │
     │   │  ├─ If found (duplicate):
     │   │  │  ├─ Return 409 Conflict with cached response
     │   │  │  └─ Do NOT re-queue
     │   │  │
     │   │  └─ If not found (new):
     │   │     └─ Continue to Step 4
     │   │
     │   └─ Idempotency checked
     │
     ├─> Step 4: Enqueue Record
     │   ├─ chunk_queue.rs::enqueue_record()
     │   │  ├─ Create Record { project, source, kind, text, metadata }
     │   │  ├─ Compute sha256 of text (for dedup)
     │   │  ├─ Add timestamp (ingest time)
     │   │  └─ Store in local queue (in-memory + RocksDB backup)
     │   │
     │   └─ Record queued
     │
     ├─> Step 5: Send to External Queue
     │   ├─ gateway_queue_adapter.rs::send_to_external_queue()
     │   │  ├─ Serialize record to JSON
     │   │  ├─ Send to external gateway (api.riotpiao.com/queue)
     │   │  │  POST /queue
     │   │  │  Authorization: <gateway_api_key>
     │   │  │  Content-Type: application/json
     │   │  │  
     │   │  │  Body: { "project": "poimen", "source": "...", ... }
     │   │  │
     │   │  ├─ Gateway stores in SQS / Redis / Kafka
     │   │  └─ Returns: { queue_id, status: "pending" }
     │   │
     │   └─ Sent to external queue (async)
     │
     ├─> Step 6: Build & Return Response (immediate)
     │   ├─ Return 201 Created
     │   ├─ Include: chunk_id, sha256, queue_status: "pending"
     │   └─ Indicate async processing
     │
     └─> Return 201 Created

RESPONSE: 201 Created
  {
    "id": "chunk-abc123def456",
    "sha256": "de12cd34ef56789abcdef0123456789abcdef01234567",
    "queue_status": "pending",
    "idempotency_key": "sess-123:de12cd34ef56",
    "enqueued_at": "2025-01-29T10:00:00Z"
  }

ASYNC PROCESSING (Background):
  
  1. queue_worker.rs::process_queue() [runs continuously]
     ├─> Poll external queue (30s visibility timeout)
     │
     ├─> For each message:
     │   ├─ Receive from queue
     │   ├─ Deserialize record
     │   │
     │   ├─ Step 1: Embedding
     │   │  ├─ embeddings.rs::embed_text(text)
     │   │  │  ├─ Send to LLM service (nomic 768-dim)
     │   │  │  └─ Get: embedding [768 floats]
     │   │  │
     │   │  └─ Embedding complete
     │   │
     │   ├─ Step 2: Insert to Postgres (PRIMARY)
     │   │  ├─ pgvector_repo.rs::insert_record()
     │   │  │  ├─ INSERT INTO memory_vector
     │   │  │  │  (project_id, level, text, embedding, source, breadcrumb,
     │   │  │  │   metadata, created_at)
     │   │  │  │  VALUES (?, ?, ?, ?, ?, ?, ?, ?)
     │   │  │  │
     │   │  │  ├─ On success:
     │   │  │  │  ├─ Get: vector_id
     │   │  │  │  └─ Update metadata: embedding_id
     │   │  │  │
     │   │  │  └─ On error:
     │   │  │     ├─ Log error
     │   │  │     ├─ Continue (try OpenSearch anyway)
     │   │  │     └─ pgvector is critical, but don't block queue
     │   │  │
     │   │  └─ Postgres insert complete
     │   │
     │   ├─ Step 3: Index to OpenSearch (SECONDARY - fail-soft)
     │   │  ├─ dual_write_indexer.rs::index_to_opensearch()
     │   │  │  ├─ Prepare OpenSearch document:
     │   │  │  │  {
     │   │  │  │    "_id": sha256,
     │   │  │  │    "project_id": "poimen",
     │   │  │  │    "level": "L1",
     │   │  │  │    "content": text,
     │   │  │  │    "breadcrumb": breadcrumb,
     │   │  │  │    "source": source,
     │   │  │  │    "created_at": timestamp,
     │   │  │  │    "metadata": metadata
     │   │  │  │  }
     │   │  │  │
     │   │  │  ├─ opensearch_client.rs::bulk_index()
     │   │  │  │  ├─ Add to bulk buffer (batch 1000 docs)
     │   │  │  │  ├─ Send Authorization: Bearer <JWT> to OpenSearch
     │   │  │  │  │  (OpenSearch validates token + checks permissions)
     │   │  │  │  │
     │   │  │  │  └─ On success:
     │   │  │  │     └─ Document indexed (available for lexical search)
     │   │  │  │
     │   │  │  └─ On error (OpenSearch unreachable):
     │   │  │     ├─ Log warning
     │   │  │     ├─ Retry with exponential backoff (1s, 2s, 4s, 8s)
     │   │  │     ├─ After max retries: continue (graceful degradation)
     │   │  │     └─ Lexical search will have gaps, but semantic works
     │   │  │
     │   │  └─ OpenSearch index complete (or gracefully degraded)
     │   │
     │   ├─ Step 4: Mark Message Complete
     │   │  ├─ Remove from queue (visibility timeout expires)
     │   │  ├─ Record processed successfully
     │   │  └─ idempotency.rs::store_processed(idempotency_key)
     │   │     ├─ Store in idempotency store with 24h TTL
     │   │     ├─ Include result: { id, sha256 }
     │   │     └─ Future duplicate requests get cached response
     │   │
     │   └─ Message processing complete
     │
     └─> Poll next message (or wait if queue empty)

ERROR PATHS (Synchronous):
  - 401 Unauthorized: JWT missing/invalid
  - 403 Forbidden: Missing "memory:write"
  - 429 Too Many Requests: Rate limit exceeded (100/hr)
  - 409 Conflict: Duplicate ingest (same idempotency key within 24h)
    └─> Return cached 201 response
  - 400 Bad Request: Missing required fields
  - 503 Service Unavailable: Cannot reach external queue
    └─> 503, but queue message still created locally (will retry)

ERROR PATHS (Asynchronous - Queue Worker):
  - LLM service unavailable: Retry embedding (exponential backoff)
  - Postgres insert fails: Log error, try to write event log, continue
  - OpenSearch unreachable: Graceful degradation (semantic works, lexical skipped)
  - Queue message corrupted: Move to dead-letter queue (DLQ)

PERFORMANCE:
  - Synchronous (return to user): < 100ms
  - Embedding (queue worker): 100-500ms
  - Postgres insert: 5-20ms
  - OpenSearch index: 10-50ms
  - Total pipeline: 500-1000ms (can ingest 100/hr, 1-2 per second)
  - Rate limit: 100/hour (1 per 36 seconds)

---

Route 6: POST /memory/rebuild

Purpose: Rebuild all indexes from event log (M2.8)

REQUEST:
  POST http://localhost:8080/memory/rebuild
  Authorization: Bearer <JWT>
  Content-Type: application/json
  
  {
    "project": "poimen",
    "dry_run": false,
    "verify_parity": true
  }

FULL CALL FLOW:
  1. http_server.rs::handle_rebuild()
     ├─> Step 1: JWT Validation
     │   ├─ jwt_validator.rs::validate_token()
     │   ├─ Check "memory:write" capability (requires write permission)
     │   └─ Deny if missing (403)
     │
     ├─> Step 2: Checkpoint Before Rebuild
     │   ├─ vault_projector.rs::compute_vault_hash()
     │   │  ├─ Read all records from Postgres
     │   │  ├─ Sort by sha256
     │   │  ├─ Compute SHA256 of sorted list
     │   │  └─ Get: checksum_before
     │   │
     │   └─ Checksum captured
     │
     ├─> Step 3: Truncate Indexes
     │   ├─ If NOT dry_run:
     │   │  ├─ pgvector_repo.rs::truncate_project(project_id)
     │   │  │  ├─ DELETE FROM memory_vector
     │   │  │  │  WHERE project_id = ?
     │   │  │  │
     │   │  │  ├─ VACUUM (reclaim space)
     │   │  │  └─ Truncate complete
     │   │  │
     │   │  ├─ opensearch_client.rs::delete_index(project_id)
     │   │  │  ├─ DELETE vault-{project_id}
     │   │  │  ├─ Send with JWT Authorization
     │   │  │  └─ Index deleted
     │   │  │
     │   │  └─ Indexes cleared
     │   │
     │   └─ Truncation complete (or skipped if dry_run)
     │
     ├─> Step 4: Replay Event Log
     │   ├─ event_log.rs::read_event_log(project_id)
     │   │  ├─ Read JSONL file from disk (sequential, all events)
     │   │  ├─ Filter for project_id
     │   │  └─ Yield events one by one
     │   │
     │   ├─ For each event:
     │   │  ├─ Deserialize JSON → Record
     │   │  ├─ Validate: project_id, source, text (non-null)
     │   │  │
     │   │  ├─ embeddings.rs::embed_text(text)
     │   │  │  ├─ Call LLM (same model as ingest)
     │   │  │  ├─ Get: embedding [768 floats]
     │   │  │  └─ Deterministic: same text → same embedding
     │   │  │
     │   │  ├─ pgvector_repo.rs::insert_record()
     │   │  │  ├─ INSERT INTO memory_vector (...)
     │   │  │  │  VALUES (project, level, text, embedding, ...)
     │   │  │  │
     │   │  │  └─ Record inserted
     │   │  │
     │   │  ├─ dual_write_indexer.rs::index_to_opensearch()
     │   │  │  ├─ Add to bulk buffer
     │   │  │  ├─ Every 1000 records: flush bulk request
     │   │  │  └─ Index updated
     │   │  │
     │   │  ├─ Track progress:
     │   │  │  ├─ records_processed += 1
     │   │  │  └─ Report every 100 records (can stream to client)
     │   │  │
     │   │  └─ Record complete
     │   │
     │   └─ All events replayed
     │
     ├─> Step 5: Checkpoint After Rebuild
     │   ├─ vault_projector.rs::compute_vault_hash()
     │   │  ├─ Same logic as Step 2, on new data
     │   │  └─ Get: checksum_after
     │   │
     │   └─ Checksum captured
     │
     ├─> Step 6: Parity Verification (M2.8 gate)
     │   ├─ If verify_parity:
     │   │  ├─ Compare checksums:
     │   │  │  ├─ if checksum_before == checksum_after:
     │   │  │  │    status = "pass"
     │   │  │  │  else:
     │   │  │  │    status = "FAIL"
     │   │  │  │
     │   │  │  └─ This detects corruption in rebuild
     │   │  │
     │   │  ├─ If status == "FAIL":
     │   │  │  ├─ Return 422 Unprocessable Entity
     │   │  │  ├─ Include: checksum_before, checksum_after
     │   │  │  └─ Client should NOT retry (indicates log corruption)
     │   │  │
     │   │  └─ Parity verified
     │   │
     │   └─ Parity check complete (or skipped if verify_parity=false)
     │
     ├─> Step 7: Build Response
     │   ├─ Collect stats:
     │   │  ├─ phase: "complete" or "dry_run"
     │   │  ├─ records_processed: count
     │   │  ├─ errors: count of failed records
     │   │  ├─ total_time_ms: elapsed time
     │   │  ├─ checksum_before: hex string
     │   │  ├─ checksum_after: hex string
     │   │  ├─ parity_verified: bool
     │   │  └─ status: "success" or "failed"
     │   │
     │   └─ Return response
     │
     └─> Return 200 OK

RESPONSE: 200 OK
  {
    "phase": "rebuilding",
    "records_processed": 542,
    "errors": 0,
    "total_time_ms": 8234,
    "checksum_before": "abc123def456789abcdef456789abc123def456",
    "checksum_after": "abc123def456789abcdef456789abc123def456",
    "parity_verified": true,
    "status": "success"
  }

DRY RUN MODE:
  If dry_run=true:
    - Skip Step 3 (don't truncate)
    - Replay log but DON'T insert (just count)
    - Report what would happen
    - Useful for validation before destructive rebuild

  Response includes: "phase": "dry_run" (not "rebuilding")

ERROR PATHS:
  - 401 Unauthorized: JWT missing/invalid
  - 403 Forbidden: Missing "memory:write"
  - 422 Unprocessable Entity: Parity check failed
    └─> { "error": "parity_check_failed", "before": "...", "after": "..." }
  - 503 Service Unavailable: Cannot connect to Postgres/OpenSearch
  - 408 Request Timeout: Rebuild takes > 60 seconds (partial results returned)

PERFORMANCE:
  - 500 records: ~5 seconds (1 per 10ms)
  - 5000 records: ~50 seconds
  - Fully deterministic (same result every time)
  - pgvector & OpenSearch stay consistent

---

Route 7: GET /memory/verify

Purpose: Run composition gates to validate system properties (M2.8, M1.8, M3.7, M8.9)

REQUEST:
  GET http://localhost:8080/memory/verify?project=poimen
  Authorization: Bearer <JWT>

FULL CALL FLOW:
  1. http_server.rs::handle_verify()
     ├─> Step 1: JWT Validation
     │   ├─ jwt_validator.rs::validate_token()
     │   ├─ Check "memory:read" capability
     │   └─ Deny if missing (403)
     │
     ├─> Step 2: Run Composition Gates
     │   ├─ verify.rs::run_verification(project_id)
     │   │
     │   ├─ GATE M1.8: Update Rate Baseline
     │   │  ├─ pg_repo.rs::get_evidence_acceptance_rate()
     │   │  │  ├─ Query: SELECT COUNT(*) as total,
     │   │  │  │           COUNT(CASE WHEN accepted=true THEN 1 END) as accepted
     │   │  │  │          FROM lessons
     │   │  │  │          WHERE project_id = ?
     │   │  │  │          AND created_at > now - 90d
     │   │  │  │
     │   │  │  ├─ Compute: rate = accepted / total
     │   │  │  └─ Return: rate (example: 0.75)
     │   │  │
     │   │  ├─ Compare with baseline:
     │   │  │  ├─ baseline = 0.70 (from M1.8)
     │   │  │  ├─ actual = 0.75
     │   │  │  │
     │   │  │  ├─ if actual >= baseline:
     │   │  │  │    status = "pass"
     │   │  │  │  else:
     │   │  │  │    status = "fail"  (regression detected)
     │   │  │  │
     │   │  │  └─ M1.8 complete
     │   │  │
     │   │  └─ Gate M1.8 result: { status, metric, description }
     │   │
     │   │
     │   ├─ GATE M2.8: Rebuild Parity
     │   │  ├─ (Same as /memory/rebuild endpoint)
     │   │  ├─ Checkpoint before
     │   │  ├─ Rebuild from log
     │   │  ├─ Checkpoint after
     │   │  │
     │   │  ├─ if checksum_before == checksum_after:
     │   │  │    status = "pass"
     │   │  │  else:
     │   │  │    status = "fail"
     │   │  │
     │   │  └─ Gate M2.8 result: { status, checksum_before, checksum_after }
     │   │
     │   │
     │   ├─ GATE M3.7: Three-Tier Retrieval
     │   │  ├─ accuracy_metrics.rs::measure_tier_distribution()
     │   │  │  ├─ Sample 100 random queries (from history)
     │   │  │  ├─ For each query, call /memory/context endpoint
     │   │  │  │
     │   │  │  ├─ Record which tier had results:
     │   │  │  │  ├─ tier_1_hits: count
     │   │  │  │  ├─ tier_2_hits: count
     │   │  │  │  └─ tier_3_hits: count
     │   │  │  │
     │   │  │  ├─ Compute rates:
     │   │  │  │  ├─ tier_1_rate = tier_1_hits / 100
     │   │  │  │  ├─ tier_2_recall = 1.0 - (queries_with_no_result / 100)
     │   │  │  │  └─ tier_3_fallback = tier_3_hits / tier_2_misses
     │   │  │  │
     │   │  │  └─ Return: { tier_1_rate, tier_2_recall, tier_3_fallback }
     │   │  │
     │   │  ├─ Check thresholds:
     │   │  │  ├─ tier_1_rate >= 0.80 ?
     │   │  │  │  └─ Gates tells if known issues are being recalled
     │   │  │  │
     │   │  │  ├─ tier_2_recall >= 0.50 ?
     │   │  │  │  └─ Gates tells if novel issues are found (at least half)
     │   │  │  │
     │   │  │  └─ tier_3_fallback <= 0.10 ?
     │   │  │     └─ Gates tells reference docs don't dominate (<=10%)
     │   │  │
     │   │  ├─ status = all thresholds pass ? "pass" : "fail"
     │   │  │
     │   │  └─ Gate M3.7 result: { status, tier_breakdown, thresholds_met }
     │   │
     │   │
     │   ├─ GATE M3.6: Reference Cycle Guard
     │   │  ├─ reference_cycle_guard.rs::test_cycle_detection()
     │   │  │  ├─ Take 10 random R (reference) chunks
     │   │  │  ├─ For each: try to find in recent transcripts
     │   │  │  ├─ If found as evidence (not marked derived):
     │   │  │  │    re_entry_count += 1
     │   │  │  │
     │   │  │  └─ Return: re_entry_count
     │   │  │
     │   │  ├─ Check threshold:
     │   │  │  ├─ if re_entry_count == 0:
     │   │  │  │    status = "pass"
     │   │  │  │  else:
     │   │  │  │    status = "fail"  (guard not working)
     │   │  │  │
     │   │  │  └─ Gate M3.6 result: { status, re_entries }
     │   │
     │   │
     │   ├─ GATE M8.9: Hybrid Search Accuracy (NDCG)
     │   │  ├─ accuracy_metrics.rs::compute_ndcg()
     │   │  │  ├─ Collect all queries from history (last 7 days)
     │   │  │  ├─ For each: rank results by score
     │   │  │  ├─ Judge relevance (ground truth):
     │   │  │  │  ├─ Perfect match: relevance = 1.0
     │   │  │  │  ├─ Related: relevance = 0.7
     │   │  │  │  └─ Unrelated: relevance = 0.0
     │   │  │  │
     │   │  │  ├─ Compute NDCG@10:
     │   │  │  │  ├─ DCG = sum of (relevance / log(position + 1))
     │   │  │  │  ├─ IDCG = ideal DCG (all perfect at top)
     │   │  │  │  └─ NDCG = DCG / IDCG (normalized 0-1)
     │   │  │  │
     │   │  │  └─ Return: ndcg_score (example: 0.88)
     │   │  │
     │   │  ├─ Check threshold:
     │   │  │  ├─ baseline = 0.85 (from M8.9)
     │   │  │  ├─ if ndcg >= baseline:
     │   │  │  │    status = "pass"
     │   │  │  │  else:
     │   │  │  │    status = "fail"
     │   │  │  │
     │   │  │  └─ Gate M8.9 result: { status, ndcg_score, baseline }
     │   │
     │   │
     │   └─ All gates complete
     │
     ├─> Step 3: Aggregate Results
     │   ├─ overall_status = "healthy" if all gates pass
     │   ├─ overall_status = "degraded" if some gates fail
     │   └─ overall_status = "critical" if key gates fail (M2.8, M3.7)
     │
     └─> Return 200 OK (or 422 if critical failures)

RESPONSE: 200 OK
  {
    "project": "poimen",
    "overall_status": "healthy",
    "checks": [
      {
        "gate": "M1.8_update_rate",
        "status": "pass",
        "metric": "0.75 >= 0.70",
        "description": "Evidence acceptance rate at baseline"
      },
      {
        "gate": "M2.8_rebuild_parity",
        "status": "pass",
        "metric": "checksum match after rebuild",
        "details": "abc123def456..."
      },
      {
        "gate": "M3.7_tier_retrieval",
        "status": "pass",
        "metric": "Tier-1 hit rate: 0.82 >= 0.80",
        "tier_breakdown": {
          "tier_1": 82,
          "tier_2": 14,
          "tier_3": 4
        }
      },
      {
        "gate": "M3.6_reference_cycle_guard",
        "status": "pass",
        "metric": "Zero re-entries detected",
        "re_entries": 0
      },
      {
        "gate": "M8.9_hybrid_search_accuracy",
        "status": "pass",
        "metric": "NDCG@10: 0.88 >= 0.85",
        "ndcg_score": 0.88,
        "baseline": 0.85
      }
    ],
    "timestamp": "2025-01-29T10:00:00Z"
  }

ERROR RESPONSE: 422 Unprocessable Entity (critical failure)
  {
    "project": "poimen",
    "overall_status": "critical",
    "failed_gates": ["M2.8_rebuild_parity"],
    "reason": "Rebuild parity check failed - indexes may be corrupted"
  }

PERFORMANCE:
  - M1.8: 5ms (database query)
  - M2.8: 5-10 seconds (full rebuild)
  - M3.7: 5-10 seconds (100 sample queries)
  - M3.6: 1 second (10 sample checks)
  - M8.9: 2-5 seconds (historical query analysis)
  - Total: 15-30 seconds

---

Authorization & Authentication

JWT Flow

Client
  │
  ├─ Get token from Authentik:
  │  POST https://authentik.riotpiao.com/application/o/token/
  │  
  │  Response: { "access_token": "eyJ0eXAi...", "expires_in": 3600 }
  │
  └─ Use token in API calls:
     GET /memory/vault
     Authorization: Bearer eyJ0eXAi...

Token Validation (per-endpoint)

1. Extract JWT from Authorization header (Bearer <token>)
2. jwt_validator.rs::validate_token()
   ├─ Split token: [header, payload, signature]
   ├─ Verify signature:
   │  ├─ Get JWKS from Authentik (cached 1hr)
   │  ├─ Find key matching "kid" in token header
   │  └─ Validate RS256 signature
   ├─ Decode payload (base64)
   ├─ Check claims:
   │  ├─ "iss" (issuer) matches config
   │  ├─ "aud" (audience) = "poimen-memory"
   │  ├─ "exp" (expiry) > now
   │  └─ "sub" (subject) present
   └─ Return: { user, roles, permissions }

3. Check capability (per-endpoint):
   ├─ GET /memory/query → needs "memory:read"
   ├─ POST /memory/ingest → needs "memory:write"
   └─ Other endpoints → needs "memory:read" or "memory:write"

4. If any check fails → 401 or 403

Rate Limiting

rate_limiter.rs::check_limit(apikey, endpoint)
  │
  ├─ Get token bucket state:
  │  ├─ Key: "{apikey}:{endpoint}"
  │  ├─ Bucket: { tokens: N, last_refill: timestamp }
  │  │
  │  └─ Limits:
  │     ├─ ingest: 100/hour (1 per 36 seconds)
  │     ├─ query: 1000/hour (1 per 3.6 seconds)
  │     └─ projects (context): 100/hour
  │
  ├─ Refill tokens:
  │  └─ tokens += (now - last_refill) * (limit / 3600 seconds)
  │
  ├─ Check available:
  │  ├─ if tokens >= 1:
  │  │    tokens -= 1
  │  │    return OK
  │  │  else:
  │  │    return 429 Too Many Requests
  │  │    ├─ Retry-After: (seconds until next token)
  │  │    └─ X-RateLimit-Remaining: 0
  │  │
  │  └─ Update last_refill
  │
  └─ Return: OK or 429

Idempotency

idempotency.rs::is_duplicate(key)
  │
  ├─ Key format: "{project}:{source}:{sha256(text)}"
  │
  ├─ Query: SELECT result FROM idempotency_store
  │          WHERE key = ?
  │          AND created_at > now - 24 hours
  │
  ├─ If found:
  │  └─ Return 409 Conflict with cached response
  │
  ├─ If not found:
  │  ├─ Process request normally
  │  └─ On success: store(key, result, ttl=24h)
  │
  └─ Prevents duplicate ingest processing

---

Error Handling & Fallbacks

Graceful Degradation

Query Path:
  ├─ Normal (both engines available):
  │  └─ Hybrid search (60% semantic + 40% lexical)
  │
  ├─ Semantic (pgvector) available, Lexical (OpenSearch) DOWN:
  │  ├─ Fall back to semantic-only
  │  ├─ Log warning
  │  └─ Return results with "degraded": true flag
  │
  ├─ Lexical (OpenSearch) available, Semantic (pgvector) DOWN:
  │  ├─ Fall back to lexical-only
  │  ├─ Log critical (semantic is primary)
  │  └─ Return results with "degraded": true flag
  │
  └─ Both DOWN:
     └─ Return 503 Service Unavailable

Retry Logic

OpenSearch Index (dual-write, fail-soft):
  ├─ Send bulk request
  │
  ├─ On timeout (> 5s):
  │  ├─ Retry with exponential backoff: 1s, 2s, 4s, 8s
  │  ├─ Max 4 retries (total 15 seconds)
  │  └─ If all fail: continue (OpenSearch was optional anyway)
  │
  └─ Log: "lexical_index_delayed" or "lexical_index_failed"

Embedding (critical path):
  ├─ Call LLM service
  │
  ├─ On timeout:
  │  ├─ Retry up to 3 times
  │  ├─ If all fail: reject ingest (400 or 503)
  │  └─ Embedding is not optional
  │
  └─ Log: "embedding_service_error"

Timeout Handling

/memory/query:
  ├─ Set timeout: 10 seconds (hard limit)
  │
  ├─ Semantic search timeout:
  │  ├─ Kill pgvector query at 5 seconds
  │  └─ Return partial results
  │
  ├─ Lexical search timeout:
  │  ├─ Kill OpenSearch query at 5 seconds
  │  └─ Return partial results from other engine
  │
  └─ Overall timeout:
     └─ Return 504 Gateway Timeout with best-effort results

/memory/context:
  ├─ Set timeout: 15 seconds (need time for Tier-3)
  │
  ├─ Tier-1 (signature): must complete (5ms)
  ├─ Tier-2 (hybrid): must complete (150ms)
  └─ Tier-3 (Obsidian): best-effort, drop if timeout (> 5s)
     └─ Return Tier-1 & Tier-2 only if Obsidian times out

---

Performance Characteristics

Latency (p95)

Endpoint	Operation	Latency
/health	DB check	5ms
/memory/vault	List files	50ms
/memory/query	Semantic only	100ms
/memory/query	Lexical only	80ms
/memory/query	Hybrid	150ms
/memory/context	Tier-1 only	10ms
/memory/context	Tier-1 + Tier-2	150ms
/memory/context	All tiers	500ms
/memory/ingest	Queue + return	50ms
/memory/rebuild	1000 records	10s
/memory/verify	All gates	30s

Throughput

Endpoint	Rate Limit	Per Second
Query	1000/hour	~0.3 req/s
Ingest	100/hour	~0.03 req/s
Context	100/hour	~0.03 req/s
Vault	—	~5 req/s

Storage

Component	Size	Scaling
pgvector index	768 floats × N records	~8KB per record
OpenSearch index	Full text × N records	~1KB per record
Event log (JSONL)	~2KB per record	10GB per 5M records
PVC (vault files)	~10GB	Grows with documentation

---

System Architecture

Component Interaction

User/Agent
  │
  ├─ HTTP/JSON
  │
  ↓
┌─────────────────────────────────────────┐
│ Memory Service Pod (Actix-web)          │
│ ├─ HTTP handlers (request routing)      │
│ ├─ JWT validation (authorize)           │
│ ├─ Rate limiting (throttle)             │
│ ├─ Hybrid search orchestration          │
│ └─ Three-tier context retrieval         │
└─────────────────────────────────────────┘
   │           │              │
   ├── (SQL) ──┤──────────────┤── (REST)
   │           │              │
   ↓           ↓              ↓
┌──────────┐ ┌──────────┐ ┌──────────────┐
│PostgreSQL│ │OpenSearch│ │ Obsidian API │
│(pgvector)│ │(BM25)    │ │(Reference)   │
└──────────┘ └──────────┘ └──────────────┘
   ↑           ↑
   └── (OIDC Token Validation)
        ↑
        │
    [Authentik JWKS]
        │
      (Cache: 1hr)

Data Flow

Ingest:
  Input → Queue → Worker → Embed → [Postgres + OpenSearch]

Query:
  Input → JWT Validate → Rate Limit → Classify → 
  [Semantic] [Lexical] (parallel) → Normalize → Fuse → Return

Context:
  Input → JWT Validate → Rate Limit → 
  [Tier-1: Signature] [Tier-2: Hybrid] [Tier-3: Obsidian] (sequential) →
  Budget-aware assemble → Return

Rebuild:
  Event Log → Replay → Embed → [Postgres + OpenSearch] → Verify Parity

---

Conclusion

The Poimen Memory system provides comprehensive API coverage across:

• ✅ Health & Monitoring: Status checks, composition gates
• ✅ Retrieval: Hybrid semantic + lexical, three-tier context
• ✅ Ingest: Async queue-based processing, idempotency
• ✅ Authz: JWT/OIDC, capability-based, per-endpoint
• ✅ Resilience: Graceful degradation, retry logic, timeouts
• ✅ Performance: 50-150ms typical queries, 1000/hr throughput

All routes fully documented with call flows, error paths, and performance metrics.