poimen-memory/docs/DEPLOYMENT_CHECKLIST.md

Deployment Checklist: Memory Service API Ready

Phase 1: API Endpoints Ready ✅

Vault Endpoints (JSON API)

• GET /memory/vault — list projects
• GET /memory/vault?project=X — file tree
• GET /memory/vault/{project}/{file} — file content (JSON)
• YAML frontmatter parsing to JSON metadata
• JWT auth on all endpoints

Search Endpoints

• GET /memory/query?method=semantic — pgvector only
• GET /memory/query?method=hybrid — semantic + OpenSearch (with fallback)
• Hybrid score fusion (60% semantic + 40% lexical)
• JWT auth required

Code Changes

• crates/mem-cli/src/http_server.rs — vault + search handlers
  • vault_browser_handler() — returns JSON projects list
  • vault_project_tree() — helper for file tree
  • vault_project_handler() — GET /{project} → file tree
  • vault_file_handler() — GET /{project}/{file} → JSON content
  • query_handler() — updated for hybrid search with OpenSearch fallback
  • AppState.opensearch_client — optional OpenSearch integration
• Environment variable: OPENSEARCH_HOSTS (optional)

---

Phase 2: OpenSearch Deployment

K8s Manifests

• k8s/infra/databases/opensearch.yaml

  • StatefulSet: 2 replicas (opensearch-0, opensearch-1)
  • Service: opensearch (headless), opensearch-internal (ClusterIP)
  • ConfigMap: opensearch.yml configuration
  • PVC: 30Gi per pod (Longhorn)
  • ServiceAccount + NetworkPolicy
  • Probes: liveness (60s), readiness (30s)
  • Security: security plugin disabled (assume K8s network isolation)

• Updated k8s/infra/databases/kustomization.yaml

  • Added - opensearch.yaml to resources

Deployment Steps

# 1. Deploy OpenSearch
kubectl apply -k k8s/infra/databases/

# 2. Wait for StatefulSet ready
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# Expected:
# opensearch-0   1/1 Running
# opensearch-1   1/1 Running

# 3. Verify cluster health
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health
# {"status":"green","...}

# 4. Configure Memory Service
kubectl set env deployment poimen-memory \
  -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200

# 5. Restart Memory Service
kubectl rollout restart deployment poimen-memory -n poimen

---

Phase 3: DNS & Ingress

DNS Records

vault.riotpiao.com     IN A <cluster-ip>
memory.riotpiao.com    IN A <cluster-ip>

Ingress Routes

vault.riotpiao.com → /memory/vault/* endpoints (vault browser) memory.riotpiao.com → full API (search, projects, skills, etc.)

Sample Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: memory-ingress
  namespace: poimen
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - vault.riotpiao.com
        - memory.riotpiao.com
      secretName: memory-tls
  rules:
    - host: vault.riotpiao.com
      http:
        paths:
          - path: /memory/vault
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port: {number: 8080}
    - host: memory.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port: {number: 8080}

---

Phase 4: Testing

Test Vault Endpoints

# Authenticate
TOKEN=$(curl -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d "client_id=poimen-memory" \
  -d "client_secret=..." \
  -d "grant_type=client_credentials" | jq -r .access_token)

# List projects
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault
# Expected: {"projects": ["poimen", ...]}

# List files in project
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault?project=poimen'
# Expected: {"project": "poimen", "files": [...]}

# Get file content
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault/poimen/index
# Expected: {"project": "poimen", "file": "index.md", "metadata": {...}, "content": "..."}

Test Search Endpoints

# Semantic only
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic'
# Expected: {"method": "semantic", "results": [...]}

# Hybrid (best)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes'
# Expected: {"method": "hybrid", "results": [...]}
# (or "semantic_fallback" if OpenSearch not ready)

Load Test

ab -n 1000 -c 10 \
  -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=test'

---

Phase 5: Frontend Deployment (Next)

Waiting on:

• React SPA build
• Vault browser UI
• Search form + result display
• GRC workflow (edit → MR → merge)
• Agent execution tracking

---

Monitoring

Endpoints Health

# Memory Service
curl https://memory.riotpiao.com/health

# OpenSearch
curl https://vault.riotpiao.com/memory/query?project=test&query=test
# If errors → check OPENSEARCH_HOSTS config

# Metrics
kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=100

Dashboard Metrics

# Query latency
histogram_quantile(0.95, http_request_duration_seconds{method="GET", endpoint="/memory/query"})

# Cache hit rate
opensearch_query_cache_hit_count / opensearch_query_cache_total

# Cluster health
opensearch_cluster_health_status  # 1 = green, 0 = red

---

Fallback Behavior

If OpenSearch is Down

✅ /memory/vault/* endpoints work (no OpenSearch dependency) ✅ /memory/query with method=semantic works ⚠️ /memory/query with method=hybrid falls back to semantic (no error) ❌ /memory/query with method=hybrid&strict=true returns 503

If pgvector is Down

❌ All endpoints fail (core dependency)

If Authentik is Down

❌ All endpoints fail with 401 (no auth)

---

Troubleshooting

OpenSearch not found

Symptom: "semantic_fallback" always returned, no hybrid scores

Fix:

# Check env var
kubectl get deployment -n poimen poimen-memory -o yaml | grep OPENSEARCH_HOSTS

# Set it
kubectl set env deployment poimen-memory -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200
kubectl rollout restart deployment poimen-memory -n poimen

# Verify connectivity from pod
kubectl exec -n poimen <pod> -- curl http://opensearch-internal:9200/_cluster/health

OpenSearch cluster red

Symptom: opensearch-1 not starting, cluster unhealthy

Fix:

# Check logs
kubectl logs -n poimen opensearch-1

# Common issues:
# 1. vm.max_map_count too low (init container should fix)
# 2. PVC not provisioned (check Longhorn)
# 3. Memory limit too low (increase to 1Gi)

# Reset cluster
kubectl delete pvc opensearch-data-opensearch-1 -n poimen
kubectl delete pod opensearch-1 -n poimen

JWT validation fails on queries

Symptom: 401 Unauthorized: JWT validation failed

Fix:

# Check Authentik JWKS accessible
curl https://authentik.riotpiao.com/application/o/poimen-memory/jwks/

# Check token not expired
jwt decode <your-token>  # look at 'exp' claim

# Check token has required claims
# Should have: iss, aud, sub, roles, permissions

---

Checklist Summary

Ready for Production

• API endpoints return JSON (not HTML)
• Vault tree endpoint working
• Hybrid search implemented (with fallback)
• OpenSearch manifests created
• JWT auth on all endpoints
• Environment variables documented
• Deployment guide written
• Frontend React app deployed
• GRC workflow endpoints implemented
• Load tested at scale
• Monitoring configured

Status: Ready to deploy OpenSearch + test API