poimen-memory/docs/DEPLOYMENT_CHECKLIST.md

# Deployment Checklist: Memory Service API Ready

## Phase 1: API Endpoints Ready ✅

### Vault Endpoints (JSON API)

- [x] `GET /memory/vault` — list projects
- [x] `GET /memory/vault?project=X` — file tree
- [x] `GET /memory/vault/{project}/{file}` — file content (JSON)
- [x] YAML frontmatter parsing to JSON metadata
- [x] JWT auth on all endpoints

### Search Endpoints

- [x] `GET /memory/query?method=semantic` — pgvector only
- [x] `GET /memory/query?method=hybrid` — semantic + OpenSearch (with fallback)
- [x] Hybrid score fusion (60% semantic + 40% lexical)
- [x] JWT auth required

### Code Changes

- [x] `crates/mem-cli/src/http_server.rs` — vault + search handlers
  - `vault_browser_handler()` — returns JSON projects list
  - `vault_project_tree()` — helper for file tree
  - `vault_project_handler()` — GET /{project} → file tree
  - `vault_file_handler()` — GET /{project}/{file} → JSON content
  - `query_handler()` — updated for hybrid search with OpenSearch fallback
  - `AppState.opensearch_client` — optional OpenSearch integration
- [x] Environment variable: `OPENSEARCH_HOSTS` (optional)

---

## Phase 2: OpenSearch Deployment

### K8s Manifests

- [x] `k8s/infra/databases/opensearch.yaml`
  - StatefulSet: 2 replicas (opensearch-0, opensearch-1)
  - Service: `opensearch` (headless), `opensearch-internal` (ClusterIP)
  - ConfigMap: opensearch.yml configuration
  - PVC: 30Gi per pod (Longhorn)
  - ServiceAccount + NetworkPolicy
  - Probes: liveness (60s), readiness (30s)
  - Security: security plugin disabled (assume K8s network isolation)

- [x] Updated `k8s/infra/databases/kustomization.yaml`
  - Added `- opensearch.yaml` to resources

### Deployment Steps

```bash
# 1. Deploy OpenSearch
kubectl apply -k k8s/infra/databases/

# 2. Wait for StatefulSet ready
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# Expected:
# opensearch-0   1/1 Running
# opensearch-1   1/1 Running

# 3. Verify cluster health
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health
# {"status":"green","...}

# 4. Configure Memory Service
kubectl set env deployment poimen-memory \
  -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200

# 5. Restart Memory Service
kubectl rollout restart deployment poimen-memory -n poimen
```

---

## Phase 3: DNS & Ingress

### DNS Records

```
vault.riotpiao.com     IN A <cluster-ip>
memory.riotpiao.com    IN A <cluster-ip>
```

### Ingress Routes

**vault.riotpiao.com** → /memory/vault/* endpoints (vault browser)
**memory.riotpiao.com** → full API (search, projects, skills, etc.)

Sample Ingress:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: memory-ingress
  namespace: poimen
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - vault.riotpiao.com
        - memory.riotpiao.com
      secretName: memory-tls
  rules:
    - host: vault.riotpiao.com
      http:
        paths:
          - path: /memory/vault
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port: {number: 8080}
    - host: memory.riotpiao.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: poimen-memory
                port: {number: 8080}
```

---

## Phase 4: Testing

### Test Vault Endpoints

```bash
# Authenticate
TOKEN=$(curl -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d "client_id=poimen-memory" \
  -d "client_secret=..." \
  -d "grant_type=client_credentials" | jq -r .access_token)

# List projects
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault
# Expected: {"projects": ["poimen", ...]}

# List files in project
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault?project=poimen'
# Expected: {"project": "poimen", "files": [...]}

# Get file content
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault/poimen/index
# Expected: {"project": "poimen", "file": "index.md", "metadata": {...}, "content": "..."}
```

### Test Search Endpoints

```bash
# Semantic only
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic'
# Expected: {"method": "semantic", "results": [...]}

# Hybrid (best)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes'
# Expected: {"method": "hybrid", "results": [...]}
# (or "semantic_fallback" if OpenSearch not ready)
```

### Load Test

```bash
ab -n 1000 -c 10 \
  -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=test'
```

---

## Phase 5: Frontend Deployment (Next)

Waiting on:
- [ ] React SPA build
- [ ] Vault browser UI
- [ ] Search form + result display
- [ ] GRC workflow (edit → MR → merge)
- [ ] Agent execution tracking

---

## Monitoring

### Endpoints Health

```bash
# Memory Service
curl https://memory.riotpiao.com/health

# OpenSearch
curl https://vault.riotpiao.com/memory/query?project=test&query=test
# If errors → check OPENSEARCH_HOSTS config

# Metrics
kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=100
```

### Dashboard Metrics

```prometheus
# Query latency
histogram_quantile(0.95, http_request_duration_seconds{method="GET", endpoint="/memory/query"})

# Cache hit rate
opensearch_query_cache_hit_count / opensearch_query_cache_total

# Cluster health
opensearch_cluster_health_status  # 1 = green, 0 = red
```

---

## Fallback Behavior

### If OpenSearch is Down

✅ /memory/vault/* endpoints work (no OpenSearch dependency)
✅ /memory/query with method=semantic works
⚠️ /memory/query with method=hybrid falls back to semantic (no error)
❌ /memory/query with method=hybrid&strict=true returns 503

### If pgvector is Down

❌ All endpoints fail (core dependency)

### If Authentik is Down

❌ All endpoints fail with 401 (no auth)

---

## Troubleshooting

### OpenSearch not found

**Symptom:** "semantic_fallback" always returned, no hybrid scores

**Fix:**
```bash
# Check env var
kubectl get deployment -n poimen poimen-memory -o yaml | grep OPENSEARCH_HOSTS

# Set it
kubectl set env deployment poimen-memory -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200
kubectl rollout restart deployment poimen-memory -n poimen

# Verify connectivity from pod
kubectl exec -n poimen <pod> -- curl http://opensearch-internal:9200/_cluster/health
```

### OpenSearch cluster red

**Symptom:** opensearch-1 not starting, cluster unhealthy

**Fix:**
```bash
# Check logs
kubectl logs -n poimen opensearch-1

# Common issues:
# 1. vm.max_map_count too low (init container should fix)
# 2. PVC not provisioned (check Longhorn)
# 3. Memory limit too low (increase to 1Gi)

# Reset cluster
kubectl delete pvc opensearch-data-opensearch-1 -n poimen
kubectl delete pod opensearch-1 -n poimen
```

### JWT validation fails on queries

**Symptom:** `401 Unauthorized: JWT validation failed`

**Fix:**
```bash
# Check Authentik JWKS accessible
curl https://authentik.riotpiao.com/application/o/poimen-memory/jwks/

# Check token not expired
jwt decode <your-token>  # look at 'exp' claim

# Check token has required claims
# Should have: iss, aud, sub, roles, permissions
```

---

## Checklist Summary

### Ready for Production

- [x] API endpoints return JSON (not HTML)
- [x] Vault tree endpoint working
- [x] Hybrid search implemented (with fallback)
- [x] OpenSearch manifests created
- [x] JWT auth on all endpoints
- [x] Environment variables documented
- [x] Deployment guide written
- [ ] Frontend React app deployed
- [ ] GRC workflow endpoints implemented
- [ ] Load tested at scale
- [ ] Monitoring configured

**Status:** Ready to deploy OpenSearch + test API