poimen-memory/docs/OPENSEARCH_DEPLOYMENT_GUIDE.md

OpenSearch + Dashboards Deployment Guide

Status: ✅ DEPLOYED

OpenSearch cluster + Dashboards UI are now running on K8s cluster poimen namespace.

Deployment Command (already run):

kubectl apply -k k8s/infra/databases/

Commit: 630a125 — deploy: OpenSearch + Dashboards StatefulSet

---

📊 What's Running

OpenSearch Cluster (2-node HA)

opensearch-0                  1/1 Running   ← Primary node
opensearch-1                  1/1 Running   ← Replica node
opensearch-internal:9200      Ready         ← API endpoint (Memory Service connects here)

Configuration:

• Image: opensearchproject/opensearch:2.11.0
• Storage: 30Gi per pod (Longhorn)
• Resources: 512Mi-1Gi memory, 250m-500m CPU each
• Network: K8s internal only (NetworkPolicy restricts access)
• Security: plugins.security.disabled (secured by K8s network)

OpenSearch Dashboards (UI)

opensearch-dashboards-*      1/1 Running   ← Dashboard pod
opensearch-dashboards:5601   Ready         ← Web UI (port-forward for local access)

Configuration:

• Image: opensearchproject/opensearch-dashboards:2.11.0
• Login: admin / admin (⚠️ change in production)
• Memory: 256Mi-512Mi
• CPU: 100m-500m

---

🚀 Next Steps (Quick Start)

1. Verify OpenSearch Cluster is Healthy

# Port-forward to OpenSearch API
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &

# Check cluster status
curl http://localhost:9200/_cluster/health

Expected Output:

{
  "cluster_name": "poimen-memory",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "active_primary_shards": 0,
  "active_shards": 0,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}

✅ green = cluster healthy ⚠️ yellow = some replicas unavailable (wait 1-2 min) ❌ red = cluster unhealthy (check pod logs)

2. Access Dashboards UI (Local Development)

# Port-forward to Dashboards
kubectl port-forward -n poimen svc/opensearch-dashboards 5601:5601 &

# Open in browser
open http://localhost:5601
# or
firefox http://localhost:5601

Login:

• Username: admin
• Password: admin

First Time Setup:

1. Dashboards auto-creates .opensearch_dashboards index
2. Accept default settings
3. Explore → Dev Tools → Console (for manual BM25 queries)

3. Configure Memory Service to Use OpenSearch

# Set environment variable
kubectl set env deployment poimen-memory -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200

# Restart Memory Service pods
kubectl rollout restart deployment poimen-memory -n poimen

# Wait for rollout
kubectl rollout status deployment poimen-memory -n poimen

Verify Connection:

# Check Memory Service logs
kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=50 | grep -i opensearch
# Should see: "Initialized OpenSearch client: opensearch-internal.poimen.svc.cluster.local:9200"

4. Test Vault Endpoints

# Get JWT token from Authentik
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d "client_id=poimen-memory" \
  -d "client_secret=$AUTHENTIK_SECRET" \
  -d "grant_type=client_credentials" | jq -r .access_token)

# List projects
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault

# List files in project
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault?project=poimen'

# Get specific file
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault/poimen/index' | jq .

Expected:

• 200 OK with JSON response
• If 401: check JWT token is valid
• If 403: check JWT has required scopes

5. Test Hybrid Search

# Semantic only (always works)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic'

# Hybrid search (now with OpenSearch)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes'

# Force strict hybrid (fail if OpenSearch down)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=hybrid&strict=true'

Expected Responses:

✅ Hybrid (Fallback to Semantic if OpenSearch unavailable):

{
  "method": "hybrid",
  "query": "kubernetes",
  "project": "poimen",
  "results": [
    {
      "level": "L1",
      "score": 0.992,
      "text": "...",
      "source": "claude",
      "provenance": ["pi-1"]
    }
  ]
}

⚠️ Semantic Fallback (if OpenSearch down):

{
  "method": "semantic_fallback",
  "query": "kubernetes",
  "project": "poimen",
  "results": [...]
}

---

🛠️ Operations

Monitor Cluster Health

# Watch pod status
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# Check OpenSearch logs
kubectl logs -n poimen opensearch-0 --tail=100
kubectl logs -n poimen opensearch-1 --tail=100

# Check Dashboards logs
kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50

Common Issues

❌ Pods not starting

Symptom: Pending or CrashLoopBackOff

Check:

kubectl describe pod -n poimen opensearch-0

Common Causes:

1. vm.max_map_count too low → Init container should fix (wait 30s)
2. PVC not provisioned → Check Longhorn: kubectl get pvc -n poimen
3. Memory limit exceeded → Increase limits.memory in StatefulSet
4. Node affinity → Check node labels: kubectl get nodes --show-labels

Fix:

# Force pod recreation
kubectl delete pod -n poimen opensearch-0
# Scheduler will restart it

# Check logs after restart
kubectl logs -n poimen opensearch-0 --tail=100

❌ Cluster status = yellow

Symptom: Only 1 node showing, shards unassigned

Cause: Waiting for second node to start (normal during initial deployment)

Fix: Wait 1-2 minutes

# Watch until green
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

❌ Cluster status = red

Symptom: Cluster health = red, both nodes showing but unhealthy

Debug:

# Check node logs
kubectl logs -n poimen opensearch-0 --tail=200 | grep -i error

# Check if nodes can communicate
kubectl exec -n poimen opensearch-0 -- curl http://opensearch-1.opensearch.poimen.svc.cluster.local:9300/

Common Causes:

• Network Policy blocking communication
• Disk/memory pressure
• JVM out of memory

Recovery:

# Reset cluster (deletes data, be careful in production!)
kubectl delete pvc opensearch-data-opensearch-0 opensearch-data-opensearch-1 -n poimen
kubectl delete pod opensearch-0 opensearch-1 -n poimen
# Wait ~3 minutes for recovery

❌ Dashboards can't connect to OpenSearch

Symptom: Dashboards UI shows "Cannot connect to Elasticsearch"

Check:

kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50

Fix:

# Verify OpenSearch is healthy
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health

# Check Dashboards config
kubectl describe cm opensearch-dashboards-config -n poimen
# Should show: opensearch.hosts = ["http://opensearch-internal.poimen.svc.cluster.local:9200"]

# Restart Dashboards
kubectl rollout restart deployment opensearch-dashboards -n poimen

Backup & Recovery

Export OpenSearch Indices

# List all indices
curl http://localhost:9200/_cat/indices

# Snapshot creation (requires S3/backup config)
# Docs: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-resilience/snapshots/snapshot-restore/

Restore from Backup

# Restore index from snapshot
curl -X POST http://localhost:9200/_snapshot/backup/snapshot-1/_restore

---

📈 Performance Tuning

OpenSearch JVM Memory

Current: 512Mi-1Gi per pod

For larger datasets (>1M vectors):

# Edit StatefulSet
kubectl edit statefulset opensearch -n poimen

# Update resources:
resources:
  requests:
    memory: "1Gi"      # was 512Mi
  limits:
    memory: "2Gi"      # was 1Gi

OpenSearch Shard Configuration

Current: Auto-configured by Dashboards

Optimize for hybrid search (many small shards):

# After indices are created
curl -X PUT http://localhost:9200/vault-poimen/_settings -d '{
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "codec": "best_compression",
    "refresh_interval": "30s"
  }
}'

IVFFlat Index Parameters (pgvector side)

-- Optimize for 1M+ vectors
CREATE INDEX ON memory_vector
USING ivfflat (embedding vector_cosine_ops)
WITH (lists=1000);  -- sqrt(1000000) ≈ 1000

-- Current: lists=100 (good for <100k vectors)

---

🔐 Security (Production Checklist)

⚠️ TODO: Production Security

• Change Dashboards password in secret:

  kubectl patch secret opensearch-dashboards-secret -n poimen \
    -p '{"stringData":{"password":"YOUR_SECURE_PASSWORD"}}'
  kubectl rollout restart deployment opensearch-dashboards -n poimen
  

• Enable OpenSearch security plugin:

  # In opensearch.yml ConfigMap
  plugins.security.disabled: "false"
  plugins.security.ssl.http.enabled: "true"
  plugins.security.ssl.http.keystore_filepath: "/usr/share/opensearch/config/certs/keystore.jks"
  # ... (requires cert generation)
  

• Setup Dashboards OAuth2/SAML:

  # opensearch_dashboards.yml
  opensearch.username: null  # Remove hardcoded auth
  opensearch.password: null
  xpack.security.auth.providers: ["saml"]
  # ... (requires IdP config)
  

• Setup NetworkPolicy for Ingress access (if exposing publicly)

• Enable audit logging:

  # opensearch.yml
  plugins.security.audit.type: internal_opensearch
  plugins.security.audit.config.http_endpoints: ["opensearch:9200"]
  

---

📚 Useful Commands

# Cluster status
curl http://localhost:9200/_cluster/health | jq .

# List indices
curl http://localhost:9200/_cat/indices | jq .

# Index stats
curl http://localhost:9200/_stats | jq .indices

# Node info
curl http://localhost:9200/_nodes | jq '.nodes | length'

# Clear query cache
curl -X POST http://localhost:9200/_cache/clear

# Force merge indices (maintenance)
curl -X POST http://localhost:9200/vault-*/_forcemerge?max_num_segments=1

# Pod resource usage
kubectl top pods -n poimen -l app.kubernetes.io/name=opensearch

# PVC usage
kubectl exec -n poimen opensearch-0 -- df -h /usr/share/opensearch/data

---

📞 Support

Check Status Anytime

# Everything OK?
kubectl get pods -n poimen -l "app.kubernetes.io/name in (opensearch, opensearch-dashboards)"

# Cluster green?
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health | jq .status
# Should show: "green"

Logs

# Real-time OpenSearch logs
kubectl logs -n poimen opensearch-0 -f

# Real-time Dashboards logs
kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards -f

# Search for errors
kubectl logs -n poimen opensearch-0 | grep -i error | tail -20

---

🎯 Integration with Memory Service

Architecture

        User Request (JWT)
                 ↓
   ┌────────────────────────────────┐
   │ poimen-memory Pod              │
   │ (2 replicas)                   │
   ├────────────────────────────────┤
   │ GET /memory/query              │
   │   ├─ Query pgvector (60%)      │
   │   └─ Query OpenSearch (40%)    │
   │       (BM25 full-text search)  │
   │   └─ Fusion & rerank           │
   └──────────┬──────────────────┬──┘
              │                  │
              ↓                  ↓
       pgvector(Postgres)  OpenSearch Cluster
       (768-dim embed)     (2 nodes, HA)
       IVFFlat index       BM25 indices

Query Flow

1. User sends: GET /memory/query?project=X&query=Y
2. Memory Service receives JWT, validates scopes
3. Parallel queries:
   • pgvector: "SELECT ... ORDER BY embedding <-> query_embedding LIMIT 50"
   • OpenSearch: "POST vault-X/_search" with BM25 query
4. Fusion: Combine top-50 results from each, score: 0.6*sem + 0.4*lex
5. Return: Top-10 merged results with method indicator (hybrid / semantic_fallback / semantic)

Graceful Degradation

• ✅ OpenSearch healthy: Hybrid search (60% + 40% fusion)
• ⚠️ OpenSearch slow: Timeout → fallback to semantic only
• ⚠️ OpenSearch down: Fallback to semantic only (no error)
• ❌ pgvector down: All queries fail (core dependency)

---

📋 Deployment Checklist Summary

Phase 1: ✅ OpenSearch Deployed

• StatefulSet: 2 replicas
• Services: opensearch, opensearch-internal
• ConfigMap: opensearch.yml
• PVC: 30Gi storage
• Dashboards: UI ready

Phase 2: 🔄 Configure Memory Service (NEXT)

• Set OPENSEARCH_HOSTS env var
• Restart Memory Service pods
• Verify pod logs show connection success

Phase 3: 🔄 Test API Endpoints

• Test vault endpoints
• Test semantic search
• Test hybrid search

Phase 4: ⏳ Production Hardening (Future)

• Change Dashboards password
• Enable OpenSearch security plugin
• Setup OAuth2/SAML for Dashboards
• Enable audit logging
• Configure backup/recovery

---

✨ What's Next

After Memory Service is configured with OPENSEARCH_HOSTS:

1. Vault Endpoints: Test vault.riotpiao.com for file browsing
2. Hybrid Search: Test memory.riotpiao.com/query with hybrid results
3. Frontend: Deploy React app for UI (vault browser, search form)
4. GRC Workflow: Implement git + merge endpoints
5. Agent Streaming: WebSocket endpoint for real-time agent execution

---

Last Updated: Commit 630a125