docs: OpenSearch Deployment & Operations Guide

Complete guide for OpenSearch + Dashboards production operations: ✅ Quick Start (5 steps): 1. Verify cluster health (curl _cluster/health) 2. Access Dashboards UI (port-forward 5601) 3. Configure Memory Service (OPENSEARCH_HOSTS env var) 4. Test vault endpoints (vault.riotpiao.com) 5. Test hybrid search (/memory/query) 📊 Operations: - Health checks and monitoring - Troubleshooting: pods not starting, yellow/red status, connection issues - Performance tuning: JVM memory, shard config - Backup & recovery procedures - Security hardening checklist (production) 🔐 Security: - TODO items for production deployment - Dashboards password change - OpenSearch security plugin enable - OAuth2/SAML integration 📈 Integration: - Architecture diagram (pgvector + OpenSearch) - Query flow explanation - Graceful degradation scenarios - Dependency management 🔧 Useful Commands: - Health status queries - Index management - Pod logs and resource usage - PVC monitoring Deployment checklist: Phase 1: ✅ OpenSearch deployed Phase 2: 🔄 Configure Memory Service (NEXT) Phase 3: 🔄 Test endpoints Phase 4: ⏳ Production hardening
2026-08-27 21:12:10 -07:00
parent 630a125778
commit 3f096e8f9c
1 changed files with 558 additions and 0 deletions
@@ -0,0 +1,558 @@
+# OpenSearch + Dashboards Deployment Guide
+
+## Status: ✅ DEPLOYED
+
+OpenSearch cluster + Dashboards UI are now running on K8s cluster `poimen` namespace.
+
+**Deployment Command (already run):**
+```bash
+kubectl apply -k k8s/infra/databases/
+```
+
+**Commit:** `630a125` — deploy: OpenSearch + Dashboards StatefulSet
+
+---
+
+## 📊 What's Running
+
+### OpenSearch Cluster (2-node HA)
+
+```
+opensearch-0                  1/1 Running   ← Primary node
+opensearch-1                  1/1 Running   ← Replica node
+opensearch-internal:9200      Ready         ← API endpoint (Memory Service connects here)
+```
+
+**Configuration:**
+- **Image:** opensearchproject/opensearch:2.11.0
+- **Storage:** 30Gi per pod (Longhorn)
+- **Resources:** 512Mi-1Gi memory, 250m-500m CPU each
+- **Network:** K8s internal only (NetworkPolicy restricts access)
+- **Security:** plugins.security.disabled (secured by K8s network)
+
+### OpenSearch Dashboards (UI)
+
+```
+opensearch-dashboards-*      1/1 Running   ← Dashboard pod
+opensearch-dashboards:5601   Ready         ← Web UI (port-forward for local access)
+```
+
+**Configuration:**
+- **Image:** opensearchproject/opensearch-dashboards:2.11.0
+- **Login:** admin / admin (⚠️ change in production)
+- **Memory:** 256Mi-512Mi
+- **CPU:** 100m-500m
+
+---
+
+## 🚀 Next Steps (Quick Start)
+
+### 1. Verify OpenSearch Cluster is Healthy
+
+```bash
+# Port-forward to OpenSearch API
+kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
+
+# Check cluster status
+curl http://localhost:9200/_cluster/health
+```
+
+**Expected Output:**
+```json
+{
+  "cluster_name": "poimen-memory",
+  "status": "green",
+  "timed_out": false,
+  "number_of_nodes": 2,
+  "number_of_data_nodes": 2,
+  "active_primary_shards": 0,
+  "active_shards": 0,
+  "relocating_shards": 0,
+  "initializing_shards": 0,
+  "unassigned_shards": 0,
+  "delayed_unassigned_shards": 0,
+  "number_of_pending_tasks": 0,
+  "number_of_in_flight_fetch": 0,
+  "task_max_waiting_in_queue_millis": 0,
+  "active_shards_percent_as_number": 100.0
+}
+```
+
+✅ **green** = cluster healthy
+⚠️  **yellow** = some replicas unavailable (wait 1-2 min)
+❌ **red** = cluster unhealthy (check pod logs)
+
+### 2. Access Dashboards UI (Local Development)
+
+```bash
+# Port-forward to Dashboards
+kubectl port-forward -n poimen svc/opensearch-dashboards 5601:5601 &
+
+# Open in browser
+open http://localhost:5601
+# or
+firefox http://localhost:5601
+```
+
+**Login:**
+- **Username:** admin
+- **Password:** admin
+
+**First Time Setup:**
+1. Dashboards auto-creates `.opensearch_dashboards` index
+2. Accept default settings
+3. Explore → Dev Tools → Console (for manual BM25 queries)
+
+### 3. Configure Memory Service to Use OpenSearch
+
+```bash
+# Set environment variable
+kubectl set env deployment poimen-memory -n poimen \
+  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200
+
+# Restart Memory Service pods
+kubectl rollout restart deployment poimen-memory -n poimen
+
+# Wait for rollout
+kubectl rollout status deployment poimen-memory -n poimen
+```
+
+**Verify Connection:**
+```bash
+# Check Memory Service logs
+kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=50 | grep -i opensearch
+# Should see: "Initialized OpenSearch client: opensearch-internal.poimen.svc.cluster.local:9200"
+```
+
+### 4. Test Vault Endpoints
+
+```bash
+# Get JWT token from Authentik
+TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
+  -d "client_id=poimen-memory" \
+  -d "client_secret=$AUTHENTIK_SECRET" \
+  -d "grant_type=client_credentials" | jq -r .access_token)
+
+# List projects
+curl -H "Authorization: Bearer $TOKEN" \
+  https://vault.riotpiao.com/memory/vault
+
+# List files in project
+curl -H "Authorization: Bearer $TOKEN" \
+  'https://vault.riotpiao.com/memory/vault?project=poimen'
+
+# Get specific file
+curl -H "Authorization: Bearer $TOKEN" \
+  'https://vault.riotpiao.com/memory/vault/poimen/index' | jq .
+```
+
+**Expected:**
+- 200 OK with JSON response
+- If 401: check JWT token is valid
+- If 403: check JWT has required scopes
+
+### 5. Test Hybrid Search
+
+```bash
+# Semantic only (always works)
+curl -H "Authorization: Bearer $TOKEN" \
+  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic'
+
+# Hybrid search (now with OpenSearch)
+curl -H "Authorization: Bearer $TOKEN" \
+  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes'
+
+# Force strict hybrid (fail if OpenSearch down)
+curl -H "Authorization: Bearer $TOKEN" \
+  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=hybrid&strict=true'
+```
+
+**Expected Responses:**
+
+✅ **Hybrid (Fallback to Semantic if OpenSearch unavailable):**
+```json
+{
+  "method": "hybrid",
+  "query": "kubernetes",
+  "project": "poimen",
+  "results": [
+    {
+      "level": "L1",
+      "score": 0.992,
+      "text": "...",
+      "source": "claude",
+      "provenance": ["pi-1"]
+    }
+  ]
+}
+```
+
+⚠️ **Semantic Fallback (if OpenSearch down):**
+```json
+{
+  "method": "semantic_fallback",
+  "query": "kubernetes",
+  "project": "poimen",
+  "results": [...]
+}
+```
+
+---
+
+## 🛠️ Operations
+
+### Monitor Cluster Health
+
+```bash
+# Watch pod status
+kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w
+
+# Check OpenSearch logs
+kubectl logs -n poimen opensearch-0 --tail=100
+kubectl logs -n poimen opensearch-1 --tail=100
+
+# Check Dashboards logs
+kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50
+```
+
+### Common Issues
+
+#### ❌ Pods not starting
+
+**Symptom:** `Pending` or `CrashLoopBackOff`
+
+**Check:**
+```bash
+kubectl describe pod -n poimen opensearch-0
+```
+
+**Common Causes:**
+1. **vm.max_map_count too low** → Init container should fix (wait 30s)
+2. **PVC not provisioned** → Check Longhorn: `kubectl get pvc -n poimen`
+3. **Memory limit exceeded** → Increase `limits.memory` in StatefulSet
+4. **Node affinity** → Check node labels: `kubectl get nodes --show-labels`
+
+**Fix:**
+```bash
+# Force pod recreation
+kubectl delete pod -n poimen opensearch-0
+# Scheduler will restart it
+
+# Check logs after restart
+kubectl logs -n poimen opensearch-0 --tail=100
+```
+
+#### ❌ Cluster status = yellow
+
+**Symptom:** Only 1 node showing, shards unassigned
+
+**Cause:** Waiting for second node to start (normal during initial deployment)
+
+**Fix:** Wait 1-2 minutes
+```bash
+# Watch until green
+kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w
+```
+
+#### ❌ Cluster status = red
+
+**Symptom:** Cluster health = red, both nodes showing but unhealthy
+
+**Debug:**
+```bash
+# Check node logs
+kubectl logs -n poimen opensearch-0 --tail=200 | grep -i error
+
+# Check if nodes can communicate
+kubectl exec -n poimen opensearch-0 -- curl http://opensearch-1.opensearch.poimen.svc.cluster.local:9300/
+```
+
+**Common Causes:**
+- Network Policy blocking communication
+- Disk/memory pressure
+- JVM out of memory
+
+**Recovery:**
+```bash
+# Reset cluster (deletes data, be careful in production!)
+kubectl delete pvc opensearch-data-opensearch-0 opensearch-data-opensearch-1 -n poimen
+kubectl delete pod opensearch-0 opensearch-1 -n poimen
+# Wait ~3 minutes for recovery
+```
+
+#### ❌ Dashboards can't connect to OpenSearch
+
+**Symptom:** Dashboards UI shows "Cannot connect to Elasticsearch"
+
+**Check:**
+```bash
+kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50
+```
+
+**Fix:**
+```bash
+# Verify OpenSearch is healthy
+kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
+curl http://localhost:9200/_cluster/health
+
+# Check Dashboards config
+kubectl describe cm opensearch-dashboards-config -n poimen
+# Should show: opensearch.hosts = ["http://opensearch-internal.poimen.svc.cluster.local:9200"]
+
+# Restart Dashboards
+kubectl rollout restart deployment opensearch-dashboards -n poimen
+```
+
+### Backup & Recovery
+
+#### Export OpenSearch Indices
+
+```bash
+# List all indices
+curl http://localhost:9200/_cat/indices
+
+# Snapshot creation (requires S3/backup config)
+# Docs: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-resilience/snapshots/snapshot-restore/
+```
+
+#### Restore from Backup
+
+```bash
+# Restore index from snapshot
+curl -X POST http://localhost:9200/_snapshot/backup/snapshot-1/_restore
+```
+
+---
+
+## 📈 Performance Tuning
+
+### OpenSearch JVM Memory
+
+**Current:** 512Mi-1Gi per pod
+
+**For larger datasets (>1M vectors):**
+```yaml
+# Edit StatefulSet
+kubectl edit statefulset opensearch -n poimen
+
+# Update resources:
+resources:
+  requests:
+    memory: "1Gi"      # was 512Mi
+  limits:
+    memory: "2Gi"      # was 1Gi
+```
+
+### OpenSearch Shard Configuration
+
+**Current:** Auto-configured by Dashboards
+
+**Optimize for hybrid search (many small shards):**
+```bash
+# After indices are created
+curl -X PUT http://localhost:9200/vault-poimen/_settings -d '{
+  "index": {
+    "number_of_shards": 3,
+    "number_of_replicas": 1,
+    "codec": "best_compression",
+    "refresh_interval": "30s"
+  }
+}'
+```
+
+### IVFFlat Index Parameters (pgvector side)
+
+```sql
+-- Optimize for 1M+ vectors
+CREATE INDEX ON memory_vector
+USING ivfflat (embedding vector_cosine_ops)
+WITH (lists=1000);  -- sqrt(1000000) ≈ 1000
+
+-- Current: lists=100 (good for <100k vectors)
+```
+
+---
+
+## 🔐 Security (Production Checklist)
+
+### ⚠️ TODO: Production Security
+
+- [ ] Change Dashboards password in secret:
+  ```bash
+  kubectl patch secret opensearch-dashboards-secret -n poimen \
+    -p '{"stringData":{"password":"YOUR_SECURE_PASSWORD"}}'
+  kubectl rollout restart deployment opensearch-dashboards -n poimen
+  ```
+
+- [ ] Enable OpenSearch security plugin:
+  ```yaml
+  # In opensearch.yml ConfigMap
+  plugins.security.disabled: "false"
+  plugins.security.ssl.http.enabled: "true"
+  plugins.security.ssl.http.keystore_filepath: "/usr/share/opensearch/config/certs/keystore.jks"
+  # ... (requires cert generation)
+  ```
+
+- [ ] Setup Dashboards OAuth2/SAML:
+  ```yaml
+  # opensearch_dashboards.yml
+  opensearch.username: null  # Remove hardcoded auth
+  opensearch.password: null
+  xpack.security.auth.providers: ["saml"]
+  # ... (requires IdP config)
+  ```
+
+- [ ] Setup NetworkPolicy for Ingress access (if exposing publicly)
+
+- [ ] Enable audit logging:
+  ```yaml
+  # opensearch.yml
+  plugins.security.audit.type: internal_opensearch
+  plugins.security.audit.config.http_endpoints: ["opensearch:9200"]
+  ```
+
+---
+
+## 📚 Useful Commands
+
+```bash
+# Cluster status
+curl http://localhost:9200/_cluster/health | jq .
+
+# List indices
+curl http://localhost:9200/_cat/indices | jq .
+
+# Index stats
+curl http://localhost:9200/_stats | jq .indices
+
+# Node info
+curl http://localhost:9200/_nodes | jq '.nodes | length'
+
+# Clear query cache
+curl -X POST http://localhost:9200/_cache/clear
+
+# Force merge indices (maintenance)
+curl -X POST http://localhost:9200/vault-*/_forcemerge?max_num_segments=1
+
+# Pod resource usage
+kubectl top pods -n poimen -l app.kubernetes.io/name=opensearch
+
+# PVC usage
+kubectl exec -n poimen opensearch-0 -- df -h /usr/share/opensearch/data
+```
+
+---
+
+## 📞 Support
+
+### Check Status Anytime
+
+```bash
+# Everything OK?
+kubectl get pods -n poimen -l "app.kubernetes.io/name in (opensearch, opensearch-dashboards)"
+
+# Cluster green?
+kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
+curl http://localhost:9200/_cluster/health | jq .status
+# Should show: "green"
+```
+
+### Logs
+
+```bash
+# Real-time OpenSearch logs
+kubectl logs -n poimen opensearch-0 -f
+
+# Real-time Dashboards logs
+kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards -f
+
+# Search for errors
+kubectl logs -n poimen opensearch-0 | grep -i error | tail -20
+```
+
+---
+
+## 🎯 Integration with Memory Service
+
+### Architecture
+
+```
+        User Request (JWT)
+                 ↓
+   ┌────────────────────────────────┐
+   │ poimen-memory Pod              │
+   │ (2 replicas)                   │
+   ├────────────────────────────────┤
+   │ GET /memory/query              │
+   │   ├─ Query pgvector (60%)      │
+   │   └─ Query OpenSearch (40%)    │
+   │       (BM25 full-text search)  │
+   │   └─ Fusion & rerank           │
+   └──────────┬──────────────────┬──┘
+              │                  │
+              ↓                  ↓
+       pgvector(Postgres)  OpenSearch Cluster
+       (768-dim embed)     (2 nodes, HA)
+       IVFFlat index       BM25 indices
+```
+
+### Query Flow
+
+1. User sends: `GET /memory/query?project=X&query=Y`
+2. Memory Service receives JWT, validates scopes
+3. **Parallel queries:**
+   - pgvector: "SELECT ... ORDER BY embedding <-> query_embedding LIMIT 50"
+   - OpenSearch: "POST vault-X/_search" with BM25 query
+4. **Fusion:** Combine top-50 results from each, score: `0.6*sem + 0.4*lex`
+5. **Return:** Top-10 merged results with method indicator (hybrid / semantic_fallback / semantic)
+
+### Graceful Degradation
+
+- ✅ **OpenSearch healthy:** Hybrid search (60% + 40% fusion)
+- ⚠️  **OpenSearch slow:** Timeout → fallback to semantic only
+- ⚠️  **OpenSearch down:** Fallback to semantic only (no error)
+- ❌ **pgvector down:** All queries fail (core dependency)
+
+---
+
+## 📋 Deployment Checklist Summary
+
+### Phase 1: ✅ OpenSearch Deployed
+- [x] StatefulSet: 2 replicas
+- [x] Services: opensearch, opensearch-internal
+- [x] ConfigMap: opensearch.yml
+- [x] PVC: 30Gi storage
+- [x] Dashboards: UI ready
+
+### Phase 2: 🔄 Configure Memory Service (NEXT)
+- [ ] Set OPENSEARCH_HOSTS env var
+- [ ] Restart Memory Service pods
+- [ ] Verify pod logs show connection success
+
+### Phase 3: 🔄 Test API Endpoints
+- [ ] Test vault endpoints
+- [ ] Test semantic search
+- [ ] Test hybrid search
+
+### Phase 4: ⏳ Production Hardening (Future)
+- [ ] Change Dashboards password
+- [ ] Enable OpenSearch security plugin
+- [ ] Setup OAuth2/SAML for Dashboards
+- [ ] Enable audit logging
+- [ ] Configure backup/recovery
+
+---
+
+## ✨ What's Next
+
+**After Memory Service is configured with OPENSEARCH_HOSTS:**
+
+1. **Vault Endpoints:** Test `vault.riotpiao.com` for file browsing
+2. **Hybrid Search:** Test `memory.riotpiao.com/query` with hybrid results
+3. **Frontend:** Deploy React app for UI (vault browser, search form)
+4. **GRC Workflow:** Implement git + merge endpoints
+5. **Agent Streaming:** WebSocket endpoint for real-time agent execution
+
+---
+
+Last Updated: Commit `630a125`