diff --git a/docs/OPENSEARCH_DEPLOYMENT_GUIDE.md b/docs/OPENSEARCH_DEPLOYMENT_GUIDE.md new file mode 100644 index 0000000..ad6b3a6 --- /dev/null +++ b/docs/OPENSEARCH_DEPLOYMENT_GUIDE.md @@ -0,0 +1,558 @@ +# OpenSearch + Dashboards Deployment Guide + +## Status: ✅ DEPLOYED + +OpenSearch cluster + Dashboards UI are now running on K8s cluster `poimen` namespace. + +**Deployment Command (already run):** +```bash +kubectl apply -k k8s/infra/databases/ +``` + +**Commit:** `630a125` — deploy: OpenSearch + Dashboards StatefulSet + +--- + +## 📊 What's Running + +### OpenSearch Cluster (2-node HA) + +``` +opensearch-0 1/1 Running ← Primary node +opensearch-1 1/1 Running ← Replica node +opensearch-internal:9200 Ready ← API endpoint (Memory Service connects here) +``` + +**Configuration:** +- **Image:** opensearchproject/opensearch:2.11.0 +- **Storage:** 30Gi per pod (Longhorn) +- **Resources:** 512Mi-1Gi memory, 250m-500m CPU each +- **Network:** K8s internal only (NetworkPolicy restricts access) +- **Security:** plugins.security.disabled (secured by K8s network) + +### OpenSearch Dashboards (UI) + +``` +opensearch-dashboards-* 1/1 Running ← Dashboard pod +opensearch-dashboards:5601 Ready ← Web UI (port-forward for local access) +``` + +**Configuration:** +- **Image:** opensearchproject/opensearch-dashboards:2.11.0 +- **Login:** admin / admin (⚠️ change in production) +- **Memory:** 256Mi-512Mi +- **CPU:** 100m-500m + +--- + +## 🚀 Next Steps (Quick Start) + +### 1. Verify OpenSearch Cluster is Healthy + +```bash +# Port-forward to OpenSearch API +kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 & + +# Check cluster status +curl http://localhost:9200/_cluster/health +``` + +**Expected Output:** +```json +{ + "cluster_name": "poimen-memory", + "status": "green", + "timed_out": false, + "number_of_nodes": 2, + "number_of_data_nodes": 2, + "active_primary_shards": 0, + "active_shards": 0, + "relocating_shards": 0, + "initializing_shards": 0, + "unassigned_shards": 0, + "delayed_unassigned_shards": 0, + "number_of_pending_tasks": 0, + "number_of_in_flight_fetch": 0, + "task_max_waiting_in_queue_millis": 0, + "active_shards_percent_as_number": 100.0 +} +``` + +✅ **green** = cluster healthy +⚠️ **yellow** = some replicas unavailable (wait 1-2 min) +❌ **red** = cluster unhealthy (check pod logs) + +### 2. Access Dashboards UI (Local Development) + +```bash +# Port-forward to Dashboards +kubectl port-forward -n poimen svc/opensearch-dashboards 5601:5601 & + +# Open in browser +open http://localhost:5601 +# or +firefox http://localhost:5601 +``` + +**Login:** +- **Username:** admin +- **Password:** admin + +**First Time Setup:** +1. Dashboards auto-creates `.opensearch_dashboards` index +2. Accept default settings +3. Explore → Dev Tools → Console (for manual BM25 queries) + +### 3. Configure Memory Service to Use OpenSearch + +```bash +# Set environment variable +kubectl set env deployment poimen-memory -n poimen \ + OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200 + +# Restart Memory Service pods +kubectl rollout restart deployment poimen-memory -n poimen + +# Wait for rollout +kubectl rollout status deployment poimen-memory -n poimen +``` + +**Verify Connection:** +```bash +# Check Memory Service logs +kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=50 | grep -i opensearch +# Should see: "Initialized OpenSearch client: opensearch-internal.poimen.svc.cluster.local:9200" +``` + +### 4. Test Vault Endpoints + +```bash +# Get JWT token from Authentik +TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \ + -d "client_id=poimen-memory" \ + -d "client_secret=$AUTHENTIK_SECRET" \ + -d "grant_type=client_credentials" | jq -r .access_token) + +# List projects +curl -H "Authorization: Bearer $TOKEN" \ + https://vault.riotpiao.com/memory/vault + +# List files in project +curl -H "Authorization: Bearer $TOKEN" \ + 'https://vault.riotpiao.com/memory/vault?project=poimen' + +# Get specific file +curl -H "Authorization: Bearer $TOKEN" \ + 'https://vault.riotpiao.com/memory/vault/poimen/index' | jq . +``` + +**Expected:** +- 200 OK with JSON response +- If 401: check JWT token is valid +- If 403: check JWT has required scopes + +### 5. Test Hybrid Search + +```bash +# Semantic only (always works) +curl -H "Authorization: Bearer $TOKEN" \ + 'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic' + +# Hybrid search (now with OpenSearch) +curl -H "Authorization: Bearer $TOKEN" \ + 'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes' + +# Force strict hybrid (fail if OpenSearch down) +curl -H "Authorization: Bearer $TOKEN" \ + 'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=hybrid&strict=true' +``` + +**Expected Responses:** + +✅ **Hybrid (Fallback to Semantic if OpenSearch unavailable):** +```json +{ + "method": "hybrid", + "query": "kubernetes", + "project": "poimen", + "results": [ + { + "level": "L1", + "score": 0.992, + "text": "...", + "source": "claude", + "provenance": ["pi-1"] + } + ] +} +``` + +⚠️ **Semantic Fallback (if OpenSearch down):** +```json +{ + "method": "semantic_fallback", + "query": "kubernetes", + "project": "poimen", + "results": [...] +} +``` + +--- + +## 🛠️ Operations + +### Monitor Cluster Health + +```bash +# Watch pod status +kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w + +# Check OpenSearch logs +kubectl logs -n poimen opensearch-0 --tail=100 +kubectl logs -n poimen opensearch-1 --tail=100 + +# Check Dashboards logs +kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50 +``` + +### Common Issues + +#### ❌ Pods not starting + +**Symptom:** `Pending` or `CrashLoopBackOff` + +**Check:** +```bash +kubectl describe pod -n poimen opensearch-0 +``` + +**Common Causes:** +1. **vm.max_map_count too low** → Init container should fix (wait 30s) +2. **PVC not provisioned** → Check Longhorn: `kubectl get pvc -n poimen` +3. **Memory limit exceeded** → Increase `limits.memory` in StatefulSet +4. **Node affinity** → Check node labels: `kubectl get nodes --show-labels` + +**Fix:** +```bash +# Force pod recreation +kubectl delete pod -n poimen opensearch-0 +# Scheduler will restart it + +# Check logs after restart +kubectl logs -n poimen opensearch-0 --tail=100 +``` + +#### ❌ Cluster status = yellow + +**Symptom:** Only 1 node showing, shards unassigned + +**Cause:** Waiting for second node to start (normal during initial deployment) + +**Fix:** Wait 1-2 minutes +```bash +# Watch until green +kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w +``` + +#### ❌ Cluster status = red + +**Symptom:** Cluster health = red, both nodes showing but unhealthy + +**Debug:** +```bash +# Check node logs +kubectl logs -n poimen opensearch-0 --tail=200 | grep -i error + +# Check if nodes can communicate +kubectl exec -n poimen opensearch-0 -- curl http://opensearch-1.opensearch.poimen.svc.cluster.local:9300/ +``` + +**Common Causes:** +- Network Policy blocking communication +- Disk/memory pressure +- JVM out of memory + +**Recovery:** +```bash +# Reset cluster (deletes data, be careful in production!) +kubectl delete pvc opensearch-data-opensearch-0 opensearch-data-opensearch-1 -n poimen +kubectl delete pod opensearch-0 opensearch-1 -n poimen +# Wait ~3 minutes for recovery +``` + +#### ❌ Dashboards can't connect to OpenSearch + +**Symptom:** Dashboards UI shows "Cannot connect to Elasticsearch" + +**Check:** +```bash +kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50 +``` + +**Fix:** +```bash +# Verify OpenSearch is healthy +kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 & +curl http://localhost:9200/_cluster/health + +# Check Dashboards config +kubectl describe cm opensearch-dashboards-config -n poimen +# Should show: opensearch.hosts = ["http://opensearch-internal.poimen.svc.cluster.local:9200"] + +# Restart Dashboards +kubectl rollout restart deployment opensearch-dashboards -n poimen +``` + +### Backup & Recovery + +#### Export OpenSearch Indices + +```bash +# List all indices +curl http://localhost:9200/_cat/indices + +# Snapshot creation (requires S3/backup config) +# Docs: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-resilience/snapshots/snapshot-restore/ +``` + +#### Restore from Backup + +```bash +# Restore index from snapshot +curl -X POST http://localhost:9200/_snapshot/backup/snapshot-1/_restore +``` + +--- + +## 📈 Performance Tuning + +### OpenSearch JVM Memory + +**Current:** 512Mi-1Gi per pod + +**For larger datasets (>1M vectors):** +```yaml +# Edit StatefulSet +kubectl edit statefulset opensearch -n poimen + +# Update resources: +resources: + requests: + memory: "1Gi" # was 512Mi + limits: + memory: "2Gi" # was 1Gi +``` + +### OpenSearch Shard Configuration + +**Current:** Auto-configured by Dashboards + +**Optimize for hybrid search (many small shards):** +```bash +# After indices are created +curl -X PUT http://localhost:9200/vault-poimen/_settings -d '{ + "index": { + "number_of_shards": 3, + "number_of_replicas": 1, + "codec": "best_compression", + "refresh_interval": "30s" + } +}' +``` + +### IVFFlat Index Parameters (pgvector side) + +```sql +-- Optimize for 1M+ vectors +CREATE INDEX ON memory_vector +USING ivfflat (embedding vector_cosine_ops) +WITH (lists=1000); -- sqrt(1000000) ≈ 1000 + +-- Current: lists=100 (good for <100k vectors) +``` + +--- + +## 🔐 Security (Production Checklist) + +### ⚠️ TODO: Production Security + +- [ ] Change Dashboards password in secret: + ```bash + kubectl patch secret opensearch-dashboards-secret -n poimen \ + -p '{"stringData":{"password":"YOUR_SECURE_PASSWORD"}}' + kubectl rollout restart deployment opensearch-dashboards -n poimen + ``` + +- [ ] Enable OpenSearch security plugin: + ```yaml + # In opensearch.yml ConfigMap + plugins.security.disabled: "false" + plugins.security.ssl.http.enabled: "true" + plugins.security.ssl.http.keystore_filepath: "/usr/share/opensearch/config/certs/keystore.jks" + # ... (requires cert generation) + ``` + +- [ ] Setup Dashboards OAuth2/SAML: + ```yaml + # opensearch_dashboards.yml + opensearch.username: null # Remove hardcoded auth + opensearch.password: null + xpack.security.auth.providers: ["saml"] + # ... (requires IdP config) + ``` + +- [ ] Setup NetworkPolicy for Ingress access (if exposing publicly) + +- [ ] Enable audit logging: + ```yaml + # opensearch.yml + plugins.security.audit.type: internal_opensearch + plugins.security.audit.config.http_endpoints: ["opensearch:9200"] + ``` + +--- + +## 📚 Useful Commands + +```bash +# Cluster status +curl http://localhost:9200/_cluster/health | jq . + +# List indices +curl http://localhost:9200/_cat/indices | jq . + +# Index stats +curl http://localhost:9200/_stats | jq .indices + +# Node info +curl http://localhost:9200/_nodes | jq '.nodes | length' + +# Clear query cache +curl -X POST http://localhost:9200/_cache/clear + +# Force merge indices (maintenance) +curl -X POST http://localhost:9200/vault-*/_forcemerge?max_num_segments=1 + +# Pod resource usage +kubectl top pods -n poimen -l app.kubernetes.io/name=opensearch + +# PVC usage +kubectl exec -n poimen opensearch-0 -- df -h /usr/share/opensearch/data +``` + +--- + +## 📞 Support + +### Check Status Anytime + +```bash +# Everything OK? +kubectl get pods -n poimen -l "app.kubernetes.io/name in (opensearch, opensearch-dashboards)" + +# Cluster green? +kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 & +curl http://localhost:9200/_cluster/health | jq .status +# Should show: "green" +``` + +### Logs + +```bash +# Real-time OpenSearch logs +kubectl logs -n poimen opensearch-0 -f + +# Real-time Dashboards logs +kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards -f + +# Search for errors +kubectl logs -n poimen opensearch-0 | grep -i error | tail -20 +``` + +--- + +## 🎯 Integration with Memory Service + +### Architecture + +``` + User Request (JWT) + ↓ + ┌────────────────────────────────┐ + │ poimen-memory Pod │ + │ (2 replicas) │ + ├────────────────────────────────┤ + │ GET /memory/query │ + │ ├─ Query pgvector (60%) │ + │ └─ Query OpenSearch (40%) │ + │ (BM25 full-text search) │ + │ └─ Fusion & rerank │ + └──────────┬──────────────────┬──┘ + │ │ + ↓ ↓ + pgvector(Postgres) OpenSearch Cluster + (768-dim embed) (2 nodes, HA) + IVFFlat index BM25 indices +``` + +### Query Flow + +1. User sends: `GET /memory/query?project=X&query=Y` +2. Memory Service receives JWT, validates scopes +3. **Parallel queries:** + - pgvector: "SELECT ... ORDER BY embedding <-> query_embedding LIMIT 50" + - OpenSearch: "POST vault-X/_search" with BM25 query +4. **Fusion:** Combine top-50 results from each, score: `0.6*sem + 0.4*lex` +5. **Return:** Top-10 merged results with method indicator (hybrid / semantic_fallback / semantic) + +### Graceful Degradation + +- ✅ **OpenSearch healthy:** Hybrid search (60% + 40% fusion) +- ⚠️ **OpenSearch slow:** Timeout → fallback to semantic only +- ⚠️ **OpenSearch down:** Fallback to semantic only (no error) +- ❌ **pgvector down:** All queries fail (core dependency) + +--- + +## 📋 Deployment Checklist Summary + +### Phase 1: ✅ OpenSearch Deployed +- [x] StatefulSet: 2 replicas +- [x] Services: opensearch, opensearch-internal +- [x] ConfigMap: opensearch.yml +- [x] PVC: 30Gi storage +- [x] Dashboards: UI ready + +### Phase 2: 🔄 Configure Memory Service (NEXT) +- [ ] Set OPENSEARCH_HOSTS env var +- [ ] Restart Memory Service pods +- [ ] Verify pod logs show connection success + +### Phase 3: 🔄 Test API Endpoints +- [ ] Test vault endpoints +- [ ] Test semantic search +- [ ] Test hybrid search + +### Phase 4: ⏳ Production Hardening (Future) +- [ ] Change Dashboards password +- [ ] Enable OpenSearch security plugin +- [ ] Setup OAuth2/SAML for Dashboards +- [ ] Enable audit logging +- [ ] Configure backup/recovery + +--- + +## ✨ What's Next + +**After Memory Service is configured with OPENSEARCH_HOSTS:** + +1. **Vault Endpoints:** Test `vault.riotpiao.com` for file browsing +2. **Hybrid Search:** Test `memory.riotpiao.com/query` with hybrid results +3. **Frontend:** Deploy React app for UI (vault browser, search form) +4. **GRC Workflow:** Implement git + merge endpoints +5. **Agent Streaming:** WebSocket endpoint for real-time agent execution + +--- + +Last Updated: Commit `630a125`