poimen-memory/docs/OPENSEARCH_DEPLOYMENT_GUIDE.md

# OpenSearch + Dashboards Deployment Guide

## Status: ✅ DEPLOYED

OpenSearch cluster + Dashboards UI are now running on K8s cluster `poimen` namespace.

**Deployment Command (already run):**
```bash
kubectl apply -k k8s/infra/databases/
```

**Commit:** `630a125` — deploy: OpenSearch + Dashboards StatefulSet

---

## 📊 What's Running

### OpenSearch Cluster (2-node HA)

```
opensearch-0                  1/1 Running   ← Primary node
opensearch-1                  1/1 Running   ← Replica node
opensearch-internal:9200      Ready         ← API endpoint (Memory Service connects here)
```

**Configuration:**
- **Image:** opensearchproject/opensearch:2.11.0
- **Storage:** 30Gi per pod (Longhorn)
- **Resources:** 512Mi-1Gi memory, 250m-500m CPU each
- **Network:** K8s internal only (NetworkPolicy restricts access)
- **Security:** plugins.security.disabled (secured by K8s network)

### OpenSearch Dashboards (UI)

```
opensearch-dashboards-*      1/1 Running   ← Dashboard pod
opensearch-dashboards:5601   Ready         ← Web UI (port-forward for local access)
```

**Configuration:**
- **Image:** opensearchproject/opensearch-dashboards:2.11.0
- **Login:** admin / admin (⚠️ change in production)
- **Memory:** 256Mi-512Mi
- **CPU:** 100m-500m

---

## 🚀 Next Steps (Quick Start)

### 1. Verify OpenSearch Cluster is Healthy

```bash
# Port-forward to OpenSearch API
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &

# Check cluster status
curl http://localhost:9200/_cluster/health
```

**Expected Output:**
```json
{
  "cluster_name": "poimen-memory",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 2,
  "number_of_data_nodes": 2,
  "active_primary_shards": 0,
  "active_shards": 0,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}
```

✅ **green** = cluster healthy
⚠️  **yellow** = some replicas unavailable (wait 1-2 min)
❌ **red** = cluster unhealthy (check pod logs)

### 2. Access Dashboards UI (Local Development)

```bash
# Port-forward to Dashboards
kubectl port-forward -n poimen svc/opensearch-dashboards 5601:5601 &

# Open in browser
open http://localhost:5601
# or
firefox http://localhost:5601
```

**Login:**
- **Username:** admin
- **Password:** admin

**First Time Setup:**
1. Dashboards auto-creates `.opensearch_dashboards` index
2. Accept default settings
3. Explore → Dev Tools → Console (for manual BM25 queries)

### 3. Configure Memory Service to Use OpenSearch

```bash
# Set environment variable
kubectl set env deployment poimen-memory -n poimen \
  OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200

# Restart Memory Service pods
kubectl rollout restart deployment poimen-memory -n poimen

# Wait for rollout
kubectl rollout status deployment poimen-memory -n poimen
```

**Verify Connection:**
```bash
# Check Memory Service logs
kubectl logs -n poimen -l app.kubernetes.io/name=poimen-memory --tail=50 | grep -i opensearch
# Should see: "Initialized OpenSearch client: opensearch-internal.poimen.svc.cluster.local:9200"
```

### 4. Test Vault Endpoints

```bash
# Get JWT token from Authentik
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d "client_id=poimen-memory" \
  -d "client_secret=$AUTHENTIK_SECRET" \
  -d "grant_type=client_credentials" | jq -r .access_token)

# List projects
curl -H "Authorization: Bearer $TOKEN" \
  https://vault.riotpiao.com/memory/vault

# List files in project
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault?project=poimen'

# Get specific file
curl -H "Authorization: Bearer $TOKEN" \
  'https://vault.riotpiao.com/memory/vault/poimen/index' | jq .
```

**Expected:**
- 200 OK with JSON response
- If 401: check JWT token is valid
- If 403: check JWT has required scopes

### 5. Test Hybrid Search

```bash
# Semantic only (always works)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=semantic'

# Hybrid search (now with OpenSearch)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes'

# Force strict hybrid (fail if OpenSearch down)
curl -H "Authorization: Bearer $TOKEN" \
  'https://memory.riotpiao.com/memory/query?project=poimen&query=kubernetes&method=hybrid&strict=true'
```

**Expected Responses:**

✅ **Hybrid (Fallback to Semantic if OpenSearch unavailable):**
```json
{
  "method": "hybrid",
  "query": "kubernetes",
  "project": "poimen",
  "results": [
    {
      "level": "L1",
      "score": 0.992,
      "text": "...",
      "source": "claude",
      "provenance": ["pi-1"]
    }
  ]
}
```

⚠️ **Semantic Fallback (if OpenSearch down):**
```json
{
  "method": "semantic_fallback",
  "query": "kubernetes",
  "project": "poimen",
  "results": [...]
}
```

---

## 🛠️ Operations

### Monitor Cluster Health

```bash
# Watch pod status
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w

# Check OpenSearch logs
kubectl logs -n poimen opensearch-0 --tail=100
kubectl logs -n poimen opensearch-1 --tail=100

# Check Dashboards logs
kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50
```

### Common Issues

#### ❌ Pods not starting

**Symptom:** `Pending` or `CrashLoopBackOff`

**Check:**
```bash
kubectl describe pod -n poimen opensearch-0
```

**Common Causes:**
1. **vm.max_map_count too low** → Init container should fix (wait 30s)
2. **PVC not provisioned** → Check Longhorn: `kubectl get pvc -n poimen`
3. **Memory limit exceeded** → Increase `limits.memory` in StatefulSet
4. **Node affinity** → Check node labels: `kubectl get nodes --show-labels`

**Fix:**
```bash
# Force pod recreation
kubectl delete pod -n poimen opensearch-0
# Scheduler will restart it

# Check logs after restart
kubectl logs -n poimen opensearch-0 --tail=100
```

#### ❌ Cluster status = yellow

**Symptom:** Only 1 node showing, shards unassigned

**Cause:** Waiting for second node to start (normal during initial deployment)

**Fix:** Wait 1-2 minutes
```bash
# Watch until green
kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch -w
```

#### ❌ Cluster status = red

**Symptom:** Cluster health = red, both nodes showing but unhealthy

**Debug:**
```bash
# Check node logs
kubectl logs -n poimen opensearch-0 --tail=200 | grep -i error

# Check if nodes can communicate
kubectl exec -n poimen opensearch-0 -- curl http://opensearch-1.opensearch.poimen.svc.cluster.local:9300/
```

**Common Causes:**
- Network Policy blocking communication
- Disk/memory pressure
- JVM out of memory

**Recovery:**
```bash
# Reset cluster (deletes data, be careful in production!)
kubectl delete pvc opensearch-data-opensearch-0 opensearch-data-opensearch-1 -n poimen
kubectl delete pod opensearch-0 opensearch-1 -n poimen
# Wait ~3 minutes for recovery
```

#### ❌ Dashboards can't connect to OpenSearch

**Symptom:** Dashboards UI shows "Cannot connect to Elasticsearch"

**Check:**
```bash
kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards --tail=50
```

**Fix:**
```bash
# Verify OpenSearch is healthy
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health

# Check Dashboards config
kubectl describe cm opensearch-dashboards-config -n poimen
# Should show: opensearch.hosts = ["http://opensearch-internal.poimen.svc.cluster.local:9200"]

# Restart Dashboards
kubectl rollout restart deployment opensearch-dashboards -n poimen
```

### Backup & Recovery

#### Export OpenSearch Indices

```bash
# List all indices
curl http://localhost:9200/_cat/indices

# Snapshot creation (requires S3/backup config)
# Docs: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-resilience/snapshots/snapshot-restore/
```

#### Restore from Backup

```bash
# Restore index from snapshot
curl -X POST http://localhost:9200/_snapshot/backup/snapshot-1/_restore
```

---

## 📈 Performance Tuning

### OpenSearch JVM Memory

**Current:** 512Mi-1Gi per pod

**For larger datasets (>1M vectors):**
```yaml
# Edit StatefulSet
kubectl edit statefulset opensearch -n poimen

# Update resources:
resources:
  requests:
    memory: "1Gi"      # was 512Mi
  limits:
    memory: "2Gi"      # was 1Gi
```

### OpenSearch Shard Configuration

**Current:** Auto-configured by Dashboards

**Optimize for hybrid search (many small shards):**
```bash
# After indices are created
curl -X PUT http://localhost:9200/vault-poimen/_settings -d '{
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "codec": "best_compression",
    "refresh_interval": "30s"
  }
}'
```

### IVFFlat Index Parameters (pgvector side)

```sql
-- Optimize for 1M+ vectors
CREATE INDEX ON memory_vector
USING ivfflat (embedding vector_cosine_ops)
WITH (lists=1000);  -- sqrt(1000000) ≈ 1000

-- Current: lists=100 (good for <100k vectors)
```

---

## 🔐 Security (Production Checklist)

### ⚠️ TODO: Production Security

- [ ] Change Dashboards password in secret:
  ```bash
  kubectl patch secret opensearch-dashboards-secret -n poimen \
    -p '{"stringData":{"password":"YOUR_SECURE_PASSWORD"}}'
  kubectl rollout restart deployment opensearch-dashboards -n poimen
  ```

- [ ] Enable OpenSearch security plugin:
  ```yaml
  # In opensearch.yml ConfigMap
  plugins.security.disabled: "false"
  plugins.security.ssl.http.enabled: "true"
  plugins.security.ssl.http.keystore_filepath: "/usr/share/opensearch/config/certs/keystore.jks"
  # ... (requires cert generation)
  ```

- [ ] Setup Dashboards OAuth2/SAML:
  ```yaml
  # opensearch_dashboards.yml
  opensearch.username: null  # Remove hardcoded auth
  opensearch.password: null
  xpack.security.auth.providers: ["saml"]
  # ... (requires IdP config)
  ```

- [ ] Setup NetworkPolicy for Ingress access (if exposing publicly)

- [ ] Enable audit logging:
  ```yaml
  # opensearch.yml
  plugins.security.audit.type: internal_opensearch
  plugins.security.audit.config.http_endpoints: ["opensearch:9200"]
  ```

---

## 📚 Useful Commands

```bash
# Cluster status
curl http://localhost:9200/_cluster/health | jq .

# List indices
curl http://localhost:9200/_cat/indices | jq .

# Index stats
curl http://localhost:9200/_stats | jq .indices

# Node info
curl http://localhost:9200/_nodes | jq '.nodes | length'

# Clear query cache
curl -X POST http://localhost:9200/_cache/clear

# Force merge indices (maintenance)
curl -X POST http://localhost:9200/vault-*/_forcemerge?max_num_segments=1

# Pod resource usage
kubectl top pods -n poimen -l app.kubernetes.io/name=opensearch

# PVC usage
kubectl exec -n poimen opensearch-0 -- df -h /usr/share/opensearch/data
```

---

## 📞 Support

### Check Status Anytime

```bash
# Everything OK?
kubectl get pods -n poimen -l "app.kubernetes.io/name in (opensearch, opensearch-dashboards)"

# Cluster green?
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200 &
curl http://localhost:9200/_cluster/health | jq .status
# Should show: "green"
```

### Logs

```bash
# Real-time OpenSearch logs
kubectl logs -n poimen opensearch-0 -f

# Real-time Dashboards logs
kubectl logs -n poimen -l app.kubernetes.io/name=opensearch-dashboards -f

# Search for errors
kubectl logs -n poimen opensearch-0 | grep -i error | tail -20
```

---

## 🎯 Integration with Memory Service

### Architecture

```
        User Request (JWT)
                 ↓
   ┌────────────────────────────────┐
   │ poimen-memory Pod              │
   │ (2 replicas)                   │
   ├────────────────────────────────┤
   │ GET /memory/query              │
   │   ├─ Query pgvector (60%)      │
   │   └─ Query OpenSearch (40%)    │
   │       (BM25 full-text search)  │
   │   └─ Fusion & rerank           │
   └──────────┬──────────────────┬──┘
              │                  │
              ↓                  ↓
       pgvector(Postgres)  OpenSearch Cluster
       (768-dim embed)     (2 nodes, HA)
       IVFFlat index       BM25 indices
```

### Query Flow

1. User sends: `GET /memory/query?project=X&query=Y`
2. Memory Service receives JWT, validates scopes
3. **Parallel queries:**
   - pgvector: "SELECT ... ORDER BY embedding <-> query_embedding LIMIT 50"
   - OpenSearch: "POST vault-X/_search" with BM25 query
4. **Fusion:** Combine top-50 results from each, score: `0.6*sem + 0.4*lex`
5. **Return:** Top-10 merged results with method indicator (hybrid / semantic_fallback / semantic)

### Graceful Degradation

- ✅ **OpenSearch healthy:** Hybrid search (60% + 40% fusion)
- ⚠️  **OpenSearch slow:** Timeout → fallback to semantic only
- ⚠️  **OpenSearch down:** Fallback to semantic only (no error)
- ❌ **pgvector down:** All queries fail (core dependency)

---

## 📋 Deployment Checklist Summary

### Phase 1: ✅ OpenSearch Deployed
- [x] StatefulSet: 2 replicas
- [x] Services: opensearch, opensearch-internal
- [x] ConfigMap: opensearch.yml
- [x] PVC: 30Gi storage
- [x] Dashboards: UI ready

### Phase 2: 🔄 Configure Memory Service (NEXT)
- [ ] Set OPENSEARCH_HOSTS env var
- [ ] Restart Memory Service pods
- [ ] Verify pod logs show connection success

### Phase 3: 🔄 Test API Endpoints
- [ ] Test vault endpoints
- [ ] Test semantic search
- [ ] Test hybrid search

### Phase 4: ⏳ Production Hardening (Future)
- [ ] Change Dashboards password
- [ ] Enable OpenSearch security plugin
- [ ] Setup OAuth2/SAML for Dashboards
- [ ] Enable audit logging
- [ ] Configure backup/recovery

---

## ✨ What's Next

**After Memory Service is configured with OPENSEARCH_HOSTS:**

1. **Vault Endpoints:** Test `vault.riotpiao.com` for file browsing
2. **Hybrid Search:** Test `memory.riotpiao.com/query` with hybrid results
3. **Frontend:** Deploy React app for UI (vault browser, search form)
4. **GRC Workflow:** Implement git + merge endpoints
5. **Agent Streaming:** WebSocket endpoint for real-time agent execution

---

Last Updated: Commit `630a125`