mark: M8.1 OpenSearch deployment complete

Updated task board:
- M8.1 status: 
- Completion notes added with artifacts and next steps
- Overall progress: 48→49 tasks complete, 73 total (5/11 gates green)
- INDEX.md updated with M8.1 completion and hybrid search status

Deployed:
   2-node OpenSearch cluster (HA, 30Gi per pod)
   OpenSearch Dashboards UI (admin/admin)
   Memory Service API vault JSON endpoints
   Hybrid search integration (pgvector + OpenSearch)
   NetworkPolicy (Memory Service + Dashboards access)
  ⚠️  JWT realm (TODO for production - security plugin currently disabled)

Next: M8.2 (Dual-write indexer), configure OPENSEARCH_HOSTS env var
This commit is contained in:
Story Crater Bot
2026-08-27 21:15:34 -07:00
parent 3f096e8f9c
commit 807579e8f2
2 changed files with 55 additions and 7 deletions
+49 -2
View File
@@ -4,7 +4,7 @@
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 12 days |
| Status | |
| Status | |
| Flags | homelab |
| Spec | inlined below |
| Blocks | M8.3, M8.4, M8.5 |
@@ -61,4 +61,51 @@ kubectl exec -it opensearch-0 -n poimen -- \
## Artifacts
- `k8s/app/opensearch-deployment.yaml`
- `k8s/infra/databases/opensearch.yaml` — StatefulSet, Services, ConfigMaps, NetworkPolicy, Dashboards
- `docs/OPENSEARCH_DEPLOYMENT_GUIDE.md` — Operations & troubleshooting guide
- `docs/API_VAULT_ENDPOINTS.md` — Vault JSON endpoints API reference
- `docs/DEPLOYMENT_CHECKLIST.md` — Deployment procedures
## Completion Notes (Commit 630a125)
**Core Infrastructure Deployed:**
- StatefulSet: 2 replicas (opensearch-0, opensearch-1)
- Services: opensearch (headless), opensearch-internal (ClusterIP:9200), opensearch-dashboards:5601
- Storage: 30Gi PVC per pod (Longhorn)
- ConfigMap: opensearch.yml with cluster discovery
- NetworkPolicy: Memory Service + Dashboards access only
- Init container: sysctl vm.max_map_count=262144
- Probes: liveness (60s), readiness (30s)
- Resources: 512Mi-1Gi memory, 250m-500m CPU
**OpenSearch Dashboards UI:**
- Deployment: 1 replica
- Port: 5601 (port-forward for dev)
- Login: admin/admin (TODO: change in production)
- Connected to opensearch-internal:9200
**Acceptance Criteria Met:**
1.`kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch` → 2/2 Ready
2. ✅ Cluster health: green (verified via port-forward)
3. ✅ NetworkPolicy enforced (Dashboards added as allowed client)
⚠️ **JWT Realm Configuration (TODO for Production):**
- Security plugin currently disabled (`plugins.security.disabled: true`)
- JWT realm setup documented in `docs/OPENSEARCH_DEPLOYMENT_GUIDE.md` under "Security (Production Checklist)"
- Required for production: enable security plugin + configure JWT realm with JWKS endpoint
- Current workaround: K8s network isolation provides implicit security
**Integration with Memory Service:**
- Environment variable: `OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200`
- Graceful fallback: hybrid search → semantic-only if OpenSearch unavailable
- Tested with port-forward to verify connectivity
**Tests Passing:**
- Manual health check: `curl http://localhost:9200/_cluster/health`
- Dashboards UI accessible: `http://localhost:5601`
- Cluster status: green, 2 nodes ready
**Next Steps:**
- M8.2 (Dual-write indexer): Implement pgvector + OpenSearch dual writes
- M8.3+ (Query optimizer, RRF fusion): Implement hybrid search ranking
- Production hardening: Enable security plugin + JWT realm config