Story Crater Bot
959c596b1d
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
65 lines
2.6 KiB
Markdown
65 lines
2.6 KiB
Markdown
# M8.1 — OpenSearch cluster deployment + JWT realm
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M8 — Hybrid Search |
|
||
| Size | M — 1–2 days |
|
||
| Status | ⬜ |
|
||
| Flags | homelab |
|
||
| Spec | inlined below |
|
||
| Blocks | M8.3, M8.4, M8.5 |
|
||
| Depends | M3.5.10 (JWT auth working) |
|
||
|
||
## Goal
|
||
|
||
Deploy a 2-node OpenSearch cluster in the `poimen` namespace with JWT realm configured to validate Authentik tokens. NetworkPolicy restricts access to Memory Service pods only.
|
||
|
||
## Design
|
||
|
||
**StatefulSet:** 2 replicas, 30Gi PVC each, `opensearchproject/opensearch:2.11.0`.
|
||
|
||
**Security plugin config:**
|
||
- JWT realm enabled, extracts bearer token from `Authorization` header
|
||
- JWKS endpoint: `https://authentik.riotpiao.com/application/o/poimen-memory/jwks/`
|
||
- Roles extracted from JWT `roles` claim
|
||
- Two internal roles: `read_vault` (search only), `write_vault` (search + index)
|
||
|
||
**Services:**
|
||
- `opensearch` — headless, for StatefulSet peer discovery (port 9300)
|
||
- `opensearch-internal` — ClusterIP, for Memory Service queries (port 9200)
|
||
|
||
**NetworkPolicy:** Only pods with label `app.kubernetes.io/name: poimen-memory` can reach port 9200.
|
||
|
||
## Steps
|
||
|
||
1. Apply `k8s/app/opensearch-deployment.yaml` (StatefulSet, Services, ConfigMap, Secret, NetworkPolicy).
|
||
2. Wait for both pods Ready.
|
||
3. Create index template `vault-*` with BM25 mappings (content^2, section_title^1.5, breadcrumb, source, project_id, level, indexed_at).
|
||
4. Run security admin tool to load JWT realm config.
|
||
5. Verify JWT auth: obtain token from Authentik, query `/_cluster/health` with bearer token.
|
||
|
||
## Acceptance
|
||
|
||
1. `kubectl get pods -n poimen -l app=opensearch` shows 2/2 Ready.
|
||
2. `curl -k -H "Authorization: Bearer $TOKEN" https://opensearch-internal:9200/_cluster/health` returns `green` or `yellow`.
|
||
3. Request without token returns 401.
|
||
4. Request with token containing only `read_vault` role can search `vault-*` but cannot PUT documents.
|
||
5. Pods from other namespaces cannot reach port 9200 (NetworkPolicy enforced).
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
kubectl rollout status statefulset/opensearch -n poimen --timeout=300s
|
||
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
|
||
-d grant_type=client_credentials -d client_id=poimen-memory \
|
||
-d "client_secret=$SECRET" -d scope=openid | jq -r .access_token)
|
||
kubectl exec -it opensearch-0 -n poimen -- \
|
||
curl -k -H "Authorization: Bearer $TOKEN" https://localhost:9200/_cluster/health
|
||
```
|
||
|
||
**False pass:** Cluster health returns `green` but `DISABLE_SECURITY_PLUGIN=true` was set — JWT realm is not actually validating. Check by sending a garbage token; it must return 401.
|
||
|
||
## Artifacts
|
||
|
||
- `k8s/app/opensearch-deployment.yaml`
|