Files

T

Story Crater Bot 959c596b1d chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)

Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.

2026-08-27 20:25:05 -07:00

2.6 KiB

Raw Blame History

M8.1 — OpenSearch cluster deployment + JWT realm

Field	Value
Phase	M8 — Hybrid Search
Size	M — 1–2 days
Status	⬜
Flags	homelab
Spec	inlined below
Blocks	M8.3, M8.4, M8.5
Depends	M3.5.10 (JWT auth working)

Goal

Deploy a 2-node OpenSearch cluster in the poimen namespace with JWT realm configured to validate Authentik tokens. NetworkPolicy restricts access to Memory Service pods only.

Design

StatefulSet: 2 replicas, 30Gi PVC each, opensearchproject/opensearch:2.11.0.

Security plugin config:

JWT realm enabled, extracts bearer token from Authorization header
JWKS endpoint: https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
Roles extracted from JWT roles claim
Two internal roles: read_vault (search only), write_vault (search + index)

Services:

opensearch — headless, for StatefulSet peer discovery (port 9300)
opensearch-internal — ClusterIP, for Memory Service queries (port 9200)

NetworkPolicy: Only pods with label app.kubernetes.io/name: poimen-memory can reach port 9200.

Steps

Apply k8s/app/opensearch-deployment.yaml (StatefulSet, Services, ConfigMap, Secret, NetworkPolicy).
Wait for both pods Ready.
Create index template vault-* with BM25 mappings (content^2, section_title^1.5, breadcrumb, source, project_id, level, indexed_at).
Run security admin tool to load JWT realm config.
Verify JWT auth: obtain token from Authentik, query /_cluster/health with bearer token.

Acceptance

kubectl get pods -n poimen -l app=opensearch shows 2/2 Ready.
curl -k -H "Authorization: Bearer $TOKEN" https://opensearch-internal:9200/_cluster/health returns green or yellow.
Request without token returns 401.
Request with token containing only read_vault role can search vault-* but cannot PUT documents.
Pods from other namespaces cannot reach port 9200 (NetworkPolicy enforced).

Verify

kubectl rollout status statefulset/opensearch -n poimen --timeout=300s
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d grant_type=client_credentials -d client_id=poimen-memory \
  -d "client_secret=$SECRET" -d scope=openid | jq -r .access_token)
kubectl exec -it opensearch-0 -n poimen -- \
  curl -k -H "Authorization: Bearer $TOKEN" https://localhost:9200/_cluster/health

False pass: Cluster health returns green but DISABLE_SECURITY_PLUGIN=true was set — JWT realm is not actually validating. Check by sending a garbage token; it must return 401.

Artifacts

k8s/app/opensearch-deployment.yaml

2.6 KiB Raw Blame History Unescape Escape