chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,443 @@
|
||||
# OpenSearch + JWT Authentication Setup
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers deploying OpenSearch with JWT authentication integrated with Authentik, providing hybrid search (semantic + lexical) for the Poimen Memory service.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ GET /memory/query + JWT Bearer token │
|
||||
└────────────┬────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Memory Service (Rust) │
|
||||
│ ├─ Validate JWT (Authentik JWKS) │
|
||||
│ ├─ pgvector semantic search │
|
||||
│ ├─ OpenSearch lexical search │
|
||||
│ └─ Combine + rerank (hybrid) │
|
||||
└────────────┬────────────────────────────┘
|
||||
│
|
||||
┌──────┴──────┐
|
||||
│ │
|
||||
↓ ↓
|
||||
pgvector OpenSearch
|
||||
(semantic) (lexical + JWT)
|
||||
│
|
||||
├─ JWT realm (validate Authentik tokens)
|
||||
├─ Role mapping (extract from JWT claims)
|
||||
└─ Index-level permissions
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster (1.24+)
|
||||
- Authentik configured with poimen-memory OAuth2 app
|
||||
- PostgreSQL with pgvector (existing)
|
||||
- Memory Service deployed
|
||||
|
||||
## Step 1: Deploy OpenSearch with JWT Auth
|
||||
|
||||
### Apply the deployment manifest
|
||||
|
||||
```bash
|
||||
kubectl apply -f k8s/app/opensearch-deployment.yaml
|
||||
```
|
||||
|
||||
This creates:
|
||||
- **StatefulSet** (2 replicas, 30Gi PVC each)
|
||||
- **ConfigMap** with security config (JWT realm)
|
||||
- **Services** (headless + internal)
|
||||
- **Secret** for admin password
|
||||
- **NetworkPolicy** (only Memory Service access)
|
||||
|
||||
### Verify deployment
|
||||
|
||||
```bash
|
||||
# Wait for pods ready
|
||||
kubectl rollout status statefulset/opensearch -n poimen
|
||||
|
||||
# Check JWT realm configuration
|
||||
kubectl logs opensearch-0 -n poimen | grep -i jwt
|
||||
|
||||
# Health check
|
||||
kubectl exec -it opensearch-0 -n poimen -- curl -k --user admin:OpenSearch@Admin123! https://localhost:9200/_cluster/health
|
||||
```
|
||||
|
||||
## Step 2: Configure OpenSearch Security
|
||||
|
||||
### Port-forward to OpenSearch
|
||||
|
||||
```bash
|
||||
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200
|
||||
```
|
||||
|
||||
### Create index template
|
||||
|
||||
```bash
|
||||
curl -k -X PUT "https://localhost:9200/_index_template/vault" \
|
||||
-u admin:OpenSearch@Admin123! \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"index_patterns": ["vault-*"],
|
||||
"settings": {
|
||||
"number_of_shards": 2,
|
||||
"number_of_replicas": 1,
|
||||
"index.codec": "best_compression"
|
||||
},
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"content": {
|
||||
"type": "text",
|
||||
"analyzer": "standard"
|
||||
},
|
||||
"source": {
|
||||
"type": "keyword"
|
||||
},
|
||||
"level": {
|
||||
"type": "keyword"
|
||||
},
|
||||
"breadcrumb": {
|
||||
"type": "keyword"
|
||||
},
|
||||
"indexed_at": {
|
||||
"type": "date"
|
||||
}
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Verify JWT realm is working
|
||||
|
||||
```bash
|
||||
# Get a JWT from Authentik
|
||||
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=poimen-memory" \
|
||||
-d "client_secret=<secret>" \
|
||||
-d "scope=openid" | jq -r .access_token)
|
||||
|
||||
# Test OpenSearch with JWT
|
||||
curl -k -X GET "https://localhost:9200/_cluster/health" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
|
||||
# Should return cluster health (if JWT is valid)
|
||||
```
|
||||
|
||||
## Step 3: Update Memory Service Configuration
|
||||
|
||||
### Add environment variables
|
||||
|
||||
```yaml
|
||||
# k8s/app/memory-deployment.yaml
|
||||
env:
|
||||
- name: OPENSEARCH_HOSTS
|
||||
value: "opensearch-internal.poimen.svc.cluster.local:9200"
|
||||
- name: OPENSEARCH_ENABLED
|
||||
value: "true"
|
||||
- name: SEARCH_METHOD
|
||||
value: "hybrid" # hybrid | semantic | lexical
|
||||
- name: HYBRID_WEIGHTS_SEMANTIC
|
||||
value: "0.6"
|
||||
- name: HYBRID_WEIGHTS_LEXICAL
|
||||
value: "0.4"
|
||||
- name: OPENSEARCH_VERIFY_TLS
|
||||
value: "false" # For self-signed certs in dev
|
||||
```
|
||||
|
||||
### Update Cargo.toml
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
# Add OpenSearch client (if not using raw HTTP)
|
||||
opensearch = "2.1"
|
||||
serde_json = "1.0"
|
||||
tokio = "1.0"
|
||||
```
|
||||
|
||||
## Step 4: Test Hybrid Search
|
||||
|
||||
### Index a test document
|
||||
|
||||
```bash
|
||||
# Get JWT
|
||||
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=poimen-memory" \
|
||||
-d "client_secret=<secret>" \
|
||||
-d "scope=openid" | jq -r .access_token)
|
||||
|
||||
# Port-forward Memory Service
|
||||
kubectl port-forward -n poimen svc/poimen-memory 8080:8080
|
||||
|
||||
# Index a document via Memory Service
|
||||
curl -X POST http://localhost:8080/memory/vault/index \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"id": "test-doc",
|
||||
"content": "kubectl port-forward service 8080",
|
||||
"source": "runbooks/port-forward.md",
|
||||
"level": "L1",
|
||||
"breadcrumb": ["runbooks"]
|
||||
}'
|
||||
```
|
||||
|
||||
### Search hybrid
|
||||
|
||||
```bash
|
||||
# Semantic + Lexical search
|
||||
curl -X POST http://localhost:8080/memory/query \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"query": "fix kubernetes port 8080",
|
||||
"method": "hybrid",
|
||||
"limit": 10
|
||||
}' | jq .
|
||||
```
|
||||
|
||||
**Expected response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "fix kubernetes port 8080",
|
||||
"results": [
|
||||
{
|
||||
"id": "test-doc",
|
||||
"chunk": "kubectl port-forward service 8080",
|
||||
"score": 0.92,
|
||||
"source": "runbooks/port-forward.md",
|
||||
"level": "L1",
|
||||
"breadcrumb": ["runbooks"],
|
||||
"method": "hybrid",
|
||||
"breakdown": {
|
||||
"semantic": 0.88,
|
||||
"lexical": 0.96
|
||||
}
|
||||
}
|
||||
],
|
||||
"total": 1,
|
||||
"search_method": "hybrid"
|
||||
}
|
||||
```
|
||||
|
||||
## Step 5: JWT Token Validation Details
|
||||
|
||||
### How OpenSearch validates JWT
|
||||
|
||||
1. **Token arrives**: `Authorization: Bearer eyJh...`
|
||||
2. **OpenSearch extracts**: Token after "Bearer "
|
||||
3. **Validates signature**: Using JWKS from Authentik
|
||||
4. **Extracts claims**: `sub`, `roles`, `permissions`
|
||||
5. **Maps to user**: Creates internal user from JWT
|
||||
6. **Checks permissions**: Verifies access to indices
|
||||
|
||||
### JWT Claims Expected
|
||||
|
||||
```json
|
||||
{
|
||||
"iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
|
||||
"aud": "opensearch",
|
||||
"sub": "[email protected]",
|
||||
"roles": ["read_vault", "write_vault"],
|
||||
"permissions": ["memory:read", "memory:write"],
|
||||
"exp": 1234567890,
|
||||
"iat": 1234567800
|
||||
}
|
||||
```
|
||||
|
||||
### Update Authentik OAuth2 App
|
||||
|
||||
Ensure the poimen-memory app includes custom claims:
|
||||
|
||||
```
|
||||
Scope: openid email profile
|
||||
Custom Claims:
|
||||
- roles: ["memory:read", "memory:write"]
|
||||
- permissions: ["memory:read", "memory:write"]
|
||||
```
|
||||
|
||||
## Step 6: Role-Based Access Control (RBAC)
|
||||
|
||||
### Available Roles in OpenSearch
|
||||
|
||||
```yaml
|
||||
read_vault:
|
||||
- Can search vault indices
|
||||
- Can read documents
|
||||
- No write permissions
|
||||
|
||||
write_vault:
|
||||
- Can index new documents
|
||||
- Can update existing
|
||||
- Can read documents
|
||||
|
||||
all_access:
|
||||
- Full cluster access
|
||||
- Admin role
|
||||
```
|
||||
|
||||
### Map JWT Roles to OpenSearch Roles
|
||||
|
||||
Edit `internal_users.yml` in ConfigMap:
|
||||
|
||||
```yaml
|
||||
authc:
|
||||
realms:
|
||||
jwt_realm:
|
||||
type: jwt
|
||||
roles_key: roles # Extract "roles" claim from JWT
|
||||
claims_mapping:
|
||||
principal: sub
|
||||
roles: roles
|
||||
```
|
||||
|
||||
### Test role enforcement
|
||||
|
||||
```bash
|
||||
# User with read_vault role only
|
||||
curl -X GET "https://localhost:9200/vault-*/_search" \
|
||||
-H "Authorization: Bearer <read-only-jwt>"
|
||||
# ✅ Success (read allowed)
|
||||
|
||||
curl -X PUT "https://localhost:9200/vault-test/_doc/123" \
|
||||
-H "Authorization: Bearer <read-only-jwt>" \
|
||||
-d '{"content": "test"}'
|
||||
# ❌ 403 Forbidden (write denied)
|
||||
```
|
||||
|
||||
## Step 7: Monitoring & Troubleshooting
|
||||
|
||||
### Check OpenSearch logs
|
||||
|
||||
```bash
|
||||
kubectl logs opensearch-0 -n poimen -f --tail=50
|
||||
```
|
||||
|
||||
### JWT validation errors
|
||||
|
||||
If you see "JWT verification failed":
|
||||
|
||||
1. Verify JWKS endpoint is accessible:
|
||||
```bash
|
||||
curl https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
|
||||
```
|
||||
|
||||
2. Check token expiry:
|
||||
```bash
|
||||
TOKEN="..."
|
||||
echo $TOKEN | cut -d. -f2 | base64 -d | jq .exp
|
||||
date +%s
|
||||
```
|
||||
|
||||
3. Verify issuer matches config:
|
||||
```bash
|
||||
echo $TOKEN | cut -d. -f2 | base64 -d | jq .iss
|
||||
# Should equal: https://authentik.riotpiao.com/application/o/poimen-memory/
|
||||
```
|
||||
|
||||
### Cluster health
|
||||
|
||||
```bash
|
||||
kubectl exec -it opensearch-0 -n poimen -- curl -k \
|
||||
--user admin:OpenSearch@Admin123! \
|
||||
https://localhost:9200/_cluster/health | jq .
|
||||
```
|
||||
|
||||
### Search latency
|
||||
|
||||
Monitor hybrid search performance:
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:8080/memory/metrics?type=search \
|
||||
-H "Authorization: Bearer $TOKEN" | jq .
|
||||
```
|
||||
|
||||
## Step 8: Migration from Elasticsearch (if applicable)
|
||||
|
||||
### Reindex Elasticsearch to OpenSearch
|
||||
|
||||
```bash
|
||||
# Export from Elasticsearch
|
||||
curl -X POST "elasticsearch:9200/_reindex" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"source": {
|
||||
"index": "vault-*"
|
||||
},
|
||||
"dest": {
|
||||
"index": "vault-"
|
||||
}
|
||||
}'
|
||||
|
||||
# Import to OpenSearch
|
||||
# (Use snapshot/restore or Logstash)
|
||||
```
|
||||
|
||||
## Security Checklist
|
||||
|
||||
- [x] OpenSearch JWT realm configured
|
||||
- [x] JWKS endpoint from Authentik is reachable
|
||||
- [x] NetworkPolicy restricts access (Memory Service only)
|
||||
- [x] TLS enabled (self-signed certs for dev, proper certs for prod)
|
||||
- [x] Admin password changed from default
|
||||
- [x] JWT token validation enabled
|
||||
- [x] Roles mapped from JWT claims
|
||||
- [x] Index-level permissions enforced
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Optimize search performance
|
||||
|
||||
```yaml
|
||||
# In opensearch.yml
|
||||
indices:
|
||||
memory:
|
||||
max_result_window: 50000 # Increase result set size
|
||||
queries:
|
||||
cache:
|
||||
size: 20% # Allocate 20% heap to query cache
|
||||
```
|
||||
|
||||
### Heap allocation
|
||||
|
||||
```yaml
|
||||
# For 2 replicas with 2Gi each
|
||||
-Xms2g -Xmx2g
|
||||
# Total: 4Gi per node
|
||||
```
|
||||
|
||||
### Shard configuration
|
||||
|
||||
```yaml
|
||||
# Index settings
|
||||
number_of_shards: 2 # Match cluster node count
|
||||
number_of_replicas: 1 # One replica per shard
|
||||
refresh_interval: 30s # Batch writes
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If OpenSearch doesn't work:
|
||||
|
||||
```bash
|
||||
# Revert to semantic-only search
|
||||
kubectl set env deployment/poimen-memory SEARCH_METHOD=semantic
|
||||
|
||||
# Keep OpenSearch pods running (no data loss)
|
||||
# No indexing to OpenSearch
|
||||
# Queries use pgvector only
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Deploy OpenSearch + JWT
|
||||
2. ✅ Configure hybrid search in Memory Service
|
||||
3. ⏳ Run end-to-end tests
|
||||
4. ⏳ Monitor metrics (latency, accuracy)
|
||||
5. ⏳ Gradual rollout (feature flag: 10% → 50% → 100%)
|
||||
Reference in New Issue
Block a user