poimen-memory/docs/OPENSEARCH_JWT_SETUP.md

OpenSearch + JWT Authentication Setup

Overview

This guide covers deploying OpenSearch with JWT authentication integrated with Authentik, providing hybrid search (semantic + lexical) for the Poimen Memory service.

Architecture

┌─────────────────────────────────────────┐
│ Frontend (React)                        │
│ GET /memory/query + JWT Bearer token    │
└────────────┬────────────────────────────┘
             │
             ↓
┌─────────────────────────────────────────┐
│ Memory Service (Rust)                   │
│ ├─ Validate JWT (Authentik JWKS)        │
│ ├─ pgvector semantic search              │
│ ├─ OpenSearch lexical search             │
│ └─ Combine + rerank (hybrid)             │
└────────────┬────────────────────────────┘
             │
      ┌──────┴──────┐
      │             │
      ↓             ↓
  pgvector     OpenSearch
  (semantic)   (lexical + JWT)
              │
              ├─ JWT realm (validate Authentik tokens)
              ├─ Role mapping (extract from JWT claims)
              └─ Index-level permissions

Prerequisites

• Kubernetes cluster (1.24+)
• Authentik configured with poimen-memory OAuth2 app
• PostgreSQL with pgvector (existing)
• Memory Service deployed

Step 1: Deploy OpenSearch with JWT Auth

Apply the deployment manifest

kubectl apply -f k8s/app/opensearch-deployment.yaml

This creates:

• StatefulSet (2 replicas, 30Gi PVC each)
• ConfigMap with security config (JWT realm)
• Services (headless + internal)
• Secret for admin password
• NetworkPolicy (only Memory Service access)

Verify deployment

# Wait for pods ready
kubectl rollout status statefulset/opensearch -n poimen

# Check JWT realm configuration
kubectl logs opensearch-0 -n poimen | grep -i jwt

# Health check
kubectl exec -it opensearch-0 -n poimen -- curl -k --user admin:OpenSearch@Admin123! https://localhost:9200/_cluster/health

Step 2: Configure OpenSearch Security

Port-forward to OpenSearch

kubectl port-forward -n poimen svc/opensearch-internal 9200:9200

Create index template

curl -k -X PUT "https://localhost:9200/_index_template/vault" \
  -u admin:OpenSearch@Admin123! \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["vault-*"],
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.codec": "best_compression"
    },
    "mappings": {
      "properties": {
        "content": {
          "type": "text",
          "analyzer": "standard"
        },
        "source": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "breadcrumb": {
          "type": "keyword"
        },
        "indexed_at": {
          "type": "date"
        }
      }
    }
  }'

Verify JWT realm is working

# Get a JWT from Authentik
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
  -d "grant_type=client_credentials" \
  -d "client_id=poimen-memory" \
  -d "client_secret=<secret>" \
  -d "scope=openid" | jq -r .access_token)

# Test OpenSearch with JWT
curl -k -X GET "https://localhost:9200/_cluster/health" \
  -H "Authorization: Bearer $TOKEN"

# Should return cluster health (if JWT is valid)

Step 3: Update Memory Service Configuration

Add environment variables

# k8s/app/memory-deployment.yaml
env:
  - name: OPENSEARCH_HOSTS
    value: "opensearch-internal.poimen.svc.cluster.local:9200"
  - name: OPENSEARCH_ENABLED
    value: "true"
  - name: SEARCH_METHOD
    value: "hybrid"  # hybrid | semantic | lexical
  - name: HYBRID_WEIGHTS_SEMANTIC
    value: "0.6"
  - name: HYBRID_WEIGHTS_LEXICAL
    value: "0.4"
  - name: OPENSEARCH_VERIFY_TLS
    value: "false"  # For self-signed certs in dev

Update Cargo.toml

[dependencies]
# Add OpenSearch client (if not using raw HTTP)
opensearch = "2.1"
serde_json = "1.0"
tokio = "1.0"

Step 4: Test Hybrid Search

Index a test document

# Get JWT
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
  -d "grant_type=client_credentials" \
  -d "client_id=poimen-memory" \
  -d "client_secret=<secret>" \
  -d "scope=openid" | jq -r .access_token)

# Port-forward Memory Service
kubectl port-forward -n poimen svc/poimen-memory 8080:8080

# Index a document via Memory Service
curl -X POST http://localhost:8080/memory/vault/index \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "test-doc",
    "content": "kubectl port-forward service 8080",
    "source": "runbooks/port-forward.md",
    "level": "L1",
    "breadcrumb": ["runbooks"]
  }'

Search hybrid

# Semantic + Lexical search
curl -X POST http://localhost:8080/memory/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "fix kubernetes port 8080",
    "method": "hybrid",
    "limit": 10
  }' | jq .

Expected response:

{
  "query": "fix kubernetes port 8080",
  "results": [
    {
      "id": "test-doc",
      "chunk": "kubectl port-forward service 8080",
      "score": 0.92,
      "source": "runbooks/port-forward.md",
      "level": "L1",
      "breadcrumb": ["runbooks"],
      "method": "hybrid",
      "breakdown": {
        "semantic": 0.88,
        "lexical": 0.96
      }
    }
  ],
  "total": 1,
  "search_method": "hybrid"
}

Step 5: JWT Token Validation Details

How OpenSearch validates JWT

1. Token arrives: Authorization: Bearer eyJh...
2. OpenSearch extracts: Token after "Bearer "
3. Validates signature: Using JWKS from Authentik
4. Extracts claims: sub, roles, permissions
5. Maps to user: Creates internal user from JWT
6. Checks permissions: Verifies access to indices

JWT Claims Expected

{
  "iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
  "aud": "opensearch",
  "sub": "[email protected]",
  "roles": ["read_vault", "write_vault"],
  "permissions": ["memory:read", "memory:write"],
  "exp": 1234567890,
  "iat": 1234567800
}

Update Authentik OAuth2 App

Ensure the poimen-memory app includes custom claims:

Scope: openid email profile
Custom Claims:
  - roles: ["memory:read", "memory:write"]
  - permissions: ["memory:read", "memory:write"]

Step 6: Role-Based Access Control (RBAC)

Available Roles in OpenSearch

read_vault:
  - Can search vault indices
  - Can read documents
  - No write permissions

write_vault:
  - Can index new documents
  - Can update existing
  - Can read documents

all_access:
  - Full cluster access
  - Admin role

Map JWT Roles to OpenSearch Roles

Edit internal_users.yml in ConfigMap:

authc:
  realms:
    jwt_realm:
      type: jwt
      roles_key: roles  # Extract "roles" claim from JWT
      claims_mapping:
        principal: sub
        roles: roles

Test role enforcement

# User with read_vault role only
curl -X GET "https://localhost:9200/vault-*/_search" \
  -H "Authorization: Bearer <read-only-jwt>"
# ✅ Success (read allowed)

curl -X PUT "https://localhost:9200/vault-test/_doc/123" \
  -H "Authorization: Bearer <read-only-jwt>" \
  -d '{"content": "test"}'
# ❌ 403 Forbidden (write denied)

Step 7: Monitoring & Troubleshooting

Check OpenSearch logs

kubectl logs opensearch-0 -n poimen -f --tail=50

JWT validation errors

If you see "JWT verification failed":

1. Verify JWKS endpoint is accessible:

   curl https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
   

2. Check token expiry:

   TOKEN="..."
   echo $TOKEN | cut -d. -f2 | base64 -d | jq .exp
   date +%s
   

3. Verify issuer matches config:

   echo $TOKEN | cut -d. -f2 | base64 -d | jq .iss
   # Should equal: https://authentik.riotpiao.com/application/o/poimen-memory/
   

Cluster health

kubectl exec -it opensearch-0 -n poimen -- curl -k \
  --user admin:OpenSearch@Admin123! \
  https://localhost:9200/_cluster/health | jq .

Search latency

Monitor hybrid search performance:

curl -X GET http://localhost:8080/memory/metrics?type=search \
  -H "Authorization: Bearer $TOKEN" | jq .

Step 8: Migration from Elasticsearch (if applicable)

Reindex Elasticsearch to OpenSearch

# Export from Elasticsearch
curl -X POST "elasticsearch:9200/_reindex" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": {
      "index": "vault-*"
    },
    "dest": {
      "index": "vault-"
    }
  }'

# Import to OpenSearch
# (Use snapshot/restore or Logstash)

Security Checklist

• OpenSearch JWT realm configured
• JWKS endpoint from Authentik is reachable
• NetworkPolicy restricts access (Memory Service only)
• TLS enabled (self-signed certs for dev, proper certs for prod)
• Admin password changed from default
• JWT token validation enabled
• Roles mapped from JWT claims
• Index-level permissions enforced

Performance Tuning

Optimize search performance

# In opensearch.yml
indices:
  memory:
    max_result_window: 50000  # Increase result set size
  queries:
    cache:
      size: 20%  # Allocate 20% heap to query cache

Heap allocation

# For 2 replicas with 2Gi each
-Xms2g -Xmx2g
# Total: 4Gi per node

Shard configuration

# Index settings
number_of_shards: 2        # Match cluster node count
number_of_replicas: 1      # One replica per shard
refresh_interval: 30s      # Batch writes

Rollback Plan

If OpenSearch doesn't work:

# Revert to semantic-only search
kubectl set env deployment/poimen-memory SEARCH_METHOD=semantic

# Keep OpenSearch pods running (no data loss)
# No indexing to OpenSearch
# Queries use pgvector only

Next Steps

1. ✅ Deploy OpenSearch + JWT
2. ✅ Configure hybrid search in Memory Service
3. ⏳ Run end-to-end tests
4. ⏳ Monitor metrics (latency, accuracy)
5. ⏳ Gradual rollout (feature flag: 10% → 50% → 100%)