poimen-memory/docs/OPENSEARCH_JWT_SETUP.md

# OpenSearch + JWT Authentication Setup

## Overview

This guide covers deploying OpenSearch with JWT authentication integrated with Authentik, providing hybrid search (semantic + lexical) for the Poimen Memory service.

## Architecture

```
┌─────────────────────────────────────────┐
│ Frontend (React)                        │
│ GET /memory/query + JWT Bearer token    │
└────────────┬────────────────────────────┘
             │
             ↓
┌─────────────────────────────────────────┐
│ Memory Service (Rust)                   │
│ ├─ Validate JWT (Authentik JWKS)        │
│ ├─ pgvector semantic search              │
│ ├─ OpenSearch lexical search             │
│ └─ Combine + rerank (hybrid)             │
└────────────┬────────────────────────────┘
             │
      ┌──────┴──────┐
      │             │
      ↓             ↓
  pgvector     OpenSearch
  (semantic)   (lexical + JWT)
              │
              ├─ JWT realm (validate Authentik tokens)
              ├─ Role mapping (extract from JWT claims)
              └─ Index-level permissions
```

## Prerequisites

- Kubernetes cluster (1.24+)
- Authentik configured with poimen-memory OAuth2 app
- PostgreSQL with pgvector (existing)
- Memory Service deployed

## Step 1: Deploy OpenSearch with JWT Auth

### Apply the deployment manifest

```bash
kubectl apply -f k8s/app/opensearch-deployment.yaml
```

This creates:
- **StatefulSet** (2 replicas, 30Gi PVC each)
- **ConfigMap** with security config (JWT realm)
- **Services** (headless + internal)
- **Secret** for admin password
- **NetworkPolicy** (only Memory Service access)

### Verify deployment

```bash
# Wait for pods ready
kubectl rollout status statefulset/opensearch -n poimen

# Check JWT realm configuration
kubectl logs opensearch-0 -n poimen | grep -i jwt

# Health check
kubectl exec -it opensearch-0 -n poimen -- curl -k --user admin:OpenSearch@Admin123! https://localhost:9200/_cluster/health
```

## Step 2: Configure OpenSearch Security

### Port-forward to OpenSearch

```bash
kubectl port-forward -n poimen svc/opensearch-internal 9200:9200
```

### Create index template

```bash
curl -k -X PUT "https://localhost:9200/_index_template/vault" \
  -u admin:OpenSearch@Admin123! \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["vault-*"],
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.codec": "best_compression"
    },
    "mappings": {
      "properties": {
        "content": {
          "type": "text",
          "analyzer": "standard"
        },
        "source": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "breadcrumb": {
          "type": "keyword"
        },
        "indexed_at": {
          "type": "date"
        }
      }
    }
  }'
```

### Verify JWT realm is working

```bash
# Get a JWT from Authentik
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
  -d "grant_type=client_credentials" \
  -d "client_id=poimen-memory" \
  -d "client_secret=<secret>" \
  -d "scope=openid" | jq -r .access_token)

# Test OpenSearch with JWT
curl -k -X GET "https://localhost:9200/_cluster/health" \
  -H "Authorization: Bearer $TOKEN"

# Should return cluster health (if JWT is valid)
```

## Step 3: Update Memory Service Configuration

### Add environment variables

```yaml
# k8s/app/memory-deployment.yaml
env:
  - name: OPENSEARCH_HOSTS
    value: "opensearch-internal.poimen.svc.cluster.local:9200"
  - name: OPENSEARCH_ENABLED
    value: "true"
  - name: SEARCH_METHOD
    value: "hybrid"  # hybrid | semantic | lexical
  - name: HYBRID_WEIGHTS_SEMANTIC
    value: "0.6"
  - name: HYBRID_WEIGHTS_LEXICAL
    value: "0.4"
  - name: OPENSEARCH_VERIFY_TLS
    value: "false"  # For self-signed certs in dev
```

### Update Cargo.toml

```toml
[dependencies]
# Add OpenSearch client (if not using raw HTTP)
opensearch = "2.1"
serde_json = "1.0"
tokio = "1.0"
```

## Step 4: Test Hybrid Search

### Index a test document

```bash
# Get JWT
TOKEN=$(curl -s -X POST http://localhost:9000/application/o/token/ \
  -d "grant_type=client_credentials" \
  -d "client_id=poimen-memory" \
  -d "client_secret=<secret>" \
  -d "scope=openid" | jq -r .access_token)

# Port-forward Memory Service
kubectl port-forward -n poimen svc/poimen-memory 8080:8080

# Index a document via Memory Service
curl -X POST http://localhost:8080/memory/vault/index \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "test-doc",
    "content": "kubectl port-forward service 8080",
    "source": "runbooks/port-forward.md",
    "level": "L1",
    "breadcrumb": ["runbooks"]
  }'
```

### Search hybrid

```bash
# Semantic + Lexical search
curl -X POST http://localhost:8080/memory/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "fix kubernetes port 8080",
    "method": "hybrid",
    "limit": 10
  }' | jq .
```

**Expected response:**

```json
{
  "query": "fix kubernetes port 8080",
  "results": [
    {
      "id": "test-doc",
      "chunk": "kubectl port-forward service 8080",
      "score": 0.92,
      "source": "runbooks/port-forward.md",
      "level": "L1",
      "breadcrumb": ["runbooks"],
      "method": "hybrid",
      "breakdown": {
        "semantic": 0.88,
        "lexical": 0.96
      }
    }
  ],
  "total": 1,
  "search_method": "hybrid"
}
```

## Step 5: JWT Token Validation Details

### How OpenSearch validates JWT

1. **Token arrives**: `Authorization: Bearer eyJh...`
2. **OpenSearch extracts**: Token after "Bearer "
3. **Validates signature**: Using JWKS from Authentik
4. **Extracts claims**: `sub`, `roles`, `permissions`
5. **Maps to user**: Creates internal user from JWT
6. **Checks permissions**: Verifies access to indices

### JWT Claims Expected

```json
{
  "iss": "https://authentik.riotpiao.com/application/o/poimen-memory/",
  "aud": "opensearch",
  "sub": "[email protected]",
  "roles": ["read_vault", "write_vault"],
  "permissions": ["memory:read", "memory:write"],
  "exp": 1234567890,
  "iat": 1234567800
}
```

### Update Authentik OAuth2 App

Ensure the poimen-memory app includes custom claims:

```
Scope: openid email profile
Custom Claims:
  - roles: ["memory:read", "memory:write"]
  - permissions: ["memory:read", "memory:write"]
```

## Step 6: Role-Based Access Control (RBAC)

### Available Roles in OpenSearch

```yaml
read_vault:
  - Can search vault indices
  - Can read documents
  - No write permissions

write_vault:
  - Can index new documents
  - Can update existing
  - Can read documents

all_access:
  - Full cluster access
  - Admin role
```

### Map JWT Roles to OpenSearch Roles

Edit `internal_users.yml` in ConfigMap:

```yaml
authc:
  realms:
    jwt_realm:
      type: jwt
      roles_key: roles  # Extract "roles" claim from JWT
      claims_mapping:
        principal: sub
        roles: roles
```

### Test role enforcement

```bash
# User with read_vault role only
curl -X GET "https://localhost:9200/vault-*/_search" \
  -H "Authorization: Bearer <read-only-jwt>"
# ✅ Success (read allowed)

curl -X PUT "https://localhost:9200/vault-test/_doc/123" \
  -H "Authorization: Bearer <read-only-jwt>" \
  -d '{"content": "test"}'
# ❌ 403 Forbidden (write denied)
```

## Step 7: Monitoring & Troubleshooting

### Check OpenSearch logs

```bash
kubectl logs opensearch-0 -n poimen -f --tail=50
```

### JWT validation errors

If you see "JWT verification failed":

1. Verify JWKS endpoint is accessible:
   ```bash
   curl https://authentik.riotpiao.com/application/o/poimen-memory/jwks/
   ```

2. Check token expiry:
   ```bash
   TOKEN="..."
   echo $TOKEN | cut -d. -f2 | base64 -d | jq .exp
   date +%s
   ```

3. Verify issuer matches config:
   ```bash
   echo $TOKEN | cut -d. -f2 | base64 -d | jq .iss
   # Should equal: https://authentik.riotpiao.com/application/o/poimen-memory/
   ```

### Cluster health

```bash
kubectl exec -it opensearch-0 -n poimen -- curl -k \
  --user admin:OpenSearch@Admin123! \
  https://localhost:9200/_cluster/health | jq .
```

### Search latency

Monitor hybrid search performance:

```bash
curl -X GET http://localhost:8080/memory/metrics?type=search \
  -H "Authorization: Bearer $TOKEN" | jq .
```

## Step 8: Migration from Elasticsearch (if applicable)

### Reindex Elasticsearch to OpenSearch

```bash
# Export from Elasticsearch
curl -X POST "elasticsearch:9200/_reindex" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": {
      "index": "vault-*"
    },
    "dest": {
      "index": "vault-"
    }
  }'

# Import to OpenSearch
# (Use snapshot/restore or Logstash)
```

## Security Checklist

- [x] OpenSearch JWT realm configured
- [x] JWKS endpoint from Authentik is reachable
- [x] NetworkPolicy restricts access (Memory Service only)
- [x] TLS enabled (self-signed certs for dev, proper certs for prod)
- [x] Admin password changed from default
- [x] JWT token validation enabled
- [x] Roles mapped from JWT claims
- [x] Index-level permissions enforced

## Performance Tuning

### Optimize search performance

```yaml
# In opensearch.yml
indices:
  memory:
    max_result_window: 50000  # Increase result set size
  queries:
    cache:
      size: 20%  # Allocate 20% heap to query cache
```

### Heap allocation

```yaml
# For 2 replicas with 2Gi each
-Xms2g -Xmx2g
# Total: 4Gi per node
```

### Shard configuration

```yaml
# Index settings
number_of_shards: 2        # Match cluster node count
number_of_replicas: 1      # One replica per shard
refresh_interval: 30s      # Batch writes
```

## Rollback Plan

If OpenSearch doesn't work:

```bash
# Revert to semantic-only search
kubectl set env deployment/poimen-memory SEARCH_METHOD=semantic

# Keep OpenSearch pods running (no data loss)
# No indexing to OpenSearch
# Queries use pgvector only
```

## Next Steps

1. ✅ Deploy OpenSearch + JWT
2. ✅ Configure hybrid search in Memory Service
3. ⏳ Run end-to-end tests
4. ⏳ Monitor metrics (latency, accuracy)
5. ⏳ Gradual rollout (feature flag: 10% → 50% → 100%)