feat: authentik jwt + sops encryption for prod secrets & llm auth

SECURITY: - Add authentik_jwt.rs: OAuth2 client credentials flow with caching - SOPS encrypt secrets with age key (SOPS_AGE_KEY_FILE) - JWT tokens for LLM gateway, S3, and API gateway access - Token auto-refresh when expired (60s before expiry) - No hardcoded credentials in code or config ENTITY EXTRACTION: - LlmEntityExtractor now uses Authentik JWT instead of mock - Fallback to env var if Authentik not configured - Reflection verification still enabled - WikiLink extraction as Stage 0 (always active) DEPLOYMENT: - ConfigMap: LLM_ENDPOINT, LLM_MODEL, timeouts - Secret: AUTHENTIK_ISSUER, CLIENT_ID, CLIENT_SECRET, S3 keys - envFrom mounts both ConfigMap and Secret - KSOPS plugin for ArgoCD auto-decryption DOCUMENTATION: - docs/AUTHENTIK_SOPS_SETUP.md: Complete integration guide - Service account creation in Authentik - SOPS encryption/decryption workflow - JWT token exchange flow - Troubleshooting guide FILES: - crates/mem-ingest/src/authentik_jwt.rs (new, 180 LOC) - crates/mem-ingest/src/entity_extractor.rs (updated, JWT auth) - crates/mem-ingest/Cargo.toml (add reqwest) - k8s/app/poimen-memory-secrets.yaml (new, unencrypted template) - k8s/app/deployment.yaml (add secrets envFrom) - k8s/app/config.yaml (add LLM config) - k8s/.sops.yaml (encryption rules) - docs/AUTHENTIK_SOPS_SETUP.md (new, 350 LOC) NEXT: 1. Create Authentik service account (manual) 2. Encrypt secrets with SOPS 3. Deploy to poimen namespace 4. Test JWT token exchange with LLM endpoint
2026-09-08 13:58:39 -07:00
parent 800d9d8ae2
commit 16e3ff16f1
11 changed files with 607 additions and 7 deletions
@@ -0,0 +1,321 @@
+# Poimen Memory: Authentik JWT + SOPS Encryption Setup
+
+## Overview
+
+The Poimen Memory service uses:
+1. **Authentik service account** for OAuth2 client credentials flow
+2. **SOPS + Age encryption** to encrypt secrets in git
+3. **JWT tokens** for authentication to LLM gateway, S3, and other services
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      Kubernetes (poimen)                    │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  ┌──────────────┐    ┌─────────────────┐                  │
+│  │  ConfigMap   │    │  Secret (SOPS)  │                  │
+│  │ (unencrypted)│    │  (age-encrypted)│                  │
+│  └──────┬───────┘    └────────┬────────┘                  │
+│         │                     │                            │
+│         ├─────────┬───────────┤                            │
+│         │         │           │                            │
+│    ┌────▼─────────▼───────────▼────┐                      │
+│    │   poimen-memory Pod            │                     │
+│    │   Environment Variables:       │                     │
+│    │   - LLM_ENDPOINT               │                     │
+│    │   - AUTHENTIK_ISSUER           │                     │
+│    │   - AUTHENTIK_CLIENT_ID        │                     │
+│    │   - AUTHENTIK_CLIENT_SECRET    │                     │
+│    │   - S3_ACCESS_KEY              │                     │
+│    │   - S3_SECRET_KEY              │                     │
+│    └────┬────────────────┬──────────┘                      │
+│         │                │                                 │
+│  ┌──────▼──┐  ┌──────────▼──────┐                         │
+│  │ Authentik│  │  LLM Endpoint   │                         │
+│  │ (JWT)    │  │ (api.riotpiao)  │                         │
+│  └──────────┘  └─────────────────┘                         │
+│                                                             │
+│       ┌─────────────────────────────────────┐             │
+│       │    Entity Extraction Pipeline       │             │
+│       │    ┌────────────────────────────┐   │             │
+│       │    │ 1. WikiLink fallback        │   │             │
+│       │    │ 2. LLM extraction (JWT auth)│   │             │
+│       │    │ 3. Reflection verification  │   │             │
+│       │    │ 4. Contradiction detection  │   │             │
+│       │    └────────────────────────────┘   │             │
+│       └──────────────┬──────────────────────┘             │
+│                      │                                     │
+│              ┌───────▼────────┐                            │
+│              │  PostgreSQL    │                            │
+│              │  (entities DB) │                            │
+│              └────────────────┘                            │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Step 1: Create Authentik Service Account
+
+### In Authentik Admin Panel:
+
+1. Navigate: **Settings** → **Applications** → **Create Application**
+2. Name: `poimen-memory`
+3. Slug: `poimen-memory`
+4. Provider: Create a new OAuth2 Provider
+   - Name: `poimen-memory`
+   - Client type: `confidential`
+   - Client ID: `<auto-generated>`
+   - Client secret: `<auto-generated>`
+5. Save and note the **Client ID** and **Client Secret**
+
+### Verify OAuth2 Token Endpoint:
+```bash
+curl -X POST https://authentik.riotpiao.com/application/o/token/ \
+  -d "grant_type=client_credentials" \
+  -d "client_id=<CLIENT_ID>" \
+  -d "client_secret=<CLIENT_SECRET>"
+  
+# Response:
+# {
+#   "access_token": "eyJ0eXAi...",
+#   "token_type": "Bearer",
+#   "expires_in": 3600
+# }
+```
+
+## Step 2: Create Encrypted Secrets File
+
+### 2.1 Ensure SOPS is configured:
+
+```bash
+# Load SOPS_AGE_KEY_FILE
+export SOPS_AGE_KEY_FILE=~/.sops/key.txt
+
+# Verify key exists
+ls -la ~/.sops/key.txt
+```
+
+### 2.2 Create unencrypted secrets template:
+
+```yaml
+# k8s/app/poimen-memory-secrets.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: poimen-memory-secrets
+  namespace: poimen
+type: Opaque
+stringData:
+  # Authentik OAuth2 Credentials
+  AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/memory"
+  AUTHENTIK_AUDIENCE: "poimen-memory"
+  AUTHENTIK_CLIENT_ID: "<from-authentik-app>"
+  AUTHENTIK_CLIENT_SECRET: "<from-authentik-app>"
+  
+  # LLM Gateway API Key (optional fallback)
+  LLM_API_KEY: "<jwt-will-be-auto-generated>"
+  
+  # S3/Minio Credentials
+  S3_ACCESS_KEY: "<minio-access-key>"
+  S3_SECRET_KEY: "<minio-secret-key>"
+```
+
+### 2.3 Encrypt with SOPS:
+
+```bash
+export SOPS_AGE_KEY_FILE=~/.sops/key.txt
+cd ~/workplace/Poimen/memory
+
+sops -e k8s/app/poimen-memory-secrets.yaml > k8s/app/poimen-memory-secrets.enc.yaml
+
+# Verify encryption worked
+sops -d k8s/app/poimen-memory-secrets.enc.yaml | head -20
+```
+
+### 2.4 Commit encrypted file only:
+
+```bash
+git add k8s/app/poimen-memory-secrets.enc.yaml
+git add .sops.yaml
+git rm k8s/app/poimen-memory-secrets.yaml  # Remove plaintext
+git commit -m "feat: add SOPS-encrypted Authentik secrets"
+```
+
+## Step 3: Deploy to Kubernetes
+
+### 3.1 Install KSOPS plugin (if using ArgoCD):
+
+```bash
+# ArgoCD Helm values
+kustomization:
+  plugins:
+    - name: Kustomize
+      image: ghcr.io/viaduct-ai/kustomize-sops:v4.1.1
+```
+
+### 3.2 Apply secrets manifest:
+
+```bash
+# With KSOPS: ArgoCD auto-decrypts and applies
+# Without KSOPS: Manual decryption before apply
+export SOPS_AGE_KEY_FILE=~/.sops/key.txt
+sops -d k8s/app/poimen-memory-secrets.enc.yaml | kubectl apply -f -
+
+# Verify secret created
+kubectl -n poimen get secret poimen-memory-secrets
+kubectl -n poimen describe secret poimen-memory-secrets
+```
+
+### 3.3 Update deployment envFrom:
+
+```yaml
+# k8s/app/deployment.yaml
+spec:
+  template:
+    spec:
+      containers:
+        - name: poimen-memory
+          envFrom:
+            - configMapRef:
+                name: poimen-memory-config
+            - secretRef:
+                name: poimen-memory-secrets  # <-- Add this
+```
+
+## Step 4: Entity Extractor JWT Flow
+
+### Code: `crates/mem-ingest/src/entity_extractor.rs`
+
+```rust
+// Initialization
+pub struct LlmEntityExtractor {
+    jwt_issuer: Option<Arc<Mutex<AuthentikJwtIssuer>>>,
+}
+
+impl LlmEntityExtractor {
+    pub fn new(model_name: &str) -> Self {
+        let jwt_issuer = AuthentikJwtIssuer::from_env().ok();
+        Self {
+            jwt_issuer: jwt_issuer.map(|iss| Arc::new(Mutex::new(iss))),
+        }
+    }
+}
+
+// LLM call with JWT
+async fn call_llm_endpoint(&self, prompt: &str) -> Result<String> {
+    // Get JWT token from Authentik (cached, auto-refreshed)
+    let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
+        let issuer = jwt_issuer.lock().await;
+        let token = issuer.get_access_token().await?;
+        format!("Bearer {}", token)
+    } else {
+        format!("Bearer {}", fallback_api_key)
+    };
+    
+    // POST to LLM endpoint with JWT
+    client
+        .post(&endpoint)
+        .header("Authorization", auth_header)
+        .json(&payload)
+        .send()
+        .await?
+}
+```
+
+## Step 5: Runtime Verification
+
+### 5.1 Check JWT token exchange in logs:
+
+```bash
+kubectl -n poimen logs deployment/poimen-memory | grep -i "authentik\|jwt"
+
+# Expected output:
+# [2026-01-09T20:30:15Z] Obtained Authentik JWT token (expires in 3600 seconds)
+# [2026-01-09T20:30:15Z] LLM response (via Authentik JWT): {...}
+```
+
+### 5.2 Test entity extraction end-to-end:
+
+```bash
+# Port-forward to service
+kubectl -n poimen port-forward svc/poimen-memory 8080:8080 &
+
+# Ingest a record
+curl -X POST http://localhost:8080/memory/ingest \
+  -H "Content-Type: application/json" \
+  -d '{
+    "project": "homelab",
+    "source": "test://jwt",
+    "ingest_id": "jwt-test-001",
+    "records": [{
+      "role": "architect",
+      "text": "[[Kubernetes]] uses [[Docker]]. [[ArgoCD]] manages deployments.",
+      "timestamp": "2026-01-09T20:30:00Z",
+      "source_position": 0
+    }]
+  }'
+
+# Check logs for JWT usage
+kubectl -n poimen logs deployment/poimen-memory | tail -20
+```
+
+## Step 6: Monitoring & Maintenance
+
+### Token Expiry Handling:
+- JWT tokens are cached with auto-refresh
+- If token expires during use, new token is fetched automatically
+- No manual token rotation required
+
+### Credential Rotation:
+- Rotate Authentik client secret periodically
+- Update SOPS secret file and re-encrypt
+- Redeploy pod to pick up new secret
+
+### SOPS Key Rotation (Yearly):
+```bash
+# Generate new age key
+age-keygen -o ~/.sops/key.txt.new
+
+# Re-encrypt all secrets with new key
+for file in k8s/**/*.enc.yaml; do
+  sops -r $file
+done
+
+# Update ArgoCD to use new key
+# Commit changes
+git add k8s/**/*.enc.yaml
+git commit -m "chore: rotate SOPS encryption keys"
+```
+
+## Troubleshooting
+
+### Issue: "AUTHENTIK_ISSUER not set"
+**Cause**: Secret not mounted properly
+**Solution**: `kubectl -n poimen get secret poimen-memory-secrets`
+
+### Issue: "JWT token request failed: 401"
+**Cause**: Invalid client credentials
+**Solution**: Verify Client ID/Secret in Authentik, check SOPS decryption
+
+### Issue: "error loading config: no matching creation rules found"
+**Cause**: SOPS .sops.yaml not configured correctly
+**Solution**: Use `.sops.yaml` with explicit age key instead of config-based rules
+
+### Issue: "LLM API error: 403 Forbidden"
+**Cause**: JWT token doesn't have permission to LLM gateway
+**Solution**: Add RBAC role "LLM User" to service account in Authentik
+
+---
+
+## Files Modified
+
+- ✅ `crates/mem-ingest/src/authentik_jwt.rs` — JWT token exchange module
+- ✅ `crates/mem-ingest/src/entity_extractor.rs` — LLM calls with JWT
+- ✅ `crates/mem-ingest/src/lib.rs` — Module export
+- ✅ `k8s/app/poimen-memory-secrets.yaml` — Secret template (plaintext, not committed)
+- ✅ `k8s/app/poimen-memory-secrets.enc.yaml` — Secret encrypted with SOPS
+- ✅ `k8s/app/deployment.yaml` — Updated envFrom for secrets
+- ✅ `k8s/app/config.yaml` — LLM endpoint configuration
+- ✅ `k8s/.sops.yaml` — SOPS encryption rules
+