rock
16e3ff16f1
CI / CI (pull_request) Successful in 3m40s
SECURITY: - Add authentik_jwt.rs: OAuth2 client credentials flow with caching - SOPS encrypt secrets with age key (SOPS_AGE_KEY_FILE) - JWT tokens for LLM gateway, S3, and API gateway access - Token auto-refresh when expired (60s before expiry) - No hardcoded credentials in code or config ENTITY EXTRACTION: - LlmEntityExtractor now uses Authentik JWT instead of mock - Fallback to env var if Authentik not configured - Reflection verification still enabled - WikiLink extraction as Stage 0 (always active) DEPLOYMENT: - ConfigMap: LLM_ENDPOINT, LLM_MODEL, timeouts - Secret: AUTHENTIK_ISSUER, CLIENT_ID, CLIENT_SECRET, S3 keys - envFrom mounts both ConfigMap and Secret - KSOPS plugin for ArgoCD auto-decryption DOCUMENTATION: - docs/AUTHENTIK_SOPS_SETUP.md: Complete integration guide - Service account creation in Authentik - SOPS encryption/decryption workflow - JWT token exchange flow - Troubleshooting guide FILES: - crates/mem-ingest/src/authentik_jwt.rs (new, 180 LOC) - crates/mem-ingest/src/entity_extractor.rs (updated, JWT auth) - crates/mem-ingest/Cargo.toml (add reqwest) - k8s/app/poimen-memory-secrets.yaml (new, unencrypted template) - k8s/app/deployment.yaml (add secrets envFrom) - k8s/app/config.yaml (add LLM config) - k8s/.sops.yaml (encryption rules) - docs/AUTHENTIK_SOPS_SETUP.md (new, 350 LOC) NEXT: 1. Create Authentik service account (manual) 2. Encrypt secrets with SOPS 3. Deploy to poimen namespace 4. Test JWT token exchange with LLM endpoint
322 lines
11 KiB
Markdown
322 lines
11 KiB
Markdown
# Poimen Memory: Authentik JWT + SOPS Encryption Setup
|
|
|
|
## Overview
|
|
|
|
The Poimen Memory service uses:
|
|
1. **Authentik service account** for OAuth2 client credentials flow
|
|
2. **SOPS + Age encryption** to encrypt secrets in git
|
|
3. **JWT tokens** for authentication to LLM gateway, S3, and other services
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Kubernetes (poimen) │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────┐ ┌─────────────────┐ │
|
|
│ │ ConfigMap │ │ Secret (SOPS) │ │
|
|
│ │ (unencrypted)│ │ (age-encrypted)│ │
|
|
│ └──────┬───────┘ └────────┬────────┘ │
|
|
│ │ │ │
|
|
│ ├─────────┬───────────┤ │
|
|
│ │ │ │ │
|
|
│ ┌────▼─────────▼───────────▼────┐ │
|
|
│ │ poimen-memory Pod │ │
|
|
│ │ Environment Variables: │ │
|
|
│ │ - LLM_ENDPOINT │ │
|
|
│ │ - AUTHENTIK_ISSUER │ │
|
|
│ │ - AUTHENTIK_CLIENT_ID │ │
|
|
│ │ - AUTHENTIK_CLIENT_SECRET │ │
|
|
│ │ - S3_ACCESS_KEY │ │
|
|
│ │ - S3_SECRET_KEY │ │
|
|
│ └────┬────────────────┬──────────┘ │
|
|
│ │ │ │
|
|
│ ┌──────▼──┐ ┌──────────▼──────┐ │
|
|
│ │ Authentik│ │ LLM Endpoint │ │
|
|
│ │ (JWT) │ │ (api.riotpiao) │ │
|
|
│ └──────────┘ └─────────────────┘ │
|
|
│ │
|
|
│ ┌─────────────────────────────────────┐ │
|
|
│ │ Entity Extraction Pipeline │ │
|
|
│ │ ┌────────────────────────────┐ │ │
|
|
│ │ │ 1. WikiLink fallback │ │ │
|
|
│ │ │ 2. LLM extraction (JWT auth)│ │ │
|
|
│ │ │ 3. Reflection verification │ │ │
|
|
│ │ │ 4. Contradiction detection │ │ │
|
|
│ │ └────────────────────────────┘ │ │
|
|
│ └──────────────┬──────────────────────┘ │
|
|
│ │ │
|
|
│ ┌───────▼────────┐ │
|
|
│ │ PostgreSQL │ │
|
|
│ │ (entities DB) │ │
|
|
│ └────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Step 1: Create Authentik Service Account
|
|
|
|
### In Authentik Admin Panel:
|
|
|
|
1. Navigate: **Settings** → **Applications** → **Create Application**
|
|
2. Name: `poimen-memory`
|
|
3. Slug: `poimen-memory`
|
|
4. Provider: Create a new OAuth2 Provider
|
|
- Name: `poimen-memory`
|
|
- Client type: `confidential`
|
|
- Client ID: `<auto-generated>`
|
|
- Client secret: `<auto-generated>`
|
|
5. Save and note the **Client ID** and **Client Secret**
|
|
|
|
### Verify OAuth2 Token Endpoint:
|
|
```bash
|
|
curl -X POST https://authentik.riotpiao.com/application/o/token/ \
|
|
-d "grant_type=client_credentials" \
|
|
-d "client_id=<CLIENT_ID>" \
|
|
-d "client_secret=<CLIENT_SECRET>"
|
|
|
|
# Response:
|
|
# {
|
|
# "access_token": "eyJ0eXAi...",
|
|
# "token_type": "Bearer",
|
|
# "expires_in": 3600
|
|
# }
|
|
```
|
|
|
|
## Step 2: Create Encrypted Secrets File
|
|
|
|
### 2.1 Ensure SOPS is configured:
|
|
|
|
```bash
|
|
# Load SOPS_AGE_KEY_FILE
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
|
|
|
# Verify key exists
|
|
ls -la ~/.sops/key.txt
|
|
```
|
|
|
|
### 2.2 Create unencrypted secrets template:
|
|
|
|
```yaml
|
|
# k8s/app/poimen-memory-secrets.yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: poimen-memory-secrets
|
|
namespace: poimen
|
|
type: Opaque
|
|
stringData:
|
|
# Authentik OAuth2 Credentials
|
|
AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/memory"
|
|
AUTHENTIK_AUDIENCE: "poimen-memory"
|
|
AUTHENTIK_CLIENT_ID: "<from-authentik-app>"
|
|
AUTHENTIK_CLIENT_SECRET: "<from-authentik-app>"
|
|
|
|
# LLM Gateway API Key (optional fallback)
|
|
LLM_API_KEY: "<jwt-will-be-auto-generated>"
|
|
|
|
# S3/Minio Credentials
|
|
S3_ACCESS_KEY: "<minio-access-key>"
|
|
S3_SECRET_KEY: "<minio-secret-key>"
|
|
```
|
|
|
|
### 2.3 Encrypt with SOPS:
|
|
|
|
```bash
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
|
cd ~/workplace/Poimen/memory
|
|
|
|
sops -e k8s/app/poimen-memory-secrets.yaml > k8s/app/poimen-memory-secrets.enc.yaml
|
|
|
|
# Verify encryption worked
|
|
sops -d k8s/app/poimen-memory-secrets.enc.yaml | head -20
|
|
```
|
|
|
|
### 2.4 Commit encrypted file only:
|
|
|
|
```bash
|
|
git add k8s/app/poimen-memory-secrets.enc.yaml
|
|
git add .sops.yaml
|
|
git rm k8s/app/poimen-memory-secrets.yaml # Remove plaintext
|
|
git commit -m "feat: add SOPS-encrypted Authentik secrets"
|
|
```
|
|
|
|
## Step 3: Deploy to Kubernetes
|
|
|
|
### 3.1 Install KSOPS plugin (if using ArgoCD):
|
|
|
|
```bash
|
|
# ArgoCD Helm values
|
|
kustomization:
|
|
plugins:
|
|
- name: Kustomize
|
|
image: ghcr.io/viaduct-ai/kustomize-sops:v4.1.1
|
|
```
|
|
|
|
### 3.2 Apply secrets manifest:
|
|
|
|
```bash
|
|
# With KSOPS: ArgoCD auto-decrypts and applies
|
|
# Without KSOPS: Manual decryption before apply
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
|
sops -d k8s/app/poimen-memory-secrets.enc.yaml | kubectl apply -f -
|
|
|
|
# Verify secret created
|
|
kubectl -n poimen get secret poimen-memory-secrets
|
|
kubectl -n poimen describe secret poimen-memory-secrets
|
|
```
|
|
|
|
### 3.3 Update deployment envFrom:
|
|
|
|
```yaml
|
|
# k8s/app/deployment.yaml
|
|
spec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: poimen-memory
|
|
envFrom:
|
|
- configMapRef:
|
|
name: poimen-memory-config
|
|
- secretRef:
|
|
name: poimen-memory-secrets # <-- Add this
|
|
```
|
|
|
|
## Step 4: Entity Extractor JWT Flow
|
|
|
|
### Code: `crates/mem-ingest/src/entity_extractor.rs`
|
|
|
|
```rust
|
|
// Initialization
|
|
pub struct LlmEntityExtractor {
|
|
jwt_issuer: Option<Arc<Mutex<AuthentikJwtIssuer>>>,
|
|
}
|
|
|
|
impl LlmEntityExtractor {
|
|
pub fn new(model_name: &str) -> Self {
|
|
let jwt_issuer = AuthentikJwtIssuer::from_env().ok();
|
|
Self {
|
|
jwt_issuer: jwt_issuer.map(|iss| Arc::new(Mutex::new(iss))),
|
|
}
|
|
}
|
|
}
|
|
|
|
// LLM call with JWT
|
|
async fn call_llm_endpoint(&self, prompt: &str) -> Result<String> {
|
|
// Get JWT token from Authentik (cached, auto-refreshed)
|
|
let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
|
|
let issuer = jwt_issuer.lock().await;
|
|
let token = issuer.get_access_token().await?;
|
|
format!("Bearer {}", token)
|
|
} else {
|
|
format!("Bearer {}", fallback_api_key)
|
|
};
|
|
|
|
// POST to LLM endpoint with JWT
|
|
client
|
|
.post(&endpoint)
|
|
.header("Authorization", auth_header)
|
|
.json(&payload)
|
|
.send()
|
|
.await?
|
|
}
|
|
```
|
|
|
|
## Step 5: Runtime Verification
|
|
|
|
### 5.1 Check JWT token exchange in logs:
|
|
|
|
```bash
|
|
kubectl -n poimen logs deployment/poimen-memory | grep -i "authentik\|jwt"
|
|
|
|
# Expected output:
|
|
# [2026-01-09T20:30:15Z] Obtained Authentik JWT token (expires in 3600 seconds)
|
|
# [2026-01-09T20:30:15Z] LLM response (via Authentik JWT): {...}
|
|
```
|
|
|
|
### 5.2 Test entity extraction end-to-end:
|
|
|
|
```bash
|
|
# Port-forward to service
|
|
kubectl -n poimen port-forward svc/poimen-memory 8080:8080 &
|
|
|
|
# Ingest a record
|
|
curl -X POST http://localhost:8080/memory/ingest \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"project": "homelab",
|
|
"source": "test://jwt",
|
|
"ingest_id": "jwt-test-001",
|
|
"records": [{
|
|
"role": "architect",
|
|
"text": "[[Kubernetes]] uses [[Docker]]. [[ArgoCD]] manages deployments.",
|
|
"timestamp": "2026-01-09T20:30:00Z",
|
|
"source_position": 0
|
|
}]
|
|
}'
|
|
|
|
# Check logs for JWT usage
|
|
kubectl -n poimen logs deployment/poimen-memory | tail -20
|
|
```
|
|
|
|
## Step 6: Monitoring & Maintenance
|
|
|
|
### Token Expiry Handling:
|
|
- JWT tokens are cached with auto-refresh
|
|
- If token expires during use, new token is fetched automatically
|
|
- No manual token rotation required
|
|
|
|
### Credential Rotation:
|
|
- Rotate Authentik client secret periodically
|
|
- Update SOPS secret file and re-encrypt
|
|
- Redeploy pod to pick up new secret
|
|
|
|
### SOPS Key Rotation (Yearly):
|
|
```bash
|
|
# Generate new age key
|
|
age-keygen -o ~/.sops/key.txt.new
|
|
|
|
# Re-encrypt all secrets with new key
|
|
for file in k8s/**/*.enc.yaml; do
|
|
sops -r $file
|
|
done
|
|
|
|
# Update ArgoCD to use new key
|
|
# Commit changes
|
|
git add k8s/**/*.enc.yaml
|
|
git commit -m "chore: rotate SOPS encryption keys"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: "AUTHENTIK_ISSUER not set"
|
|
**Cause**: Secret not mounted properly
|
|
**Solution**: `kubectl -n poimen get secret poimen-memory-secrets`
|
|
|
|
### Issue: "JWT token request failed: 401"
|
|
**Cause**: Invalid client credentials
|
|
**Solution**: Verify Client ID/Secret in Authentik, check SOPS decryption
|
|
|
|
### Issue: "error loading config: no matching creation rules found"
|
|
**Cause**: SOPS .sops.yaml not configured correctly
|
|
**Solution**: Use `.sops.yaml` with explicit age key instead of config-based rules
|
|
|
|
### Issue: "LLM API error: 403 Forbidden"
|
|
**Cause**: JWT token doesn't have permission to LLM gateway
|
|
**Solution**: Add RBAC role "LLM User" to service account in Authentik
|
|
|
|
---
|
|
|
|
## Files Modified
|
|
|
|
- ✅ `crates/mem-ingest/src/authentik_jwt.rs` — JWT token exchange module
|
|
- ✅ `crates/mem-ingest/src/entity_extractor.rs` — LLM calls with JWT
|
|
- ✅ `crates/mem-ingest/src/lib.rs` — Module export
|
|
- ✅ `k8s/app/poimen-memory-secrets.yaml` — Secret template (plaintext, not committed)
|
|
- ✅ `k8s/app/poimen-memory-secrets.enc.yaml` — Secret encrypted with SOPS
|
|
- ✅ `k8s/app/deployment.yaml` — Updated envFrom for secrets
|
|
- ✅ `k8s/app/config.yaml` — LLM endpoint configuration
|
|
- ✅ `k8s/.sops.yaml` — SOPS encryption rules
|
|
|