322 lines
11 KiB
Markdown
322 lines
11 KiB
Markdown
|
# Poimen Memory: Authentik JWT + SOPS Encryption Setup
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
The Poimen Memory service uses:
|
||
|
|
1. **Authentik service account** for OAuth2 client credentials flow
|
||
|
|
2. **SOPS + Age encryption** to encrypt secrets in git
|
||
|
|
3. **JWT tokens** for authentication to LLM gateway, S3, and other services
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
```
|
||
|
|
┌─────────────────────────────────────────────────────────────┐
|
||
|
|
│ Kubernetes (poimen) │
|
||
|
|
├─────────────────────────────────────────────────────────────┤
|
||
|
|
│ │
|
||
|
|
│ ┌──────────────┐ ┌─────────────────┐ │
|
||
|
|
│ │ ConfigMap │ │ Secret (SOPS) │ │
|
||
|
|
│ │ (unencrypted)│ │ (age-encrypted)│ │
|
||
|
|
│ └──────┬───────┘ └────────┬────────┘ │
|
||
|
|
│ │ │ │
|
||
|
|
│ ├─────────┬───────────┤ │
|
||
|
|
│ │ │ │ │
|
||
|
|
│ ┌────▼─────────▼───────────▼────┐ │
|
||
|
|
│ │ poimen-memory Pod │ │
|
||
|
|
│ │ Environment Variables: │ │
|
||
|
|
│ │ - LLM_ENDPOINT │ │
|
||
|
|
│ │ - AUTHENTIK_ISSUER │ │
|
||
|
|
│ │ - AUTHENTIK_CLIENT_ID │ │
|
||
|
|
│ │ - AUTHENTIK_CLIENT_SECRET │ │
|
||
|
|
│ │ - S3_ACCESS_KEY │ │
|
||
|
|
│ │ - S3_SECRET_KEY │ │
|
||
|
|
│ └────┬────────────────┬──────────┘ │
|
||
|
|
│ │ │ │
|
||
|
|
│ ┌──────▼──┐ ┌──────────▼──────┐ │
|
||
|
|
│ │ Authentik│ │ LLM Endpoint │ │
|
||
|
|
│ │ (JWT) │ │ (api.riotpiao) │ │
|
||
|
|
│ └──────────┘ └─────────────────┘ │
|
||
|
|
│ │
|
||
|
|
│ ┌─────────────────────────────────────┐ │
|
||
|
|
│ │ Entity Extraction Pipeline │ │
|
||
|
|
│ │ ┌────────────────────────────┐ │ │
|
||
|
|
│ │ │ 1. WikiLink fallback │ │ │
|
||
|
|
│ │ │ 2. LLM extraction (JWT auth)│ │ │
|
||
|
|
│ │ │ 3. Reflection verification │ │ │
|
||
|
|
│ │ │ 4. Contradiction detection │ │ │
|
||
|
|
│ │ └────────────────────────────┘ │ │
|
||
|
|
│ └──────────────┬──────────────────────┘ │
|
||
|
|
│ │ │
|
||
|
|
│ ┌───────▼────────┐ │
|
||
|
|
│ │ PostgreSQL │ │
|
||
|
|
│ │ (entities DB) │ │
|
||
|
|
│ └────────────────┘ │
|
||
|
|
│ │
|
||
|
|
└─────────────────────────────────────────────────────────────┘
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 1: Create Authentik Service Account
|
||
|
|
|
||
|
|
### In Authentik Admin Panel:
|
||
|
|
|
||
|
|
1. Navigate: **Settings** → **Applications** → **Create Application**
|
||
|
|
2. Name: `poimen-memory`
|
||
|
|
3. Slug: `poimen-memory`
|
||
|
|
4. Provider: Create a new OAuth2 Provider
|
||
|
|
- Name: `poimen-memory`
|
||
|
|
- Client type: `confidential`
|
||
|
|
- Client ID: `<auto-generated>`
|
||
|
|
- Client secret: `<auto-generated>`
|
||
|
|
5. Save and note the **Client ID** and **Client Secret**
|
||
|
|
|
||
|
|
### Verify OAuth2 Token Endpoint:
|
||
|
|
```bash
|
||
|
|
curl -X POST https://authentik.riotpiao.com/application/o/token/ \
|
||
|
|
-d "grant_type=client_credentials" \
|
||
|
|
-d "client_id=<CLIENT_ID>" \
|
||
|
|
-d "client_secret=<CLIENT_SECRET>"
|
||
|
|
|
||
|
|
# Response:
|
||
|
|
# {
|
||
|
|
# "access_token": "eyJ0eXAi...",
|
||
|
|
# "token_type": "Bearer",
|
||
|
|
# "expires_in": 3600
|
||
|
|
# }
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 2: Create Encrypted Secrets File
|
||
|
|
|
||
|
|
### 2.1 Ensure SOPS is configured:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Load SOPS_AGE_KEY_FILE
|
||
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
||
|
|
|
||
|
|
# Verify key exists
|
||
|
|
ls -la ~/.sops/key.txt
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2.2 Create unencrypted secrets template:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
# k8s/app/poimen-memory-secrets.yaml
|
||
|
|
apiVersion: v1
|
||
|
|
kind: Secret
|
||
|
|
metadata:
|
||
|
|
name: poimen-memory-secrets
|
||
|
|
namespace: poimen
|
||
|
|
type: Opaque
|
||
|
|
stringData:
|
||
|
|
# Authentik OAuth2 Credentials
|
||
|
|
AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/memory"
|
||
|
|
AUTHENTIK_AUDIENCE: "poimen-memory"
|
||
|
|
AUTHENTIK_CLIENT_ID: "<from-authentik-app>"
|
||
|
|
AUTHENTIK_CLIENT_SECRET: "<from-authentik-app>"
|
||
|
|
|
||
|
|
# LLM Gateway API Key (optional fallback)
|
||
|
|
LLM_API_KEY: "<jwt-will-be-auto-generated>"
|
||
|
|
|
||
|
|
# S3/Minio Credentials
|
||
|
|
S3_ACCESS_KEY: "<minio-access-key>"
|
||
|
|
S3_SECRET_KEY: "<minio-secret-key>"
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2.3 Encrypt with SOPS:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
||
|
|
cd ~/workplace/Poimen/memory
|
||
|
|
|
||
|
|
sops -e k8s/app/poimen-memory-secrets.yaml > k8s/app/poimen-memory-secrets.enc.yaml
|
||
|
|
|
||
|
|
# Verify encryption worked
|
||
|
|
sops -d k8s/app/poimen-memory-secrets.enc.yaml | head -20
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2.4 Commit encrypted file only:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
git add k8s/app/poimen-memory-secrets.enc.yaml
|
||
|
|
git add .sops.yaml
|
||
|
|
git rm k8s/app/poimen-memory-secrets.yaml # Remove plaintext
|
||
|
|
git commit -m "feat: add SOPS-encrypted Authentik secrets"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 3: Deploy to Kubernetes
|
||
|
|
|
||
|
|
### 3.1 Install KSOPS plugin (if using ArgoCD):
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# ArgoCD Helm values
|
||
|
|
kustomization:
|
||
|
|
plugins:
|
||
|
|
- name: Kustomize
|
||
|
|
image: ghcr.io/viaduct-ai/kustomize-sops:v4.1.1
|
||
|
|
```
|
||
|
|
|
||
|
|
### 3.2 Apply secrets manifest:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# With KSOPS: ArgoCD auto-decrypts and applies
|
||
|
|
# Without KSOPS: Manual decryption before apply
|
||
|
|
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
|
||
|
|
sops -d k8s/app/poimen-memory-secrets.enc.yaml | kubectl apply -f -
|
||
|
|
|
||
|
|
# Verify secret created
|
||
|
|
kubectl -n poimen get secret poimen-memory-secrets
|
||
|
|
kubectl -n poimen describe secret poimen-memory-secrets
|
||
|
|
```
|
||
|
|
|
||
|
|
### 3.3 Update deployment envFrom:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
# k8s/app/deployment.yaml
|
||
|
|
spec:
|
||
|
|
template:
|
||
|
|
spec:
|
||
|
|
containers:
|
||
|
|
- name: poimen-memory
|
||
|
|
envFrom:
|
||
|
|
- configMapRef:
|
||
|
|
name: poimen-memory-config
|
||
|
|
- secretRef:
|
||
|
|
name: poimen-memory-secrets # <-- Add this
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 4: Entity Extractor JWT Flow
|
||
|
|
|
||
|
|
### Code: `crates/mem-ingest/src/entity_extractor.rs`
|
||
|
|
|
||
|
|
```rust
|
||
|
|
// Initialization
|
||
|
|
pub struct LlmEntityExtractor {
|
||
|
|
jwt_issuer: Option<Arc<Mutex<AuthentikJwtIssuer>>>,
|
||
|
|
}
|
||
|
|
|
||
|
|
impl LlmEntityExtractor {
|
||
|
|
pub fn new(model_name: &str) -> Self {
|
||
|
|
let jwt_issuer = AuthentikJwtIssuer::from_env().ok();
|
||
|
|
Self {
|
||
|
|
jwt_issuer: jwt_issuer.map(|iss| Arc::new(Mutex::new(iss))),
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
// LLM call with JWT
|
||
|
|
async fn call_llm_endpoint(&self, prompt: &str) -> Result<String> {
|
||
|
|
// Get JWT token from Authentik (cached, auto-refreshed)
|
||
|
|
let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
|
||
|
|
let issuer = jwt_issuer.lock().await;
|
||
|
|
let token = issuer.get_access_token().await?;
|
||
|
|
format!("Bearer {}", token)
|
||
|
|
} else {
|
||
|
|
format!("Bearer {}", fallback_api_key)
|
||
|
|
};
|
||
|
|
|
||
|
|
// POST to LLM endpoint with JWT
|
||
|
|
client
|
||
|
|
.post(&endpoint)
|
||
|
|
.header("Authorization", auth_header)
|
||
|
|
.json(&payload)
|
||
|
|
.send()
|
||
|
|
.await?
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 5: Runtime Verification
|
||
|
|
|
||
|
|
### 5.1 Check JWT token exchange in logs:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
kubectl -n poimen logs deployment/poimen-memory | grep -i "authentik\|jwt"
|
||
|
|
|
||
|
|
# Expected output:
|
||
|
|
# [2026-01-09T20:30:15Z] Obtained Authentik JWT token (expires in 3600 seconds)
|
||
|
|
# [2026-01-09T20:30:15Z] LLM response (via Authentik JWT): {...}
|
||
|
|
```
|
||
|
|
|
||
|
|
### 5.2 Test entity extraction end-to-end:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Port-forward to service
|
||
|
|
kubectl -n poimen port-forward svc/poimen-memory 8080:8080 &
|
||
|
|
|
||
|
|
# Ingest a record
|
||
|
|
curl -X POST http://localhost:8080/memory/ingest \
|
||
|
|
-H "Content-Type: application/json" \
|
||
|
|
-d '{
|
||
|
|
"project": "homelab",
|
||
|
|
"source": "test://jwt",
|
||
|
|
"ingest_id": "jwt-test-001",
|
||
|
|
"records": [{
|
||
|
|
"role": "architect",
|
||
|
|
"text": "[[Kubernetes]] uses [[Docker]]. [[ArgoCD]] manages deployments.",
|
||
|
|
"timestamp": "2026-01-09T20:30:00Z",
|
||
|
|
"source_position": 0
|
||
|
|
}]
|
||
|
|
}'
|
||
|
|
|
||
|
|
# Check logs for JWT usage
|
||
|
|
kubectl -n poimen logs deployment/poimen-memory | tail -20
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 6: Monitoring & Maintenance
|
||
|
|
|
||
|
|
### Token Expiry Handling:
|
||
|
|
- JWT tokens are cached with auto-refresh
|
||
|
|
- If token expires during use, new token is fetched automatically
|
||
|
|
- No manual token rotation required
|
||
|
|
|
||
|
|
### Credential Rotation:
|
||
|
|
- Rotate Authentik client secret periodically
|
||
|
|
- Update SOPS secret file and re-encrypt
|
||
|
|
- Redeploy pod to pick up new secret
|
||
|
|
|
||
|
|
### SOPS Key Rotation (Yearly):
|
||
|
|
```bash
|
||
|
|
# Generate new age key
|
||
|
|
age-keygen -o ~/.sops/key.txt.new
|
||
|
|
|
||
|
|
# Re-encrypt all secrets with new key
|
||
|
|
for file in k8s/**/*.enc.yaml; do
|
||
|
|
sops -r $file
|
||
|
|
done
|
||
|
|
|
||
|
|
# Update ArgoCD to use new key
|
||
|
|
# Commit changes
|
||
|
|
git add k8s/**/*.enc.yaml
|
||
|
|
git commit -m "chore: rotate SOPS encryption keys"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
### Issue: "AUTHENTIK_ISSUER not set"
|
||
|
|
**Cause**: Secret not mounted properly
|
||
|
|
**Solution**: `kubectl -n poimen get secret poimen-memory-secrets`
|
||
|
|
|
||
|
|
### Issue: "JWT token request failed: 401"
|
||
|
|
**Cause**: Invalid client credentials
|
||
|
|
**Solution**: Verify Client ID/Secret in Authentik, check SOPS decryption
|
||
|
|
|
||
|
|
### Issue: "error loading config: no matching creation rules found"
|
||
|
|
**Cause**: SOPS .sops.yaml not configured correctly
|
||
|
|
**Solution**: Use `.sops.yaml` with explicit age key instead of config-based rules
|
||
|
|
|
||
|
|
### Issue: "LLM API error: 403 Forbidden"
|
||
|
|
**Cause**: JWT token doesn't have permission to LLM gateway
|
||
|
|
**Solution**: Add RBAC role "LLM User" to service account in Authentik
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Files Modified
|
||
|
|
|
||
|
|
- ✅ `crates/mem-ingest/src/authentik_jwt.rs` — JWT token exchange module
|
||
|
|
- ✅ `crates/mem-ingest/src/entity_extractor.rs` — LLM calls with JWT
|
||
|
|
- ✅ `crates/mem-ingest/src/lib.rs` — Module export
|
||
|
|
- ✅ `k8s/app/poimen-memory-secrets.yaml` — Secret template (plaintext, not committed)
|
||
|
|
- ✅ `k8s/app/poimen-memory-secrets.enc.yaml` — Secret encrypted with SOPS
|
||
|
|
- ✅ `k8s/app/deployment.yaml` — Updated envFrom for secrets
|
||
|
|
- ✅ `k8s/app/config.yaml` — LLM endpoint configuration
|
||
|
|
- ✅ `k8s/.sops.yaml` — SOPS encryption rules
|
||
|
|
|