poimen-memory/docs/AUTHENTIK_SOPS_SETUP.md

Poimen Memory: Authentik JWT + SOPS Encryption Setup

Overview

The Poimen Memory service uses:

1. Authentik service account for OAuth2 client credentials flow
2. SOPS + Age encryption to encrypt secrets in git
3. JWT tokens for authentication to LLM gateway, S3, and other services

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Kubernetes (poimen)                    │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐    ┌─────────────────┐                  │
│  │  ConfigMap   │    │  Secret (SOPS)  │                  │
│  │ (unencrypted)│    │  (age-encrypted)│                  │
│  └──────┬───────┘    └────────┬────────┘                  │
│         │                     │                            │
│         ├─────────┬───────────┤                            │
│         │         │           │                            │
│    ┌────▼─────────▼───────────▼────┐                      │
│    │   poimen-memory Pod            │                     │
│    │   Environment Variables:       │                     │
│    │   - LLM_ENDPOINT               │                     │
│    │   - AUTHENTIK_ISSUER           │                     │
│    │   - AUTHENTIK_CLIENT_ID        │                     │
│    │   - AUTHENTIK_CLIENT_SECRET    │                     │
│    │   - S3_ACCESS_KEY              │                     │
│    │   - S3_SECRET_KEY              │                     │
│    └────┬────────────────┬──────────┘                      │
│         │                │                                 │
│  ┌──────▼──┐  ┌──────────▼──────┐                         │
│  │ Authentik│  │  LLM Endpoint   │                         │
│  │ (JWT)    │  │ (api.riotpiao)  │                         │
│  └──────────┘  └─────────────────┘                         │
│                                                             │
│       ┌─────────────────────────────────────┐             │
│       │    Entity Extraction Pipeline       │             │
│       │    ┌────────────────────────────┐   │             │
│       │    │ 1. WikiLink fallback        │   │             │
│       │    │ 2. LLM extraction (JWT auth)│   │             │
│       │    │ 3. Reflection verification  │   │             │
│       │    │ 4. Contradiction detection  │   │             │
│       │    └────────────────────────────┘   │             │
│       └──────────────┬──────────────────────┘             │
│                      │                                     │
│              ┌───────▼────────┐                            │
│              │  PostgreSQL    │                            │
│              │  (entities DB) │                            │
│              └────────────────┘                            │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Step 1: Create Authentik Service Account

In Authentik Admin Panel:

1. Navigate: Settings → Applications → Create Application
2. Name: poimen-memory
3. Slug: poimen-memory
4. Provider: Create a new OAuth2 Provider
   • Name: poimen-memory
   • Client type: confidential
   • Client ID: <auto-generated>
   • Client secret: <auto-generated>
5. Save and note the Client ID and Client Secret

Verify OAuth2 Token Endpoint:

curl -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d "grant_type=client_credentials" \
  -d "client_id=<CLIENT_ID>" \
  -d "client_secret=<CLIENT_SECRET>"
  
# Response:
# {
#   "access_token": "eyJ0eXAi...",
#   "token_type": "Bearer",
#   "expires_in": 3600
# }

Step 2: Create Encrypted Secrets File

2.1 Ensure SOPS is configured:

# Load SOPS_AGE_KEY_FILE
export SOPS_AGE_KEY_FILE=~/.sops/key.txt

# Verify key exists
ls -la ~/.sops/key.txt

2.2 Create unencrypted secrets template:

# k8s/app/poimen-memory-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: poimen-memory-secrets
  namespace: poimen
type: Opaque
stringData:
  # Authentik OAuth2 Credentials
  AUTHENTIK_ISSUER: "https://authentik.riotpiao.com/application/o/memory"
  AUTHENTIK_AUDIENCE: "poimen-memory"
  AUTHENTIK_CLIENT_ID: "<from-authentik-app>"
  AUTHENTIK_CLIENT_SECRET: "<from-authentik-app>"
  
  # LLM Gateway API Key (optional fallback)
  LLM_API_KEY: "<jwt-will-be-auto-generated>"
  
  # S3/Minio Credentials
  S3_ACCESS_KEY: "<minio-access-key>"
  S3_SECRET_KEY: "<minio-secret-key>"

2.3 Encrypt with SOPS:

export SOPS_AGE_KEY_FILE=~/.sops/key.txt
cd ~/workplace/Poimen/memory

sops -e k8s/app/poimen-memory-secrets.yaml > k8s/app/poimen-memory-secrets.enc.yaml

# Verify encryption worked
sops -d k8s/app/poimen-memory-secrets.enc.yaml | head -20

2.4 Commit encrypted file only:

git add k8s/app/poimen-memory-secrets.enc.yaml
git add .sops.yaml
git rm k8s/app/poimen-memory-secrets.yaml  # Remove plaintext
git commit -m "feat: add SOPS-encrypted Authentik secrets"

Step 3: Deploy to Kubernetes

3.1 Install KSOPS plugin (if using ArgoCD):

# ArgoCD Helm values
kustomization:
  plugins:
    - name: Kustomize
      image: ghcr.io/viaduct-ai/kustomize-sops:v4.1.1

3.2 Apply secrets manifest:

# With KSOPS: ArgoCD auto-decrypts and applies
# Without KSOPS: Manual decryption before apply
export SOPS_AGE_KEY_FILE=~/.sops/key.txt
sops -d k8s/app/poimen-memory-secrets.enc.yaml | kubectl apply -f -

# Verify secret created
kubectl -n poimen get secret poimen-memory-secrets
kubectl -n poimen describe secret poimen-memory-secrets

3.3 Update deployment envFrom:

# k8s/app/deployment.yaml
spec:
  template:
    spec:
      containers:
        - name: poimen-memory
          envFrom:
            - configMapRef:
                name: poimen-memory-config
            - secretRef:
                name: poimen-memory-secrets  # <-- Add this

Step 4: Entity Extractor JWT Flow

Code: crates/mem-ingest/src/entity_extractor.rs

// Initialization
pub struct LlmEntityExtractor {
    jwt_issuer: Option<Arc<Mutex<AuthentikJwtIssuer>>>,
}

impl LlmEntityExtractor {
    pub fn new(model_name: &str) -> Self {
        let jwt_issuer = AuthentikJwtIssuer::from_env().ok();
        Self {
            jwt_issuer: jwt_issuer.map(|iss| Arc::new(Mutex::new(iss))),
        }
    }
}

// LLM call with JWT
async fn call_llm_endpoint(&self, prompt: &str) -> Result<String> {
    // Get JWT token from Authentik (cached, auto-refreshed)
    let auth_header = if let Some(jwt_issuer) = &self.jwt_issuer {
        let issuer = jwt_issuer.lock().await;
        let token = issuer.get_access_token().await?;
        format!("Bearer {}", token)
    } else {
        format!("Bearer {}", fallback_api_key)
    };
    
    // POST to LLM endpoint with JWT
    client
        .post(&endpoint)
        .header("Authorization", auth_header)
        .json(&payload)
        .send()
        .await?
}

Step 5: Runtime Verification

5.1 Check JWT token exchange in logs:

kubectl -n poimen logs deployment/poimen-memory | grep -i "authentik\|jwt"

# Expected output:
# [2026-01-09T20:30:15Z] Obtained Authentik JWT token (expires in 3600 seconds)
# [2026-01-09T20:30:15Z] LLM response (via Authentik JWT): {...}

5.2 Test entity extraction end-to-end:

# Port-forward to service
kubectl -n poimen port-forward svc/poimen-memory 8080:8080 &

# Ingest a record
curl -X POST http://localhost:8080/memory/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "project": "homelab",
    "source": "test://jwt",
    "ingest_id": "jwt-test-001",
    "records": [{
      "role": "architect",
      "text": "[[Kubernetes]] uses [[Docker]]. [[ArgoCD]] manages deployments.",
      "timestamp": "2026-01-09T20:30:00Z",
      "source_position": 0
    }]
  }'

# Check logs for JWT usage
kubectl -n poimen logs deployment/poimen-memory | tail -20

Step 6: Monitoring & Maintenance

Token Expiry Handling:

• JWT tokens are cached with auto-refresh
• If token expires during use, new token is fetched automatically
• No manual token rotation required

Credential Rotation:

• Rotate Authentik client secret periodically
• Update SOPS secret file and re-encrypt
• Redeploy pod to pick up new secret

SOPS Key Rotation (Yearly):

# Generate new age key
age-keygen -o ~/.sops/key.txt.new

# Re-encrypt all secrets with new key
for file in k8s/**/*.enc.yaml; do
  sops -r $file
done

# Update ArgoCD to use new key
# Commit changes
git add k8s/**/*.enc.yaml
git commit -m "chore: rotate SOPS encryption keys"

Troubleshooting

Issue: "AUTHENTIK_ISSUER not set"

Cause: Secret not mounted properly Solution: kubectl -n poimen get secret poimen-memory-secrets

Issue: "JWT token request failed: 401"

Cause: Invalid client credentials Solution: Verify Client ID/Secret in Authentik, check SOPS decryption

Issue: "error loading config: no matching creation rules found"

Cause: SOPS .sops.yaml not configured correctly Solution: Use .sops.yaml with explicit age key instead of config-based rules

Issue: "LLM API error: 403 Forbidden"

Cause: JWT token doesn't have permission to LLM gateway Solution: Add RBAC role "LLM User" to service account in Authentik

---

Files Modified

• ✅ crates/mem-ingest/src/authentik_jwt.rs — JWT token exchange module
• ✅ crates/mem-ingest/src/entity_extractor.rs — LLM calls with JWT
• ✅ crates/mem-ingest/src/lib.rs — Module export
• ✅ k8s/app/poimen-memory-secrets.yaml — Secret template (plaintext, not committed)
• ✅ k8s/app/poimen-memory-secrets.enc.yaml — Secret encrypted with SOPS
• ✅ k8s/app/deployment.yaml — Updated envFrom for secrets
• ✅ k8s/app/config.yaml — LLM endpoint configuration
• ✅ k8s/.sops.yaml — SOPS encryption rules