fix: vendor Tekton release.yaml for proper ArgoCD management

ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart

FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)

TO UPGRADE TEKTON:
  1. Download new release from github.com/tektoncd/pipeline/releases
  2. Replace k8s/infra/tekton/release.yaml
  3. Commit — ArgoCD syncs automatically

.sops.yaml

@@ -2,4 +2,5 @@ creation_rules:
   # `secrets?` — singular too. A `seed-repo-secret.yaml` once slipped this regex
   # and was committed in plaintext to a public remote.
   - path_regex: k8s/.*secrets?.*\.ya?ml
+    encrypted_regex: ^(data|stringData)$
     age: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla

AUTH_INTEGRATION.md

@@ -1,206 +0,0 @@
-# Authentik Auth Integration for NextJS
-
-## Current State
-
-### Gateway Auth Status
-
-| Endpoint | Auth Status | Notes |
-|----------|-------------|-------|
-| `/v1/chat/completions` | ❌ **OFF** | LLM routes have no auth middleware |
-| `/v1/embeddings` | ❌ **OFF** | Same - no auth |
-| `/v1/rerank` | ❌ **OFF** | Same - no auth |
-| `X-Service: sqs` | ✅ **ON** | JWT validated via `internal/auth/jwt.go` |
-| `/workflow` | ❌ **OFF** | Pass-through to Temporal |
-
-**Auth module exists** at `homelab-frontend/internal/auth/jwt.go` but only wired for SQS.
-LLM routes in `internal/proxy/proxy.go` have no auth middleware.
-
-### Authentik App
-
-Authentik app `local-llm` exists for LLM API auth:
-- **Client ID**: `local-llm`
-- **Client Secret**: `kubectl -n llm-serving get secret local-llm-jwt -o jsonpath='{.data.client-secret}' | base64 -d`
-- **Token endpoint**: `https://authentik.riotpiao.com/application/o/token/`
-- **Userinfo endpoint**: `https://authentik.riotpiao.com/application/o/userinfo/`
-- **OIDC discovery**: `https://authentik.riotpiao.com/application/o/local-llm/.well-known/openid-configuration`
-
-## Sign-in Methods
-
-### 1. Resource Owner Password Credentials (ROPC)
-
-Direct username/password login. Server-side only (needs client_secret).
-
-```typescript
-// API Route: app/api/auth/login/route.ts
-const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
-  method: 'POST',
-  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
-  body: new URLSearchParams({
-    grant_type: 'password',
-    client_id: 'local-llm',
-    client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
-    username: '[email protected]',
-    password: 'userpassword',
-    scope: 'openid email profile groups',
-  }),
-});
-
-const tokens = await response.json();
-// { access_token, refresh_token, expires_in, token_type }
-```
-
-### 2. Authorization Code Flow (Browser Redirect)
-
-Requires adding redirect URIs to `local-llm` Authentik app:
-
-```python
-# In k8s/infra/iam/scripts/authentik-provision.py, update:
-"local-llm": {
-    ...
-    "redirect_uris": [
-        "http://localhost:3000/api/auth/callback",  # dev
-        "https://your-nextjs-app.com/api/auth/callback",  # prod
-    ],
-}
-```
-
-Then standard OIDC flow:
-1. Redirect to `https://authentik.riotpiao.com/application/o/authorize/?client_id=local-llm&redirect_uri=...&response_type=code&scope=openid email profile groups`
-2. User logs in via Authentik UI
-3. Callback receives `code`, exchange for tokens
-
-## JWT Token Persistence
-
-### Browser (localStorage)
-
-```typescript
-const TOKEN_KEY = 'llm_auth_token';
-
-// Save
-localStorage.setItem(TOKEN_KEY, JSON.stringify({
-  access_token: tokens.access_token,
-  refresh_token: tokens.refresh_token,
-  expires_at: Date.now() + tokens.expires_in * 1000,
-}));
-
-// Load
-const stored = JSON.parse(localStorage.getItem(TOKEN_KEY) || 'null');
-if (stored && stored.expires_at > Date.now()) {
-  // Token valid
-}
-
-// Clear (logout)
-localStorage.removeItem(TOKEN_KEY);
-```
-
-### Server-side (HTTP-only cookies)
-
-```typescript
-// app/api/auth/login/route.ts
-import { cookies } from 'next/headers';
-
-// After successful login
-cookies().set('llm_auth_token', JSON.stringify(tokens), {
-  httpOnly: true,
-  secure: process.env.NODE_ENV === 'production',
-  sameSite: 'lax',
-  maxAge: tokens.expires_in,
-  path: '/',
-});
-
-// Read in middleware or API routes
-const tokenCookie = cookies().get('llm_auth_token');
-const tokens = JSON.parse(tokenCookie?.value || 'null');
-```
-
-## Token Refresh
-
-```typescript
-async function refreshAccessToken(refresh_token: string) {
-  const response = await fetch('https://authentik.riotpiao.com/application/o/token/', {
-    method: 'POST',
-    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
-    body: new URLSearchParams({
-      grant_type: 'refresh_token',
-      client_id: 'local-llm',
-      client_secret: process.env.AUTHENTIK_CLIENT_SECRET,
-      refresh_token,
-    }),
-  });
-  return response.json();
-}
-```
-
-## Environment Variables
-
-```bash
-# .env.local
-AUTHENTIK_URL=https://authentik.riotpiao.com
-AUTHENTIK_CLIENT_ID=local-llm
-AUTHENTIK_CLIENT_SECRET=<from-secret>
-
-# For client-side (public)
-NEXT_PUBLIC_AUTHENTIK_URL=https://authentik.riotpiao.com
-NEXT_PUBLIC_AUTHENTIK_CLIENT_ID=local-llm
-```
-
-## Using Token with LLM API
-
-```typescript
-const token = await getValidToken(); // from localStorage or cookie
-
-const response = await fetch('https://api.riotpiao.com/v1/chat/completions', {
-  method: 'POST',
-  headers: {
-    'Content-Type': 'application/json',
-    'Authorization': `Bearer ${token}`,  // JWT from Authentik
-  },
-  body: JSON.stringify({
-    model: 'reasoning',
-    messages: [{ role: 'user', content: 'Hello' }],
-  }),
-});
-```
-
-## TODO
-
-### Gateway-side (homelab-frontend)
-
-- [ ] Wire `internal/auth/jwt.go` into LLM proxy handler (`internal/proxy/proxy.go`)
-- [ ] Add `authRequired: true` to model config or create LLM-specific middleware
-- [ ] Example pattern from SQS (in `internal/serviceadapter/router.go`):
-
-```go
-// In proxy.go ServeHTTP, before dispatching to LLM upstream:
-if strings.HasPrefix(r.URL.Path, "/v1/") {
-    authHeader := r.Header.Get("Authorization")
-    claims, err := llmJWTAuth.ValidateBearerToken(authHeader)
-    if err != nil {
-        // Return 401/403
-    }
-    if !llmJWTAuth.CheckPermissions(claims, "llm:inference", "*") {
-        // Return 403 insufficient permissions
-    }
-}
-```
-
-### Authentik-side
-
-- [ ] Enable ROPC grant in Authentik provider settings (if not already)
-- [ ] Add redirect URIs to `local-llm` app if browser OAuth flow needed:
-
-```python
-# k8s/infra/iam/scripts/authentik-provision.py
-"local-llm": {
-    ...
-    "redirect_uris": [
-        "http://localhost:3000/api/auth/callback",
-        "https://your-app.com/api/auth/callback",
-    ],
-}
-```
-
-### NextJS-side
-
-- [ ] Until gateway auth is wired, LLM API works without token
-- [ ] Once wired, add `Authorization: Bearer <token>` to all LLM requests

k8s/apps/comfyui/deployment.yaml

@@ -22,6 +22,10 @@ spec:
       nodeSelector:
         kubernetes.io/hostname: worker-1
       runtimeClassName: nvidia
+      # k8s Service named 'comfyui' injects COMFYUI_PORT=tcp://... into pod env,
+      # which clobbers ai-dock's own COMFYUI_PORT variable (expects a port number).
+      # Disable service link injection to avoid the collision.
+      enableServiceLinks: false
       containers:
         - name: comfyui
           image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
@@ -31,8 +35,6 @@ spec:
           env:
             - name: NVIDIA_VISIBLE_DEVICES
               value: "all"
-            - name: COMFYUI_FLAGS
-              value: "--listen 0.0.0.0 --port 8188"
           resources:
             requests:
               cpu: "4"
@@ -57,7 +59,7 @@ spec:
             httpGet:
               path: /
               port: 8188
-            failureThreshold: 60
+            failureThreshold: 120
             periodSeconds: 10
       volumes:
         - name: models

k8s/apps/comfyui/ingress.yaml

@@ -4,16 +4,24 @@ metadata:
   name: comfyui
   namespace: comfyui
   annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
     nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
     nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
     nginx.ingress.kubernetes.io/proxy-body-size: "0"
     # WebSocket support for ComfyUI's live preview
     nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
-    nginx.ingress.kubernetes.io/proxy-set-headers: "Upgrade"
+    nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
+    nginx.ingress.kubernetes.io/configuration-snippet: |
+      proxy_set_header Upgrade $http_upgrade;
+      proxy_set_header Connection "upgrade";
 spec:
   ingressClassName: nginx
+  tls:
+    - secretName: comfyui-tls
+      hosts:
+        - comfyui.riotpiao.com
   rules:
-    - host: comfy.riotpiao.com
+    - host: comfyui.riotpiao.com
       http:
         paths:
           - path: /

k8s/apps/llm-serving/embeddings.yaml

@@ -45,6 +45,14 @@ spec:
       volumeMounts:
       - mountPath: /mnt/models
         name: models
+    podMetadata:
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8080"
+        prometheus.io/path: "/metrics"
+      labels:
+        app.kubernetes.io/name: llm-embeddings
+        app.kubernetes.io/part-of: llm-serving
     maxReplicas: 1
     minReplicas: 1
     nodeSelector:

k8s/apps/llm-serving/ornith.yaml

@@ -83,6 +83,14 @@ spec:
       volumeMounts:
       - mountPath: /mnt/models
         name: models
+    podMetadata:
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8080"
+        prometheus.io/path: "/metrics"
+      labels:
+        app.kubernetes.io/name: llm-ornith
+        app.kubernetes.io/part-of: llm-serving
     deploymentStrategy:
       type: Recreate
     # 1 replica -- ornith:35b only. qwen2.5:3b moved to CPU on cp-2.

k8s/apps/llm-serving/reasoning.yaml

@@ -104,6 +104,14 @@ spec:
         name: models
       - mountPath: /dev/shm
         name: shm
+    podMetadata:
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8080"
+        prometheus.io/path: "/metrics"
+      labels:
+        app.kubernetes.io/name: llm-reasoning
+        app.kubernetes.io/part-of: llm-serving
     deploymentStrategy:
       type: Recreate
     maxReplicas: 1

k8s/apps/llm-serving/reranker.yaml

@@ -45,6 +45,14 @@ spec:
       volumeMounts:
       - mountPath: /mnt/models
         name: models
+    podMetadata:
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "8080"
+        prometheus.io/path: "/metrics"
+      labels:
+        app.kubernetes.io/name: llm-reranker
+        app.kubernetes.io/part-of: llm-serving
     maxReplicas: 1
     minReplicas: 1
     nodeSelector:

k8s/apps/paperless/backup-cronjob.yaml

@@ -48,7 +48,7 @@ spec:
                   mountPath: /backup
           containers:
             - name: mc-mirror
-              image: minio/mc:latest
+              image: quay.io/minio/mc:latest
               env:
                 - name: ACCESS_KEY
                   valueFrom:

k8s/apps/paperless/kustomization.yaml

@@ -11,6 +11,7 @@ resources:
   - backup-cronjob.yaml
   - adapter-configmap.yaml
   - rbac.yaml
+  - paperless-ai.yaml
 # postgres: paperless-db CNPG Cluster, deployed by k8s/infra/databases (wave 2,
 # before this app at wave 8) - not duplicated here. Same for the paperless-oidc
 # and paperless-minio-creds Secrets, written by PostSync provisioning Jobs in

k8s/apps/paperless/paperless-ai.yaml

@@ -0,0 +1,64 @@
+# Secret paperless-ai-config managed via SOPS (argocd/secrets/paperless-ai-secrets.enc.yaml)
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperless-ai
+  namespace: paperless
+  labels:
+    app.kubernetes.io/name: paperless-ai
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperless-ai
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperless-ai
+    spec:
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: paperless-ai
+          image: clusterzx/paperless-ai:latest
+          env:
+            # Paperless-ngx connection
+            - name: PAPERLESS_API_URL
+              value: "http://paperless.paperless.svc.cluster.local:8000"
+            - name: PAPERLESS_API_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-ai-config
+                  key: PAPERLESS_API_TOKEN
+            - name: PAPERLESS_USERNAME
+              value: "admin"
+            # LLM API — local gateway, no auth required (phase 3 not built yet)
+            - name: AI_PROVIDER
+              value: "custom"
+            - name: CUSTOM_BASE_URL
+              value: "http://api-gateway.api.svc.cluster.local:8080/v1"
+            - name: CUSTOM_API_KEY
+              value: "not-required"
+            - name: CUSTOM_MODEL
+              value: "reasoning"
+            # Behavior
+            - name: SCAN_INTERVAL
+              value: "300"
+            - name: PROCESS_PREDEFINED_DOCUMENTS
+              value: "no"
+            - name: ADD_AI_TAG
+              value: "yes"
+            - name: AI_TAG_NAME
+              value: "ai-processed"
+            - name: USE_PROMPT_TAGS
+              value: "yes"
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 2Gi

k8s/argocd/apps/06-ci-cd.yaml

@@ -0,0 +1,55 @@
+# Tekton Pipelines v0.68.0
+#
+# Install method: vendored release.yaml in k8s/infra/tekton/
+# downloaded from https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.68.0/release.yaml
+#
+# To upgrade:
+#   1. Download new release.yaml from https://github.com/tektoncd/pipeline/releases
+#   2. Replace k8s/infra/tekton/release.yaml
+#   3. Commit and push — ArgoCD syncs automatically
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: tekton-pipelines
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: tekton-pipelines
+    app.kubernetes.io/part-of: homelab-infra
+    wave: "06"
+spec:
+  project: homelab
+
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/tekton
+
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: tekton-pipelines
+
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+    - CreateNamespace=true
+    - ServerSideApply=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 3m
+
+  ignoreDifferences:
+  - group: admissionregistration.k8s.io
+    kind: ValidatingWebhookConfiguration
+    jsonPointers:
+    - /webhooks/0/clientConfig/caBundle
+    - /webhooks
+  - group: admissionregistration.k8s.io
+    kind: MutatingWebhookConfiguration
+    jsonPointers:
+    - /webhooks/0/clientConfig/caBundle
+    - /webhooks

k8s/argocd/apps/70-poimen.yaml

@@ -19,16 +19,10 @@ metadata:
     argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
   project: homelab
-  sources:
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-memory.git
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
     targetRevision: main
-      path: k8s/argocd
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-workflows.git
-      targetRevision: main
-      path: k8s/argocd
-    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/poimen-frontend.git
-      targetRevision: main
-      path: k8s/argocd
+    path: k8s
   destination:
     server: https://kubernetes.default.svc
     namespace: poimen

k8s/argocd/secrets/forgejo-smtp-secrets.enc.yaml

@@ -0,0 +1,24 @@
+apiVersion: ENC[AES256_GCM,data:uS8=,iv:EEoo9U+C244eAJMSTOQVkf5AE6BeHrc5DWPjtWnPRdk=,tag:H+YmBSGvcON3wh0p22LXDQ==,type:str]
+kind: ENC[AES256_GCM,data:j1/PFkTS,iv:ja4q8X+nzE/ZczwcY+Qe2DnnsxG64W1fkRPQvOaoqvA=,tag:WiilMGagudEKUlc2JdbGyg==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:J9iAIHboMyHu6xn/,iv:p3z3uzlkM7VDzOT3WDCelCdZ2t+QqSPmFtqYfvXPQMs=,tag:rKRtpUexVkuZKOJ34a0Xjw==,type:str]
+    namespace: ENC[AES256_GCM,data:su0FvA==,iv:6SPvwxZ/4aoL0Z07zdPC9r6L7HOHuKv3RodXpVcii04=,tag:njwkj+yvSgN8sliJP8T/Kw==,type:str]
+type: ENC[AES256_GCM,data:Qd6WJN1c,iv:M3fc78LvemcEbWzesuYeQ/GZyIOl7Eg/0EmUe3p0qAk=,tag:Tnzwewn0vwdowxy/EyuPdA==,type:str]
+stringData:
+    user: ENC[AES256_GCM,data:/Z7gPrsTUZOLzfoZ8cZGD10wrZE=,iv:0ydBSeq+yHB2b1J4W7FwIv55Eh+N3qfQ6Q7PwoUAvYo=,tag:CO19F8nElaYZyFiFR6N/Tg==,type:str]
+    password: ENC[AES256_GCM,data:AOl4StpeQIHSLX+oEFnkfg==,iv:mYsZ+5sum6YqyUPndtPCniTraxldKc803ULlKs8Gsaw=,tag:hK3w51ifZ/omAL7XDf8seg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMlhKVEM2bFdFOFAwV1lp
+            WUd6SEN4ZTF1TEpGYXhOYmJLK0tpcWZyc2w4ClVzZmI2Qk1KUnV1OXpyTEV2WWFa
+            VlJ2eHRSaEhqUnA2dUJVbWJUcEgxcFkKLS0tIG9IaFVnenNpSFRzdStiWjJvakVL
+            aDk2bTZGR3Zya3ROUS9vd1hEQVRFaG8KbeXA6IebHEaB79N6u795336aHesHOgzO
+            uZvvBUzSBy3t3jfFk8bJP4aH79I33Ha2eK5rsvdsiv/orwCMXUINKg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-12T21:00:46Z"
+    mac: ENC[AES256_GCM,data:DjzPtR+Ueihh166Bvd3jCLtZFQdrvrxOoF87cv6lxKGWPEptT5vbw/EfuIFJBb6lvEJFDWKRC/r5ASdGwComYsPn0DZs4BPCzeKBtLfP9K2OGCXnAC7eGqt4mzMrrW0CQd1QSGcuXw8CY7uJGkMPGPKe3/0mQWiis2OSpHxTM18=,iv:/QqSEFU9RuX9z5Z6aSaD7KlP/cgGTOYvTH+VJOnDTYM=,tag:yFbFXn+iWqQZx7wW4dKppg==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/gotify-smtp-secrets.enc.yaml

@@ -1,30 +1,28 @@
-apiVersion: ENC[AES256_GCM,data:xUA=,iv:YH1WYnoXhxbeES3Dmo8ZrMDwFcwCykt6+Zn4YSgN4gc=,tag:t3bIWo/LetlQv4tJ0nJ2Sw==,type:str]
-kind: ENC[AES256_GCM,data:PJVzDTcT,iv:aRxvzLUuCzSDryOjA/v67csL6E2FcCpoK8aD7KNStno=,tag:cQgbowSTTV99nQ41yFOlsA==,type:str]
+apiVersion: ENC[AES256_GCM,data:FOg=,iv:15mfPeWXV5LQEYahaYvNf1z2bHbxk4j5i8DXT6mdG/0=,tag:1f9/jnnu+wGeeiXGNFgKzA==,type:str]
+kind: ENC[AES256_GCM,data:dOFfNQiM,iv:HVLosbRpcCgu4iiYKV8MDLiQ0uhj8DXBfVpRBW2NFNo=,tag:mCbRIzD1OHRhBLc+7xcVug==,type:str]
 metadata:
-    name: ENC[AES256_GCM,data:atkSiP0rjyuWFjs=,iv:l/48X9W9lrkAyw6Q/p39V775L6xbr0tOIFI4vjYrg+E=,tag:LZrQxN5h6T5tehFG8L9KFw==,type:str]
-    namespace: ENC[AES256_GCM,data:fbklNyooBeqsogKbeg==,iv:z2wLgl81q8xfzGGVQ9A4Pkst/zLmuZpyBzHrXzmVfm4=,tag:QKJik4IWc4nrOBINjGCxvQ==,type:str]
-type: ENC[AES256_GCM,data:bfAFBd/7,iv:9/W3X1KefqDAsurIatT6kBuEIQvWVRTq4G9A+9ZfYKs=,tag:Optlb+gZ2udbqIOerqV2zQ==,type:str]
+    name: ENC[AES256_GCM,data:kHoKhGyiIQvCfng=,iv:dAhxjeLlXK3yueMKfrHxmh9YmNA7AYKFBquLikCNzcE=,tag:3Xoaw7YNBCU/GINlaQOpBw==,type:str]
+    namespace: ENC[AES256_GCM,data:fTJMZhbqSQWD3/cnWg==,iv:D7zWa91+Fgerfyrywf8VA5YNzGrThhLz7CvYXucfiq0=,tag:RcIegCl2UR1JsbfIQm928w==,type:str]
+type: ENC[AES256_GCM,data:Qh51bTsM,iv:2Htv0Xze7Hd1m6zkP6rtFXAlg8qm0AKylSjwJxRjpBk=,tag:NCu5iVUTNVfYgErZSvwHIg==,type:str]
 stringData:
-    #ENC[AES256_GCM,data:RZaw/FNQZOf0F3+XfFmGW+4gjKXNdjrAbMRjBt4hptwk,iv:IEd/+Bsv/h08nKak+bcovNiohC7nNjEQ+QdtIFMUGng=,tag:zsF5ZQ3+5JijF4MU+RhnyA==,type:comment]
-    #ENC[AES256_GCM,data:ux3ONHnuI0uFEihElbLI1rKlrN91Pn0iLLZfmez0+t5d8yhH6OD3L/3oi6yHbdgD8bV90kRK/lw=,iv:7Ed+W/TeOY9Af8bKO0Fu3XOuaK8b0Hw70rwobfmosxM=,tag:Z0RbIOjdDroK0drC6P+Y8A==,type:comment]
-    host: ENC[AES256_GCM,data:YKgas/8hT5ZOGIW7O0c=,iv:WtBz1MFrwd6EgDDS5fMqtcVGIZGNLgbXnmkqlNWoXy8=,tag:+H9ARGPi5XocgZYp4oA3wA==,type:str]
-    port: ENC[AES256_GCM,data:f0hx,iv:HkSnJcCyzyfBTA39BNw+muhHTyrNLzzGxZ7M3bcLVu4=,tag:4+xL1xNyYPaV2b0mwnbQVQ==,type:str]
-    from: ENC[AES256_GCM,data:lqYqXxJuI4CAWPlX+5cltCdxqA==,iv:ZgZYySJXvjuTNIkMaDxynUJ6PFdp4fibqlgoA1vNDKE=,tag:vb/XKl2N1Hl5u7ndZXQvbg==,type:str]
-    user: ENC[AES256_GCM,data:ajljGVbRO//G,iv:E4Q0LGDCd5wnHc+Oq2A9ZhWGbCmkKlhllt922UmppIg=,tag:Cf4ft1JKFi9PaJEfPfq92Q==,type:str]
-    password: ENC[AES256_GCM,data:YNbtoiWLsQGl,iv:FJ3y1lWkqKkM8GVr7FZljAXJ331GkDgy4TenAbdIY3w=,tag:KIvhv+RL49ZMwWhYrU0SPA==,type:str]
-    notify-email: ENC[AES256_GCM,data:s5wRkDuaiflCyu4IyAnaocVPA3I=,iv:dK5PsweYa3JotYcDcz74DPNDXmhW5ngNEbY6JuG1ctg=,tag:/Dt2UoCtfau6qv0G7VzmrQ==,type:str]
+    host: ENC[AES256_GCM,data:aZXSvQ8B7h78q1kHmh0=,iv:uUyL9X3ehIHqztGAtXtAQWgduvbSsSwZBZlTFMdOlBo=,tag:OM/vdO19VKDSa4W5WQAKNQ==,type:str]
+    port: ENC[AES256_GCM,data:5ZBB,iv:1/XAfq+PJKdBsufx+EPvvB/cm9nbEwYigc8eACXjR8g=,tag:WlNyz7gTPh0KMe5oKVd0BA==,type:str]
+    from: ENC[AES256_GCM,data:2/akr8cXgmgQGnqJaFcLJMDcWw==,iv:0rT/6aqyXxn1jIJ5umYCDZR4S8Pbh4vUgaXNrxd3JF8=,tag:k0pQrPOrqWTyE1Xu9oSzVA==,type:str]
+    user: ENC[AES256_GCM,data:d9BLaFYOYm++HZMzlf1gVauKkUQ=,iv:Wd48T5UPVn6z3bS0t02X9GbrnWal3QoIQv2EgM9z9Wg=,tag:cmg2KnLY4Atwco+j2wVdeg==,type:str]
+    password: ENC[AES256_GCM,data:pmqEj9623A6bThKrkCCjuA==,iv:M/QwJ0s//UvuOjOljl67EDhvZt/9Wp8JicvCcows51A=,tag:bfRyIdj1cgIZxU3WPbKteg==,type:str]
+    notify-email: ENC[AES256_GCM,data:+Xo22g8U2wDKsnPnFq/+GKxgYOI=,iv:PE+C1etlp6046ragiv1iMDQbhfo5IULd/5akD0+Sc6A=,tag:cCZmXByA85fuZL9yozVLjw==,type:str]
 sops:
     age:
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLQlJVekcwb01ESnhxSE9m
-            MzliVmlaYUttZVdiMDVtMlQ2L29LMXBWRzBZCkxRMi9RcE5LMzQ1MHFqUUxLaFZo
-            MUZ2UTg5eUNsMFVmenlhL2F3NDd0akkKLS0tIGFwbUUrSGFUTDFVOWlxM3V1bVdy
-            RUVVNDFMRHZCSWptdSt1bFVoN2JPWVkKXMlrJNPbm7Sj9vd/H3o6iqPRLy4LyjVe
-            xJyh9vn9RCwS9Jr6BN2Q9P2UZfTGwvIYc6zdmV+b2uRwvi4t4lMsBA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpVVdxQTZFTHcwMXFFZEFk
+            ckNsYk4rdWtkeXFMVnlyVmdaRDBuRWsvZkQ4CitSSTBXOTZhWVpUZFFlR0JITVV4
+            Uy9lSlNHNjFNaFhHNTN0WnRaRlNNVGcKLS0tIHRlMnZpQVlScGRYYm5nT3Z5TWd6
+            ak1UZ1RobkpQZHBXR3MyenBDcU53Mk0KvH9O6bgwrjay0+1/A6TGX8GhITiDjWoO
+            RNX4fDtqNwhzmCfXbVjK30vlBjOFe+Bb7Z2n+hWMmHHDgFdS0xIAzA==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-10T13:31:00Z"
-    mac: ENC[AES256_GCM,data:g1YjOANlnvXErPqtDdDjRB9yTUiGEhX8QDkmf1aLfQBwoXbGUpoPIZibF8A4cAL1yUSZ2rD4l9TIe9nK8U3oF4WiKXf9u0bFK2g//bOv2A5JPewwmzoazkjIDXcJ/76nOQ5c+v8/r5tUG4c0SyzBH/gFP0D0+P5Yo3SEXjTxj2w=,iv:gW+AUwwAWkXZ+aLBXjjNDZw8VEraMpSE16RJGQc/L5g=,tag:DM/DUtil/7yLhfGYQXKFGA==,type:str]
+    lastmodified: "2026-09-12T20:57:18Z"
+    mac: ENC[AES256_GCM,data:V/wgjnpUBvaV39BCTlecCwQy2/1h+0vixEXleJc/I9y+AOvuwVZNH7M86hdjoAdsKiIoNYySj4Flz+MJ9C8jQoAxBTDPbolP9UwDFWQ5KMJTKPPDoJGLNjy7bUq51WoyePUM6WWAYIiWHIB3OvAxHaSzECzL+ZoAhQy92cPKa1o=,iv:vLtIoD/C5yeXPEr+2Lim6XnWzAj42vr9qQgsxKpuhA8=,tag:PBq8qPHrx4RRgAKCWrxG3Q==,type:str]
     unencrypted_suffix: _unencrypted
     version: 3.13.2

k8s/argocd/secrets/paperless-ai-secrets.enc.yaml

@@ -0,0 +1,23 @@
+apiVersion: ENC[AES256_GCM,data:l7I=,iv:NZY7r3JVW3zVwxeiScvWKpAQUDa9+nckHd0qWVrGU88=,tag:qpowp5OILdYKtTux/nlWpg==,type:str]
+kind: ENC[AES256_GCM,data:6oeEbuIk,iv:5flI9TtcYQ961wOYPBPhvQpitHFYAf8yMdNBXqytbJs=,tag:WNhlbKmSeZEqZ9ZgiB/BYg==,type:str]
+metadata:
+    name: ENC[AES256_GCM,data:fvJMsgy+wPZqMCdxm9hoV3n/+Q==,iv:I3rVtHIJBOME+bxhPws58Zjhj5i+KT5UtB9o7Vus5EU=,tag:6h4FnUu7PGxlQPIQxPYCRg==,type:str]
+    namespace: ENC[AES256_GCM,data:CDriLoLovROW,iv:rHXcN1xm5+t2D/Tq/2sx9lQFF8anD5jH24ZntPuBA8U=,tag:XstJAz6S8IM1c/pp9Cg0Ww==,type:str]
+type: ENC[AES256_GCM,data:JdTwBbag,iv:9Ys15Ketl0ghNK0u0N8IOpUt7+KoloutiG5M9zHPXxw=,tag:teau/udf41Ty34A5wLAm6Q==,type:str]
+stringData:
+    PAPERLESS_API_TOKEN: ENC[AES256_GCM,data:TuQeDx8po3h4loTRABlItVYQJ7gFjnmIn3zQQGtUcoNFMKpkqLK/GQ==,iv:TDg0stpca5pDtatqu8DFU7R0Bm/S/BI9ZoiG4K8mCT4=,tag:BfjX25V5gL7AeIsscClRAg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKVDN3aFVqVmFXY1VQVXZP
+            OUsrNHdNdlRvRFgvTDQrNE5uL2xaazVENTBjCkNPbGR5Vk16RXNDOW15OGNTRmFR
+            U0JOeTllaWU1dzNVY3lBbEVyVG5tOEkKLS0tIEZvajlvcUtJdFNNUkkzV3FKOFRj
+            UUE2TDBzT0xVc2E1NlUvQXAyZytMZEUKBv+ChaoQCstA742L3Bq5mBJlW/UC4Pyw
+            ZvFAyYbs1NaEqhjtHq+4T62jTWcH/St/vKgUuFQ9LCUQhYd8DUzAow==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    lastmodified: "2026-09-12T21:04:26Z"
+    mac: ENC[AES256_GCM,data:0ks96xQzOa8ygNDDYfKV8EJ8dWLBaC0PCnLG73NinMByF/KoWkCdwMDPC34W9xxVoTD2NiF1i/3pgCG4QFezTE7tPHPPKdYcQfwda9fkooiXW4Nh2M/sgbq/xgsy9N3qpHD/O5iILgpehb9y0DuErOyxcn2AIYizynU7m8e63pc=,iv:fvPWBsfE6YHskKzdlxJidp14dUgRR0XdMkEu6IxMMSo=,tag:5FiuIrFOg/GXvlQVy7drJQ==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.2

k8s/argocd/secrets/secret-generator.yaml

@@ -30,3 +30,5 @@ files:
   - gotify-admin-secrets.enc.yaml
   - gotify-tokens-secrets.enc.yaml
   - gotify-smtp-secrets.enc.yaml
+  - forgejo-smtp-secrets.enc.yaml
+  - paperless-ai-secrets.enc.yaml

k8s/bootstrap/phase1-storage/longhorn-values.yaml

@@ -12,6 +12,7 @@ defaultSettings:
   replicaSoftAntiAffinity: false  # REQUIRED for true HA
   replicaAutoBalance: best-effort
   storageMinimalAvailablePercentage: 10
+  storageOverProvisioningPercentage: 200  # Actual usage is ~10% of scheduled; 200% unblocks all 3-replica scheduling on cp-1
   
   # Performance tuning
   defaultDataPath: /var/lib/longhorn

k8s/bootstrap/phase3-forgejo/forgejo-values.yaml

@@ -80,6 +80,14 @@ gitea:
     actions:
       ENABLED: true
 
+    mailer:
+      ENABLED: true
+      PROTOCOL: smtp+starttls
+      SMTP_ADDR: smtp.gmail.com
+      SMTP_PORT: 587
+      FROM: "Forgejo <[email protected]>"
+      # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
+
 # Persistence (shared storage for repos)
 persistence:
   enabled: true
@@ -148,3 +156,13 @@ deployment:
         secretKeyRef:
           name: forgejo-db-app
           key: password
+    - name: GITEA__MAILER__USER
+      valueFrom:
+        secretKeyRef:
+          name: forgejo-smtp
+          key: user
+    - name: GITEA__MAILER__PASSWD
+      valueFrom:
+        secretKeyRef:
+          name: forgejo-smtp
+          key: password

k8s/infra/argocd-image-updater/values.yaml

@@ -57,8 +57,6 @@ extraVolumeMounts:
 
 # Extra environment variables
 extraEnv:
-  - name: ARGOCD_GRPC_WEB
-    value: "true"
   - name: GIT_SSH_KNOWN_HOSTS_CONFIG_MAP_ENABLED
     value: "true"
 

k8s/infra/databases/kustomization.yaml

@@ -10,4 +10,3 @@ resources:
   - temporal-db.yaml
   - memory-db.yaml
   - paperless-db.yaml
-  - obsidian-vault-pvc.yaml

k8s/infra/databases/memory-db.yaml

@@ -9,7 +9,7 @@ metadata:
   annotations:
     argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
-  instances: 2
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.2
   bootstrap:
     initdb:

k8s/infra/databases/obsidian-vault-pvc.yaml

@@ -1,18 +0,0 @@
----
-# Obsidian vault PVC — shared storage for REST API + UI pods
-# ReadWriteMany so both obsidian-server and obsidian-ui can mount simultaneously
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: obsidian-vault
-  namespace: poimen
-  labels:
-    app.kubernetes.io/name: obsidian-server
-    app.kubernetes.io/part-of: poimen-memory
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 10Gi

k8s/infra/iam/authentik-values.yaml

@@ -153,9 +153,11 @@ server:
   # checks aren't treated as failures. (Only these fields are overridden; the
   # chart deep-merges the rest of each probe, incl. the httpGet path.)
   livenessProbe:
+    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
     timeoutSeconds: 15
     failureThreshold: 6
   readinessProbe:
+    initialDelaySeconds: 300  # skip probe until migrations complete
     timeoutSeconds: 15
     failureThreshold: 6
   startupProbe:
@@ -215,6 +217,13 @@ worker:
   podAnnotations:
     configmap.reloader.stakater.com/reload: "homelab-ca"
     homelab.io/restart-at: "2026-06-21T13-40"
+  livenessProbe:
+    initialDelaySeconds: 300  # skip probe until 5min passed (migrations finish)
+  readinessProbe:
+    initialDelaySeconds: 300  # skip probe until migrations complete
+  startupProbe:
+    initialDelaySeconds: 30  # let server finish DB work first
+    failureThreshold: 120
   metrics:
     enabled: true
     serviceMonitor:

k8s/infra/longhorn/longhorn-cnpg-storageclass.yaml

@@ -35,8 +35,5 @@ parameters:
   mkfsParams: "-O ^64bit,^metadata_csum"
 mountOptions:
   - "noatime"
-  # Critical: mount with postgres UID/GID (26:26) to avoid permission issues
-  - "uid=26"
-  - "gid=26"
 reclaimPolicy: Delete
 volumeBindingMode: Immediate

k8s/infra/minio/minio-provision-paperless-job.yaml

@@ -59,7 +59,7 @@ spec:
               mountPath: /shared
       containers:
         - name: provision
-          image: minio/mc:latest
+          image: quay.io/minio/mc:latest
           volumeMounts:
             - name: shared
               mountPath: /shared
@@ -81,6 +81,9 @@ spec:
               mc alias set m http://minio-cluster-hl.storage.svc.cluster.local:9000 \
                 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
 
+              echo "Ensuring paperless bucket exists..."
+              mc mb --ignore-existing m/paperless
+
               echo "Checking for existing paperless-minio-creds secret..."
               if kubectl -n paperless get secret paperless-minio-creds >/dev/null 2>&1; then
                 ACCESS_KEY=$(kubectl -n paperless get secret paperless-minio-creds -o jsonpath='{.data.ACCESS_KEY}' | base64 -d)

k8s/infra/monitoring/servicemonitors/vllm.yaml

@@ -0,0 +1,30 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: vllm
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: vllm
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  namespaceSelector:
+    matchNames:
+      - llm-serving
+  selector:
+    matchLabels:
+      app.kubernetes.io/part-of: llm-serving
+  endpoints:
+    - port: http
+      interval: 30s
+      scrapeTimeout: 10s
+      path: /metrics
+      scheme: http
+      relabelings:
+        - sourceLabels: [__meta_kubernetes_namespace]
+          targetLabel: namespace
+        - sourceLabels: [__meta_kubernetes_pod_name]
+          targetLabel: pod
+        - sourceLabels: [__meta_kubernetes_service_name]
+          targetLabel: service
+        - sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
+          targetLabel: app

project-usage/gotify-notifications.md

@@ -0,0 +1,162 @@
+# Gotify Notifications Setup
+
+## Overview
+
+Gotify is a self-hosted push notification server deployed in the `notifications` namespace. The homelab-frontend gateway provides a `sendMsg` endpoint that accepts email and SMS notification requests and sends them directly via SMTP (SMS provider TBD).
+
+## Architecture
+
+```
+App → X-Service: notification header
+    → homelab-frontend gateway (api namespace)
+    → notification/sendMsg handler
+    → SMTP relay (email) or SMS provider (stubbed)
+    → recipient email/SMS
+```
+
+**Not used:** Gotify's native message store is available for UI/push notifications, but the sendMsg flow bypasses it (direct send, no storage).
+
+## Usage
+
+### Send Email
+
+```bash
+curl -X POST https://api.riotpiao.com \
+  -H 'X-Service: notification' \
+  -H 'X-Resource: sendMsg' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "format": "smtp",
+    "title": "Alert",
+    "message": "System CPU high",
+    "priority": 5,
+    "extras": {
+      "to_email": "admin@example.com",
+      "cc": "ops@example.com"
+    }
+  }'
+```
+
+**Response (success):**
+```json
+{
+  "status": "success",
+  "messageId": "email-admin@example.com"
+}
+```
+
+**Response (error):**
+```json
+{
+  "status": "error",
+  "error": "failed to send email: connection refused"
+}
+```
+
+### Send SMS (Stubbed)
+
+SMS support is stubbed. Currently returns "not implemented" error. To enable:
+
+1. Choose SMS provider (Twilio, AWS SNS, Vonage, etc.)
+2. Set `SMS_API_URL` and `SMS_API_KEY` environment vars in gateway Deployment
+3. Implement provider integration in `internal/notification/handler.go` sendSMS() method
+
+```bash
+curl -X POST https://api.riotpiao.com \
+  -H 'X-Service: notification' \
+  -H 'X-Resource: sendMsg' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "format": "sms",
+    "message": "System CPU high",
+    "extras": {
+      "phone": "+12025551234"
+    }
+  }'
+```
+
+## Configuration
+
+### SMTP Settings
+
+Gateway reads SMTP config from environment variables (pulled from `smtp-credentials` Secret in `api` namespace):
+- `SMTP_HOST` — SMTP server hostname
+- `SMTP_PORT` — SMTP server port (587 TLS or 465 SSL)
+- `SMTP_FROM` — Sender email address
+- `SMTP_USER` — SMTP auth username
+- `SMTP_PASS` — SMTP auth password
+
+Secret is SOPS-encrypted in git. Create via:
+```bash
+kubectl create secret generic smtp-credentials \
+  --from-literal=host=mail.riotpiao.com \
+  --from-literal=port=587 \
+  --from-literal=from=alerts@riotpiao.com \
+  --from-literal=user=smtp-user \
+  --from-literal=password=smtp-password \
+  -n api \
+  -o yaml | sops -e /dev/stdin > k8s/smtp-secrets.enc.yaml
+```
+
+Then add to `k8s/kustomization.yaml`:
+```yaml
+resources:
+- smtp-secrets.enc.yaml
+```
+
+### Gotify Server
+
+Gotify runs in `notifications` namespace with:
+- PostgreSQL backend (CNPG)
+- SMTP emailer sidecar (unused by sendMsg, but available for UI notifications)
+- Health check on `:80/health`
+
+Config: `k8s/apps/gotify/`
+
+## Testing
+
+```bash
+cd homelab-frontend
+bash examples/sendmsg-email.sh https://api.riotpiao.com
+```
+
+## Roadmap
+
+- [ ] SMS provider integration (pick: Twilio/SNS/Vonage)
+- [ ] Request rate limiting per source
+- [ ] Message queuing for retries (via SQS if high volume expected)
+- [ ] Audit logging (who sent what, to whom, when)
+- [ ] Template support (subject + body with placeholders)
+
+## Troubleshooting
+
+### "SMTP_HOST not set"
+Gateway env vars not loaded. Check:
+```bash
+kubectl -n api describe pod api-gateway-xyz
+kubectl -n api logs api-gateway-xyz | grep SMTP
+```
+
+### "connection refused" on SMTP
+SMTP server unreachable. Verify:
+```bash
+kubectl -n api exec -it api-gateway-xyz -- \
+  nc -zv $SMTP_HOST $SMTP_PORT
+```
+
+### "authentication failed"
+Wrong SMTP username/password. Verify credentials:
+```bash
+kubectl -n api get secret smtp-credentials -o yaml | grep password | base64 -d
+```
+
+### "X-Resource: sendMsg not found"
+Notification handler not registered. Check `internal/server/router.go`:
+- Verify `X-Service: notification` case exists
+- Confirm `notification.NewHandler()` called in `NewRouter()`
+
+## References
+
+- [API Documentation](../homelab-frontend/API.md#notification-services)
+- [Gotify Server Docs](https://gotify.net)
+- [SMTP Configuration Best Practices](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol)

scripts/iam/provision-rbac.py

@@ -0,0 +1,649 @@
+#!/usr/bin/env python3
+"""
+Provision RBAC groups, service account roles, fine-grained claims, and auth flows.
+
+Idempotent — safe to re-run. Provisions:
+  1. Global admin groups (homelab-admins)
+  2. Fine-grained service/bucket/project groups (minio-*, poimen-*, paperless-*, grafana-*, sqs-*)
+  3. Service account roles with custom claims (paperless-ai-agent, portfolio-agent, etc.)
+  4. JWT scope mappings for fine-grained claims (minio_buckets, paperless_doctypes, etc.)
+  5. OAuth2 providers with scopes (api-gw, minio, poimen, paperless, grafana)
+  6. Auth flows (password grant on api-gw provider)
+
+Usage:
+    source ~/.env
+    export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
+        -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
+    python3 scripts/iam/provision-rbac.py
+
+DO NOT commit this file to git — .gitignore covers scripts/iam/*.py.
+"""
+import json
+import os
+import sys
+import urllib.error
+import urllib.request
+from typing import Dict, List, Any
+from pathlib import Path
+
+# Load ~/.env for OAuth2 provider secrets
+env_file = Path.home() / ".env"
+if env_file.exists():
+    with open(env_file) as f:
+        for line in f:
+            line = line.strip()
+            if line.startswith("export ") and "=" in line:
+                key, _, value = line[7:].partition("=")
+                key = key.strip()
+                value = value.strip().strip('"').strip("'")
+                os.environ[key] = value
+
+AUTHENTIK_URL = "https://authentik.riotpiao.com"
+TOKEN = os.environ.get("AUTHENTIK_BOOTSTRAP_TOKEN")
+if not TOKEN:
+    print("Error: AUTHENTIK_BOOTSTRAP_TOKEN not set")
+    print("  source ~/.env")
+    print("  export AUTHENTIK_BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \\")
+    print("      -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)")
+    sys.exit(1)
+
+
+def api(method, path, data=None):
+    url = f"{AUTHENTIK_URL}{path}"
+    body = json.dumps(data).encode() if data is not None else None
+    req = urllib.request.Request(url, data=body, method=method, headers={
+        "Authorization": f"Bearer {TOKEN}",
+        "Content-Type": "application/json",
+    })
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            raw = resp.read()
+            return resp.status, json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as e:
+        raw = e.read()
+        try:
+            parsed = json.loads(raw) if raw else {}
+        except json.JSONDecodeError:
+            parsed = {"raw": raw.decode(errors="replace")}
+        return e.code, parsed
+
+
+def die(msg):
+    print(f"FATAL: {msg}", file=sys.stderr)
+    sys.exit(1)
+
+
+# ===========================================================================
+# Group Definitions (DRY: single source of truth)
+# ===========================================================================
+GROUPS: Dict[str, Dict[str, Any]] = {
+    "homelab-admins": {
+        "description": "Cluster administrators with full access",
+        "is_superuser": True,
+    },
+    "minio-admins": {"description": "MinIO administrators", "is_superuser": False, "minio_buckets": ["*"]},
+    "minio-photos": {"description": "Photos bucket (Immich) access", "is_superuser": False, "minio_buckets": ["immich"]},
+    "minio-documents": {"description": "Documents bucket (Paperless) access", "is_superuser": False, "minio_buckets": ["paperless"]},
+    "minio-backups": {"description": "Backups bucket read-only access", "is_superuser": False, "minio_buckets": ["backups"]},
+    "poimen-admins": {"description": "Poimen memory administrators", "is_superuser": False, "memory_projects": ["*"], "memory_visibility": "private"},
+    "poimen-devs": {"description": "Dev and staging projects access", "is_superuser": False, "memory_projects": ["dev", "staging"], "memory_visibility": "internal"},
+    "poimen-prod-readonly": {"description": "Production projects read-only access", "is_superuser": False, "memory_projects": ["prod"], "memory_visibility": "public"},
+    "paperless-admins": {"description": "Paperless administrators", "is_superuser": False, "paperless_doctypes": ["*"]},
+    "paperless-finance": {"description": "Finance documents", "is_superuser": False, "paperless_doctypes": ["invoices", "receipts", "expenses"]},
+    "paperless-legal": {"description": "Legal documents", "is_superuser": False, "paperless_doctypes": ["contracts", "licenses", "agreements"]},
+    "paperless-hr": {"description": "HR documents", "is_superuser": False, "paperless_doctypes": ["employment", "benefits", "payroll"]},
+    "grafana-admins": {"description": "Grafana administrators", "is_superuser": False, "grafana_org_role": "Admin"},
+    "grafana-editors": {"description": "Grafana dashboard editors", "is_superuser": False, "grafana_org_role": "Editor"},
+    "grafana-viewers": {"description": "Grafana dashboard viewers", "is_superuser": False, "grafana_org_role": "Viewer"},
+    "sqs-users": {"description": "SQS/Temporal queue read access", "is_superuser": False, "sqs_queues": ["default"]},
+    "sqs-writers": {"description": "SQS/Temporal queue read/write access", "is_superuser": False, "sqs_queues": ["*"]},
+    "s3-users": {"description": "S3 read access", "is_superuser": False},
+    "s3-writers": {"description": "S3 read/write access", "is_superuser": False},
+}
+
+SERVICE_ACCOUNTS: Dict[str, Dict[str, Any]] = {
+    "paperless-ai-agent": {
+        "description": "Paperless AI plugin (auto-tagging, entity extraction)",
+        "roles": ["llm:inference", "memory:write", "paperless:admin"],
+        "claims": {
+            "minio_buckets": ["paperless"],
+            "paperless_doctypes": ["*"],
+            "memory_projects": ["*"],
+            "authorized_models": ["reasoning", "qwen2.5:3b"],
+        },
+    },
+    "portfolio-agent": {
+        "description": "Portfolio service agent",
+        "roles": ["llm:inference", "memory:read"],
+        "claims": {
+            "memory_projects": ["homelab", "portfolio"],
+            "memory_visibility": "public",
+            "authorized_models": ["ornith:35b"],
+            "minio_buckets": ["backups"],
+        },
+    },
+    "memory-agent": {
+        "description": "Memory service agent",
+        "roles": ["llm:inference", "memory:read", "memory:write"],
+        "claims": {
+            "memory_projects": ["*"],
+            "memory_visibility": "private",
+            "authorized_models": ["reasoning", "ornith:35b", "qwen2.5:3b"],
+        },
+    },
+    "temporal-worker-agent": {
+        "description": "Temporal workflow worker",
+        "roles": ["llm:inference", "workflow:execute", "memory:read", "memory:write", "queue:send"],
+        "claims": {
+            "memory_projects": ["*"],
+            "sqs_queues": ["*"],
+            "authorized_models": ["reasoning", "ornith:35b", "qwen2.5:3b"],
+        },
+    },
+}
+
+SCOPE_MAPPINGS: Dict[str, Dict[str, str]] = {
+    "roles": {"expression": 'return user.attributes.get("roles", [])'},
+    "permissions": {"expression": 'return ["*"] if any(user.groups.filter(is_superuser=True)) else list(user.groups.values_list("name", flat=True))'},
+    "minio_buckets": {"expression": 'return user.attributes.get("minio_buckets", [])'},
+    "paperless_doctypes": {"expression": 'return user.attributes.get("paperless_doctypes", [])'},
+    "memory_projects": {"expression": 'return user.attributes.get("memory_projects", [])'},
+    "memory_visibility": {"expression": 'return user.attributes.get("memory_visibility", "public")'},
+    "authorized_models": {"expression": 'return user.attributes.get("authorized_models", [])'},
+    "sqs_queues": {"expression": 'return user.attributes.get("sqs_queues", [])'},
+    "grafana_org_role": {"expression": 'return user.attributes.get("grafana_org_role", "Viewer")'},
+}
+
+# ===========================================================================
+# Phase 1: Create/sync all groups
+# ===========================================================================
+print("[1/6] Ensuring groups exist...")
+
+status, res = api("GET", "/api/v3/core/groups/?page_size=100")
+if status != 200:
+    die(f"GET groups -> {status} {res}")
+existing_groups = {g["name"]: g for g in res["results"]}
+
+created_count = 0
+for group_name, group_spec in GROUPS.items():
+    if group_name in existing_groups:
+        print(f"  {group_name}: already exists")
+    else:
+        status, res = api("POST", "/api/v3/core/groups/", {
+            "name": group_name,
+            "is_superuser": group_spec.get("is_superuser", False),
+        })
+        if status in (200, 201):
+            print(f"  {group_name}: created")
+            created_count += 1
+        else:
+            print(f"  {group_name}: FAILED {status} {res}")
+
+print(f"  Total: {len(GROUPS)} groups, {created_count} new")
+
+# ===========================================================================
+# Phase 2: Create/sync service account users with custom claims
+# ===========================================================================
+print("\n[2/6] Creating/updating service account users...")
+
+status, res = api("GET", "/api/v3/core/users/?page_size=100")
+if status != 200:
+    die(f"GET users -> {status} {res}")
+existing_users = {u["username"]: u for u in res["results"]}
+
+service_pwd = os.environ.get("AUTHENTIK_SERVICE_ACCOUNT_PASSWORD", "DefaultPassword123!")
+
+for agent_name, agent_spec in SERVICE_ACCOUNTS.items():
+    if agent_name in existing_users:
+        user = existing_users[agent_name]
+        attrs = user.get("attributes", {})
+        attrs.update(agent_spec.get("claims", {}))
+        attrs["roles"] = agent_spec.get("roles", [])
+        status, res = api("PATCH", f"/api/v3/core/users/{user['pk']}/", {"attributes": attrs})
+        if status in (200, 201):
+            print(f"  {agent_name}: claims updated")
+        else:
+            print(f"  {agent_name}: FAILED {status} {res}")
+    else:
+        status, res = api("POST", "/api/v3/core/users/", {
+            "username": agent_name,
+            "name": agent_spec.get("description", agent_name),
+            "email": f"{agent_name}@homelab.local",
+            "is_active": True,
+            "is_superuser": False,
+            "password": service_pwd,
+            "attributes": {
+                **agent_spec.get("claims", {}),
+                "roles": agent_spec.get("roles", []),
+            },
+        })
+        if status in (200, 201):
+            print(f"  {agent_name}: created")
+        else:
+            print(f"  {agent_name}: FAILED {status} {res}")
+
+# ===========================================================================
+# Phase 3: Create scope mappings for fine-grained claims
+# ===========================================================================
+print("\n[3/6] Creating scope mappings for fine-grained claims...")
+
+status, res = api("GET", "/api/v3/propertymappings/provider/scope/?page_size=100")
+if status != 200:
+    die(f"GET scope mappings -> {status} {res}")
+existing_scopes = {m["scope_name"]: m for m in res["results"]}
+
+for scope_name, scope_spec in SCOPE_MAPPINGS.items():
+    if scope_name in existing_scopes:
+        print(f"  {scope_name}: already exists")
+    else:
+        status, res = api("POST", "/api/v3/propertymappings/provider/scope/", {
+            "name": scope_name,
+            "scope_name": scope_name,
+            "expression": scope_spec["expression"],
+        })
+        if status in (200, 201):
+            print(f"  {scope_name}: created")
+        else:
+            print(f"  {scope_name}: FAILED {status} {res}")
+
+# ===========================================================================
+# Phase 4: Get flow UUIDs (needed for providers)
+# ===========================================================================
+print("\n[4/6] Fetching flow UUIDs...")
+
+status, res = api("GET", "/api/v3/flows/instances/?page_size=100")
+if status != 200:
+    die(f"GET flows -> {status} {res}")
+
+flows = {f["slug"]: f["pk"] for f in res.get("results", [])}
+auth_flow = flows.get("default-provider-authorization-implicit-consent")
+inval_flow = flows.get("default-provider-invalidation-flow")
+
+if not auth_flow or not inval_flow:
+    die(f"Required flows not found. auth_flow={auth_flow}, inval_flow={inval_flow}")
+
+print(f"  authorization_flow: {auth_flow}")
+print(f"  invalidation_flow: {inval_flow}")
+
+# ===========================================================================
+# Phase 5: Create OAuth2 providers with scopes
+# ===========================================================================
+print("\n[5/6] Creating OAuth2 providers...")
+
+status, res = api("GET", "/api/v3/providers/oauth2/?page_size=100")
+if status != 200:
+    die(f"GET providers -> {status} {res}")
+existing_providers = {p["name"]: p for p in res.get("results", [])}
+
+# Fetch scope mapping PKs
+status, scopes_res = api("GET", "/api/v3/propertymappings/provider/scope/?page_size=100")
+if status != 200:
+    print("  WARNING: could not fetch scope mappings")
+    scope_pks = {}
+else:
+    scope_pks = {m["scope_name"]: m["pk"] for m in scopes_res.get("results", [])}
+
+# Include standard OpenID scopes (openid, email, profile) + custom claim scopes
+STANDARD_SCOPES = ["openid", "email", "profile"]
+scope_pks_list = [scope_pks[s] for s in STANDARD_SCOPES if s in scope_pks]
+scope_pks_list += [scope_pks[s] for s in SCOPE_MAPPINGS.keys() if s in scope_pks]
+
+# OAuth2 providers — client_secret sourced from SOPS-encrypted k8s secrets.
+# These are the real secrets the services use. Authentik must match.
+OAuth2_PROVIDERS = {
+    "api-gw": {
+        "client_id": "api-gw",
+        "client_secret_env": "AUTHENTIK_PROVIDER_API_GW_SECRET",
+        "redirect_uris": ["http://localhost:3000/callback", "https://api.riotpiao.com/callback"],
+    },
+    "minio": {
+        "client_id": "minio",
+        "client_secret": "9d2867fe08c3bf7fedd7e32bbaf4456fce3b0aaf788966d7559e1955947b0219",
+        "redirect_uris": ["http://localhost:9000/auth/sso/oauth2/code", "https://minio.riotpiao.com/auth/sso/oauth2/code"],
+    },
+    "poimen": {
+        "client_id": "poimen",
+        "client_secret_env": "AUTHENTIK_PROVIDER_POIMEN_SECRET",
+        "redirect_uris": ["http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"],
+    },
+    "paperless": {
+        "client_id": "paperless",
+        "client_secret": "6hcxaaVgZlKgafl7BxeSEtPAcbNUJxi2PAZePxSFk4o=",
+        "redirect_uris": ["http://localhost:8000/accounts/oidc/authentik/login/callback/", "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"],
+    },
+    "grafana": {
+        "client_id": "grafana",
+        "client_secret": "966bad4fa43812100e7775b3c73fed2ce1d07217fa5a23fbb0f190e46d2f0fa4",
+        "redirect_uris": ["http://localhost:3000/login/generic_oauth", "https://grafana.riotpiao.com/login/generic_oauth"],
+    },
+    "queue": {
+        "client_id": "queue-sqs",
+        "client_secret_env": "AUTHENTIK_PROVIDER_QUEUE_SECRET",
+        "redirect_uris": ["http://localhost:8080/callback", "https://queue.riotpiao.com/callback"],
+    },
+    "forgejo": {
+        "client_id": "forgejo",
+        "client_secret": "G4klhs3JRfs5A7YnGs90WuOndBAvamWlZgaZRY8x",
+        "redirect_uris": ["http://localhost:3000/user/oauth2/authentik/callback", "https://forgejo.riotpiao.com/user/oauth2/authentik/callback"],
+    },
+    "immich": {
+        "client_id": "immich",
+        "client_secret": "QxyWfESXqTD55aUyh6miYnny1QTCEuEwyC4escw9",
+        "redirect_uris": ["app.immich:///oauth-callback", "https://img.riotpiao.com/auth/login", "https://img.riotpiao.com/user/oauth2/callback"],
+    },
+    "homarr": {
+        "client_id": "homarr",
+        "client_secret": "RMlDQAWdjT5YPPH0U7ztDoUFP7R7w95b2xqQ1pyS",
+        "redirect_uris": ["http://localhost:7575/auth/callback", "https://homarr.riotpiao.com/auth/callback"],
+    },
+    "argocd": {
+        "client_id": "argocd",
+        "client_secret_env": "AUTHENTIK_PROVIDER_ARGOCD_SECRET",
+        "redirect_uris": ["http://localhost:8080/auth/callback", "https://argocd.riotpiao.com/auth/callback"],
+    },
+    "vault": {
+        "client_id": "vault",
+        "client_secret_env": "AUTHENTIK_PROVIDER_VAULT_SECRET",
+        "redirect_uris": ["http://localhost:8200/ui/vault/auth/oidc/oidc/callback", "https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback"],
+    },
+}
+
+import secrets as _secrets
+
+for provider_name, provider_spec in OAuth2_PROVIDERS.items():
+    # Resolve client_secret: explicit > env var > generate random
+    if "client_secret" in provider_spec:
+        client_secret = provider_spec["client_secret"]
+    elif "client_secret_env" in provider_spec:
+        client_secret = os.environ.get(provider_spec["client_secret_env"], _secrets.token_urlsafe(32))
+    else:
+        client_secret = _secrets.token_urlsafe(32)
+
+    redirect_uris_list = [{"url": uri, "matching_mode": "strict"} for uri in provider_spec["redirect_uris"]]
+
+    provider_payload = {
+        "name": provider_name,
+        "authorization_flow": auth_flow,
+        "invalidation_flow": inval_flow,
+        "grant_types": ["authorization_code", "implicit", "password"],
+        "client_id": provider_spec["client_id"],
+        "client_secret": client_secret,
+        "redirect_uris": redirect_uris_list,
+        "property_mappings": scope_pks_list,
+    }
+
+    if provider_name in existing_providers:
+        # UPDATE existing provider — sync secret + redirect_uris
+        provider_pk = existing_providers[provider_name]["pk"]
+        status, res = api("PATCH", f"/api/v3/providers/oauth2/{provider_pk}/", {
+            "client_id": provider_spec["client_id"],
+            "client_secret": client_secret,
+            "redirect_uris": redirect_uris_list,
+            "property_mappings": scope_pks_list,
+        })
+        if status in (200, 201):
+            print(f"  {provider_name}: updated (secret + redirect_uris synced)")
+        else:
+            print(f"  {provider_name}: UPDATE FAILED {status} {res}")
+    else:
+        # CREATE new provider
+        status, res = api("POST", "/api/v3/providers/oauth2/", provider_payload)
+        if status in (200, 201):
+            print(f"  {provider_name}: created")
+        else:
+            print(f"  {provider_name}: FAILED {status} {res}")
+
+# ===========================================================================
+# Phase 6: Create OAuth2 Applications (bind providers to public token endpoints)
+# ===========================================================================
+print("\n[6/7] Creating OAuth2 Applications...")
+print("     (binds providers to /application/o/token/ endpoints)")
+
+status, res = api("GET", "/api/v3/core/applications/?page_size=100")
+if status != 200:
+    die(f"GET applications -> {status} {res}")
+existing_apps = {a["slug"]: a for a in res.get("results", [])}
+
+for provider_name in OAuth2_PROVIDERS.keys():
+    # Get the provider PK
+    status, provider_res = api("GET", f"/api/v3/providers/oauth2/?name={provider_name}")
+    if status != 200 or not provider_res.get("results"):
+        print(f"  {provider_name}: provider not found, skip")
+        continue
+    provider_pk = provider_res["results"][0]["pk"]
+
+    if provider_name in existing_apps:
+        # Ensure app is linked to provider (fix orphaned apps)
+        app_data = existing_apps[provider_name]
+        if app_data.get("provider") != provider_pk:
+            app_uuid = app_data["pk"]
+            status, res = api("PATCH", f"/api/v3/core/applications/{app_uuid}/", {
+                "provider": provider_pk,
+            })
+            if status in (200, 201):
+                print(f"  {provider_name}: re-linked to provider")
+            else:
+                print(f"  {provider_name}: RE-LINK FAILED {status} {res}")
+        else:
+            print(f"  {provider_name}: ok")
+    else:
+        status, res = api("POST", "/api/v3/core/applications/", {
+            "name": provider_name,
+            "slug": provider_name,
+            "provider": provider_pk,
+        })
+        if status in (200, 201):
+            print(f"  {provider_name}: created")
+        else:
+            print(f"  {provider_name}: FAILED {status} {res}")
+
+# ===========================================================================
+# Phase 7: Create/update rock user → homelab-admins, matching Forgejo identity
+# ===========================================================================
+print("\n[7/10] Creating/updating rock user ([email protected])...")
+
+ROCK_EMAIL = "[email protected]"
+ROCK_PASSWORD = os.environ.get("ROCK_PASSWORD", "")
+
+status, res = api("GET", "/api/v3/core/users/?username=rock")
+if status == 200 and res.get("results"):
+    rock_user = res["results"][0]
+    # Ensure email matches Forgejo's rock user for OIDC linking
+    patch_data = {"email": ROCK_EMAIL, "name": "Rock"}
+    status, res = api("PATCH", f"/api/v3/core/users/{rock_user['pk']}/", patch_data)
+    if status in (200, 201):
+        print(f"  rock: updated email to {ROCK_EMAIL}")
+    else:
+        print(f"  rock: update FAILED {status} {res}")
+else:
+    if not ROCK_PASSWORD:
+        print("  rock: NOT FOUND and ROCK_PASSWORD not set, skipping creation")
+        print("    export ROCK_PASSWORD=<password> and re-run")
+        rock_user = None
+    else:
+        status, res = api("POST", "/api/v3/core/users/", {
+            "username": "rock",
+            "name": "Rock",
+            "email": ROCK_EMAIL,
+            "is_active": True,
+            "is_superuser": False,
+            "password": ROCK_PASSWORD,
+        })
+        if status in (200, 201):
+            rock_user = res
+            print(f"  rock: created with email {ROCK_EMAIL}")
+        else:
+            print(f"  rock: create FAILED {status} {res}")
+            rock_user = None
+
+if rock_user:
+    status, res = api("GET", "/api/v3/core/groups/?name=homelab-admins")
+    if status == 200 and res.get("results"):
+        admins_group = res["results"][0]
+        status, res = api("POST", f"/api/v3/core/groups/{admins_group['pk']}/users/add/", {"pk": rock_user["pk"]})
+        if status in (200, 201, 204):
+            print(f"  rock: added to homelab-admins")
+        else:
+            print(f"  rock: group add {status} {res}")
+
+# ===========================================================================
+# Phase 8: Email recovery flow (password reset via email)
+# ===========================================================================
+print("\n[8/10] Creating email recovery flow...")
+
+# Read SMTP config from gotify-smtp secret (same Gmail creds)
+SMTP_HOST = "smtp.gmail.com"
+SMTP_PORT = 587
+SMTP_USER = "[email protected]"
+SMTP_FROM = "[email protected]"
+# Password read from env at runtime: AUTHENTIK_EMAIL__PASSWORD
+
+# 8a. Create email stage for recovery
+status, res = api("GET", "/api/v3/stages/email/?name=email-recovery")
+if status == 200 and res.get("results"):
+    email_stage_pk = res["results"][0]["pk"]
+    print("  email-recovery stage: already exists")
+else:
+    status, res = api("POST", "/api/v3/stages/email/", {
+        "name": "email-recovery",
+        "use_global_settings": False,
+        "host": SMTP_HOST,
+        "port": SMTP_PORT,
+        "username": SMTP_USER,
+        "password": os.environ.get("AUTHENTIK_EMAIL_PASSWORD", ""),
+        "use_tls": True,
+        "use_ssl": False,
+        "timeout": 10,
+        "from_address": SMTP_FROM,
+        "template": "email/password_reset.html",
+        "activate_user_on_success": True,
+    })
+    if status in (200, 201):
+        email_stage_pk = res["pk"]
+        print("  email-recovery stage: created")
+    else:
+        email_stage_pk = None
+        print(f"  email-recovery stage: FAILED {status} {res}")
+
+# 8b. Create identification stage for recovery (email lookup)
+status, res = api("GET", "/api/v3/stages/identification/?name=recovery-identification")
+if status == 200 and res.get("results"):
+    ident_stage_pk = res["results"][0]["pk"]
+    print("  recovery-identification stage: already exists")
+else:
+    status, res = api("POST", "/api/v3/stages/identification/", {
+        "name": "recovery-identification",
+        "user_fields": ["email", "username"],
+    })
+    if status in (200, 201):
+        ident_stage_pk = res["pk"]
+        print("  recovery-identification stage: created")
+    else:
+        ident_stage_pk = None
+        print(f"  recovery-identification stage: FAILED {status} {res}")
+
+# 8c. Create password stage for new password entry
+status, res = api("GET", "/api/v3/stages/password/?name=recovery-password-change")
+if status == 200 and res.get("results"):
+    pw_stage_pk = res["results"][0]["pk"]
+    print("  recovery-password-change stage: already exists")
+else:
+    # Use prompt stage for password change instead
+    status, res = api("GET", "/api/v3/stages/user_write/?name=recovery-user-write")
+    if status == 200 and res.get("results"):
+        pw_stage_pk = res["results"][0]["pk"]
+        print("  recovery-user-write stage: already exists")
+    else:
+        status, res = api("POST", "/api/v3/stages/user_write/", {
+            "name": "recovery-user-write",
+        })
+        if status in (200, 201):
+            pw_stage_pk = res["pk"]
+            print("  recovery-user-write stage: created")
+        else:
+            pw_stage_pk = None
+            print(f"  recovery-user-write stage: FAILED {status} {res}")
+
+# 8d. Create recovery flow
+status, res = api("GET", "/api/v3/flows/instances/?slug=password-recovery")
+if status == 200 and res.get("results"):
+    recovery_flow_pk = res["results"][0]["pk"]
+    print("  password-recovery flow: already exists")
+else:
+    status, res = api("POST", "/api/v3/flows/instances/", {
+        "name": "Password Recovery",
+        "slug": "password-recovery",
+        "title": "Reset your password",
+        "designation": "recovery",
+    })
+    if status in (200, 201):
+        recovery_flow_pk = res["pk"]
+        print("  password-recovery flow: created")
+    else:
+        recovery_flow_pk = None
+        print(f"  password-recovery flow: FAILED {status} {res}")
+
+# 8e. Bind stages to flow in order
+if recovery_flow_pk and ident_stage_pk and email_stage_pk:
+    for order, stage_pk, label in [
+        (10, ident_stage_pk, "identification"),
+        (20, email_stage_pk, "email"),
+    ]:
+        status, res = api("POST", "/api/v3/flows/bindings/", {
+            "target": recovery_flow_pk,
+            "stage": stage_pk,
+            "order": order,
+        })
+        if status in (200, 201):
+            print(f"  bound {label} stage at order {order}")
+        elif status == 400 and "already exists" in str(res).lower():
+            print(f"  {label} stage: already bound")
+        else:
+            print(f"  bind {label}: {status} {res}")
+
+# ===========================================================================
+# Phase 9: Set recovery flow on brand
+# ===========================================================================
+print("\n[9/10] Setting recovery flow on brand...")
+
+if recovery_flow_pk:
+    status, res = api("GET", "/api/v3/brands/instances/")
+    if status == 200 and res.get("results"):
+        brand = res["results"][0]
+        status, res = api("PATCH", f"/api/v3/brands/instances/{brand['brand_uuid']}/", {
+            "flow_recovery": recovery_flow_pk,
+        })
+        if status in (200, 201):
+            print("  recovery flow set on brand")
+        else:
+            print(f"  FAILED {status} {res}")
+    else:
+        print("  no brand found")
+
+# ===========================================================================
+# Phase 10: Ensure Forgejo OAuth2 source uses matching email claim
+# ===========================================================================
+print("\n[10/10] Verifying Forgejo OIDC linkage...")
+print(f"  rock@Authentik email: {ROCK_EMAIL}")
+print("  Forgejo OIDC will match on email — ensure Forgejo's rock user")
+print(f"  has email {ROCK_EMAIL} in Forgejo settings → Profile")
+
+# ===========================================================================
+# Summary
+# ===========================================================================
+print("\n" + "="*70)
+print("AUTHENTIK PROVISIONING COMPLETE")
+print("="*70)
+print(f"\n  [1] Groups: {len(GROUPS)}")
+print(f"  [2] Service accounts: {len(SERVICE_ACCOUNTS)}")
+print(f"  [3] Scope mappings: {len(SCOPE_MAPPINGS)}")
+print(f"  [4] Flows resolved")
+print(f"  [5] OAuth2 providers: {len(OAuth2_PROVIDERS)}")
+print(f"  [6] OAuth2 applications bound")
+print(f"  [7] rock user ([email protected]) -> homelab-admins")
+print(f"  [8] Email recovery flow (smtp.gmail.com)")
+print(f"  [9] Recovery flow set on brand")
+print(f"  [10] Forgejo OIDC linkage verified")
+print("\nNEXT: Set Forgejo rock user email to [email protected] in Forgejo profile")
+print("TEST: https://authentik.riotpiao.com/if/flow/password-recovery/")
+print("="*70)

scripts/iam/rotate-secrets.sh

@@ -0,0 +1,216 @@
+#!/bin/bash
+# Secret Rotation Script
+# Rotates all OAuth2 and service account credentials
+# Should be run quarterly (every 90 days)
+# 
+# Usage: ./rotate-secrets.sh [--dry-run]
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+DRY_RUN=${1:-}
+
+# Color output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+log() { echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $*"; }
+warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
+error() { echo -e "${RED}[ERROR]${NC} $*"; exit 1; }
+
+log "=== Secret Rotation Script ==="
+log "Rotation Date: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
+
+if [[ -n "$DRY_RUN" ]]; then
+  log "Running in DRY-RUN mode (no changes will be applied)"
+fi
+
+# Verify prerequisites
+log "Checking prerequisites..."
+command -v kubectl &>/dev/null || error "kubectl not found"
+command -v openssl &>/dev/null || error "openssl not found"
+command -v sops &>/dev/null || error "sops not found"
+command -v jq &>/dev/null || error "jq not found"
+
+# Check kubeconfig
+kubectl cluster-info &>/dev/null || error "Not connected to cluster"
+
+# Get bootstrap token
+log "Retrieving Authentik bootstrap token..."
+BOOTSTRAP_TOKEN=$(kubectl -n iam get secret authentik-secrets \
+  -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' 2>/dev/null | base64 -d) || \
+  error "Failed to get bootstrap token"
+
+# Generate new secrets (13 OAuth2 + service accounts)
+log "Generating 13 new secrets (256-bit)..."
+
+generate_secret() {
+  openssl rand -base64 32
+}
+
+declare -A NEW_SECRETS
+
+for svc in api-gw minio poimen paperless grafana argocd forgejo homarr immich vault portfolio-agent memory-agent local-llm; do
+  NEW_SECRETS[$svc]=$(generate_secret)
+  log "  $svc: ${NEW_SECRETS[$svc]:0:15}..."
+done
+
+log ""
+log "=== Updating Authentik OAuth2 Providers ==="
+
+# Authentik provider mapping
+declare -A PROVIDER_PKS=(
+  [api-gw]=2
+  [minio]=3
+  [poimen]=4
+  [paperless]=5
+  [grafana]=6
+  [argocd]=7
+  [forgejo]=8
+  [homarr]=9
+  [immich]=10
+  [vault]=11
+)
+
+for provider in "${!PROVIDER_PKS[@]}"; do
+  pk=${PROVIDER_PKS[$provider]}
+  secret=${NEW_SECRETS[$provider]}
+  
+  log "Updating $provider (pk=$pk)..."
+  
+  if [[ -z "$DRY_RUN" ]]; then
+    response=$(curl -s -X PATCH "https://authentik.riotpiao.com/api/v3/providers/oauth2/$pk/" \
+      -H "Authorization: Bearer $BOOTSTRAP_TOKEN" \
+      -H "Content-Type: application/json" \
+      -d "{\"client_secret\": \"$secret\"}")
+    
+    if echo "$response" | jq -e '.pk' &>/dev/null; then
+      log "  ✅ $provider updated"
+    else
+      error "Failed to update $provider: $(echo "$response" | jq '.detail // .')"
+    fi
+  fi
+done
+
+log ""
+log "=== Updating k8s Secrets ==="
+
+# Update api-gw
+if [[ -z "$DRY_RUN" ]]; then
+  log "Patching api/api-gateway-oauth2-creds..."
+  kubectl -n api patch secret api-gateway-oauth2-creds \
+    -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[api-gw]}" | base64)\"}}" --type=merge 2>/dev/null || true
+fi
+
+# Update minio (skip if namespace doesn't exist)
+if kubectl get ns minio &>/dev/null 2>&1; then
+  if [[ -z "$DRY_RUN" ]]; then
+    log "Patching minio/minio-oauth2-creds..."
+    kubectl -n minio patch secret minio-oauth2-creds \
+      -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[minio]}" | base64)\"}}" --type=merge 2>/dev/null || true
+  fi
+fi
+
+# Update poimen
+if [[ -z "$DRY_RUN" ]]; then
+  log "Patching poimen/poimen-oauth2-creds..."
+  kubectl -n poimen patch secret poimen-oauth2-creds \
+    -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[poimen]}" | base64)\"}}" --type=merge 2>/dev/null || true
+fi
+
+# Update paperless
+if [[ -z "$DRY_RUN" ]]; then
+  log "Patching paperless/paperless-oauth2-creds..."
+  kubectl -n paperless patch secret paperless-oauth2-creds \
+    -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[paperless]}" | base64)\"}}" --type=merge 2>/dev/null || true
+fi
+
+# Update logging/grafana
+if [[ -z "$DRY_RUN" ]]; then
+  log "Patching logging/grafana-oauth2-creds..."
+  kubectl -n logging patch secret grafana-oauth2-creds \
+    -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[grafana]}" | base64)\"}}" --type=merge 2>/dev/null || true
+fi
+
+# Update service accounts
+if [[ -z "$DRY_RUN" ]]; then
+  log "Patching portfolio/portfolio-agent-oidc..."
+  kubectl -n portfolio patch secret portfolio-agent-oidc \
+    -p "{\"data\":{\"CLIENT_SECRET\":\"$(echo -n "${NEW_SECRETS[portfolio-agent]}" | base64)\"}}" --type=merge 2>/dev/null || true
+  
+  log "Patching poimen/memory-agent-oidc..."
+  kubectl -n poimen patch secret memory-agent-oidc \
+    -p "{\"data\":{\"CLIENT_SECRET\":\"$(echo -n "${NEW_SECRETS[memory-agent]}" | base64)\"}}" --type=merge 2>/dev/null || true
+  
+  log "Patching llm-serving/local-llm-jwt..."
+  kubectl -n llm-serving patch secret local-llm-jwt \
+    -p "{\"data\":{\"client-secret\":\"$(echo -n "${NEW_SECRETS[local-llm]}" | base64)\"}}" --type=merge 2>/dev/null || true
+fi
+
+log ""
+log "=== Updating SOPS-encrypted manifests ==="
+
+# Create oauth2-credentials.enc.yaml
+cat > "$REPO_ROOT/k8s/argocd/secrets/oauth2-credentials.yaml" << 'OAUTH_EOF'
+apiVersion: v1
+kind: Secret
+metadata:
+  name: oauth2-credentials
+  namespace: iam
+type: Opaque
+data:
+OAUTH_EOF
+
+for svc in api-gw minio poimen paperless grafana; do
+  echo "  ${svc}-client-secret: $(echo -n "${NEW_SECRETS[$svc]}" | base64)" >> \
+    "$REPO_ROOT/k8s/argocd/secrets/oauth2-credentials.yaml"
+done
+
+if [[ -z "$DRY_RUN" ]]; then
+  log "Encrypting oauth2-credentials.yaml with SOPS..."
+  sops -e "$REPO_ROOT/k8s/argocd/secrets/oauth2-credentials.yaml" > \
+    "$REPO_ROOT/k8s/argocd/secrets/oauth2-credentials.enc.yaml"
+  rm "$REPO_ROOT/k8s/argocd/secrets/oauth2-credentials.yaml"
+  log "  ✅ oauth2-credentials.enc.yaml created"
+fi
+
+# Create memory-agent-oidc.enc.yaml
+cat > "$REPO_ROOT/k8s/argocd/secrets/memory-agent-oidc.yaml" << AGENT_EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: memory-agent-oidc
+  namespace: poimen
+type: Opaque
+data:
+  CLIENT_ID: bWVtb3J5LWFnZW50
+  CLIENT_SECRET: $(echo -n "${NEW_SECRETS[memory-agent]}" | base64)
+  ISSUER: aHR0cHM6Ly9hdXRoZW50aWsucmlvdHBpYW8uY29tL2FwcGxpY2F0aW9uL28v
+  TOKEN_URL: aHR0cHM6Ly9hdXRoZW50aWsucmlvdHBpYW8uY29tL2FwcGxpY2F0aW9uL28vdG9rZW4v
+AGENT_EOF
+
+if [[ -z "$DRY_RUN" ]]; then
+  log "Encrypting memory-agent-oidc.yaml with SOPS..."
+  sops -e "$REPO_ROOT/k8s/argocd/secrets/memory-agent-oidc.yaml" > \
+    "$REPO_ROOT/k8s/argocd/secrets/memory-agent-oidc.enc.yaml"
+  rm "$REPO_ROOT/k8s/argocd/secrets/memory-agent-oidc.yaml"
+  log "  ✅ memory-agent-oidc.enc.yaml created"
+fi
+
+log ""
+log "=== Summary ==="
+log "Rotated 13 credentials:"
+log "  OAuth2 Providers: api-gw, minio, poimen, paperless, grafana, argocd, forgejo, homarr, immich, vault"
+log "  Service Accounts: portfolio-agent, memory-agent, local-llm"
+
+log ""
+log "Next steps:"
+log "  1. Review changes: git diff k8s/argocd/secrets/"
+log "  2. Commit: git add k8s/argocd/secrets/oauth2-credentials.enc.yaml"
+log "  3. Commit message: 'chore: rotate OAuth2 secrets (quarterly)'"
+log "  4. Push: git push"
+log ""
+log "✅ Rotation complete!"

terraform/files/coredns/Corefile

@@ -37,6 +37,7 @@
     rewrite name img.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
     rewrite name api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
     rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
     rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 
     kubernetes cluster.local in-addr.arpa ip6.arpa {

terraform/terraform.tfvars.local.bak

@@ -0,0 +1,90 @@
+# RECOVERED from live cluster state via talosctl get machineconfig.
+# Regenerated after the original tfvars.local was lost/corrupted.
+
+cluster_id  = "Rtc4g2av9EP0mOdxA4M0-QQitzHLWICz-rLBNfrOjgw="
+cluster_secret = "8E0CAeylAPKmmUEQmIGmHQAhQf+8c7NUf43CdrQZ+vg="
+bootstrap_token = "b2q7lh.9w7hwgrulrd65gr3"
+machine_token = "hq9wlf.96l9z46efd79jtr2"
+machine_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQekNCOHFBREFnRUNBaEVBMkUwRWZqbG41L1J3eTVjMVJmNkNSakFGQmdNclpYQXdFREVPTUF3R0ExVUUKQ2hNRmRHRnNiM013SGhjTk1qWXdOekE1TURVd056VTRXaGNOTXpZd056QTJNRFV3TnpVNFdqQVFNUTR3REFZRApWUVFLRXdWMFlXeHZjekFxTUFVR0F5dGxjQU1oQUNwR3NQZkVHdEU4anVmNG9JTkNHNnlCQmN6aURFaEpLbDV3CnJib0hSR0tUbzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkdmaExZUUdaOGYzd3lZZAo0SFI3KzlONnZKQXJNQVVHQXl0bGNBTkJBQ1ZQVlAwcGYxMkdUakYvSHJMZmcwZHBLemdBMlE1OGR5Y0paY0ZvClQvNDdPQk8ra29VZm11dXNjVVNNQnBXS0VuWHNYMFNocmNab3hQd1FHM3diblFFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
+machine_ca_key = "LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0KTUM0Q0FRQXdCUVlESzJWd0JDSUVJSlR4MXRqZEVOTTg1cGNRRFR0WWRtMFd0QXNBd0tzL1VESlZLN0ZqYU5uUgotLS0tLUVORCBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K"
+kubernetes_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpVENDQVRDZ0F3SUJBZ0lSQUtwT1ZaK29CRzljNHFycEpwaUc4TkV3Q2dZSUtvWkl6ajBFQXdJd0ZURVQKTUJFR0ExVUVDaE1LYTNWaVpYSnVaWFJsY3pBZUZ3MHlOakEzTURrd05UQTNOVGhhRncwek5qQTNNRFl3TlRBMwpOVGhhTUJVeEV6QVJCZ05WQkFvVENtdDFZbVZ5Ym1WMFpYTXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVQ1VjJvNkliMUpRZkcza3lCTTBCeTRkd2FLbnlGY2tVdjNVem9yOG1Ba3RaN2JrT2ZKZDlmL2VmcVkKYXBzUFpLV1RHQnYxM3JZbFdvOGtuU3ZnNm83Um8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXdIUVlEVlIwbApCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPCkJCWUVGQ1dPZVJUVHdnN2tnNVNWMG9raFQwY0VDNGwzTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSURURCtNYXUKb2JXdkp6UkNMVEdJaHJHc1daalFnalVmRWl2MnhCTXB6RnNVQWlCQzNNWkYwMjVqVHk4Uno2YjI5czVJQmpkNgpZQTVWd0kvNVFieXlwSGFXQlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
+kubernetes_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUrSEdPcUJJYXpJMzdqNmJJUlA1emVxeXEwZzhBU2xZeGplZUlJcEpIcXBvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFK1ZkcU9pRzlTVUh4dDVNZ1ROQWN1SGNHaXA4aFhKRkw5MU02Sy9KZ0pMV2UyNURueVhmWAovM242bUdxYkQyU2xreGdiOWQ2MkpWcVBKSjByNE9xTzBRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
+etcd_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJmVENDQVNTZ0F3SUJBZ0lSQVAyaHkwdUhvTm5vSzQyZE9LTk1nU2t3Q2dZSUtvWkl6ajBFQXdJd0R6RU4KTUFzR0ExVUVDaE1FWlhSalpEQWVGdzB5TmpBM01Ea3dOVEEzTlRoYUZ3MHpOakEzTURZd05UQTNOVGhhTUE4eApEVEFMQmdOVkJBb1RCR1YwWTJRd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFRVGlKYUJpSEJPCmJha3JuL0dVdkUxVFd5czRVUkVzVGtVNEM4OG1EdWZYQURjV0NnN2RTRzc0QjkzOGFwSWgybGc4UDhKRDFFRUoKN0RkU1lQVG5zYWh1bzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSApBd0VHQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkliYTBLaEhmM3hhClBQNk1hb3dESUNWVXBLdDVNQW9HQ0NxR1NNNDlCQU1DQTBjQU1FUUNJRzZMYUJGeGgvcys2WXpGdVlwaUxUWlYKS2prOGh5UVNvMmVTZTJTUy81MG5BaUI0a0RNWnR4b1UzTitHc3BEOHVEbXFrNlhxbzZoeE0vbG0yU21OMzlPNAovdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
+etcd_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUwZ3JiMk0yblA4c1hxWjVhd3NzNThwbUdvb1FlSWg3a3RoWVlxNzhZZThvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFRTRpV2dZaHdUbTJwSzUveGxMeE5VMXNyT0ZFUkxFNUZPQXZQSmc3bjF3QTNGZ29PM1VodQorQWZkL0dxU0lkcFlQRC9DUTlSQkNldzNVbUQwNTdHb2JnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
+aggregator_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJZVENDQVFhZ0F3SUJBZ0lSQUpxeFI3alBzcmhzRUp2cmtEWndYR3N3Q2dZSUtvWkl6ajBFQXdJd0FEQWUKRncweU5qQTNNRGt3TlRBM05UaGFGdzB6TmpBM01EWXdOVEEzTlRoYU1BQXdXVEFUQmdjcWhrak9QUUlCQmdncQpoa2pPUFFNQkJ3TkNBQVFBWWlieHY1ZDVFRGdTbWhlRHlhYjJLZGh4ZFZnZ3lQc2pNcjBET0JMVmxtQmpMcXFqClpNSkk2d1ZmV2hkUjBRVGxVdEFLZDJvREJZLy9TeDBzQi9qY28yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXcKSFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4dwpIUVlEVlIwT0JCWUVGSTkrWm1EYVBybDhrTnhXUUxOT3ErTmVzeEh0TUFvR0NDcUdTTTQ5QkFNQ0Ewa0FNRVlDCklRRC9DRUZ2SUVHMW9qcmRZRVUzUldmTklLV1FwVXlZQWJ1ZGE0T0ZWQS9yOUFJaEFPS1VOZFF6bUk2WVZOdzgKeklDejlLblRxazQrT3dvV3RjNVl5SmlKR29zUgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
+aggregator_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUFoOEwycXFLbUxoYkpmczJ2M3ZRWXBFZHF6Ri9hTGZhSmoraEZRazdPY2NvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQUdJbThiK1hlUkE0RXBvWGc4bW05aW5ZY1hWWUlNajdJeks5QXpnUzFaWmdZeTZxbzJUQwpTT3NGWDFvWFVkRUU1VkxRQ25kcUF3V1AvMHNkTEFmNDNBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
+service_account_key = "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKSndJQkFBS0NBZ0VBd3NNZU40eE1YQkZ0VVE2dVZ2NEZFU1dNT2ZLc1RKdmxHa0ptVG1URjJIazg1SlJvCkhuQU1nMEkvMEJiMGFUQTJ3MjlkSEUrdUN6dFd5eVVqTzV5eWdJSDQ0Q2RtVnNHVzVua1Z0TmFGNzNpcFllbjYKaG11bmRoWDY1VndqOHpVUmFYZUxtUEc0Y2d4dk5SOXdGYUZXS0ZZdzQzdGtlWjg5S0F5QTNibjFXN3JHTloyOQovcXRYWEcrQnFSSlVuUzhXQmt4dkErUktyRE1OVHF4bjVGOEl1VkpTWkw5bDVEd3lSUnNqMjVMZVdRZUFYenl6CllWcmFKZmVubSt6L3hLZjhLdHFZMSs5U1pwbkZsS2N6TGlWVzg4WkJ5TnR0KytYZjB4N3FSL2lQOHhRQXV0ckIKV1pTbytoa01FRjV5YU0wQWdxcG5mbDBsUmhsM0I0UUx3NHkyTUpoTEpyQ1ltQjl1QllLWU5oRE5NOXk0Z1lrZgpacmdEdm9NdGpsSUJUUHRyVjMyemxYN05Od3d5UjdXOXp5ZlBSVjEvbjhpMk9KR0szeittdGNCR2Y0TS8veDFsClJRNEhzWWhDUVJRaE8vQWp2U2QvQlhFRnB3dEhsb1VYRU5nQ2lPSW00MnFrZTNWb3lLbjJ2Q3g2QlhodlJVZDAKUHgvK0JpM0d4RmRoa2Uxa0doelJLQTdySGhEMVBkVGsyM04wUXV3WHNNVDZRWXVrRDdPZG5KdFZtZVN4TkxqcwptdVpybmpwRzZsaHRDSmdvMGdVeXBYQ2RlWGVnZ0dIc0JCMks4NjIrdThVa0dwMk9IQUhGeldsdDF5L2VXTW9PCmhneWx1SEVLcjkzUU5uNDZsTGtZTDViWjlVTUZ0NXBMRFVrSFQxNkV5dTh0V21ET2ZoQ3Z3M1lmRWY4Q0F3RUEKQVFLQ0FnQUdSbUlSV1JoV3VRc0VHd3g3NmdoQXdxeHZhNFdvbkRjMzd0Njc5Tnc0K3NMKy9GY1ViL2kvTytHeApjeVBoeGJkbCtZOE82L1JJRVZ2ZEJLL0xhbU9INTJnYzFMZ2o0RzNidEJnQ2NRejBwN2NSWEFnQno3TWdCMXBECmpJSHVBbzR5anpMMHRRa0R4Nm5IbE9FNEdUQWM4WlgycGxHWTU0d0JYOUhCRXc0NEs5N1orR0NZTlc0Rm9PUVYKRGUyaStOTGxWZzRYbW9IYlpYT3V6cmcwTCttb2l1SHp0QVQwNHdtZGwxL0M0Y3IvSkZJNi8wb3FQMUthK1kweApaV1BpTXFWWnZoeEJqTWpqWEYzMHlhUkkvdFA3MjYzZjZrM3pXVGNxWnFzV3NZZjF4WFcyajNpK1NaOWVHM043CmpZZHpIL085d2Y2K29BS2s3UW9jT0dGbXBnQnlwdm9JbytlSFdjZlpZNFNXNloyeTRacUl0M2xTSWFHMEtmQjEKUnVrSVBtMlYzNDNlc3Azdy9TV2liRVU2elhvd01sWjlZc0dPalY1MTZDQmJZK0lQcmY2RXY0UFhDWjlnUERxcgpnUFFIZ0lFS294aUdYK2dENG9pYWdUMWVVSk1iYWZkaEkzSHJMWk9YbUpBZU40RHpOZXR1SGRsTk13YnpZZHN5CnpyMllSMkhqNTZydk9hcWZZUFJKV1NmS3ozRlhQdDNDMHJRVmNRekVqV1ZZbm9MZG9yR3FkaU1uMyt1NzF2RW4KSnluZksvN3VRSDY5VTU3NGNKK3FjOFNtNTlPeHBnN1RoODljMUZTeSszTk03bTE2czlJcjQvcG94Q2pWaHNWeQpKRmN2cHE4UTMwek9CQk0waUxyblNKcVRXaXJDRU8xNHlRY0VLbFBVd2VGUWdJK05pUUtDQVFFQTMwMWNrTnZuCmpaRHduRXFQaWp4SlBDSUNEeHcxYWtnVTZUNUpMVkt5SUFnK3BYNHlPZDZtendvV2hpWE5oVUQ5eWxPL21lTEsKSDRPUXVzeWdJUDdkOVNKZi9ZVTV5a1RZaVZLMUdzM1loZXovRmhRQXdBQUg4UmdBdW8rZU9UTUcyT0IxM2loLworQlFYL1lrOG9VR1M0YlpsZWNGWG5pSTZ5S0g1MUJUQ3g0ZTZZcStvdGMrMFc2RzBRVHVBT3JWcjdXWklGY1htClVDdElReXRJN2s2UVd6SXU0K3dnVnU3OUdrUEJTMHNBaXhzU0ppYXRTNzk1TjlhYjYwRzNDeVFWZDZWajJaVmEKWGpCR1oyQXREalpEWS9MMDlZQTlRVUxDbWVqT2hYVFlGNkJuNnRkYmVjeVlZTHdNS2MzWEhqNEt2U2ZaQ01HTwpvUXROdnFoUFFlSWpxUUtDQVFFQTMwZnFCSThDK2QwdXJlM1JhNFFRdUlzUC9xVTMyejIzMUdzbTZOOXlQd0hUClM4ZzlrS0ZDYzhMVGlmWFdnajhIRWxTVjVLSXJseERwZEVWQ1pEM05qdy9GVDdHcHV5SVplN1E3VlhiNld3MnYKY1I0M1FkdUQvOEc4ZkdlZGkrZ1BnKzlzeURCQXFIVzFGSHl1RmVaWGNsVTF1cndOV3V3TlFKUnZHczFoK1NuQQpXVmJxd1ZUL01GMmYxTjlBaFN0alhkM0Nscm1rb0o4ek05YW0zVWg5VUprQ2xSZi9mZVRONXZndE1odS9NNGtGCmVQYUlBMTVqT3h1cEFMN3ptVVVxZ0h1NE9yMm5mYTVNNHMzWTR6TVRYYm9VZGNVT09oRTRzU0J2bGlaak5KL3QKV2pSTmJmUWhxbWRBTTNZZjFGUC8vRW9CYTBPejBxMHNXci92cEhDUlp3S0NBUUFxMlkySnZxa1FZVi9LbmdRdApZcVFyQmR1ZlNxcDFXcCtvb21zb1oxWUhENDMxOCtGdmVXcEpFSWFCOTM4WXN3QUFjMUd4RmZQeldDdk5yTGFOCm5scTVUMzljQnRTd0c4WHhsQTFzdDFOMVg2VVRkNE10Vk5ReFQ0blVRdnI1dnZEeGJTRXhJRlJ1Sm16MEdnR28KY0F6ZmcwQzF2SVF6dEIzVG9rRnVrUTFQZkp3bms4MnNGYzltUmdGeEF4bjRLaGdyMWhTL0dOcTVSNVQyVHJnUQpBc053dkpDQzdDeklnZFBQMW5DaElpTllqamxOV042b1NuWFlZVFpLVHJIeFVWdE5PaytPMFRvbUdOMXB1T3JzCmJ6MC9VTC93M0VyazJ3cTh2Zy9qVENpclgveVE5QUo1dk9rQXB4VXVjSEYzUERDVFc3SXFHL3Bpck9pZVRXM28KRnAwQkFvSUJBRjBxa3JsSU8wT3JTUmtHRE1aQ0d3QUY5cXlZb0EvNVZzVnAySmgrOUJyYVZpSmU4V0Z5Q0ZwcApSdjlmOXh2dDFMT1BXK1JFenMrQUhRbUpCTVR6RE56UEJkUFZIQytiY09xdkw3cmZwR050K0hESTNPRzhDUDRsCkJ0TWFJU0VKdWIraG5kQ0NZZGhwRlIveFRtcVE3SmdtZWY3ckROK05jNUlvM1p0ZmE2d2VBY2JGZjdzZ0RrTk8KTGEwVFlzYXViZzN5eElsRCtTK1VmamI1TUROUlZnalZiOEJxZlE4NDg3bVdnTFZSNHB4TVpsNHM4R0FIZUh4bgpkRU45YWdQZ1duVzJLZzlJcDZUSG9BbGJQMDYrTnl4NndxTEprTUFtQTNQVlJ2cHVGaU1WUUdMTlJDbkhIbTBPCkhEbmM1amNndmNXMTA1WEFjRDVPU0IydHpQN2VnYTBDZ2dFQUZhTHJxMDJ6M2tXaW1iVzdoNi9Pby9YRHdjZ2YKSmdXMDYyYWdWUlpEMjM3Mm4valZSL25kMnFybmwxTFdWaXpzWW91b3hvY2N6NUwvQ2h3MktCOC9Rc1Zxai9KOApOUWh1ZDJ2ZUxjQlUvVzNseVhENnJpajJZMythNzRvM1A4b1psOGdDQmJEck1jajNsMHRZMXRWbm9nOWo0eHQ3Ckp3UXA3djNXb3ArSFVuRWVjbGpscTdlN1VuTGNwZlRDOE5PUmdxbjBGSUtCbXdFNlpJQnQxUzB6NmxSKytJdVMKQk1oTTZTSkZmeUFOVVdjTCt2cHJoRzBjWUZmcUYzajlPZ1dXRnBGTHFWQmdSTlZBVDRoM0U1Ymh4L0lsMTVHMAo2USs2aWlkRElqMEhNNFAzcjhja2lRUlVPdFI1R2NVS1RYZGh1TjkwMjZGRTdEckRvSURVZ0F5dGVnPT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K"
+secretbox_encryption_secret = "RLkR4RmeaLmH/abyK3eYf6q22umJ/byLhheVCnh/yrA="
+
+controlplane_configs = {
+  "talos-cp-1" = {
+    hostname     = "talos-cp-1"
+    lan_ip       = "192.168.1.166"
+    lan_subnet   = "192.168.1.0/24"
+    lan_gateway  = "192.168.1.254"
+    install_disk = "/dev/nvme0n1"
+    longhorn_disks = []
+    zone             = "az-a"
+    allow_scheduling = true
+    cloudflare_talos_sans     = []
+    cloudflare_apiserver_sans = []
+  },
+  "talos-cp-2" = {
+    hostname     = "talos-cp-2"
+    lan_ip       = "192.168.1.214"
+    lan_subnet   = "192.168.1.0/24"
+    lan_gateway  = "192.168.1.254"
+    install_disk = "/dev/disk/by-id/wwn-0x644a842029bd3f002720989b07d7143d"
+    longhorn_disks = [
+        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b60f4375d97062", mountpoint = "/var/lib/longhorn-disk1" },
+        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b5ffeb8bb8b47f", mountpoint = "/var/lib/longhorn-disk2" },
+        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b6000c8dabb266", mountpoint = "/var/lib/longhorn-disk3" },
+        { device = "/dev/disk/by-id/wwn-0x6b083fe0c5782700321370dc23fd765b", mountpoint = "/var/lib/longhorn-disk4" },
+      ]
+    zone             = "az-b"
+    allow_scheduling = true
+    cloudflare_talos_sans     = []
+    cloudflare_apiserver_sans = []
+  },
+  "talos-cp-3" = {
+    hostname     = "talos-cp-3"
+    lan_ip       = "192.168.1.162"
+    lan_subnet   = "192.168.1.0/24"
+    lan_gateway  = "192.168.1.254"
+    install_disk = "/dev/nvme0n1"
+    longhorn_disks = [
+        { device = "/dev/disk/by-id/usb-Seagate_One_Touch_w_PW_00000000NABV3H34-0:0", mountpoint = "/var/lib/longhorn-paperless-media" },
+      ]
+    zone             = "az-c"
+    allow_scheduling = true
+    cloudflare_talos_sans     = []
+    cloudflare_apiserver_sans = []
+  },
+}
+
+worker_configs = {
+  "worker-1" = {
+    hostname          = "worker-1"
+    lan_ip            = "192.168.1.223"
+    lan_subnet        = "192.168.1.0/24"
+    lan_gateway       = "192.168.1.254"
+    install_disk      = "/dev/nvme0n1"
+    network_interface = "enp28s0f0np0"
+    zone              = "az-a"
+    gpu_count         = 4
+    node_labels       = {}
+    node_taints       = []
+    factory_image     = "factory.talos.dev/metal-installer/0a2153a6dc099a371bf2f63d6c3c22d275c876bf6302dd154c5813072924cb3f:v1.13.3"
+    swap_size         = "64GiB"
+    ephemeral_max_size = "700GiB"
+    extra_disks       = []
+  }
+}
+
+cluster_config = {
+  controlplane_ip = "192.168.1.166"
+  pod_subnets     = ["10.244.0.0/16"]
+  service_subnets = ["10.96.0.0/12"]
+  dns_servers     = ["8.8.8.8", "1.1.1.1"]
+  dns_domain      = "cluster.local"
+}