fix(sops): drop the orphan forgejo-registry-pat file

ksops failed the whole sops-secrets application on this file: it declares Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists. The failure blocks every other file in the generator list, including zitadel-masterkey. The same Secret name is already generated in namespace api by forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace forgejo was never created. Remove the file and its generator entry. ArgoCD never created the Secret, so pruning has nothing to remove.
Merge pull request 'feat(iam): wire Zitadel as the Authentik replacement' (#63 ) from harness/zitadel-iam-wiring into main
2026-10-24 03:46:29 -07:00 · 2026-10-24 10:41:17 +00:00 · 2026-10-24 10:41:09 +00:00 · 2026-10-24 03:38:21 -07:00 · 2026-10-24 03:38:21 -07:00 · 2026-10-24 03:38:21 -07:00
32 changed files with 1808 additions and 66 deletions
@@ -31,6 +31,17 @@ AUTHENTIK_BOOTSTRAP_TOKEN=
 #   openssl rand -base64 24
 AUTHENTIK_PG_PASSWORD=

+# ── ZITADEL (Authentik replacement) ─────────────────────────────────────────────
+# Instance URL, same hostname as the OIDC issuer.
+ZITADEL_URL="https://zitadel.riotpiao.com"
+
+# PAT of the 'iam-admin' machine user, created by the chart's setup job on the
+# first sync. Leave empty until then. Read it with:
+#   kubectl -n iam get secret iam-admin-pat -o jsonpath='{.data.pat}' | base64 -d
+# scripts/iam/zitadel-provision.py reads this variable first, then the cluster
+# Secret. The masterkey and the client secrets never live in this file.
+ZITADEL_BOOTSTRAP_TOKEN=
+
 # ── MinIO ──────────────────────────────────────────────────────────────────────
 MINIO_ROOT_USER=
 MINIO_ROOT_PASSWORD=
@@ -47,6 +47,11 @@ k8s/storage/test/test
 terraform/.terraform/
 terraform/*.tfstate
 terraform/*.tfstate.*
+# The Zitadel root module keeps its own state (see terraform/zitadel/providers.tf).
+# That state carries generated client secrets and machine keys.
+terraform/zitadel/*.tfstate
+terraform/zitadel/*.tfstate.*
+terraform/zitadel/*.tfvars
 terraform.tfvars.local
 skills-lock.json
 secrets-plaintext.yaml
@@ -69,3 +74,10 @@ k8s/**/*-secret.yaml

 # IAM provisioning scripts contain credential references — never commit
 scripts/iam/*.py
+# Exception: this one reads every credential from the cluster at run time and
+# embeds none. Keep it reviewable in git.
+!scripts/iam/zitadel-provision.py
+
+# Runtime credential export from the old Authentik provisioning run. Holds live
+# M2M tokens; never commit it.
+/authentik-m2m-tokens.env
@@ -4,7 +4,9 @@ metadata:
  name: comfyui
  namespace: comfyui
  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # No cert-manager annotation and no tls: block. This host is covered by
+    # the *.riotpiao.com wildcard, which nginx serves as its
+    # default-ssl-certificate (ingress-nginx/riotpiao-com-tls).
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
@@ -16,10 +18,6 @@ metadata:
      proxy_set_header Connection "upgrade";
 spec:
  ingressClassName: nginx
-  tls:
-    - secretName: comfyui-tls
-      hosts:
-        - comfyui.riotpiao.com
  rules:
    - host: comfyui.riotpiao.com
      http:
@@ -57,6 +57,55 @@ spec:
      prune: true
      selfHeal: true
 ---
+# ZITADEL — the Authentik replacement. Coexistence, not cutover: Authentik keeps
+# serving in the same `iam` namespace until the last consumer moves, then this
+# becomes the only IdP. Migration plan (outside git):
+# ~/workplace/homelab-doc/ZITADEL_MIGRATION.html
+#
+# Wave 3 ordering: sops-secrets (wave 0) creates the `zitadel-masterkey` Secret,
+# databases (wave 2) creates the `zitadel-db` CNPG cluster and its
+# `zitadel-db-app` DSN secret. Both must exist before the init job starts.
+#
+# automated sync is deliberately absent for the first sync — same reason as
+# forgejo. The chart ships `pre-install,pre-upgrade` Helm hooks (zitadel-init,
+# zitadel-setup), which ArgoCD maps to PreSync and re-runs on every sync, manual
+# or automated. A human reads the first sync's hook output before any repeat can
+# run. The init job runs `init zitadel` only (CNPG owns the database and the app
+# user), and the setup job creates the first org and the `iam-admin` machine key
+# that Terraform consumes. Read the SECOND-SYNC HAZARD block at the top of
+# k8s/infra/iam/zitadel-values.yaml before enabling automated sync.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: zitadel
+  namespace: argocd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+  annotations:
+    argocd.argoproj.io/sync-wave: "3"
+spec:
+  project: homelab
+  revisionHistoryLimit: 3
+  sources:
+    - repoURL: https://charts.zitadel.com
+      chart: zitadel
+      # Pinned: the chart trails the upstream app release, and the verified
+      # masterkey/passwap facts were read at v4.19.x. A floating tag would
+      # change the app version without a diff anyone reviews.
+      targetRevision: 10.4.0
+      helm:
+        valueFiles:
+          - $values/k8s/infra/iam/zitadel-values.yaml
+    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: iam
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+---
 # Raw IAM manifests: key-rotation cronjob + authentik migration job.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: oidc-bootstrap
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
+    targetRevision: main
+    path: k8s/infra/oidc-bootstrap
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: kube-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
@@ -1,23 +0,0 @@
-apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
-kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
-metadata:
-    name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
-    namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
-type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
-stringData:
-    REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
-            K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
-            aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
-            R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
-            be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-08-23T23:14:53Z"
-    mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2
@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: forgejo-registry-token
+    namespace: cicd
+data:
+    username: ENC[AES256_GCM,data:oRNecs0a7rs=,iv:/5EFdNqpSjoxe7XDmAEDMd6lXd0SMsRiMmBM9SIwDHY=,tag:Wi+ggoP+9/vgz9rHrxB9nA==,type:str]
+    password: ENC[AES256_GCM,data:HbmwnWk3ST5SDWXwIhu+M5f/FOfFbJHaG0Q/IWrO63HRXmJTmLMpbjtWIXC9IMFoHeDZ1MNAV8A=,iv:ztJZgpaS//yYCiyr6/ubs/UiXkbGMnokvy4Pe45jQMk=,tag:WMpawc/SVAV4mxsCd6XqAA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmhxN2E3ZnNTdkR6U2c1
+            ZStsQktkeWUwMEdmdFIyMkRYRVk1ZGppWlZ3CmUzR2JubWw2VjdWUGt6S2lpNmQ3
+            cDdSWjJiYzRIWS9UVERpVDFJenFnaDAKLS0tIGhDSmNMSjZwcDVPR3U0dUcwNk9Q
+            WmdGS1lyS3dkSXIvOFhyRklGRll2UlUK8NGts6a6Fv10c6gxIIPqIIqfgzQ3EEyk
+            SFrMThwbdaOqLY4AMeDrg+k0Y2t1Xj/XOvdNbRCRictDwZUsptvO7A==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:imebczVzG9YWd42ER4a+w+Bi3dxbN0HFm/1igxzJt6Z0HG/q6i0s59swpGVruk9dQJWrlO8KKqX739ATp2KGC5szrEuRFvPemF594XFCg6ykW+ZeP5VtTvpJg2nHv7/Pu+71ljKKi4gqOJ4SJDn8owAbSN+su6PiT323wax930A=,iv:iG/G2+qQkvbBU95w/GG2PrbAHDOaamuvjqMDBxSW6zI=,tag:wy9Vn72HBDHqCyzTcV15/Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: memory-agent-oidc
+    namespace: poimen
+data:
+    CLIENT_ID: ENC[AES256_GCM,data:c/RvtV6anofX8h1BmlKu9w==,iv:3lQ1BR8kVvF1RHVfKstmb0Y7u/uX5OrvmGiCi/xTpjw=,tag:9JC2WkfQFWaiZroAEmPenw==,type:str]
+    CLIENT_SECRET: ENC[AES256_GCM,data:I258/6O43PccyllYN/wfoKvTgNgZONTivM/KFIr8TGqSl+2XQkwBXvK2VQc5Xgc5cpwlrgnjk8pjQbqr,iv:MZyA+V5z9Po2VoxqvOSlWh1fZp1lrkcbYMx7VcSIaJs=,tag:gXH0BpA8KOZiVd+4RnhRzg==,type:str]
+    ISSUER: ENC[AES256_GCM,data:GX/uQPDoeJWXXQZlzD6oOIvxo7H4LRAv4MrPAiCgmv6WxW59+KKkymuDT/T9MiDm4bNhxVPgB7nHbDcMHfPvU2zCNM2TWYUsNOgGwxNAqM8=,iv:5OEsdrUZvMLO2C8Ea2i5alsyFGtmEk3othTemk8fOk4=,tag:8CMlRY2JCe4z4fv4WuWjRQ==,type:str]
+    TOKEN_URL: ENC[AES256_GCM,data:KeU1pQzvQ2/1lQhOqNtcR7t5TGVx1J7NY8ogxCXS6IuHhVe0zWYIbC4BYFc2H9PAjsMJwCCg75N1FrIiCYBvpfK8ln0=,iv:NbPiJC/B7W1Zm+JwnEGDikE2cb3L4l6TBeQTKEb2boM=,tag:FRmllFLZzQW03lsCu9mAUA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmK0dPSmhKT01FSitDdUNW
+            Um1XQU9FVnBUeDdoWXpCcXRzVXh0VXkvNGxNClZaQ3R5U2NkYTJjaFdHRWhUVmFS
+            K0tHZVRFRDZGREVRZzZqcWJ6dnRpUUEKLS0tIE15UUNtMStKbWwzb3dEVGEwN1VQ
+            cktlVTM2SUhpOGJZMXJOOG84RnhCQzAKOdM0pThEaN37SJtRrKIivncMu8Z91Btq
+            es6NOlJdBUhzIhnEJWfDzZ9aA6I/fYzugKbC+AfUwdGgThAgHx9crg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:/BTQ984D+hsk0LmnjD3hxopB65NAR54LhoAPNcATWA7MCTJXfdnYRu3gNxUzyiyCBd+rDaPp77TBnak4jVjMnOiYJsYLj1R7hrrhTZopz8pT+YNMupperwsirKZwK8it4Mu+fnpCyox2mtGED0hk+Nh/b6IteTRt5lS7noumo2M=,iv:RWIrVkNYj+8mqMSPiLueToexabzAtVbIR525m6bUrlg=,tag:5+wcNoNYsjGKpFlML+N70Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: oauth2-credentials
+    namespace: iam
+data:
+    api-gw-client-secret: ENC[AES256_GCM,data:YiUCk61cCsTJE8gTLZWhqVPsCSu3b4OnG+Oh456A+mnxip0FjLTTqV+kYBdfZivIkn9ua4RcVW+rYILu,iv:K08m1EaB7/SIVNWZ3FfeRP4tbCAo6KeiNC0NpE9VO1o=,tag:ZXV9YRWUOPJ/vdoqMtvQeA==,type:str]
+    minio-client-secret: ENC[AES256_GCM,data:vcwLo1cJqbZr2MwkAKO09xPjb09ss7+s6HIvEfaLxt8ka5PUAlXQ37pTkynuktlKq/t0DvSk+x1+5G7QqVD1JJlwyDGM6yuOhBQ3MwucLNl+T4RDKtgGuA==,iv:sCdd+UNX2TfTAM6Zslrh8KrvBSzqpe67F3YMGrl4YDU=,tag:9Tpzm9NEXNuXyCd7l7GeHw==,type:str]
+    poimen-client-secret: ENC[AES256_GCM,data:dYtCiMGZaOFokihfGD2SXJDfXrhfwEQcTOPNu7JNHDUnx0hqwAdYAPAJTEN21p0OpHYDINMs5lwUpMwY,iv:ll8USuzs0881ryHImVjHGO0iFxUzqg5XtI6Ekd+EFXE=,tag:I/4ZfB9Ss4zEP57sRtRTXQ==,type:str]
+    paperless-client-secret: ENC[AES256_GCM,data:XMj9rHkI20lI37vChq9wS5lUq7l95SisRX0LChuj09O5w76Zt9NdG6OEUYQFf1z8vCfMJxBInIEEIq0e,iv:UpYy2ZEZ4AgBACEyrlxEcr13V+8Ksu4p8JHYApmyYHo=,tag:o9Z+/b6T9POgowIsks56kw==,type:str]
+    grafana-client-secret: ENC[AES256_GCM,data:V/j0+P3HoDLqYjRalWK4v1hvJTuVmzxVra47HsF1/BZvWZjIyEIH3Pt8BekT3O1dUFynxjcaxdZMBVYg,iv:9pMp+vIsp/sA7GP9wqNDf5DdZVLDow46OURW0mMRXIc=,tag:i7VJFA5o0hToZEJ5JPWIvw==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2amZPendIOU9PeStOcmJG
+            c1dzOFVvd3RoT3dXQlJUNkdCSytEZUVGVEJJCklMUkNjUytBMXh0OVUrNUlpMWw3
+            aHRFT1hBZTZHNHVtaGJCTEFyT2ZWNWsKLS0tIENxR3FKMEplV0V5Y0dsZzBkODVG
+            U0ZlT2R0VFdEQmpsakVvZ1FJRkRQYTAKESQcSlMAuCG0/tLWWZaeoPayf1EKxSaf
+            e4o8TmjmSgPEhepQ/LmNXl6obyuqUxztkh4Mm6pocqDp+gfzt7X/1g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:AI/znHiUa/Rwg0YRz65qUZ+b6V73oi3ZfAG/V5uZZvrGFrA0EISzD5LaP5Kl4GUU1svo4G5EI4aftPDgpdPptz21CCa8iYkXGe9pTDmtaE5ZICQAUbRcVhGMTfuWFhrgEhMfCJdOpfHaIQVEsQJQJgrnkXQVuUe9aT75LQz7pnY=,iv:KbIaYT0VeBT2WeSU5V5ZB4e+04JICYaY+egGN1Pv5bM=,tag:pPbhDxeLUx6V1XC/LPwxTg==,type:str]
+    version: 3.13.2
@@ -11,7 +11,6 @@ files:
    - agent-pod-ssh-key.enc.yaml
    - authentik-secrets.enc.yaml
    - cloudflare-secrets.enc.yaml
-  - forgejo-registry-pat.enc.yaml
    - forgejo-registry-pat-api.enc.yaml
    - forgejo-registry-pull.enc.yaml
    - forgejo-runner-token.enc.yaml
@@ -33,5 +32,9 @@ files:
    - gotify-smtp-secrets.enc.yaml
    - forgejo-smtp-secrets.enc.yaml
    - paperless-ai-secrets.enc.yaml
+    - memory-agent-oidc.enc.yaml
+    - oauth2-credentials.enc.yaml
+    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
+  - zitadel-masterkey.enc.yaml
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: zitadel-masterkey
+    namespace: iam
+type: Opaque
+stringData:
+    masterkey: ENC[AES256_GCM,data:dGp0wSCUJORP6W5PHLKYaQEqrZsgiAwU1GqcuvuZV+g=,iv:Wwt6E/XXwsoVEQxe4gG1m5oJLOcNUXngftADQvZQepw=,tag:1Vget5gs+t455U9zLavdwg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBld0tUWHNWem13cDlxRG14
+            QWRlTjA5SzZnS2pBd2M0WWkzQWxKZW9HZWdJCk00QWJJcWZFM1ZTSUlReUlGZG4z
+            RFBSUWRlVUx2NGN2OFAwazBYRTVVNkEKLS0tIHpSSW9IWHhNa0d3SmEydXU4akVj
+            MGJkN2U4WmtLbVFMZm55U1hEZElZMlUKKV9twnC9GC7MSmNLxEBHbPl4RN5Hsd+z
+            NViCKxUoQAB8apn2XVHZcDnyfKNfr1P3aSa3IIHyVY3prgo1OFDpdw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-24T08:57:27Z"
+    mac: ENC[AES256_GCM,data:hXDAddVmnlf/yinY2c2Q8yfsjqi6n9cRbEbwykqYScd6mDPfdF7BriTiu9K+CV/IvIT4zcMTZbhBeQ/vTCAh0TZDp1lOVglwtNH/imgagyzPmcCvbCIkZKwF5vwWQIGHcDI6pk4/zVQkqt7zKTf8+Qj0pNh5UrAjVeEFE7cuJkQ=,iv:oIQvROFGbiESmMTthHOJiNM+Gy5BCfM5XtDWbszOPmc=,tag:AaYrJrwXuKD7w0p+TTa7FA==,type:str]
+    version: 3.13.2
@@ -8,6 +8,19 @@
 # depend on them existing first.
 installCRDs: true

+# Point the ACME DNS-01 self-check at the public resolvers directly.
+#
+# Without these flags cert-manager resolves through the pod resolver
+# (CoreDNS at 10.96.0.10). The self-check then never saw the TXT record and
+# reported "Waiting for DNS-01 challenge propagation" forever. The wildcard
+# certificate renewal stalled on 2026-09-23 and the cert expired on
+# 2026-10-23, which broke every Cloudflare Tunnel origin with
+# "x509: certificate has expired".
+#
+extraArgs:
+  - --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
+  - --dns01-recursive-nameservers-only
+
 # Tolerate the control-plane taint (talos-cp-1 is the only schedulable node).
 tolerations:
  - key: node-role.kubernetes.io/control-plane
@@ -1,13 +1,21 @@
 # k8s/bootstrap/ingress/riotpiao-com-cert.yaml
 # Public wildcard cert for *.riotpiao.com, signed by Let's Encrypt (DNS-01 via
-# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml). Distinct from
-# homelab-tls (self-signed homelab-ca, internal *.riotpiao.com) —
-# reference riotpiao-com-tls explicitly on ingresses that expose riotpiao.com
-# hosts to the internet; it is not nginx's default-ssl-certificate.
+# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml).
 #
-# issuerRef starts on letsencrypt-staging. Switch to letsencrypt-prod once the
-# staging cert issues cleanly, then delete this Certificate's Secret to force
-# a re-issue against prod (staging certs aren't browser-trusted).
+# nginx serves this secret as its default-ssl-certificate
+# (../ingress/nginx-values.yaml), so every Ingress that omits a tls: block uses
+# it. Do not add per-host tls: blocks for *.riotpiao.com hosts: they create a
+# second, single-host certificate and split the wildcard coverage.
+#
+# Renewal is automatic and does not drop traffic:
+#   - cert-manager starts a new order 30 days before expiry (renewBefore).
+#   - cert-manager writes the renewed pair into the same secret.
+#   - ingress-nginx watches the secret and hot-reloads. No pod restart.
+#   - The old pair stays valid until the new one is written, so no gap exists.
+#
+# The DNS-01 self-check must reach the public resolvers directly. Without the
+# --dns01-recursive-nameservers flags in ../../cert-manager/cert-manager-values.yaml
+# the self-check never sees the TXT record and renewal stalls until expiry.
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
@@ -107,7 +107,9 @@ ingress:
  enabled: true
  className: nginx
  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # No cert-manager annotation. forgejo.riotpiao.com is covered by the
+    # *.riotpiao.com wildcard, served as nginx's default-ssl-certificate.
+    #
    # This chart Ingress and the hand-written one in
    # k8s/bootstrap/ingress/ingress.yaml both claim forgejo.riotpiao.com.
    # ingress-nginx breaks the tie by oldest creationTimestamp, and the chart's
@@ -125,10 +127,9 @@ ingress:
      paths:
        - path: /
          pathType: Prefix
-  tls:
-    - secretName: forgejo-tls
-      hosts:
-        - forgejo.riotpiao.com
+  # tls: intentionally omitted. The chart would create a second, single-host
+  # certificate (forgejo-tls) that expired on 2026-10-23 and competed with the
+  # wildcard. The wildcard covers this host.

 # Resources
 resources:
@@ -7,5 +7,6 @@ kind: Kustomization
 resources:
  - namespaces.yaml
  - authentik-db.yaml
+  - zitadel-db.yaml
  - temporal-db.yaml
  - paperless-db.yaml
@@ -0,0 +1,42 @@
+# Dedicated CNPG Postgres for ZITADEL (GitOps, wave 2 — before the zitadel
+# chart at wave 3). Same shape as authentik-db.yaml: CNPG generates the secret
+# `zitadel-db-app` and the service `zitadel-db-rw` in ns iam, and
+# k8s/infra/iam/zitadel-values.yaml reads the DSN from that secret locally (no
+# cross-namespace secret copy).
+#
+# ZITADEL requires PostgreSQL 14+; the repo convention image is 16.2.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: zitadel-db
+  namespace: iam
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
+  bootstrap:
+    initdb:
+      database: zitadel
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+  storage:
+    size: 10Gi
+    storageClass: longhorn-cnpg
+  monitoring:
+    enablePodMonitor: true
+  affinity:
+    # preferred + tolerations: HA across nodes without deadlocking a partly-
+    # schedulable 3-CP (same as authentik-db and forgejo-db).
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule
@@ -0,0 +1,134 @@
+# k8s/infra/iam/zitadel-values.yaml
+# ZITADEL — Authentik replacement. Coexistence is deliberate: Authentik keeps
+# running, read-only, in the same `iam` namespace until the last consumer moves.
+# Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
+#
+# Chart: zitadel/zitadel 10.4.0, app v4.19.2. The plan cites v4.19.4; the chart
+# has not published it yet. Pin the chart, never `*`, until the v4.19.4 chart
+# exists — a silent app bump invalidates the verified masterkey/passwap facts.
+#
+# URL: https://zitadel.riotpiao.com. TLS terminates at ingress-nginx, which
+# serves the wildcard riotpiao-com-tls as its default-ssl-certificate, so
+# ExternalSecure=true and TLS.Enabled=false (no per-rule tls: block needed).
+#
+# Database: CNPG cluster zitadel-db (wave 2). The DSN comes from CNPG's
+# generated `zitadel-db-app` secret. When ZITADEL_DATABASE_POSTGRES_DSN is set,
+# ZITADEL ignores the discrete Database.Postgres.* fields, so neither a
+# password nor a host is written into this file.
+#
+# Masterkey: Secret `zitadel-masterkey` (k8s/argocd/secrets/
+# zitadel-masterkey.enc.yaml), key `masterkey`, exactly 32 bytes. It wraps the
+# rows in system.encryption_keys, which encrypt every client secret and private
+# key. Losing it makes those values unreadable: keep the offline sealed copy.
+#
+# Ingress uses the chart's own objects (controller: nginx adds the gRPC backend
+# annotation ZITADEL needs for its HTTP/2 API). Do not hand-write a second
+# Ingress for this host.
+
+zitadel:
+  configmapConfig:
+    ExternalSecure: true
+    ExternalDomain: zitadel.riotpiao.com
+    TLS:
+      Enabled: false
+
+  # Rendered into a Secret by the chart; read by pods and init jobs alike.
+  masterkeySecretName: zitadel-masterkey
+
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+
+  # FirstInstance defaults stay for the FIRST sync: the chart creates org
+  # `zitadel` plus the `iam-admin` machine user and its machine-key Secret.
+  # Terraform consumes that key to declare projects, roles, OIDC apps and the
+  # groups action.
+  #
+  # SECOND-SYNC HAZARD — read before enabling automated sync.
+  # The chart marks eight objects as `pre-install,pre-upgrade` hooks (two
+  # ServiceAccounts, two ConfigMaps, Role, RoleBinding, zitadel-init,
+  # zitadel-setup), and one cleanup Job as `post-delete`. ArgoCD maps the first
+  # eight to PreSync with `before-hook-creation`: it deletes and re-creates them
+  # on EVERY sync, not only on a chart change.
+  #   - zitadel-init  (`init zitadel`) — schema step
+  #   - zitadel-setup (`setup`)        — first instance + iam-admin machine key
+  # On an initialized instance the setup step can fail, which aborts the sync
+  # before the Deployment is applied. If it succeeds instead, its
+  # machinekeyWriter container re-applies the iam-admin key Secret that
+  # Terraform reads, which silently rotates a credential in use.
+  # After the first sync reports healthy:
+  #   1. set configmapConfig.FirstInstance.Skip: true here — permanently: the
+  #      setup steps are schema-only once the instance exists, and Skip alone
+  #      stops the machinekeyWriter rotation
+  #   2. set initJob.enabled: false below
+  #   3. commit, sync once, then enable automated+selfHeal on the Application
+  # For a later schema upgrade, re-enable initJob.enabled only — never set
+  # FirstInstance.Skip back to false.
+  # ArgoCD never prunes a hook Job, so a Job that failed before step 2 stays and
+  # keeps ZitadelSetupJobFailed firing until a human deletes it.
+
+# DSN mode (chart renders this through the zitadel.dbEnv helper). CNPG's `uri`
+# key is a postgresql:// URL, which ZITADEL accepts as a DSN.
+env:
+  - name: ZITADEL_DATABASE_POSTGRES_DSN
+    valueFrom:
+      secretKeyRef:
+        name: zitadel-db-app
+        key: uri
+
+# Bundled Bitnami Postgres is disabled: the CNPG cluster above is the system of
+# record. Enabling it would create a second, unmanaged database.
+postgresql:
+  enabled: false
+
+# CNPG already created the database, the `app` user and its grants, so the init
+# job runs only the ZITADEL-internal step. Running the full default command
+# would try to create a database and a superuser it does not own.
+initJob:
+  command: "zitadel"
+  activeDeadlineSeconds: 600
+
+# Two API replicas: the read path survives one node drain, and both share the
+# same masterkey and database. The login UI stays at one replica — a session
+# cookie is not shared, so a second pod buys nothing.
+replicaCount: 2
+revisionHistoryLimit: 3
+
+ingress:
+  enabled: true
+  className: nginx
+  controller: nginx
+
+login:
+  enabled: true
+  replicaCount: 1
+  ingress:
+    enabled: true
+    className: nginx
+
+# Chart-owned ServiceMonitor. The cluster's Prometheus runs with
+# serviceMonitorSelectorNilUsesHelmValues=false, so the release label is not
+# required for discovery. Endpoint is /debug/metrics on port http2-server.
+metrics:
+  enabled: true
+  serviceMonitor:
+    enabled: true
+    scrapeInterval: 30s
+    scrapeTimeout: 10s
+
+# The 3-CP / 2-worker cluster mixes control-plane taints with worker capacity.
+# Without the toleration a rolling upgrade can leave both replicas Pending.
+tolerations:
+  - key: node-role.kubernetes.io/control-plane
+    operator: Exists
+    effect: NoSchedule
+
+affinity:
+  podAntiAffinity:
+    preferredDuringSchedulingIgnoredDuringExecution:
+      - weight: 100
+        podAffinityTerm:
+          topologyKey: kubernetes.io/hostname
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: zitadel
@@ -0,0 +1,68 @@
+# ZITADEL alert parity with the Authentik rules it will replace.
+#
+# Only two signals are ported at this point. The metrics endpoint
+# (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
+# no request or error counters that this cluster has verified, so no error-rate
+# rule is written here. Writing one against a guessed series name would produce
+# a permanently-firing or permanently-silent alert — both worse than none.
+# AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
+# carries zero providers and is deleted with the migration, not migrated.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: zitadel-rules
+  namespace: iam
+spec:
+  groups:
+    - name: zitadel.rules
+      interval: 30s
+      rules:
+        - alert: ZitadelDown
+          # absent() is required: equality alone cannot fire when the series is
+          # gone (ServiceMonitor deleted, Service selector changed, pods scaled
+          # to zero). Those are exactly the outages a target-down rule exists for.
+          expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "ZITADEL API target is down"
+            description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
+
+        - alert: ZitadelLoginUIDown
+          expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "ZITADEL login UI has no available replica"
+            description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
+
+        - alert: ZitadelRestarting
+          # pod!~"zitadel-db-.*" matters: the CNPG database pods share the
+          # `zitadel` name prefix, and a database restart must not page as an
+          # IdP crash loop.
+          expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "ZITADEL pod restarts more than 3 times per hour"
+            description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
+
+        # The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
+        # restartPolicy: Never, so their restart counter stays 0. Without this
+        # rule a failed schema init or first-instance setup reaches nobody but
+        # the person who runs `kubectl -n iam get job`.
+        # No absent() guard here on purpose: the chart ships only
+        # `before-hook-creation`, so a successful hook Job is replaced at the
+        # next PreSync. Once the jobs are retired, absent() would fire on a
+        # healthy cluster.
+        - alert: ZitadelSetupJobFailed
+          expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "ZITADEL init/setup job failed"
+            description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"
@@ -8,6 +8,7 @@ resources:
  - alerts/ingress-alerts.yaml
  - alerts/svc-argocd-rules.yaml
  - alerts/svc-authentik-rules.yaml
+  - alerts/svc-zitadel-rules.yaml
  - alerts/svc-forgejo-rules.yaml
  - alerts/svc-grafana-rules.yaml
  - alerts/svc-minio-rules.yaml
@@ -0,0 +1,59 @@
+# One-shot bootstrap: restart the control-plane apiservers ONCE so they pick up the
+# OIDC authenticator (they last started before the Authentik provider existed, so OIDC
+# tokens were never accepted -> every login got 401). Delete this whole directory after.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: oidc-bootstrap
+  namespace: kube-system
+subjects:
+  - kind: ServiceAccount
+    name: oidc-bootstrap
+    namespace: kube-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: oidc-bootstrap
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-bootstrap-restart-apiserver-v2
+  namespace: kube-system
+spec:
+  backoffLimit: 4
+  ttlSecondsAfterFinished: 7200
+  template:
+    spec:
+      serviceAccountName: oidc-bootstrap
+      restartPolicy: Never
+      containers:
+        - name: restart
+          image: docker.io/curlimages/curl:8.22.0
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -u
+              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+              for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
+                curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN"                   -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0"                   -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
+              done
+              echo "deletes requested; kubelet recreates each static pod in ~20-40s"
@@ -0,0 +1,141 @@
+# One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
+#
+# Two earlier mechanisms failed and are documented so nobody repeats them:
+#   1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
+#   2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
+#      so every pod SIGTERMed itself (exit 143) before reaching the real process.
+# This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
+# impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
+# three control planes by 2 minutes each, so only one API server is down at a time.
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp1-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-1
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 10s before restarting the API server on talos-cp-1"
+              sleep 10
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp2-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-2
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 120s before restarting the API server on talos-cp-2"
+              sleep 120
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: oidc-restart-apiserver-cp3-v2
+  namespace: cicd
+spec:
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 3600
+  template:
+    spec:
+      hostPID: true
+      restartPolicy: Never
+      nodeSelector:
+        kubernetes.io/hostname: talos-cp-3
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: kill
+          image: docker.io/library/busybox:1.36
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              echo "sleeping 240s before restarting the API server on talos-cp-3"
+              sleep 240
+              self=$$
+              killed=0
+              for d in /proc/[0-9]*; do
+                pid=${d#/proc/}
+                [ "$pid" = "$self" ] && continue
+                comm=$(cat "$d/comm" 2>/dev/null)
+                [ "$comm" = "kube-apiserver" ] || continue
+                echo "SIGTERM -> pid $pid"
+                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
+              done
+              echo "signalled $killed API server process(es); kubelet recreates the static pod"
+
@@ -0,0 +1,488 @@
+#!/usr/bin/env python3
+"""Provision ZITADEL end to end: machine key, Terraform, SOPS secrets, users.
+
+One command replaces the manual sequence the Authentik script
+(scripts/iam/provision-rbac.py) used to do in a single pass. Phases run in a
+fixed order and every phase is safe to re-run:
+
+  keys         extract the iam-admin machine key and PAT from namespace iam
+  apply        terraform apply terraform/zitadel (project, roles, apps, machines)
+  secrets      write generated client ids/secrets back into the SOPS files
+  users        import one human user, converting the Django password hash
+  verify       check discovery, client count and the groups-claim gate
+
+`all` runs keys, apply, secrets, verify. It never runs users: the password hash
+must be supplied by a human, one user at a time.
+
+Nothing here talks to Authentik. Authentik keeps serving until a consumer is
+moved deliberately.
+
+Usage:
+  scripts/iam/zitadel-provision.py keys
+  scripts/iam/zitadel-provision.py apply --org-id <id>
+  scripts/iam/zitadel-provision.py secrets --dry-run
+  scripts/iam/zitadel-provision.py secrets --apply-config
+  scripts/iam/zitadel-provision.py users --django-hash '<pbkdf2_sha256$...>'
+  scripts/iam/zitadel-provision.py verify
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import re
+import signal
+import stat
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+REPO = Path(__file__).resolve().parents[2]
+TF_DIR = REPO / "terraform" / "zitadel"
+SOPS_DIR = REPO / "k8s" / "argocd" / "secrets"
+SOPS_CONFIG = REPO / ".sops.yaml"
+CONF_DIR = Path(os.environ.get("ZITADEL_CONF_DIR", Path.home() / ".config" / "zitadel"))
+NAMESPACE = os.environ.get("ZITADEL_NAMESPACE", "iam")
+BASE_URL = os.environ.get("ZITADEL_BASE_URL", "https://zitadel.riotpiao.com")
+
+_TEMP_FILES: set[str] = set()
+
+
+def _cleanup_temps(*_a) -> None:
+    for path in list(_TEMP_FILES):
+        try:
+            os.unlink(path)
+        except OSError:
+            pass
+        _TEMP_FILES.discard(path)
+
+
+for _sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP):
+    signal.signal(_sig, lambda s, f: (_cleanup_temps(), sys.exit(130)))
+
+
+def die(msg: str) -> None:
+    _cleanup_temps()
+    print(f"FATAL: {msg}", file=sys.stderr)
+    sys.exit(1)
+
+
+def run(cmd: list[str], **kw) -> subprocess.CompletedProcess:
+    return subprocess.run(cmd, check=False, text=True, capture_output=True, **kw)
+
+
+def need(binary: str) -> None:
+    if subprocess.run(["sh", "-c", f"command -v {binary}"], capture_output=True).returncode != 0:
+        die(f"{binary} is not on PATH")
+
+
+def write_private(path: Path, data: bytes) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    try:
+        os.fchmod(fd, 0o600)  # open() mode applies at creation only
+        with os.fdopen(fd, "wb") as fh:
+            fh.write(data)
+    finally:
+        pass
+
+
+def secret_value(name: str, key: str) -> str:
+    cp = run(["kubectl", "-n", NAMESPACE, "get", "secret", name, "-o", f"jsonpath={{{key}}}"])
+    if cp.returncode != 0:
+        die(f"cannot read secret {NAMESPACE}/{name}: {cp.stderr.strip()}")
+    raw = cp.stdout.strip()
+    if not raw:
+        die(f"secret {NAMESPACE}/{name} has no key {key}")
+    return base64.b64decode(raw).decode()
+
+
+def load_env_file(path: Path = REPO / ".env") -> None:
+    """Load KEY=VALUE lines from .env without overriding the real environment.
+
+    The file is git-ignored and holds the bootstrap token a human pastes in.
+    Sourcing it in the shell would work too, but then every phase depends on the
+    caller remembering to do it.
+    """
+    if not path.exists():
+        return
+    for line in path.read_text().splitlines():
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, _, value = line.partition("=")
+        key = key.strip()
+        value = value.strip().strip('"').strip("'")
+        if key and key not in os.environ:
+            os.environ[key] = value
+
+
+def bootstrap_token() -> str | None:
+    """Resolve the iam-admin PAT: environment, then cached file, then cluster.
+
+    Order matters. ZITADEL_BOOTSTRAP_TOKEN in .env is filled by a human after the
+    first sync and wins. The cached file under ~/.config/zitadel keeps a run
+    working when the API server is unreachable. The cluster Secret is the last
+    resort and the only source that always reflects a rotation.
+    """
+    token = os.environ.get("ZITADEL_BOOTSTRAP_TOKEN", "").strip()
+    if token:
+        return token
+    cached = CONF_DIR / "iam-admin.pat"
+    if cached.exists():
+        return cached.read_text().strip()
+    try:
+        return secret_value("iam-admin-pat", ".data.pat").strip()
+    except SystemExit:
+        return None
+
+
+def curl_json(method: str, url: str, token: str | None = None, body: dict | None = None,
+              timeout: int = 20) -> tuple[int, str]:
+    """Call the API without putting a secret in the process argv.
+
+    `curl --config -` reads the options from stdin, so neither the bearer token
+    nor the password hash of a user import shows up in `ps`.
+    """
+    def esc(value: str) -> str:
+        return value.replace("\\", "\\\\").replace('"', '\\"')
+
+    cfg = [f'url = "{esc(url)}"', f'request = "{method}"', "silent", "show-error",
+           f"max-time = {timeout}", 'write-out = "\\n%{http_code}"',
+           'header = "Content-Type: application/json"']
+    if token:
+        cfg.append(f'header = "Authorization: Bearer {esc(token)}"')
+    if body is not None:
+        cfg.append(f'data = "{esc(json.dumps(body))}"')
+    cp = run(["curl", "--config", "-"], input="\n".join(cfg) + "\n")
+    if cp.returncode != 0:
+        die(f"{method} {url} failed: {cp.stderr.strip()}")
+    payload, _, code = cp.stdout.rpartition("\n")
+    return int(code.strip() or 0), payload
+
+
+# ── phase: keys ────────────────────────────────────────────────────────────
+
+def phase_keys(_args: argparse.Namespace) -> int:
+    write_private(CONF_DIR / "iam-admin.json", secret_value("iam-admin", ".data.iam-admin\\.json").encode())
+    write_private(CONF_DIR / "iam-admin.pat", secret_value("iam-admin-pat", ".data.pat").strip().encode())
+    print(f"machine key  -> {CONF_DIR / 'iam-admin.json'} (0600)")
+    print(f"PAT          -> {CONF_DIR / 'iam-admin.pat'} (0600)")
+    print("both files stay outside the repo; never commit them")
+    return 0
+
+
+# ── phase: apply ───────────────────────────────────────────────────────────
+
+def phase_apply(args: argparse.Namespace) -> int:
+    need("terraform")
+    if not args.org_id:
+        die("apply requires --org-id (ZITADEL console -> Organization settings -> Resource ID)")
+    key = CONF_DIR / "iam-admin.json"
+    if not key.exists():
+        die(f"{key} missing — run the keys phase first")
+    env = dict(os.environ, ZITADEL_JWT_PROFILE_FILE=str(key))
+
+    for cmd in (["init", "-input=false"], ["fmt", "-check", "-recursive"]):
+        cp = run(["terraform", f"-chdir={TF_DIR}", *cmd], env=env)
+        if cp.returncode != 0:
+            die(f"terraform {' '.join(cmd)} failed: {cp.stderr.strip() or cp.stdout.strip()}")
+
+    cp = run(["terraform", f"-chdir={TF_DIR}", "apply", "-input=false", "-auto-approve",
+              f"-var=org_id={args.org_id}"], env=env)
+    sys.stdout.write(cp.stdout)
+    sys.stderr.write(cp.stderr)
+    if cp.returncode != 0:
+        die("terraform apply failed — no SOPS value was written. Objects created before the "
+            "failure stay in ZITADEL; re-run apply once the cause is fixed")
+    return 0
+
+
+# ── phase: secrets ─────────────────────────────────────────────────────────
+
+# client -> [(sops file, Secret name inside the file, key, kind)]
+# Only secrets with a git-tracked SOPS file appear here. Every other client is
+# printed as manual: guessing a path writes a credential into the wrong Secret.
+SECRET_MAP = {
+    "grafana": [("grafana-oidc-secrets.enc.yaml", "grafana-oidc",
+                 "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET", "secret")],
+    "minio": [("minio-secrets.enc.yaml", "minio-oidc",
+               "MINIO_IDENTITY_OPENID_CLIENT_SECRET", "secret")],
+    "paperless-ai-agent": [("paperless-ai-secrets.enc.yaml", "paperless-ai-config",
+                            "LLM_AUTH_CLIENT_SECRET", "secret")],
+}
+
+# Clients whose client_id lives in a plain (unencrypted) manifest. They need the
+# id update as much as the secret: ZITADEL generates a new client_id for every
+# application, so a secret-only re-point yields invalid_client.
+CLIENT_ID_MAP = {
+    "grafana": ("k8s/infra/logging/grafana-values.yaml",
+                r"(?m)^(\s*client_id:\s*)\S+\s*$", r"\g<1>{value}"),
+    "minio": ("k8s/infra/minio/minio-tenant.yaml",
+              r"(?m)^(\s*value:\s*)\"minio\"\s*$", r'\g<1>"{value}"'),
+    "paperless-ai-agent": ("k8s/apps/paperless/paperless-ai.yaml",
+                           r"client_id=paperless-ai-agent", "client_id={value}"),
+}
+
+# Secrets that a live workload mounts but no SOPS file in git defines. The phase
+# cannot re-point them safely, so it names them.
+MANUAL_ONLY = {
+    "homarr": "Secret homarr-oidc (keys client-id, client-secret) has no SOPS file in git — "
+              "adopt it first, then add it to SECRET_MAP. The repo also carries an orphan "
+              "homarr-auth-oidc.enc.yaml that nothing consumes.",
+}
+
+
+def terraform_outputs() -> dict:
+    need("terraform")
+    key = CONF_DIR / "iam-admin.json"
+    if not key.exists():
+        die(f"{key} missing — run the keys phase first")
+    env = dict(os.environ, ZITADEL_JWT_PROFILE_FILE=str(key))
+    cp = run(["terraform", f"-chdir={TF_DIR}", "output", "-json"], env=env)
+    if cp.returncode != 0:
+        die(f"terraform output failed: {cp.stderr.strip()}")
+    return json.loads(cp.stdout or "{}")
+
+
+def sops_env() -> dict:
+    env = dict(os.environ)
+    env.setdefault("SOPS_AGE_KEY_FILE", str(Path.home() / ".sops" / "key.txt"))
+    return env
+
+
+def write_secret(rel_path: str, secret_name: str, key: str, value: str,
+                 dry_run: bool) -> None:
+    path = SOPS_DIR / rel_path
+    if not path.exists():
+        die(f"{path} does not exist — the mapping and the repo disagree")
+    env = sops_env()
+    dec = run(["sops", "--config", str(SOPS_CONFIG), "decrypt", str(path)], env=env)
+    if dec.returncode != 0:
+        die(f"cannot decrypt {rel_path}: {dec.stderr.strip()}")
+    docs = [d for d in yaml.safe_load_all(dec.stdout) if d]
+    matching = [d for d in docs if (d.get("metadata") or {}).get("name") == secret_name]
+    if len(matching) != 1:
+        die(f"{rel_path}: expected exactly one document named {secret_name}, found {len(matching)}")
+
+    current = (matching[0].get("stringData") or {}).get(key)
+    if current == value:
+        print(f"  unchanged {rel_path}:{secret_name}:{key}")
+        return
+    if dry_run:
+        print(f"  would set {rel_path}:{secret_name}:{key} (len {len(value)})")
+        return
+
+    if len(docs) == 1:
+        cp = run(["sops", "--config", str(SOPS_CONFIG), "set", "--value-stdin", "--idempotent",
+                  str(path), f'["stringData"]["{key}"]'], env=env, input=json.dumps(value))
+        if cp.returncode != 0:
+            die(f"sops set failed on {rel_path}: {cp.stderr.strip()}")
+        print(f"  wrote {rel_path}:{secret_name}:{key}")
+        return
+
+    # Multi-document file. `sops set` cannot address one document, so the whole
+    # file is rewritten. sops encrypt reads the creation rule from the FILENAME,
+    # so the temp path is overridden with the repo-relative path; without that
+    # override sops finds no matching rule and refuses.
+    matching[0].setdefault("stringData", {})[key] = value
+    plain = "---\n".join(yaml.safe_dump(d, sort_keys=False) for d in docs)
+    fd, tmp = tempfile.mkstemp(suffix=".yaml", dir=tempfile.gettempdir())
+    _TEMP_FILES.add(tmp)
+    try:
+        with os.fdopen(fd, "w") as fh:
+            fh.write(plain)
+        enc = run(["sops", "--config", str(SOPS_CONFIG), "encrypt",
+                   "--filename-override", str(path), tmp], env=env)
+        if enc.returncode != 0:
+            die(f"sops encrypt failed for {rel_path}: {enc.stderr.strip()}")
+        mode = stat.S_IMODE(path.stat().st_mode)
+        fd2, staged = tempfile.mkstemp(suffix=".enc.yaml", dir=str(path.parent))
+        try:
+            with os.fdopen(fd2, "w") as fh:
+                fh.write(enc.stdout)
+            os.chmod(staged, mode)
+            os.replace(staged, path)  # same directory: atomic, never EXDEV
+        finally:
+            if os.path.exists(staged):
+                os.unlink(staged)
+        print(f"  wrote {rel_path}:{secret_name}:{key} (multi-document rewrite)")
+    finally:
+        if os.path.exists(tmp):
+            os.unlink(tmp)
+        _TEMP_FILES.discard(tmp)
+
+
+def update_client_id(client: str, value: str, dry_run: bool, apply_config: bool) -> None:
+    spec = CLIENT_ID_MAP.get(client)
+    if not spec:
+        return
+    rel_path, pattern, repl = spec
+    path = REPO / rel_path
+    if not path.exists():
+        die(f"{path} does not exist — the client-id mapping is stale")
+    text = path.read_text()
+    matches = list(re.finditer(pattern, text))
+    if len(matches) != 1:
+        die(f"{rel_path}: expected 1 match for the {client} client id, found {len(matches)}")
+    new_line = matches[0].expand(repl.format(value=value))
+    if matches[0].group(0) == new_line:
+        print(f"  unchanged {rel_path}: {client} client id")
+        return
+    if dry_run or not apply_config:
+        print(f"  would change {rel_path}: {matches[0].group(0).strip()} -> {new_line.strip()}")
+        if not apply_config:
+            print("    re-run with --apply-config to write this plaintext manifest edit")
+        return
+    path.write_text(text[:matches[0].start()] + new_line + text[matches[0].end():])
+    print(f"  wrote {rel_path}: {client} client id")
+
+
+def phase_secrets(args: argparse.Namespace) -> int:
+    need("sops")
+    out = terraform_outputs()
+    ids = out.get("client_ids", {}).get("value", {})
+    secrets = out.get("client_secrets", {}).get("value", {})
+    machine_keys = out.get("machine_keys", {}).get("value", {})
+
+    if not ids and not machine_keys:
+        die("terraform has no ZITADEL outputs yet — run the apply phase first")
+
+    print("[1/3] client secrets into SOPS")
+    for client, entries in SECRET_MAP.items():
+        for rel_path, secret_name, key, kind in entries:
+            value = ids.get(client) if kind == "id" else secrets.get(client)
+            if value:
+                write_secret(rel_path, secret_name, key, value, args.dry_run)
+
+    print("[2/3] client ids in plain manifests")
+    for client in list(CLIENT_ID_MAP) + list(SECRET_MAP):
+        if ids.get(client):
+            update_client_id(client, ids[client], args.dry_run, args.apply_config)
+
+    print("[3/3] machine keys")
+    key_dir = CONF_DIR / "machine-keys"
+    for name, key_json in machine_keys.items():
+        if args.dry_run:
+            print(f"  would write {key_dir / (name + '.json')}")
+        else:
+            write_private(key_dir / f"{name}.json", key_json.encode())
+            print(f"  wrote {key_dir / (name + '.json')} (0600)")
+
+    unhandled = sorted(set(ids) - set(SECRET_MAP) - set(CLIENT_ID_MAP) - set(MANUAL_ONLY))
+    if MANUAL_ONLY or unhandled:
+        print("\n  MANUAL — no safe write path exists for these clients:")
+        for client, why in MANUAL_ONLY.items():
+            print(f"    {client}: {why}")
+        for client in unhandled:
+            print(f"    {client}: set its client_id and secret in its own manifest or Secret file")
+    return 0
+
+
+# ── phase: users ───────────────────────────────────────────────────────────
+
+HASH_RE = re.compile(r"^pbkdf2_sha256\$(?P<rounds>\d+)\$(?P<salt>[^$]+)\$(?P<hash>[^$]+)$")
+
+
+def convert_hash(django_hash: str) -> str:
+    """Django pbkdf2_sha256 -> passwap MCF $pbkdf2-sha256$...
+
+    ZITADEL imports one MCF string via hashedPassword.hash (max 200 chars) and
+    pins passwap, whose pbkdf2 format is $pbkdf2-sha256$<rounds>$<salt>$<hash>
+    with alt-base64 (standard base64, + -> ., no padding). Rounds, salt and hash
+    bytes are unchanged, so the password still verifies.
+    """
+    m = HASH_RE.match(django_hash.strip())
+    if not m:
+        die("unsupported hash format — only pbkdf2_sha256 is convertible")
+    alt = m.group("hash").rstrip("=").replace("+", ".")
+    mcf = f"$pbkdf2-sha256${m.group('rounds')}${m.group('salt')}${alt}"
+    if len(mcf) > 200:
+        die(f"converted hash is {len(mcf)} chars, ZITADEL accepts 200")
+    return mcf
+
+
+def phase_users(args: argparse.Namespace) -> int:
+    token = bootstrap_token()
+    if not token:
+        die("no bootstrap token — set ZITADEL_BOOTSTRAP_TOKEN in .env, or run the keys phase")
+    if not args.django_hash:
+        die("pass --django-hash with the pbkdf2_sha256 value exported from Authentik")
+    body = {
+        "username": args.username,
+        "profile": {"givenName": args.given_name, "familyName": args.family_name},
+        "email": {"email": args.email, "isVerified": True},
+        "hashedPassword": {"hash": convert_hash(args.django_hash)},
+        "passwordChangeRequired": False,
+    }
+    code, payload = curl_json("POST", f"{BASE_URL}/v2/users/human", token=token, body=body)
+    if code in (200, 201):
+        print(f"  {args.username}: created")
+        return 0
+    if code == 409:
+        print(f"  {args.username}: already exists — the password was NOT updated")
+        return 1
+    die(f"create {args.username} -> {code} {payload[:300]}")
+
+
+# ── phase: verify ──────────────────────────────────────────────────────────
+
+def phase_verify(_args: argparse.Namespace) -> int:
+    code, _ = curl_json("GET", f"{BASE_URL}/.well-known/openid-configuration", timeout=10)
+    print(f"discovery      : {code}")
+    if code != 200:
+        die("discovery endpoint is not 200 — the instance or the ingress is not ready")
+
+    out = terraform_outputs()
+    ids = out.get("client_ids", {}).get("value", {})
+    print(f"client apps    : {len(ids)}")
+    enabled = bool(out.get("groups_claim_enabled", {}).get("value"))
+    print(f"groups action  : {enabled}")
+    if not enabled:
+        print("BLOCKED for cutover: the groups claim action is disabled. Every consumer")
+        print("that reads `groups` (kube-apiserver, MinIO, Grafana) sees no groups.")
+        print("Enable it only after the live gate passes, with the script path set.")
+    return 0
+
+
+PHASES = {"keys": phase_keys, "apply": phase_apply, "secrets": phase_secrets,
+          "users": phase_users, "verify": phase_verify}
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Provision ZITADEL end to end.")
+    parser.add_argument("phase", choices=[*PHASES, "all"])
+    parser.add_argument("--org-id", help="ZITADEL org id (required for apply)")
+    parser.add_argument("--dry-run", action="store_true", help="print actions, write nothing")
+    parser.add_argument("--apply-config", action="store_true",
+                        help="also rewrite client ids inside plain manifests")
+    parser.add_argument("--username", default="rock")
+    parser.add_argument("--email", default="[email protected]")
+    parser.add_argument("--given-name", default="Rock")
+    parser.add_argument("--family-name", default="Liang")
+    parser.add_argument("--django-hash", help="pbkdf2_sha256 value exported from Authentik")
+    args = parser.parse_args()
+
+    load_env_file()
+    need("kubectl")
+    if args.phase == "all":
+        if not args.org_id:
+            die("all requires --org-id")
+        for name in ("keys", "apply"):
+            rc = PHASES[name](args)
+            if rc:
+                return rc
+        rc = phase_secrets(args)
+        if rc:
+            return rc
+        return phase_verify(args)
+    return PHASES[args.phase](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -23,6 +23,7 @@
    # via nginx (not the backend service) is critical: a direct rewrite bypasses
    # nginx TLS and exposes each app's own self-signed cert, trusted by nothing.
    rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    rewrite name zitadel.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name minio-api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
@@ -0,0 +1,58 @@
+# The groups claim — the single load-bearing decision of the migration.
+#
+# ZITADEL does not issue a `groups` claim. Authentik did, and these consumers
+# read it: kube-apiserver RBAC (`--oidc-groups-claim=groups` plus 11 RoleBindings
+# on oidc:<group>), MinIO (`MINIO_IDENTITY_OPENID_CLAIM_NAME=groups`), Grafana
+# role mapping, and every service that filters on a group name.
+#
+# Plan §1.3 Option A keeps those consumers unchanged by synthesizing the claim.
+# Option B refactors all of them. This module takes responsibility only for the
+# ZITADEL side of Option A: the action resource and its trigger.
+#
+# The script body is NOT shipped by default. The go/no-go gate (§1.3, §1.5 task
+# 1.3) is: prove on the live instance that the action runtime can read project
+# roles plus user metadata and write both `groups` and the per-agent claims,
+# for a human token AND a machine-user token. Until that proof exists, enabling
+# this resource would move every consumer onto an unverified claim source:
+#
+#   terraform apply -var enable_groups_claim_action=true \
+#                   -var groups_claim_script=./groups-claim.js
+#
+# After it applies, verify with a real token exchange before moving a consumer:
+#
+#   curl -s -X POST https://zitadel.riotpiao.com/oauth/v2/token \
+#     -d grant_type=refresh_token -d refresh_token=<t> \
+#     -d client_id=kubernetes | jq -R 'split(".")[1] | @base64d | fromjson'
+#
+# The decoded access token must contain `"groups":["homelab-admins", ...]`.
+
+resource "zitadel_action" "groups_claim" {
+  count = var.enable_groups_claim_action ? 1 : 0
+
+  org_id          = data.zitadel_org.default.id
+  name            = "homelab-groups-claim"
+  script          = file(var.groups_claim_script)
+  timeout         = "5s"
+  allowed_to_fail = false
+}
+
+# Both triggers are needed. The access-token trigger fills the JWT that
+# Kubernetes and MinIO read. The userinfo trigger fills the response that
+# Grafana, paperless and Forgejo read.
+resource "zitadel_trigger_actions" "groups_claim_access_token" {
+  count = var.enable_groups_claim_action ? 1 : 0
+
+  org_id       = data.zitadel_org.default.id
+  flow_type    = "FLOW_TYPE_CUSTOMISE_TOKEN"
+  trigger_type = "TRIGGER_TYPE_PRE_ACCESS_TOKEN_CREATION"
+  action_ids   = [zitadel_action.groups_claim[0].id]
+}
+
+resource "zitadel_trigger_actions" "groups_claim_userinfo" {
+  count = var.enable_groups_claim_action ? 1 : 0
+
+  org_id       = data.zitadel_org.default.id
+  flow_type    = "FLOW_TYPE_CUSTOMISE_TOKEN"
+  trigger_type = "TRIGGER_TYPE_PRE_USERINFO_CREATION"
+  action_ids   = [zitadel_action.groups_claim[0].id]
+}
@@ -0,0 +1,18 @@
+# API applications — the machine-token surface.
+#
+# ZITADEL keeps two different objects apart: an OIDC application is for a
+# browser flow, and an API application is what a client_credentials / JWT-profile
+# caller presents. Authentik modelled both with one provider, so the plan §1.1
+# row 1 ("OIDC app + machine user") needs one of each here.
+#
+# The gateway signs its token request with the machine key from machines.tf, so
+# the auth method is PRIVATE_KEY_JWT. No secret is stored for it.
+
+resource "zitadel_application_api" "machine" {
+  for_each = { for name, spec in local.machine_users : name => spec if contains(keys(var.api_app_consumers), name) }
+
+  org_id           = data.zitadel_org.default.id
+  project_id       = zitadel_project_v2.homelab.id
+  name             = "${each.key}-machine"
+  auth_method_type = "API_AUTH_METHOD_TYPE_PRIVATE_KEY_JWT"
+}
@@ -0,0 +1,136 @@
+# OIDC applications — one per consumer that logs a browser in.
+#
+# Client IDs: the plan §3.2 wants every client_id byte-identical after the move,
+# so that a consumer change is only the issuer URL plus the secret.
+#
+# THAT GOAL IS NOT ACHIEVABLE HERE, verified twice on 2026-10-24:
+#   - ZITADEL v4.19.2 CreateOIDCApplicationRequest has no client_id field, and
+#     CreateOIDCApplicationResponse documents the value as "generated"
+#     (proto/zitadel/app/v2beta/app_service.proto:540-650).
+#   - zitadel provider 3.8.7 exposes no settable application id: `id` on
+#     `zitadel_application_v2` is not a valid argument (terraform validate
+#     rejects it), and `client_id` on `zitadel_application_oidc` is computed.
+# Consequence: every consumer takes a NEW client_id as well as a new secret.
+# Budget the extra config change per consumer — this is not the one-line
+# re-point the plan assumed.
+#
+# The only way to keep the old value is to create each app through the v2 REST
+# API with an explicit body id, then import it:
+#   POST /v2/projects/<project_id>/apps
+#   {"id":"<old-client-id>","name":"<old-client-id>","oidcRequest":{...}}
+# Not implemented here: it would put app creation outside Terraform state.
+#
+# Secrets are generated by ZITADEL, never written here. `terraform output -json
+# client_ids` and `client_secrets` feed the SOPS re-point (§3.2, §6.9). Do not
+# commit the tfstate: it carries those values.
+
+locals {
+  oidc_clients = {
+    # Gateway. Browser sessions for the console, plus a machine path
+    # (machines.tf) for callers that need client_credentials.
+    "api-gw" = {
+      redirect_uris = ["http://localhost:3000/callback", "https://api.riotpiao.com/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "argocd" = {
+      redirect_uris = ["http://localhost:8080/auth/callback", "https://argocd.riotpiao.com/auth/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "forgejo" = {
+      redirect_uris = ["http://localhost:3000/user/oauth2/authentik/callback", "https://forgejo.riotpiao.com/user/oauth2/authentik/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "grafana" = {
+      redirect_uris = ["http://localhost:3000/login/generic_oauth", "https://grafana.riotpiao.com/login/generic_oauth"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "homarr" = {
+      redirect_uris = ["http://localhost:7575/auth/callback", "https://homarr.riotpiao.com/auth/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "immich" = {
+      redirect_uris = ["app.immich:///oauth-callback", "https://img.riotpiao.com/auth/login", "https://img.riotpiao.com/user/oauth2/callback"]
+      app_type      = "NATIVE"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "kubernetes" = {
+      redirect_uris = ["http://localhost:8000", "http://localhost:18000"]
+      app_type      = "WEB"
+      auth_method   = "NONE"
+      dev_mode      = true
+    }
+    "minio" = {
+      redirect_uris = ["http://localhost:9000/auth/sso/oauth2/code", "https://minio.riotpiao.com/auth/sso/oauth2/code"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "paperless" = {
+      redirect_uris = ["http://localhost:8000/accounts/oidc/authentik/login/callback/", "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "poimen" = {
+      redirect_uris = ["http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "queue-sqs" = {
+      redirect_uris = ["http://localhost:8080/callback", "https://queue.riotpiao.com/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+    "vault" = {
+      redirect_uris = ["http://localhost:8200/ui/vault/auth/oidc/oidc/callback", "https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback"]
+      app_type      = "WEB"
+      auth_method   = "BASIC"
+      dev_mode      = false
+    }
+  }
+}
+
+resource "zitadel_application_v2" "oidc" {
+  for_each = local.oidc_clients
+
+  org_id     = data.zitadel_org.default.id
+  project_id = zitadel_project_v2.homelab.id
+  name       = each.key
+
+  oidc {
+    redirect_uris  = each.value.redirect_uris
+    response_types = ["OIDC_RESPONSE_TYPE_CODE"]
+    grant_types = [
+      "OIDC_GRANT_TYPE_AUTHORIZATION_CODE",
+      "OIDC_GRANT_TYPE_REFRESH_TOKEN",
+    ]
+    app_type         = "OIDC_APP_TYPE_${each.value.app_type}"
+    auth_method_type = "OIDC_AUTH_METHOD_TYPE_${each.value.auth_method}"
+    version          = "OIDC_VERSION_1_0"
+
+    # JWT access tokens: Kubernetes RBAC, MinIO policy mapping and the Go
+    # gateway all read claims, and an opaque bearer token carries none.
+    access_token_type = "OIDC_TOKEN_TYPE_JWT"
+
+    # Assert roles without the client having to request the scope. Every
+    # consumer here maps roles or groups to local permissions.
+    access_token_role_assertion = true
+    id_token_role_assertion     = true
+    id_token_userinfo_assertion = true
+
+    dev_mode = each.value.dev_mode
+  }
+}
@@ -0,0 +1,139 @@
+# Machine users and their project grants (plan §1.4, §5.3).
+#
+# Authentik gave each agent an app password or a client_credentials grant built
+# on a user password. ZITADEL has no user password on a machine user: it issues
+# a JWT profile key (`KEY_TYPE_JSON`) that the SDK signs its token request with.
+# That removes the password path entirely, which is the point — see §5.3.
+#
+# Per-agent claim values are copied from `SERVICE_ACCOUNTS` in
+# scripts/iam/provision-rbac.py. They live in user metadata as JSON strings,
+# because the groups-claim action (actions.tf) is the only thing that turns them
+# into token claims. Without that action a consumer sees roles but not
+# `minio_buckets`, `authorized_models`, `sqs_queues`, `memory_projects`,
+# `memory_visibility` or `paperless_doctypes`.
+#
+# Keys are generated per machine user and land in `terraform output -json
+# machine_keys` (sensitive). Feed each one to the owning workload as a mounted
+# file, never as an environment variable dumps-able through `ps`.
+
+locals {
+  machine_users = {
+    "memory-agent" = {
+      description      = "Memory service agent"
+      project_roles    = ["poimen-admins"]
+      authenticated_by = "poimen memory (Rust)"
+      metadata = {
+        roles             = jsonencode(["llm:inference", "memory:read", "memory:write", "notification:write", "sessions:read", "sessions:write"])
+        memory_projects   = jsonencode(["*"])
+        memory_visibility = "private"
+        authorized_models = jsonencode(["reasoning", "ornith:35b", "qwen2.5:3b"])
+      }
+    }
+    "paperless-ai-agent" = {
+      description      = "Paperless AI plugin (auto-tagging, entity extraction)"
+      project_roles    = ["poimen-devs"]
+      authenticated_by = "paperless-ai"
+      metadata = {
+        roles              = jsonencode(["llm:inference", "memory:write", "paperless:admin", "notification:write"])
+        minio_buckets      = jsonencode(["paperless"])
+        paperless_doctypes = jsonencode(["*"])
+        memory_projects    = jsonencode(["*"])
+        authorized_models  = jsonencode(["reasoning", "qwen2.5:3b"])
+      }
+    }
+    "portfolio-agent" = {
+      description      = "Portfolio service agent"
+      project_roles    = ["poimen-prod-readonly"]
+      authenticated_by = "portfolio service"
+      metadata = {
+        roles             = jsonencode(["llm:inference", "memory:read"])
+        memory_projects   = jsonencode(["homelab", "portfolio"])
+        memory_visibility = "public"
+        authorized_models = jsonencode(["ornith:35b"])
+        minio_buckets     = jsonencode(["backups"])
+      }
+    }
+    "temporal-worker-agent" = {
+      description      = "Temporal workflow worker"
+      project_roles    = ["poimen-admins"]
+      authenticated_by = "Temporal worker"
+      metadata = {
+        roles             = jsonencode(["llm:inference", "workflow:execute", "memory:read", "memory:write", "queue:send", "notification:write"])
+        memory_projects   = jsonencode(["*"])
+        sqs_queues        = jsonencode(["*"])
+        authorized_models = jsonencode(["reasoning", "ornith:35b", "qwen2.5:3b"])
+      }
+    }
+    # The gateway is the one consumer that used client_credentials on a machine
+    # token, so it needs both the OIDC app (clients.tf) and a machine user
+    # (here) plus its API application (api_apps.tf). Its capability check reads
+    # the `roles` claim, which arrives as metadata.
+    "api-gw" = {
+      description      = "Go LLM gateway machine identity"
+      project_roles    = []
+      authenticated_by = "api-gw"
+      metadata = {
+        roles = jsonencode(["llm:inference"])
+      }
+    }
+    # pi-worker has no anchor in git (plan §1.1: "not in git") and no group in
+    # the Authentik provisioning script. It gets a machine user and no project
+    # role until the owner names the consumer configuration that holds its old
+    # credentials. Creating a role grant blind would widen access, not preserve it.
+    "pi-worker" = {
+      description      = "pi sandbox worker (no git anchor for its consumer config)"
+      project_roles    = []
+      authenticated_by = "pi sandbox worker"
+      metadata = {
+        roles = jsonencode([])
+      }
+    }
+  }
+
+  machine_metadata = merge([
+    for user, spec in local.machine_users : {
+      for key, value in spec.metadata : "${user}/${key}" => {
+        user  = user
+        key   = key
+        value = value
+      }
+    }
+  ]...)
+}
+
+resource "zitadel_machine_user" "agent" {
+  for_each = local.machine_users
+
+  org_id      = data.zitadel_org.default.id
+  user_name   = each.key
+  name        = each.key
+  description = "${each.value.description} — consumer: ${each.value.authenticated_by}"
+  with_secret = false
+}
+
+resource "zitadel_machine_key" "agent" {
+  for_each = local.machine_users
+
+  org_id          = data.zitadel_org.default.id
+  user_id         = zitadel_machine_user.agent[each.key].id
+  key_type        = "KEY_TYPE_JSON"
+  expiration_date = var.machine_key_expiry
+}
+
+resource "zitadel_project_member" "agent" {
+  for_each = { for user, spec in local.machine_users : user => spec if length(spec.project_roles) > 0 }
+
+  org_id     = data.zitadel_org.default.id
+  project_id = zitadel_project_v2.homelab.id
+  user_id    = zitadel_machine_user.agent[each.key].id
+  roles      = each.value.project_roles
+}
+
+resource "zitadel_user_metadata" "agent_claim" {
+  for_each = local.machine_metadata
+
+  org_id  = data.zitadel_org.default.id
+  user_id = zitadel_machine_user.agent[each.value.user].id
+  key     = each.value.key
+  value   = each.value.value
+}
@@ -0,0 +1,46 @@
+# Outputs feed two things: the SOPS re-point of consumer secrets (§3.2, §6.9)
+# and the machine-key handoff to each agent workload (§5.3). Every value that can
+# authenticate a client is marked sensitive, so `terraform output` prints a
+# placeholder until you ask for it explicitly.
+
+output "org_id" {
+  description = "Organization that owns the project, roles and applications."
+  value       = data.zitadel_org.default.id
+}
+
+output "project_id" {
+  description = "Project ID, which ZITADEL puts in the access-token `aud` claim."
+  value       = zitadel_project_v2.homelab.id
+}
+
+output "client_ids" {
+  description = "client_id per consumer. Compare against the Authentik values: any difference means the consumer config must change too."
+  value       = { for name, app in zitadel_application_v2.oidc : name => app.oidc[0].client_id }
+}
+
+output "client_secrets" {
+  description = "Generated client secret per consumer. Write these into the existing SOPS files; never into git."
+  value       = { for name, app in zitadel_application_v2.oidc : name => app.oidc[0].client_secret }
+  sensitive   = true
+}
+
+output "api_app_client_ids" {
+  description = "client_id of each API application (machine-token surface)."
+  value       = { for name, app in zitadel_application_api.machine : name => app.client_id }
+}
+
+output "machine_user_ids" {
+  description = "ZITADEL user ID per machine user."
+  value       = { for name, user in zitadel_machine_user.agent : name => user.id }
+}
+
+output "machine_keys" {
+  description = "JWT profile key JSON per machine user. Mount as a file; do not pass through the environment."
+  value       = { for name, key in zitadel_machine_key.agent : name => key.key_details }
+  sensitive   = true
+}
+
+output "groups_claim_enabled" {
+  description = "Whether the groups-claim action and its triggers exist. False means Option A is not yet in place and no consumer may move."
+  value       = var.enable_groups_claim_action
+}
@@ -0,0 +1,62 @@
+# Project, roles and default-org lookup.
+#
+# ZITADEL has no group object. The plan §1.3 maps every Authentik group to a
+# project role, and the groups-claim action (actions.tf) reconstructs the
+# `groups` claim from those role keys. Role keys therefore EQUAL the existing
+# Authentik group names, including the hyphens, so that the resulting claim is
+# byte-identical and no consumer or RBAC file changes. The v4.19.2 API allows
+# role keys of 1-200 characters with no character-set restriction (proto
+# zitadel/project/v2/project_service.proto, AddProjectRole.role_key).
+
+data "zitadel_org" "default" {
+  id = var.org_id
+}
+
+locals {
+  # Single source: the 19 groups of scripts/iam/provision-rbac.py GROUPS.
+  # Descriptions are carried over so the console reads the same as before.
+  roles = {
+    "homelab-admins"       = "Cluster administrators with full access"
+    "minio-admins"         = "MinIO administrators"
+    "minio-photos"         = "Photos bucket (Immich) access"
+    "minio-documents"      = "Documents bucket (Paperless) access"
+    "minio-backups"        = "Backups bucket read-only access"
+    "poimen-admins"        = "Poimen memory administrators"
+    "poimen-devs"          = "Dev and staging projects access"
+    "poimen-prod-readonly" = "Production projects read-only access"
+    "paperless-admins"     = "Paperless administrators"
+    "paperless-finance"    = "Finance documents"
+    "paperless-legal"      = "Legal documents"
+    "paperless-hr"         = "HR documents"
+    "grafana-admins"       = "Grafana administrators"
+    "grafana-editors"      = "Grafana dashboard editors"
+    "grafana-viewers"      = "Grafana dashboard viewers"
+    "sqs-users"            = "SQS/Temporal queue read access"
+    "sqs-writers"          = "SQS/Temporal queue read/write access"
+    "s3-users"             = "S3 read access"
+    "s3-writers"           = "S3 read/write access"
+  }
+}
+
+resource "zitadel_project_v2" "homelab" {
+  org_id             = data.zitadel_org.default.id
+  name               = var.project_name
+  project_id         = var.project_id
+  project_role_check = false
+
+  # Roles must reach the tokens. Every consumer authorizes on them:
+  # kube-apiserver RBAC reads `groups`, MinIO reads bucket policies, Grafana
+  # reads grafana_org_role. Role assertion on the ID token is what most OIDC
+  # clients inspect.
+  project_role_assertion = true
+}
+
+resource "zitadel_project_role" "groups" {
+  for_each = local.roles
+
+  org_id       = data.zitadel_org.default.id
+  project_id   = zitadel_project_v2.homelab.id
+  role_key     = each.key
+  display_name = each.key
+  group        = "homelab"
+}
@@ -0,0 +1,53 @@
+# terraform/zitadel — identity configuration for the ZITADEL instance
+#
+# Scope: this root module declares the objects INSIDE ZITADEL (org, project,
+# roles, OIDC applications, machine users, grants). It does NOT deploy ZITADEL
+# itself — that is ArgoCD's job (`k8s/argocd/apps/30-security.yaml`, values in
+# `k8s/infra/iam/zitadel-values.yaml`).
+#
+# Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
+# This file implements §1.3 (roles), §1.4 (machine users), §2.4 (grants) and
+# §6.2 (Terraform provider route). The claim-injection action of §1.3 lives in
+# `actions.tf` and stays disabled until the live go/no-go gate proves it.
+#
+# Authentication to ZITADEL uses the `iam-admin` machine key that the chart's
+# setup job writes to the `iam-admin` Secret in namespace iam:
+#
+#   kubectl -n iam get secret iam-admin -o jsonpath='{.data.iam-admin\.json}' \
+#     | base64 -d > ~/.config/zitadel/iam-admin.json
+#   chmod 600 ~/.config/zitadel/iam-admin.json
+#
+#   export ZITADEL_JWT_PROFILE_FILE=$HOME/.config/zitadel/iam-admin.json
+#   terraform -chdir=terraform/zitadel init
+#   terraform -chdir=terraform/zitadel plan
+#
+# State is LOCAL and git-ignored (`terraform/*.tfstate` in .gitignore). That is
+# a deliberate first step, not the final answer: §8 risk 8 of the plan names
+# state drift as a blast-radius item. Move the state to MinIO before a second
+# operator ever runs `apply`. No secret is written to state except the generated
+# client secrets and machine keys, which are marked sensitive — they still land
+# in the state file, so treat that file as a secret.
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    zitadel = {
+      source  = "zitadel/zitadel"
+      version = "3.8.7"
+    }
+  }
+}
+
+provider "zitadel" {
+  domain = var.zitadel_domain
+  port   = var.zitadel_port
+
+  # JWT profile of the iam-admin machine user. Preferred over a PAT: it is
+  # short-lived per call and never sits in the environment as a bearer token.
+  # Set the path through ZITADEL_JWT_PROFILE_FILE.
+  jwt_profile_file = var.jwt_profile_file
+
+  # The instance serves plain HTTP behind ingress-nginx, which terminates TLS.
+  # Turn this on only if the provider must call the pod directly.
+  insecure = var.zitadel_insecure
+}
@@ -0,0 +1,103 @@
+variable "zitadel_domain" {
+  description = "External domain of the ZITADEL instance, without scheme."
+  type        = string
+  default     = "zitadel.riotpiao.com"
+}
+
+variable "zitadel_port" {
+  description = "Port for the ZITADEL API. 443 behind ingress, 8080 for a direct pod port-forward."
+  type        = string
+  default     = "443"
+}
+
+variable "zitadel_insecure" {
+  description = "Call the API over plain HTTP (set true only when port-forwarding the pod)."
+  type        = bool
+  default     = false
+}
+
+variable "jwt_profile_file" {
+  description = "Path to the iam-admin machine key JSON extracted from the iam-admin Secret."
+  type        = string
+  default     = null
+}
+
+variable "org_id" {
+  description = <<-EOT
+    ID of the ZITADEL organization that holds the project. Read it from the
+    console: https://zitadel.riotpiao.com -> Organization settings -> Resource
+    ID. The chart's setup job creates the default org; this module never creates
+    one, because creating a second org would split the role and grant space.
+  EOT
+  type        = string
+}
+
+variable "project_id" {
+  description = <<-EOT
+    Stable project ID. ZITADEL puts the project ID in the access token `aud`
+    claim, so pinning it keeps audience checks readable across a rebuild.
+    Every consumer that validates `aud` against a client ID must be re-pointed
+    to this value — see the plan §5.2.
+  EOT
+  type        = string
+  default     = "homelab"
+}
+
+variable "project_name" {
+  description = "Display name of the project that carries every role and application."
+  type        = string
+  default     = "homelab"
+}
+
+variable "machine_key_expiry" {
+  description = "Expiry of every machine key, RFC3339. Matches the chart's default iam-admin key window; rotate by changing this value and re-applying."
+  type        = string
+  default     = "2029-01-01T00:00:00Z"
+}
+
+variable "api_app_consumers" {
+  description = <<-EOT
+    Machine users that also need an API application (the client_credentials
+    replacement). Only the gateway does: every other agent signs a JWT with its
+    machine key and does not need a client_credentials grant.
+  EOT
+  type        = map(string)
+  default     = { api-gw = "Go LLM gateway" }
+}
+
+variable "enable_groups_claim_action" {
+  description = <<-EOT
+    Create the Actions v1 script and its trigger that synthesizes the `groups`
+    claim. Keep this false until the live go/no-go gate passes: the claim is
+    load-bearing for kube-apiserver RBAC, MinIO policies, Grafana role mapping
+    and every service that reads `groups`. Turning it on with an unverified
+    script can break every consumer at once.
+  EOT
+  type        = bool
+  default     = false
+}
+
+variable "groups_claim_script" {
+  description = <<-EOT
+    Path to the JavaScript body for the groups-claim action, read only when
+    enable_groups_claim_action is true. The body is deliberately not shipped
+    blind: the ZITADEL v1 Actions runtime API must be confirmed against the live
+    instance before a script is committed, because a wrong call silently emits
+    no claim instead of failing loudly.
+  EOT
+  type        = string
+  default     = null
+
+  validation {
+    # try() is required: HCL does not short-circuit `||`, so fileexists(null)
+    # would fail the whole plan even with the action disabled.
+    condition     = try(fileexists(var.groups_claim_script), true)
+    error_message = "groups_claim_script must name an existing file when it is set."
+  }
+
+  # Enabling the action without a script would reach file(null) inside actions.tf.
+  validation {
+    condition     = !var.enable_groups_claim_action || var.groups_claim_script != null
+    error_message = "enable_groups_claim_action requires groups_claim_script."
+  }
+}
Author	SHA1	Message	Date
rock	6ee22401fd	fix(sops): drop the orphan forgejo-registry-pat file ksops failed the whole sops-secrets application on this file: it declares Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists. The failure blocks every other file in the generator list, including zitadel-masterkey. The same Secret name is already generated in namespace api by forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace forgejo was never created. Remove the file and its generator entry. ArgoCD never created the Secret, so pruning has nothing to remove.	2026-10-24 03:46:29 -07:00
rock	e9817283e6	Merge pull request 'feat(iam): wire Zitadel as the Authentik replacement' (#63 ) from harness/zitadel-iam-wiring into main	2026-10-24 10:41:17 +00:00
rock	6e9586aa15	Merge pull request 'chore(argocd): delete the azhi-riotpiao Application' (#64 ) from chore/remove-azhi-riotpiao-app into main	2026-10-24 10:41:09 +00:00
rock	979a413a5b	feat(iam): read the Zitadel bootstrap token from .env ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the same shape the Authentik variables use. The line is present and empty: the PAT does not exist until the chart's setup job creates the iam-admin machine user on the first sync, so a value cannot be minted ahead of time. The provisioning script loads .env itself instead of relying on the caller to source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN, then the cached file under ~/.config/zitadel, then the cluster Secret. The .env value wins so a human can paste a rotated token without touching the cluster; the cluster Secret remains the only source that always reflects a rotation. .env stays git-ignored. It already holds the live Authentik secrets, and this change does not move them. Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.	2026-10-24 03:38:21 -07:00
rock	2f62351731	feat(iam): one script to provision Zitadel end to end scripts/iam/zitadel-provision.py replaces the manual sequence that provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users, verify. `all` runs keys -> apply -> secrets -> verify; users stays manual because the password hash must come from a human, one user at a time. - keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel at 0600, outside the repo. - apply: terraform apply terraform/zitadel with the machine key. - secrets: write generated client secrets into the SOPS files, write client ids into the plain manifests that carry them, write machine keys per agent. Requires an exactly-one-match regex before it edits a manifest, prints the diff without --apply-config, and is a no-op when the value already matches. - users: import a human user with the Django pbkdf2_sha256 hash converted to the passwap MCF form ZITADEL accepts. - verify: discovery endpoint, client count, and it refuses to clear cutover while the groups-claim action is disabled. Design points that came out of review: - Multi-document SOPS files (minio) are rewritten whole, because `sops set` cannot address one document. sops reads the creation rule from the filename, so the temp file is passed with --filename-override; without it sops finds no matching rule and refuses. Ciphertext is staged in the destination directory and installed with os.replace, so the mode is preserved and EXDEV is impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file. - homarr is listed as manual: the live workload mounts Secret homarr-oidc, which no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no consumer. Writing the new secret into the orphan file would report success while the dashboard kept the old one. - Secrets never enter argv: curl reads its config from stdin, so the PAT and the password hash do not appear in ps. .gitignore: allow this one script through the scripts/iam/*.py ignore rule (it embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M tokens and was one `git add -A` away from being committed. Checks: py_compile clean, --help lists every phase, convert_hash verified on a synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents survive, value set, second run unchanged, mode 644 kept, no temp left), all three client-id regexes match exactly once, dry-run leaves the manifests untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr Secret, the missing client-id writes and argv exposure; all fixed and re-reviewed to SIGN-OFF. Not run against a live ZITADEL: no instance exists yet.	2026-10-24 03:38:21 -07:00
rock	f81ff0e2fc	feat(iam): declare Zitadel identity objects in Terraform Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open. It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users plus keys, one API application for the gateway, project grants, per-agent claim metadata, and a disabled groups-claim action. Facts established while writing it: - Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest has no client_id field, and provider 3.8.7 rejects a settable application id (terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a client_id-preserving re-point; every consumer now also changes client ID. - Role keys equal the existing group names, so the groups claim can be byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200 characters with no character-set restriction. - Machine users get JWT profile keys, not passwords. Per-agent claims move to user metadata and reach tokens only through the groups-claim action. - The action stays gated: enable_groups_claim_action defaults to false because the action runtime API is unverified. Two validations stop a null script from reaching file(). Checks: terraform fmt -check -recursive clean, terraform validate 0 errors, plan reaches only the expected missing-credentials error, structural gate 0 error / 0 warning, and one reviewer round: BLOCK on a plan-time fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed to SIGN-OFF. State is local and git-ignored; it carries generated client secrets and machine keys. Move it to MinIO before a second operator applies. Not applied: no Zitadel instance is running yet.	2026-10-24 03:38:21 -07:00
rock	cb1219bf96	feat(iam): wire Zitadel into namespace iam under ArgoCD Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam. Authentik keeps serving; no consumer is moved and no Authentik object changes. - k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances - k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat - k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3, automated sync deliberately absent for the first sync (chart PreSync hook re-run risk) - k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS - k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI, crash-loop and PreSync-job-failed rules - terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1, authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and kube-state-metrics has 1925 restarts. Two manual steps still gate the first sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned). The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it to sealed storage and delete it before the first sync. Losing it makes every encrypted ZITADEL value unreadable.	2026-10-24 03:38:21 -07:00
rock	8e9ac37583	chore(argocd): delete the azhi-riotpiao Application The Application pointed at github.com/ToyoGlobal/azhi-infra, branch feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled: status stayed Unknown because the source repository could not be read, and the destination namespace azi-riotpiao was never created. ArgoCD reported it as an OutOfSync managed resource of homelab-root with nothing behind it. Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from that repository is referenced any more, so the allowlist entry is dead weight. Not touched here, because it is not in git: Secret azhi-infra-github in namespace argocd holds the read-only GitHub credential for that repository. Delete it separately once no other Application needs the repo. The live Application object was deleted by hand. Until this commit reaches main, the next homelab-root sync recreates it, because main still declares it.	2026-10-24 03:35:52 -07:00
rock	3c851b16c9	Merge branch 'fix/certmanager-dns01' Restores wildcard renewal: pin the ACME DNS-01 self-check to the public resolvers and drop the per-host TLS blocks that competed with the wildcard.	2026-10-24 02:39:38 -07:00
rock	e93d4ba86c	fix(cert-manager): restore wildcard renewal and drop per-host TLS The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with "x509: certificate has expired or is not yet valid". Changes: - cert-manager: pin the self-check to the public resolvers with --dns01-recursive-nameservers and --dns01-recursive-nameservers-only. Verified against Let's Encrypt staging: the challenge went valid and the certificate issued. - comfyui and forgejo: drop the per-host tls blocks and the cert-manager.io/cluster-issuer annotations. The wildcard covers both hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired with the wildcard. - riotpiao-com-cert: document the wildcard as nginx's default certificate and the zero-downtime renewal path. Renewal now runs 30 days before expiry, writes the same secret, and nginx hot-reloads without a pod restart.	2026-10-24 02:30:48 -07:00
rock	7a1213d504	Merge pull request 'fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2' (#62 ) from fix/retire-longhorn-disk4 into main	2026-10-13 00:10:01 +00:00
rock	40aa23dfdf	Merge pull request 'test(argocd): point azhi-riotpiao at the CNPG branch' (#61 ) from feat/azhi-cnpg-test into main	2026-10-12 13:41:38 +00:00
rock	71fce30506	test(argocd): point azhi-riotpiao at the CNPG branch The StatefulSet database could not attach its Longhorn volume, so the branch runs the database as a CNPG Cluster instead. This pins the Application to that branch for the test; it returns to the default branch once the branch merges.	2026-10-12 21:24:47 +08:00
rock	0666715a5e	Merge pull request 'feat(argocd): reconcile azhi-infra from its private GitHub repo' (#60 ) from feat/azhi-riotpiao-argocd into main	2026-10-12 12:54:43 +00:00
rock	949ae5cb38	feat(argocd): let ArgoCD reconcile azhi-infra from its private GitHub repo The homelab AppProject allowlist had no azhi repository, so an Application pointing at ToyoGlobal/azhi-infra fails admission with 'repository not permitted'. Adding the URL is the whole gate. The Application itself targets the repository default branch (harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That overlay renders postgres and the API into namespace azi-riotpiao. The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR pull credential and azhi-riotpiao-app-env carries the database credentials, the JWT secrets and the AES key.	2026-10-12 20:54:00 +08:00
rock	e150969070	Merge pull request #59 from fix/etcd-defrag-main	2026-10-12 12:27:28 +00:00
rock	abfdda5f57	fix(etcd): add a working 15-day defrag CronJob under the backend quota etcd's backend file only grows: the apiserver compacts every 5m, but compaction marks superseded revisions free inside the file, and only defrag returns that space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached 2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets could not post node status (3 nodes NotReady), apiserver writes returned 500 and the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after compaction - the rest was dead pages. Compaction had also stalled while the alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have reclaimed nothing. Compact must come first. An earlier revision of this CronJob never ran, for two independent reasons, both fixed here: - its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] failed with exec: "/bin/sh": no such file or directory - it existed only on a side branch while the ArgoCD application cluster-maintenance tracks main, so the cluster never had it Contents: - CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the day-of-month field fires on the 1st, 16th and 31st, putting two runs one day apart. - Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4 against the 3.6.11 servers. - Compaction uses the MINIMUM revision across members, so a lagging member is never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat ~1.2M entries behind the leader with a frozen raft term and needed a reboot to rejoin. - Defrag skips any member whose applied index lags the leader by 100k+, and runs one member at a time since defrag pauses that member's apply loop. - Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a client certificate signed by the cluster's etcd CA, valid to 2036-10.	2026-10-07 17:12:39 +08:00
rock	2d9e3abee8	fix(oidc): match /proc/<pid>/comm instead of cmdline when signalling the API server Exit code 143 proved the previous job killed itself: the script's own cmdline contains the string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver') and skipping $$ removes that failure mode. Staggered 2 minutes apart.	2026-10-07 04:07:37 +00:00
rock	9fa8d9cf60	fix(oidc): restart the apiservers by signalling the process, not by deleting the mirror pod Static pods survive mirror-pod deletion, so the first bootstrap job had no effect. These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself, staggered by 100s so only one control plane restarts at a time.	2026-10-07 03:59:22 +00:00
rock	2e4fad3bf7	fix(oidc): use curlimages/curl for the bootstrap job (bitnami/kubectl tag did not exist) The first job could not pull its image; this one deletes the apiserver pods via the in-cluster API with the mounted service-account token instead of a kubectl image.	2026-10-07 03:52:10 +00:00
rock	5d22691c3e	chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator k8s/argocd/apps/99-oidc-bootstrap.yaml	2026-10-07 03:46:01 +00:00
rock	4eee161e0f	chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator k8s/infra/oidc-bootstrap/bootstrap.yaml	2026-10-07 03:45:56 +00:00
rock	719fa77521	Merge pull request 'fix(secrets): populate the empty sops placeholders and drop the unused one' (#55 ) from fix/secrets-placeholders into main	2026-10-06 04:48:51 +00:00
rock	64cd1f2c4e	fix(minio): freeze the community image by digest and alert on pull failure (#57 ) Co-authored-by: rock <[email protected]>	2026-10-06 03:43:12 +00:00
rock	f780e31f9c	Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent)	2026-10-06 10:05:45 +08:00
rock	c93a24f786	fix(secrets): populate the empty sops placeholders and drop the unused one (cherry picked from commit `f6207ff2b0`)	2026-10-06 10:03:04 +08:00