feat(iam): wire Zitadel into namespace iam under ArgoCD

Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.

- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
  secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
  automated sync deliberately absent for the first sync (chart PreSync hook
  re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
  crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host

Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).

The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.

k8s/infra/databases/kustomization.yaml

@@ -7,5 +7,6 @@ kind: Kustomization
 resources:
   - namespaces.yaml
   - authentik-db.yaml
+  - zitadel-db.yaml
   - temporal-db.yaml
   - paperless-db.yaml

k8s/infra/databases/zitadel-db.yaml

@@ -0,0 +1,42 @@
+# Dedicated CNPG Postgres for ZITADEL (GitOps, wave 2 — before the zitadel
+# chart at wave 3). Same shape as authentik-db.yaml: CNPG generates the secret
+# `zitadel-db-app` and the service `zitadel-db-rw` in ns iam, and
+# k8s/infra/iam/zitadel-values.yaml reads the DSN from that secret locally (no
+# cross-namespace secret copy).
+#
+# ZITADEL requires PostgreSQL 14+; the repo convention image is 16.2.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: zitadel-db
+  namespace: iam
+  annotations:
+    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
+  bootstrap:
+    initdb:
+      database: zitadel
+      owner: app
+      encoding: UTF8
+      localeCollate: C
+      localeCType: C
+  enableSuperuserAccess: false
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+  storage:
+    size: 10Gi
+    storageClass: longhorn-cnpg
+  monitoring:
+    enablePodMonitor: true
+  affinity:
+    # preferred + tolerations: HA across nodes without deadlocking a partly-
+    # schedulable 3-CP (same as authentik-db and forgejo-db).
+    podAntiAffinityType: preferred
+    topologyKey: kubernetes.io/hostname
+    tolerations:
+      - key: node-role.kubernetes.io/control-plane
+        operator: Exists
+        effect: NoSchedule

k8s/infra/iam/zitadel-values.yaml

@@ -0,0 +1,134 @@
+# k8s/infra/iam/zitadel-values.yaml
+# ZITADEL — Authentik replacement. Coexistence is deliberate: Authentik keeps
+# running, read-only, in the same `iam` namespace until the last consumer moves.
+# Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
+#
+# Chart: zitadel/zitadel 10.4.0, app v4.19.2. The plan cites v4.19.4; the chart
+# has not published it yet. Pin the chart, never `*`, until the v4.19.4 chart
+# exists — a silent app bump invalidates the verified masterkey/passwap facts.
+#
+# URL: https://zitadel.riotpiao.com. TLS terminates at ingress-nginx, which
+# serves the wildcard riotpiao-com-tls as its default-ssl-certificate, so
+# ExternalSecure=true and TLS.Enabled=false (no per-rule tls: block needed).
+#
+# Database: CNPG cluster zitadel-db (wave 2). The DSN comes from CNPG's
+# generated `zitadel-db-app` secret. When ZITADEL_DATABASE_POSTGRES_DSN is set,
+# ZITADEL ignores the discrete Database.Postgres.* fields, so neither a
+# password nor a host is written into this file.
+#
+# Masterkey: Secret `zitadel-masterkey` (k8s/argocd/secrets/
+# zitadel-masterkey.enc.yaml), key `masterkey`, exactly 32 bytes. It wraps the
+# rows in system.encryption_keys, which encrypt every client secret and private
+# key. Losing it makes those values unreadable: keep the offline sealed copy.
+#
+# Ingress uses the chart's own objects (controller: nginx adds the gRPC backend
+# annotation ZITADEL needs for its HTTP/2 API). Do not hand-write a second
+# Ingress for this host.
+
+zitadel:
+  configmapConfig:
+    ExternalSecure: true
+    ExternalDomain: zitadel.riotpiao.com
+    TLS:
+      Enabled: false
+
+  # Rendered into a Secret by the chart; read by pods and init jobs alike.
+  masterkeySecretName: zitadel-masterkey
+
+  resources:
+    requests: { memory: "512Mi", cpu: "250m" }
+    limits: { memory: "2Gi", cpu: "1" }
+
+  # FirstInstance defaults stay for the FIRST sync: the chart creates org
+  # `zitadel` plus the `iam-admin` machine user and its machine-key Secret.
+  # Terraform consumes that key to declare projects, roles, OIDC apps and the
+  # groups action.
+  #
+  # SECOND-SYNC HAZARD — read before enabling automated sync.
+  # The chart marks eight objects as `pre-install,pre-upgrade` hooks (two
+  # ServiceAccounts, two ConfigMaps, Role, RoleBinding, zitadel-init,
+  # zitadel-setup), and one cleanup Job as `post-delete`. ArgoCD maps the first
+  # eight to PreSync with `before-hook-creation`: it deletes and re-creates them
+  # on EVERY sync, not only on a chart change.
+  #   - zitadel-init  (`init zitadel`) — schema step
+  #   - zitadel-setup (`setup`)        — first instance + iam-admin machine key
+  # On an initialized instance the setup step can fail, which aborts the sync
+  # before the Deployment is applied. If it succeeds instead, its
+  # machinekeyWriter container re-applies the iam-admin key Secret that
+  # Terraform reads, which silently rotates a credential in use.
+  # After the first sync reports healthy:
+  #   1. set configmapConfig.FirstInstance.Skip: true here — permanently: the
+  #      setup steps are schema-only once the instance exists, and Skip alone
+  #      stops the machinekeyWriter rotation
+  #   2. set initJob.enabled: false below
+  #   3. commit, sync once, then enable automated+selfHeal on the Application
+  # For a later schema upgrade, re-enable initJob.enabled only — never set
+  # FirstInstance.Skip back to false.
+  # ArgoCD never prunes a hook Job, so a Job that failed before step 2 stays and
+  # keeps ZitadelSetupJobFailed firing until a human deletes it.
+
+# DSN mode (chart renders this through the zitadel.dbEnv helper). CNPG's `uri`
+# key is a postgresql:// URL, which ZITADEL accepts as a DSN.
+env:
+  - name: ZITADEL_DATABASE_POSTGRES_DSN
+    valueFrom:
+      secretKeyRef:
+        name: zitadel-db-app
+        key: uri
+
+# Bundled Bitnami Postgres is disabled: the CNPG cluster above is the system of
+# record. Enabling it would create a second, unmanaged database.
+postgresql:
+  enabled: false
+
+# CNPG already created the database, the `app` user and its grants, so the init
+# job runs only the ZITADEL-internal step. Running the full default command
+# would try to create a database and a superuser it does not own.
+initJob:
+  command: "zitadel"
+  activeDeadlineSeconds: 600
+
+# Two API replicas: the read path survives one node drain, and both share the
+# same masterkey and database. The login UI stays at one replica — a session
+# cookie is not shared, so a second pod buys nothing.
+replicaCount: 2
+revisionHistoryLimit: 3
+
+ingress:
+  enabled: true
+  className: nginx
+  controller: nginx
+
+login:
+  enabled: true
+  replicaCount: 1
+  ingress:
+    enabled: true
+    className: nginx
+
+# Chart-owned ServiceMonitor. The cluster's Prometheus runs with
+# serviceMonitorSelectorNilUsesHelmValues=false, so the release label is not
+# required for discovery. Endpoint is /debug/metrics on port http2-server.
+metrics:
+  enabled: true
+  serviceMonitor:
+    enabled: true
+    scrapeInterval: 30s
+    scrapeTimeout: 10s
+
+# The 3-CP / 2-worker cluster mixes control-plane taints with worker capacity.
+# Without the toleration a rolling upgrade can leave both replicas Pending.
+tolerations:
+  - key: node-role.kubernetes.io/control-plane
+    operator: Exists
+    effect: NoSchedule
+
+affinity:
+  podAntiAffinity:
+    preferredDuringSchedulingIgnoredDuringExecution:
+      - weight: 100
+        podAffinityTerm:
+          topologyKey: kubernetes.io/hostname
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: zitadel

k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml

@@ -0,0 +1,68 @@
+# ZITADEL alert parity with the Authentik rules it will replace.
+#
+# Only two signals are ported at this point. The metrics endpoint
+# (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
+# no request or error counters that this cluster has verified, so no error-rate
+# rule is written here. Writing one against a guessed series name would produce
+# a permanently-firing or permanently-silent alert — both worse than none.
+# AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
+# carries zero providers and is deleted with the migration, not migrated.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: zitadel-rules
+  namespace: iam
+spec:
+  groups:
+    - name: zitadel.rules
+      interval: 30s
+      rules:
+        - alert: ZitadelDown
+          # absent() is required: equality alone cannot fire when the series is
+          # gone (ServiceMonitor deleted, Service selector changed, pods scaled
+          # to zero). Those are exactly the outages a target-down rule exists for.
+          expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "ZITADEL API target is down"
+            description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
+
+        - alert: ZitadelLoginUIDown
+          expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "ZITADEL login UI has no available replica"
+            description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
+
+        - alert: ZitadelRestarting
+          # pod!~"zitadel-db-.*" matters: the CNPG database pods share the
+          # `zitadel` name prefix, and a database restart must not page as an
+          # IdP crash loop.
+          expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "ZITADEL pod restarts more than 3 times per hour"
+            description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
+
+        # The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
+        # restartPolicy: Never, so their restart counter stays 0. Without this
+        # rule a failed schema init or first-instance setup reaches nobody but
+        # the person who runs `kubectl -n iam get job`.
+        # No absent() guard here on purpose: the chart ships only
+        # `before-hook-creation`, so a successful hook Job is replaced at the
+        # next PreSync. Once the jobs are retired, absent() would fire on a
+        # healthy cluster.
+        - alert: ZitadelSetupJobFailed
+          expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "ZITADEL init/setup job failed"
+            description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"

k8s/infra/monitoring/kustomization.yaml

@@ -8,6 +8,7 @@ resources:
   - alerts/ingress-alerts.yaml
   - alerts/svc-argocd-rules.yaml
   - alerts/svc-authentik-rules.yaml
+  - alerts/svc-zitadel-rules.yaml
   - alerts/svc-forgejo-rules.yaml
   - alerts/svc-grafana-rules.yaml
   - alerts/svc-minio-rules.yaml