feat(iam): wire Zitadel into namespace iam under ArgoCD
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam. Authentik keeps serving; no consumer is moved and no Authentik object changes. - k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances - k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat - k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3, automated sync deliberately absent for the first sync (chart PreSync hook re-run risk) - k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS - k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI, crash-loop and PreSync-job-failed rules - terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1, authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and kube-state-metrics has 1925 restarts. Two manual steps still gate the first sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned). The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it to sealed storage and delete it before the first sync. Losing it makes every encrypted ZITADEL value unreadable.
This commit is contained in:
@@ -57,6 +57,55 @@ spec:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
---
|
||||
# ZITADEL — the Authentik replacement. Coexistence, not cutover: Authentik keeps
|
||||
# serving in the same `iam` namespace until the last consumer moves, then this
|
||||
# becomes the only IdP. Migration plan (outside git):
|
||||
# ~/workplace/homelab-doc/ZITADEL_MIGRATION.html
|
||||
#
|
||||
# Wave 3 ordering: sops-secrets (wave 0) creates the `zitadel-masterkey` Secret,
|
||||
# databases (wave 2) creates the `zitadel-db` CNPG cluster and its
|
||||
# `zitadel-db-app` DSN secret. Both must exist before the init job starts.
|
||||
#
|
||||
# automated sync is deliberately absent for the first sync — same reason as
|
||||
# forgejo. The chart ships `pre-install,pre-upgrade` Helm hooks (zitadel-init,
|
||||
# zitadel-setup), which ArgoCD maps to PreSync and re-runs on every sync, manual
|
||||
# or automated. A human reads the first sync's hook output before any repeat can
|
||||
# run. The init job runs `init zitadel` only (CNPG owns the database and the app
|
||||
# user), and the setup job creates the first org and the `iam-admin` machine key
|
||||
# that Terraform consumes. Read the SECOND-SYNC HAZARD block at the top of
|
||||
# k8s/infra/iam/zitadel-values.yaml before enabling automated sync.
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: zitadel
|
||||
namespace: argocd
|
||||
finalizers:
|
||||
- resources-finalizer.argocd.argoproj.io
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "3"
|
||||
spec:
|
||||
project: homelab
|
||||
revisionHistoryLimit: 3
|
||||
sources:
|
||||
- repoURL: https://charts.zitadel.com
|
||||
chart: zitadel
|
||||
# Pinned: the chart trails the upstream app release, and the verified
|
||||
# masterkey/passwap facts were read at v4.19.x. A floating tag would
|
||||
# change the app version without a diff anyone reviews.
|
||||
targetRevision: 10.4.0
|
||||
helm:
|
||||
valueFiles:
|
||||
- $values/k8s/infra/iam/zitadel-values.yaml
|
||||
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
|
||||
targetRevision: main
|
||||
ref: values
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: iam
|
||||
syncPolicy:
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
---
|
||||
# Raw IAM manifests: key-rotation cronjob + authentik migration job.
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
|
||||
@@ -38,3 +38,4 @@ files:
|
||||
- forgejo-registry-token.enc.yaml
|
||||
- pi-sandbox-secrets.enc.yaml
|
||||
- etcd-defrag-creds.enc.yaml
|
||||
- zitadel-masterkey.enc.yaml
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: zitadel-masterkey
|
||||
namespace: iam
|
||||
type: Opaque
|
||||
stringData:
|
||||
masterkey: ENC[AES256_GCM,data:dGp0wSCUJORP6W5PHLKYaQEqrZsgiAwU1GqcuvuZV+g=,iv:Wwt6E/XXwsoVEQxe4gG1m5oJLOcNUXngftADQvZQepw=,tag:1Vget5gs+t455U9zLavdwg==,type:str]
|
||||
sops:
|
||||
age:
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBld0tUWHNWem13cDlxRG14
|
||||
QWRlTjA5SzZnS2pBd2M0WWkzQWxKZW9HZWdJCk00QWJJcWZFM1ZTSUlReUlGZG4z
|
||||
RFBSUWRlVUx2NGN2OFAwazBYRTVVNkEKLS0tIHpSSW9IWHhNa0d3SmEydXU4akVj
|
||||
MGJkN2U4WmtLbVFMZm55U1hEZElZMlUKKV9twnC9GC7MSmNLxEBHbPl4RN5Hsd+z
|
||||
NViCKxUoQAB8apn2XVHZcDnyfKNfr1P3aSa3IIHyVY3prgo1OFDpdw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
lastmodified: "2026-10-24T08:57:27Z"
|
||||
mac: ENC[AES256_GCM,data:hXDAddVmnlf/yinY2c2Q8yfsjqi6n9cRbEbwykqYScd6mDPfdF7BriTiu9K+CV/IvIT4zcMTZbhBeQ/vTCAh0TZDp1lOVglwtNH/imgagyzPmcCvbCIkZKwF5vwWQIGHcDI6pk4/zVQkqt7zKTf8+Qj0pNh5UrAjVeEFE7cuJkQ=,iv:oIQvROFGbiESmMTthHOJiNM+Gy5BCfM5XtDWbszOPmc=,tag:AaYrJrwXuKD7w0p+TTa7FA==,type:str]
|
||||
version: 3.13.2
|
||||
@@ -7,5 +7,6 @@ kind: Kustomization
|
||||
resources:
|
||||
- namespaces.yaml
|
||||
- authentik-db.yaml
|
||||
- zitadel-db.yaml
|
||||
- temporal-db.yaml
|
||||
- paperless-db.yaml
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# Dedicated CNPG Postgres for ZITADEL (GitOps, wave 2 — before the zitadel
|
||||
# chart at wave 3). Same shape as authentik-db.yaml: CNPG generates the secret
|
||||
# `zitadel-db-app` and the service `zitadel-db-rw` in ns iam, and
|
||||
# k8s/infra/iam/zitadel-values.yaml reads the DSN from that secret locally (no
|
||||
# cross-namespace secret copy).
|
||||
#
|
||||
# ZITADEL requires PostgreSQL 14+; the repo convention image is 16.2.
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: zitadel-db
|
||||
namespace: iam
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
|
||||
spec:
|
||||
instances: 3
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: zitadel
|
||||
owner: app
|
||||
encoding: UTF8
|
||||
localeCollate: C
|
||||
localeCType: C
|
||||
enableSuperuserAccess: false
|
||||
resources:
|
||||
requests: { memory: "512Mi", cpu: "250m" }
|
||||
limits: { memory: "2Gi", cpu: "1" }
|
||||
storage:
|
||||
size: 10Gi
|
||||
storageClass: longhorn-cnpg
|
||||
monitoring:
|
||||
enablePodMonitor: true
|
||||
affinity:
|
||||
# preferred + tolerations: HA across nodes without deadlocking a partly-
|
||||
# schedulable 3-CP (same as authentik-db and forgejo-db).
|
||||
podAntiAffinityType: preferred
|
||||
topologyKey: kubernetes.io/hostname
|
||||
tolerations:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
@@ -0,0 +1,134 @@
|
||||
# k8s/infra/iam/zitadel-values.yaml
|
||||
# ZITADEL — Authentik replacement. Coexistence is deliberate: Authentik keeps
|
||||
# running, read-only, in the same `iam` namespace until the last consumer moves.
|
||||
# Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
|
||||
#
|
||||
# Chart: zitadel/zitadel 10.4.0, app v4.19.2. The plan cites v4.19.4; the chart
|
||||
# has not published it yet. Pin the chart, never `*`, until the v4.19.4 chart
|
||||
# exists — a silent app bump invalidates the verified masterkey/passwap facts.
|
||||
#
|
||||
# URL: https://zitadel.riotpiao.com. TLS terminates at ingress-nginx, which
|
||||
# serves the wildcard riotpiao-com-tls as its default-ssl-certificate, so
|
||||
# ExternalSecure=true and TLS.Enabled=false (no per-rule tls: block needed).
|
||||
#
|
||||
# Database: CNPG cluster zitadel-db (wave 2). The DSN comes from CNPG's
|
||||
# generated `zitadel-db-app` secret. When ZITADEL_DATABASE_POSTGRES_DSN is set,
|
||||
# ZITADEL ignores the discrete Database.Postgres.* fields, so neither a
|
||||
# password nor a host is written into this file.
|
||||
#
|
||||
# Masterkey: Secret `zitadel-masterkey` (k8s/argocd/secrets/
|
||||
# zitadel-masterkey.enc.yaml), key `masterkey`, exactly 32 bytes. It wraps the
|
||||
# rows in system.encryption_keys, which encrypt every client secret and private
|
||||
# key. Losing it makes those values unreadable: keep the offline sealed copy.
|
||||
#
|
||||
# Ingress uses the chart's own objects (controller: nginx adds the gRPC backend
|
||||
# annotation ZITADEL needs for its HTTP/2 API). Do not hand-write a second
|
||||
# Ingress for this host.
|
||||
|
||||
zitadel:
|
||||
configmapConfig:
|
||||
ExternalSecure: true
|
||||
ExternalDomain: zitadel.riotpiao.com
|
||||
TLS:
|
||||
Enabled: false
|
||||
|
||||
# Rendered into a Secret by the chart; read by pods and init jobs alike.
|
||||
masterkeySecretName: zitadel-masterkey
|
||||
|
||||
resources:
|
||||
requests: { memory: "512Mi", cpu: "250m" }
|
||||
limits: { memory: "2Gi", cpu: "1" }
|
||||
|
||||
# FirstInstance defaults stay for the FIRST sync: the chart creates org
|
||||
# `zitadel` plus the `iam-admin` machine user and its machine-key Secret.
|
||||
# Terraform consumes that key to declare projects, roles, OIDC apps and the
|
||||
# groups action.
|
||||
#
|
||||
# SECOND-SYNC HAZARD — read before enabling automated sync.
|
||||
# The chart marks eight objects as `pre-install,pre-upgrade` hooks (two
|
||||
# ServiceAccounts, two ConfigMaps, Role, RoleBinding, zitadel-init,
|
||||
# zitadel-setup), and one cleanup Job as `post-delete`. ArgoCD maps the first
|
||||
# eight to PreSync with `before-hook-creation`: it deletes and re-creates them
|
||||
# on EVERY sync, not only on a chart change.
|
||||
# - zitadel-init (`init zitadel`) — schema step
|
||||
# - zitadel-setup (`setup`) — first instance + iam-admin machine key
|
||||
# On an initialized instance the setup step can fail, which aborts the sync
|
||||
# before the Deployment is applied. If it succeeds instead, its
|
||||
# machinekeyWriter container re-applies the iam-admin key Secret that
|
||||
# Terraform reads, which silently rotates a credential in use.
|
||||
# After the first sync reports healthy:
|
||||
# 1. set configmapConfig.FirstInstance.Skip: true here — permanently: the
|
||||
# setup steps are schema-only once the instance exists, and Skip alone
|
||||
# stops the machinekeyWriter rotation
|
||||
# 2. set initJob.enabled: false below
|
||||
# 3. commit, sync once, then enable automated+selfHeal on the Application
|
||||
# For a later schema upgrade, re-enable initJob.enabled only — never set
|
||||
# FirstInstance.Skip back to false.
|
||||
# ArgoCD never prunes a hook Job, so a Job that failed before step 2 stays and
|
||||
# keeps ZitadelSetupJobFailed firing until a human deletes it.
|
||||
|
||||
# DSN mode (chart renders this through the zitadel.dbEnv helper). CNPG's `uri`
|
||||
# key is a postgresql:// URL, which ZITADEL accepts as a DSN.
|
||||
env:
|
||||
- name: ZITADEL_DATABASE_POSTGRES_DSN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: zitadel-db-app
|
||||
key: uri
|
||||
|
||||
# Bundled Bitnami Postgres is disabled: the CNPG cluster above is the system of
|
||||
# record. Enabling it would create a second, unmanaged database.
|
||||
postgresql:
|
||||
enabled: false
|
||||
|
||||
# CNPG already created the database, the `app` user and its grants, so the init
|
||||
# job runs only the ZITADEL-internal step. Running the full default command
|
||||
# would try to create a database and a superuser it does not own.
|
||||
initJob:
|
||||
command: "zitadel"
|
||||
activeDeadlineSeconds: 600
|
||||
|
||||
# Two API replicas: the read path survives one node drain, and both share the
|
||||
# same masterkey and database. The login UI stays at one replica — a session
|
||||
# cookie is not shared, so a second pod buys nothing.
|
||||
replicaCount: 2
|
||||
revisionHistoryLimit: 3
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
controller: nginx
|
||||
|
||||
login:
|
||||
enabled: true
|
||||
replicaCount: 1
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
|
||||
# Chart-owned ServiceMonitor. The cluster's Prometheus runs with
|
||||
# serviceMonitorSelectorNilUsesHelmValues=false, so the release label is not
|
||||
# required for discovery. Endpoint is /debug/metrics on port http2-server.
|
||||
metrics:
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
scrapeInterval: 30s
|
||||
scrapeTimeout: 10s
|
||||
|
||||
# The 3-CP / 2-worker cluster mixes control-plane taints with worker capacity.
|
||||
# Without the toleration a rolling upgrade can leave both replicas Pending.
|
||||
tolerations:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
- weight: 100
|
||||
podAffinityTerm:
|
||||
topologyKey: kubernetes.io/hostname
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: zitadel
|
||||
@@ -0,0 +1,68 @@
|
||||
# ZITADEL alert parity with the Authentik rules it will replace.
|
||||
#
|
||||
# Only two signals are ported at this point. The metrics endpoint
|
||||
# (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
|
||||
# no request or error counters that this cluster has verified, so no error-rate
|
||||
# rule is written here. Writing one against a guessed series name would produce
|
||||
# a permanently-firing or permanently-silent alert — both worse than none.
|
||||
# AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
|
||||
# carries zero providers and is deleted with the migration, not migrated.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: zitadel-rules
|
||||
namespace: iam
|
||||
spec:
|
||||
groups:
|
||||
- name: zitadel.rules
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: ZitadelDown
|
||||
# absent() is required: equality alone cannot fire when the series is
|
||||
# gone (ServiceMonitor deleted, Service selector changed, pods scaled
|
||||
# to zero). Those are exactly the outages a target-down rule exists for.
|
||||
expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "ZITADEL API target is down"
|
||||
description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
|
||||
|
||||
- alert: ZitadelLoginUIDown
|
||||
expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "ZITADEL login UI has no available replica"
|
||||
description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
|
||||
|
||||
- alert: ZitadelRestarting
|
||||
# pod!~"zitadel-db-.*" matters: the CNPG database pods share the
|
||||
# `zitadel` name prefix, and a database restart must not page as an
|
||||
# IdP crash loop.
|
||||
expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "ZITADEL pod restarts more than 3 times per hour"
|
||||
description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
|
||||
|
||||
# The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
|
||||
# restartPolicy: Never, so their restart counter stays 0. Without this
|
||||
# rule a failed schema init or first-instance setup reaches nobody but
|
||||
# the person who runs `kubectl -n iam get job`.
|
||||
# No absent() guard here on purpose: the chart ships only
|
||||
# `before-hook-creation`, so a successful hook Job is replaced at the
|
||||
# next PreSync. Once the jobs are retired, absent() would fire on a
|
||||
# healthy cluster.
|
||||
- alert: ZitadelSetupJobFailed
|
||||
expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "ZITADEL init/setup job failed"
|
||||
description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"
|
||||
@@ -8,6 +8,7 @@ resources:
|
||||
- alerts/ingress-alerts.yaml
|
||||
- alerts/svc-argocd-rules.yaml
|
||||
- alerts/svc-authentik-rules.yaml
|
||||
- alerts/svc-zitadel-rules.yaml
|
||||
- alerts/svc-forgejo-rules.yaml
|
||||
- alerts/svc-grafana-rules.yaml
|
||||
- alerts/svc-minio-rules.yaml
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
# via nginx (not the backend service) is critical: a direct rewrite bypasses
|
||||
# nginx TLS and exposes each app's own self-signed cert, trusted by nothing.
|
||||
rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
|
||||
rewrite name zitadel.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
|
||||
rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
|
||||
rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
|
||||
rewrite name minio-api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
|
||||
|
||||
Reference in New Issue
Block a user