feat(iam): wire Zitadel as the Authentik replacement #63

2026-10-24T10:38:41Z

rock commented

2026-10-24 10:38:41 +00:00

What this adds

Zitadel as the Authentik replacement, wired but not yet authoritative. Authentik keeps serving: no consumer is moved, no Authentik object changes.

Three layers:

ArgoCD wiring — zitadel Application (wave 3), zitadel-db CNPG cluster (wave 2), SOPS masterkey, Helm values, metrics ServiceMonitor, four alerts, CoreDNS rewrite.
Terraform — terraform/zitadel/: 1 project, 19 project roles, 12 OIDC applications, 5 machine users with JWT keys, 1 API application for the gateway, project grants, per-agent claim metadata, gated groups-claim action.
Provisioning script — scripts/iam/zitadel-provision.py: keys, apply, secrets, users, verify.

Facts that change the plan

Client IDs change. ZITADEL v4.19.2 CreateOIDCApplicationRequest has no client_id field, and provider 3.8.7 rejects a settable application id. The migration plan assumed a secret-only re-point. Every consumer needs its client ID updated too.
The groups claim is unproven. ZITADEL issues no groups claim, and kube-apiserver RBAC, MinIO and Grafana all read it. The action resource exists and stays disabled until the live gate passes.
The chart ships PreSync hooks. zitadel-init and zitadel-setup re-run on every sync. zitadel-values.yaml documents the two follow-up edits that stop the setup job from rewriting the iam-admin key Terraform reads.

Not applied by this PR

The Application has no automated sync. Merging creates zitadel-db through the databases app and creates the Application object. Someone then syncs it by hand.

Verification

helm template renders 17 objects, all in namespace iam.
kubectl apply --dry-run=server accepts the CNPG cluster and the full chart render.
kubectl kustomize builds k8s/infra/databases/, k8s/infra/monitoring/, k8s/infra/iam/.
terraform fmt -check -recursive clean, terraform validate 0 errors.
structural-check.sh 0 error / 0 warning on every changed manifest.
Reviewer rounds: two BLOCK verdicts, all findings fixed, then SIGN-OFF.

Manual steps after merge

Cloudflare Tunnel public hostname for zitadel.riotpiao.com plus a proxied CNAME to the tunnel.
terraform apply and make apply-cp for the CoreDNS rewrite (Talos-owned, not ArgoCD).
Move the offline masterkey copy (~/zitadel-masterkey.offline) into sealed storage.

## What this adds Zitadel as the Authentik replacement, wired but not yet authoritative. Authentik keeps serving: no consumer is moved, no Authentik object changes. Three layers: 1. **ArgoCD wiring** — `zitadel` Application (wave 3), `zitadel-db` CNPG cluster (wave 2), SOPS masterkey, Helm values, metrics ServiceMonitor, four alerts, CoreDNS rewrite. 2. **Terraform** — `terraform/zitadel/`: 1 project, 19 project roles, 12 OIDC applications, 5 machine users with JWT keys, 1 API application for the gateway, project grants, per-agent claim metadata, gated groups-claim action. 3. **Provisioning script** — `scripts/iam/zitadel-provision.py`: keys, apply, secrets, users, verify. ## Facts that change the plan - **Client IDs change.** ZITADEL v4.19.2 `CreateOIDCApplicationRequest` has no `client_id` field, and provider 3.8.7 rejects a settable application id. The migration plan assumed a secret-only re-point. Every consumer needs its client ID updated too. - **The groups claim is unproven.** ZITADEL issues no `groups` claim, and kube-apiserver RBAC, MinIO and Grafana all read it. The action resource exists and stays disabled until the live gate passes. - **The chart ships PreSync hooks.** `zitadel-init` and `zitadel-setup` re-run on every sync. `zitadel-values.yaml` documents the two follow-up edits that stop the setup job from rewriting the `iam-admin` key Terraform reads. ## Not applied by this PR The Application has no automated sync. Merging creates `zitadel-db` through the `databases` app and creates the Application object. Someone then syncs it by hand. ## Verification - `helm template` renders 17 objects, all in namespace `iam`. - `kubectl apply --dry-run=server` accepts the CNPG cluster and the full chart render. - `kubectl kustomize` builds `k8s/infra/databases/`, `k8s/infra/monitoring/`, `k8s/infra/iam/`. - `terraform fmt -check -recursive` clean, `terraform validate` 0 errors. - `structural-check.sh` 0 error / 0 warning on every changed manifest. - Reviewer rounds: two BLOCK verdicts, all findings fixed, then SIGN-OFF. ## Manual steps after merge 1. Cloudflare Tunnel public hostname for `zitadel.riotpiao.com` plus a proxied CNAME to the tunnel. 2. `terraform apply` and `make apply-cp` for the CoreDNS rewrite (Talos-owned, not ArgoCD). 3. Move the offline masterkey copy (`~/zitadel-masterkey.offline`) into sealed storage.

rock added 4 commits 2026-10-24 10:38:42 +00:00

feat(iam): wire Zitadel into namespace iam under ArgoCD cb1219bf96

Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.

- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
  secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
  automated sync deliberately absent for the first sync (chart PreSync hook
  re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
  crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host

Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).

The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.

feat(iam): declare Zitadel identity objects in Terraform f81ff0e2fc

Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open.
It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users
plus keys, one API application for the gateway, project grants, per-agent claim
metadata, and a disabled groups-claim action.

Facts established while writing it:

- Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest
  has no client_id field, and provider 3.8.7 rejects a settable application id
  (terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a
  client_id-preserving re-point; every consumer now also changes client ID.
- Role keys equal the existing group names, so the groups claim can be
  byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200
  characters with no character-set restriction.
- Machine users get JWT profile keys, not passwords. Per-agent claims move to
  user metadata and reach tokens only through the groups-claim action.
- The action stays gated: enable_groups_claim_action defaults to false because
  the action runtime API is unverified. Two validations stop a null script from
  reaching file().

Checks: terraform fmt -check -recursive clean, terraform validate 0 errors,
plan reaches only the expected missing-credentials error, structural gate
0 error / 0 warning, and one reviewer round: BLOCK on a plan-time
fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed
to SIGN-OFF.

State is local and git-ignored; it carries generated client secrets and machine
keys. Move it to MinIO before a second operator applies.

Not applied: no Zitadel instance is running yet.

feat(iam): one script to provision Zitadel end to end 2f62351731

scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.

- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
  at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
  into the plain manifests that carry them, write machine keys per agent.
  Requires an exactly-one-match regex before it edits a manifest, prints the
  diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
  passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
  while the groups-claim action is disabled.

Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
  cannot address one document. sops reads the creation rule from the filename,
  so the temp file is passed with --filename-override; without it sops finds no
  matching rule and refuses. Ciphertext is staged in the destination directory
  and installed with os.replace, so the mode is preserved and EXDEV is
  impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
  no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
  consumer. Writing the new secret into the orphan file would report success
  while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
  password hash do not appear in ps.

.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.

Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.

Not run against a live ZITADEL: no instance exists yet.

feat(iam): read the Zitadel bootstrap token from .env 979a413a5b

ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.

The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.

.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.

Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.

rock merged commit e9817283e6 into main

2026-10-24 10:41:18 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#63