forceMFA never took effect because of the JSON key. ZITADEL's v1 endpoint speaks
protojson, which accepts "forceMfa" (lowerCamel of force_mfa). The spelling
"forceMFA" is treated as an unknown field and silently dropped, so every earlier
run returned 200 with a new sequence while force_mfa stayed false in
projections.login_policies5.
Send forceMfa and forceMfaLocalOnly. Verified in the database:
force_mfa = t | force_mfa_local_only = t | mfa_init_skip_lifetime = 0
and in the login flow: after the password step the browser lands on
/ui/login/mfa/prompt with the "2-Factor Setup" page, so a second factor is now
required instead of optional.
Add a login-policy phase: password plus a second factor, with
mfaInitSkipLifetime 0s so enrolment is demanded at login rather than skipped for
30 days. Second factors stay OTP and U2F (authenticator app or hardware key).
Multi-factor is already configured for U2F-with-verification, so moving 2FA to
2MFA later needs no further policy change.
forceMFA does not persist through the v1 policy endpoint: it reads back absent
even after a successful PUT, which is why the skip lifetime is the lever used.
A repeat run returns 400 "Default Login Policy has not been changed"; treat that
as success so provisioning stays idempotent.
Full apply now completes end to end:
login page : built-in
smtp : already active
login policy : password + 2FA, enrolment required
roles : 21 ensured
oidc apps : 11 ensured
ZITADEL sends verification codes and password resets by SMTP only, and nothing
delivered until the instance had an *active* SMTP config. Two traps, both now
handled here instead of by hand:
- POST /admin/v1/smtp creates the config INACTIVE (state 3). An inactive config
makes GET /admin/v1/smtp answer 404 "SMTP configuration not found", so the API
reports success while every send is dropped. The _activate call is mandatory.
- The target is the in-cluster smtp-gotify relay, which forwards to Gotify.
The phase is idempotent: an active config short-circuits and only logs.
Also fix the role-exists check: ZITADEL returns `key`, not `roleKey`, so the set
was always empty and every re-run died on the first 409 "Role already exists".
Verified: `zitadel-provision.py apply` now completes, reporting
`smtp : already active`, `roles : 21 ensured`, `oidc apps : 11 ensured`.
A resend then produced Gotify message 1185 ("Verify email, Code ZVYLQG") and
submitting that code set verified_email for user riotpiao.
Forgejo keeps login sources in its database and exposes no REST endpoint for
them, so neither GitOps nor `tea` (whose admin command only manages users) can
set them. The in-pod CLI is the only supported writer.
This script wraps `gitea admin auth update-oauth` so the rename from the
Authentik-era label to `zitadel` is repeatable instead of a one-off command in
a shell history. Note in the header: the source name is both the button label
and the callback path, so ZITADEL must have both redirect URIs registered
before a rename.
loginV2.required lives in the instance database, not in chart values, so
disabling the hosted Login UI in values is only half the change: the server
kept redirecting /oauth/v2/authorize to /ui/v2/login, which no longer had a
backend and returned 503.
Set loginV2.required=false in the provisioning run so the server serves its
own login page. Verified: authorize now redirects to /ui/login/login and the
page renders the "Welcome Back!" login form.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
The api-gw gates /v1/* on a capability named in the access token. ZITADEL
had no such role, so every inference call was 403 once the token came from
ZITADEL instead of Authentik.
- add llm:inference (Authentik spelling, keeps consumer manifests unchanged)
and llm-inference (dash alias) to the project roles
- enable projectRoleAssertion on the project, which is what puts
urn:zitadel:iam:org:project:roles into the access token; without it the
role list never reaches the resource server
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.
The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.
.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.
Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.
- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
into the plain manifests that carry them, write machine keys per agent.
Requires an exactly-one-match regex before it edits a manifest, prints the
diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
while the groups-claim action is disabled.
Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
cannot address one document. sops reads the creation rule from the filename,
so the temp file is passed with --filename-override; without it sops finds no
matching rule and refuses. Ciphertext is staged in the destination directory
and installed with os.replace, so the mode is preserved and EXDEV is
impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
consumer. Writing the new secret into the orphan file would report success
while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
password hash do not appear in ps.
.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.
Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.
Not run against a live ZITADEL: no instance exists yet.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
rock
pi