Adds svc-caller, a capability that names a machine-to-machine caller, separate from
the human roles so a service account can be identified by its grant alone.
Adds MACHINE_CALLERS: one ZITADEL machine user per caller, created once with its own
key and granted its roles. poimen-llm is the first, granted llm:inference and
svc-caller. Its key is separate from every other caller's, so revoking one service
leaves the rest working and ZITADEL can overlap keys during a rotation.
The role is granted even though ZITADEL does not assert roles into a machine-user
access token on this instance. The grant is the record of what the account may do, and
it takes effect the moment that behaviour changes. Until then the gateway authorises
the caller from its allow-list entry.
The frontend's callback path is /auth/callback. The old /callback spelling stays in
the list so an image built before this change still completes a login.
Generates a new secret in ZITADEL, writes it into the Kubernetes Secret, rolls the
consumer and checks it comes back. This is the manual execution of a KeyRotation,
and the reference the controller must match.
Two traps it records: the app resource id differs from the client id, and the
secret endpoint is oidc_config/_generate_client_secret.
comfyui.riotpiao.com is served with no authentication at all: anyone who can
resolve the host reaches the UI and its queue. ComfyUI has no OIDC support, so the
front door is oauth2-proxy, and nginx asks it for a decision on every request.
Nothing inside the cluster calls this host, so the whole host is gated rather than
carved into open and closed paths. If a client ever needs direct access, add a skip
rule for its exact paths.
The ZITADEL app is created by the provisioning script, and the client secret,
cookie secret and client id live in a SOPS secret written by ksops.
Grafana sign-in failed with "Failed to get token from provider". The deployed
client secret no longer matched, because _ensure_oidc_apps called
_generate_client_secret for every app on every run, and ZITADEL stores exactly one
client secret per app: each call replaced the previous value and invalidated every
deployed copy.
Two changes:
1. Generate a secret only when the app is created. For an app that already exists,
reuse the value from state.json, because the API cannot read a secret back.
2. Re-encrypt grafana-oidc-secrets.enc.yaml with the live secret, so Grafana's
token exchange succeeds again.
Verified with the introspection endpoint:
old deployed secret -> {"error":"unauthorized_client"}
new secret -> {"active":false} (credentials accepted)
Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put
oauth2-proxy in front of it and let nginx ask the proxy for every request.
The Gotify REST API keeps its own client-token check. /message, /application,
/client, /user, /version, /stream and /health are skipped by the proxy, so
Forgejo webhooks, the harness, poimen-sender and the Android client keep working
without a browser session. Only the web UI requires SSO.
The /oauth2 path needs its own ingress: the auth_request annotation would
otherwise also guard /oauth2/start, and the login entry point would demand a
login first.
The client id and secret come from project role provisioning:
`zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is
SOPS-encrypted under k8s/argocd/secrets.
Gotify's own login remains behind the proxy. ZITADEL decides who reaches the
service, Gotify decides what that person may do inside it.
forceMFA never took effect because of the JSON key. ZITADEL's v1 endpoint speaks
protojson, which accepts "forceMfa" (lowerCamel of force_mfa). The spelling
"forceMFA" is treated as an unknown field and silently dropped, so every earlier
run returned 200 with a new sequence while force_mfa stayed false in
projections.login_policies5.
Send forceMfa and forceMfaLocalOnly. Verified in the database:
force_mfa = t | force_mfa_local_only = t | mfa_init_skip_lifetime = 0
and in the login flow: after the password step the browser lands on
/ui/login/mfa/prompt with the "2-Factor Setup" page, so a second factor is now
required instead of optional.
Add a login-policy phase: password plus a second factor, with
mfaInitSkipLifetime 0s so enrolment is demanded at login rather than skipped for
30 days. Second factors stay OTP and U2F (authenticator app or hardware key).
Multi-factor is already configured for U2F-with-verification, so moving 2FA to
2MFA later needs no further policy change.
forceMFA does not persist through the v1 policy endpoint: it reads back absent
even after a successful PUT, which is why the skip lifetime is the lever used.
A repeat run returns 400 "Default Login Policy has not been changed"; treat that
as success so provisioning stays idempotent.
Full apply now completes end to end:
login page : built-in
smtp : already active
login policy : password + 2FA, enrolment required
roles : 21 ensured
oidc apps : 11 ensured
ZITADEL sends verification codes and password resets by SMTP only, and nothing
delivered until the instance had an *active* SMTP config. Two traps, both now
handled here instead of by hand:
- POST /admin/v1/smtp creates the config INACTIVE (state 3). An inactive config
makes GET /admin/v1/smtp answer 404 "SMTP configuration not found", so the API
reports success while every send is dropped. The _activate call is mandatory.
- The target is the in-cluster smtp-gotify relay, which forwards to Gotify.
The phase is idempotent: an active config short-circuits and only logs.
Also fix the role-exists check: ZITADEL returns `key`, not `roleKey`, so the set
was always empty and every re-run died on the first 409 "Role already exists".
Verified: `zitadel-provision.py apply` now completes, reporting
`smtp : already active`, `roles : 21 ensured`, `oidc apps : 11 ensured`.
A resend then produced Gotify message 1185 ("Verify email, Code ZVYLQG") and
submitting that code set verified_email for user riotpiao.
Forgejo keeps login sources in its database and exposes no REST endpoint for
them, so neither GitOps nor `tea` (whose admin command only manages users) can
set them. The in-pod CLI is the only supported writer.
This script wraps `gitea admin auth update-oauth` so the rename from the
Authentik-era label to `zitadel` is repeatable instead of a one-off command in
a shell history. Note in the header: the source name is both the button label
and the callback path, so ZITADEL must have both redirect URIs registered
before a rename.
loginV2.required lives in the instance database, not in chart values, so
disabling the hosted Login UI in values is only half the change: the server
kept redirecting /oauth/v2/authorize to /ui/v2/login, which no longer had a
backend and returned 503.
Set loginV2.required=false in the provisioning run so the server serves its
own login page. Verified: authorize now redirects to /ui/login/login and the
page renders the "Welcome Back!" login form.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
The api-gw gates /v1/* on a capability named in the access token. ZITADEL
had no such role, so every inference call was 403 once the token came from
ZITADEL instead of Authentik.
- add llm:inference (Authentik spelling, keeps consumer manifests unchanged)
and llm-inference (dash alias) to the project roles
- enable projectRoleAssertion on the project, which is what puts
urn:zitadel:iam:org:project:roles into the access token; without it the
role list never reaches the resource server
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.
The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.
.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.
Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.
- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
into the plain manifests that carry them, write machine keys per agent.
Requires an exactly-one-match regex before it edits a manifest, prints the
diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
while the groups-claim action is disabled.
Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
cannot address one document. sops reads the creation rule from the filename,
so the temp file is passed with --filename-override; without it sops finds no
matching rule and refuses. Ciphertext is staged in the destination directory
and installed with os.replace, so the mode is preserved and EXDEV is
impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
consumer. Writing the new secret into the orphan file would report success
while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
password hash do not appear in ps.
.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.
Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.
Not run against a live ZITADEL: no instance exists yet.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
rock
pi