storage/minio-oidc is owned by the KeyRotation object iam/minio now, so its value
leaves git. minio-creds and minio-user-ollama stay: they are the root console
credential and an application key, and neither rotates through ZITADEL yet.
The file keeps a header that says so, because a prune of the sops-secrets application
would delete that Secret and the controller recreates it on the next rotation.
Two defects that a real rotation exposed:
merge writeSecret replaced the whole Secret map, so every key a consumer
keeps beside the credential disappeared. paperless lost
SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then
failed to start while the rotation reported success.
backoff a Failed attempt returned 0, so the controller retried in a tight loop.
It now waits two minutes and then retries.
All eight objects carry both identifiers ZITADEL needs: the app resource id for the
secret endpoint and the client id for introspection.
Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui,
argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload
exists for it.
Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
Three faults stopped the controller from running, each found by running it:
image arch the first build was arm64 and every node is amd64, so the pod died
with 'exec format error'. imagePullPolicy IfNotPresent kept the
cached wrong-arch layer, so it is now Always.
leases leader election needs coordination.k8s.io/leases. Without it the
controller never acquired its lock and never reconciled.
secrets list the manager caches Secrets, so it needs list and watch, not only
get, create, update and patch.
go.mod and go.sum gained the procfs entry the build needs.
Forgejo reports zero workflows for this repository even though
.forgejo/workflows/key-rotation-controller.yaml is on main. The poimen-frontend
workflows that do run live under .gitea/workflows/, so this instance reads that path
only. Move the file there.
The frontend's callback path is /auth/callback. The old /callback spelling stays in
the list so an image built before this change still completes a login.
Dockerfile: static binary on distroless, no shell, user 65532.
Workflow runs on the golang runner for a push to main that touches the controller.
It builds two tags, the commit sha and 0.1.0, which is the tag the Deployment asks
for, and runs the image with --help so a broken binary fails the job instead of the
cluster.
The runners were dead before this: every task fetch logged 'lookup
forgejo-gitea-http.cicd.svc.cluster.local: no such host'. DNS resolved from a fresh
pod and the Service had a healthy endpoint, so the runner pods held stale network
state. A rollout restart cleared it and all three poll without error.
grafana-oidc and comfyui-oauth2-proxy held nothing but the rotated credential, so
both files are gone. oauth2-credentials survives with the two secrets that were not
rotated (api-gw, poimen) and drops the three that were (grafana, minio, paperless).
Leaving them would be worse than untidy: sops-secrets ran with selfHeal, so a stale
file silently rewrites a rotated secret back to a value ZITADEL no longer accepts.
32 files remain, down from 34.
Generates a new secret in ZITADEL, writes it into the Kubernetes Secret, rolls the
consumer and checks it comes back. This is the manual execution of a KeyRotation,
and the reference the controller must match.
Two traps it records: the app resource id differs from the client id, and the
secret endpoint is oidc_config/_generate_client_secret.
Eight objects cover grafana, argocd, comfyui, minio, paperless, immich, forgejo and
api-gw. Each names its ZITADEL app, the Secret it writes, the consumer to roll and
whether the consumer reloads hot or needs a restart.
All are suspended: the controller image does not exist yet, and an unsuspended
object would claim an intent that no process acts on.
The README records the retirement target for the 34 .enc.yaml files: six
root-of-trust secrets that cannot rotate themselves survive, everything else
becomes a KeyRotation and leaves git.
Gating /oauth2 with the app redirects /oauth2/start to itself and the browser
loops. ingress-nginx applies an ingress's annotations to the locations that
ingress defines, so the login path belongs on its own object without auth-url.
This split failed once before, and the split was not the reason: the app ingress
carried a configuration-snippet that the controller rejects, which dropped that
whole object and left only the login path rendered.
The controller logs "annotation group ConfigurationSnippet contains risky
annotation" and a rejected annotation group drops the entire ingress from
nginx.conf. comfyui.riotpiao.com therefore answered 404 for every request with no
line in the access log, which is why the host looked routed but served nothing.
ingress-nginx sets the Upgrade and Connection headers itself for a websocket
upgrade, so the snippet was never needed.
Two Ingresses on one host do not merge annotations. nginx applies the annotations
of a single ingress per host, so the separate /oauth2 object silently dropped the
auth_request on the app ingress: every request fell through to the backend and
ComfyUI answered 404 for the site root.
Move /oauth2 and / into the same ingress, so the auth annotations and both paths
belong to one object.
comfyui.riotpiao.com is served with no authentication at all: anyone who can
resolve the host reaches the UI and its queue. ComfyUI has no OIDC support, so the
front door is oauth2-proxy, and nginx asks it for a decision on every request.
Nothing inside the cluster calls this host, so the whole host is gated rather than
carved into open and closed paths. If a client ever needs direct access, add a skip
rule for its exact paths.
The ZITADEL app is created by the provisioning script, and the client secret,
cookie secret and client id live in a SOPS secret written by ksops.
Replace the dead secret-rotation scaffold (an Argo app that could not render, a
CRD that was never installed, and no controller code) with a working design.
One resource describes one credential:
spec.provider zitadel-machine-key | zitadel-oidc-client
spec.strategy overlap | atomic
spec.target Secret to write, consumers to roll, reload hot|restart
spec.verify introspection or token exchange
The state machine is shared by every provider:
Prepare -> Publish -> Verify -> Retire
Order is the safety property. Retire runs only for an overlapping provider and
only after the new credential obtained a token. A failure before that point leaves
the working credential in place, which is exactly what went wrong by hand today
when a regenerated Grafana secret was never re-pointed.
Machine keys overlap in ZITADEL, so that provider rotates with no gap. Client
secrets do not overlap, so that provider is atomic: it publishes, rolls the
consumer, verifies, and restores the previous value from a shadow key if the
introspection check fails.
The controller authenticates with its own machine key, the single bootstrap
credential, read from a mounted Secret. Everything else it creates itself, so no
credential value needs to live in git.
Verified: go build and go vet clean, 12 tests pass covering scheduling, the shadow
key rollback path, provider strategies and assertion signing.
Not yet wired to Argo CD: the image must be built and pushed first, so the app is
not added until a Deployment can actually pull.
The live cluster has no Authentik workload, no DNS record and no tunnel route,
but two declarations still pointed at it.
controlplane.tftpl still set kube-apiserver oidc-issuer-url to
https://authentik.riotpiao.com/application/o/kubernetes/ and pinned that hostname
in extraHostEntries. The live API server carries zero OIDC arguments, so the
template and reality had diverged: the next machine-config render would have
re-added an issuer that does not resolve.
management-service exported KMSVC_AUTHENTIK_ISSUER_URL and
KMSVC_AUTHENTIK_AUDIENCE, both empty. Rename them to KMSVC_ZITADEL_* so the names
describe the identity provider that actually exists.
The CoreDNS Corefile kept one rewrite for authentik.riotpiao.com. The Terraform
source file no longer contains it, so the live ConfigMap was stale; that line is
removed live and the next render converges without it.
Grafana sign-in failed with "Failed to get token from provider". The deployed
client secret no longer matched, because _ensure_oidc_apps called
_generate_client_secret for every app on every run, and ZITADEL stores exactly one
client secret per app: each call replaced the previous value and invalidated every
deployed copy.
Two changes:
1. Generate a secret only when the app is created. For an app that already exists,
reuse the value from state.json, because the API cannot read a secret back.
2. Re-encrypt grafana-oidc-secrets.enc.yaml with the live secret, so Grafana's
token exchange succeeds again.
Verified with the introspection endpoint:
old deployed secret -> {"error":"unauthorized_client"}
new secret -> {"active":false} (credentials accepted)
Grafana gains one contact point, `gotify`, of type webhook. It posts to the
bridge in the notifications namespace because Gotify rejects a Grafana payload
with 400 "Field 'message' is required".
maxAlerts 0 keeps every alert in one request, and the bridge names the extra
count in the message body instead of dropping alerts.
Gotify rejects a Grafana payload with 400 "Field 'message' is required", so a
translation step is mandatory. This adds a small standard-library Python
service: it accepts Grafana's JSON, builds a Gotify message, and posts it with an
app token.
Mapping:
firing -> priority 8, title "[FIRING] <alertname>"
resolved -> priority 3, title "[RESOLVED] <alertname>"
body -> the summary annotation plus severity, namespace, pod, instance, job
A multi-alert notification names the alert count instead of dropping the rest.
The script ships as a ConfigMap so no image build is needed. The app token comes
from a SOPS secret and belongs to Gotify application `grafana`.
The bridge runs server-side, so it cannot run a browser flow and ZITADEL has no
part in it. Give it a Gotify client token instead: GOTIFY_URLS points at the
public host, GOTIFY_CLIENT_TOKENS comes from a SOPS secret.
The token belongs to user `rock`, is named igotify-assist, and has
expiresAfterInactivitySeconds 0 so it never idles out. Delete the client row in
Gotify to revoke it.
Verified with the token: GET /client returns 200 and GET /message returns the
latest message.
The first value was a base32 string, so oauth2-proxy rejected it at startup:
cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 56
bytes.
Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put
oauth2-proxy in front of it and let nginx ask the proxy for every request.
The Gotify REST API keeps its own client-token check. /message, /application,
/client, /user, /version, /stream and /health are skipped by the proxy, so
Forgejo webhooks, the harness, poimen-sender and the Android client keep working
without a browser session. Only the web UI requires SSO.
The /oauth2 path needs its own ingress: the auth_request annotation would
otherwise also guard /oauth2/start, and the login entry point would demand a
login first.
The client id and secret come from project role provisioning:
`zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is
SOPS-encrypted under k8s/argocd/secrets.
Gotify's own login remains behind the proxy. ZITADEL decides who reaches the
service, Gotify decides what that person may do inside it.
forceMFA never took effect because of the JSON key. ZITADEL's v1 endpoint speaks
protojson, which accepts "forceMfa" (lowerCamel of force_mfa). The spelling
"forceMFA" is treated as an unknown field and silently dropped, so every earlier
run returned 200 with a new sequence while force_mfa stayed false in
projections.login_policies5.
Send forceMfa and forceMfaLocalOnly. Verified in the database:
force_mfa = t | force_mfa_local_only = t | mfa_init_skip_lifetime = 0
and in the login flow: after the password step the browser lands on
/ui/login/mfa/prompt with the "2-Factor Setup" page, so a second factor is now
required instead of optional.
Add a login-policy phase: password plus a second factor, with
mfaInitSkipLifetime 0s so enrolment is demanded at login rather than skipped for
30 days. Second factors stay OTP and U2F (authenticator app or hardware key).
Multi-factor is already configured for U2F-with-verification, so moving 2FA to
2MFA later needs no further policy change.
forceMFA does not persist through the v1 policy endpoint: it reads back absent
even after a successful PUT, which is why the skip lifetime is the lever used.
A repeat run returns 400 "Default Login Policy has not been changed"; treat that
as success so provisioning stays idempotent.
Full apply now completes end to end:
login page : built-in
smtp : already active
login policy : password + 2FA, enrolment required
roles : 21 ensured
oidc apps : 11 ensured
ZITADEL sends verification codes and password resets by SMTP only, and nothing
delivered until the instance had an *active* SMTP config. Two traps, both now
handled here instead of by hand:
- POST /admin/v1/smtp creates the config INACTIVE (state 3). An inactive config
makes GET /admin/v1/smtp answer 404 "SMTP configuration not found", so the API
reports success while every send is dropped. The _activate call is mandatory.
- The target is the in-cluster smtp-gotify relay, which forwards to Gotify.
The phase is idempotent: an active config short-circuits and only logs.
Also fix the role-exists check: ZITADEL returns `key`, not `roleKey`, so the set
was always empty and every re-run died on the first 409 "Role already exists".
Verified: `zitadel-provision.py apply` now completes, reporting
`smtp : already active`, `roles : 21 ensured`, `oidc apps : 11 ensured`.
A resend then produced Gotify message 1185 ("Verify email, Code ZVYLQG") and
submitting that code set verified_email for user riotpiao.
The chart declares its ServiceAccount, ConfigMap, Role and RoleBinding as
pre-install/pre-upgrade Helm hooks with hook-delete-policy before-hook-creation.
ArgoCD maps those to PreSync and prunes them when a sync ends, so every sync
deleted the ServiceAccount and ConfigMap the ZITADEL Deployment mounts. New pods
failed with `serviceaccount "zitadel" not found`, leaving the IdP one restart
from an outage.
Add a zatadel-core Application owning hook-free copies, keeping them as ordinary
desired state that survives every sync. Prune is off for this path: deleting a
ServiceAccount here would recreate the outage.
The ConfigMap also carries the notification settings, pointing
DefaultInstance.SMTPConfiguration at the smtp-gotify relay, with
SMTPSenderAddressMatchesInstanceDomain false because the sender
[email protected] differs from the instance domain zatadel.riotpiao.com.
Drop the envVarsSecret approach: ZITADEL created no SMTP provider from those env
vars (no instance.smtp.config.added event).
ZITADEL must send verification codes, and it only speaks SMTP. Route that at
the smtp-gotify relay so codes reach Gotify and the phone.
The settings go in as a ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_* envFrom
secret, not as configmapConfig. Editing configmapConfig makes ArgoCD stop
managing the rendered zitadel-config-yaml ConfigMap, which the Deployment still
mounts; that twice left the IdP one restart away from failing. Env vars override
the same keys and leave that ConfigMap alone.
Validated with helm template 10.4.0: envFrom appears on both ZITADEL containers
and zitadel-config-yaml still renders.
Point DefaultInstance.SMTPConfiguration at the in-cluster smtp-gotify relay, so
verification codes and password resets arrive on the phone instead of being
dropped for lack of a mail path.
Validated with `helm template zitadel/zitadel 10.4.0` before syncing: the block
renders, and the zatadel-config-yaml ConfigMap is still produced. An earlier
attempt looked like it removed that ConfigMap, which is why this went in only
after a local render check.
The relay stages each message in /tmp before pushing. readOnlyRootFilesystem
left it with a read-only /tmp, so it answered '250 queued' and then dropped the
message: no Gotify push, no error line. Mount an emptyDir at /tmp.
The sync removed zitadel-config-yaml while the Deployment still mounts it, so
the next pod restart would have failed. Restore the previous render first; the
relay stays deployed and can be wired once the chart accepts the block.
Two defects in the first cut, both silent from the sender's side because the
relay answers 250 either way:
- GOTIFY_URL needs a trailing slash; without it the push has no path.
- GOTIFY_TITLE_TEMPLATE referenced .Subject, which the relay does not expose,
so template execution failed.
The relay's default title already carries the subject, so drop the template.
Also register the SOPS file in secret-generator.yaml: the sops-secrets app only
renders files listed there, so the Secret was never created and the relay sat in
CreateContainerConfigError.
Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped
being synced. The new smtp-gotify relay would have sat in git and never been
created.
Add an iam-support Application for that path, and drop the stale
rbac-dashboard-rolebinding reference from its kustomization: the file went away
with Authentik, and a missing resource fails the whole app.
ZITADEL sends verification codes and password resets by SMTP only, and this
cluster has no mail path. Every human user therefore sits at
verified_email = empty and can never finish a first login, at ZITADEL or at
Forgejo.
Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525
and forwards each message to the existing Gotify instance, so codes arrive on
the phone through gotify/igotify instead of email.
Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because
the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app
token in a SOPS-encrypted Secret; values never carry the token.
Forgejo keeps login sources in its database and exposes no REST endpoint for
them, so neither GitOps nor `tea` (whose admin command only manages users) can
set them. The in-pod CLI is the only supported writer.
This script wraps `gitea admin auth update-oauth` so the rename from the
Authentik-era label to `zitadel` is repeatable instead of a one-off command in
a shell history. Note in the header: the source name is both the button label
and the callback path, so ZITADEL must have both redirect URIs registered
before a rename.
loginV2.required lives in the instance database, not in chart values, so
disabling the hosted Login UI in values is only half the change: the server
kept redirecting /oauth/v2/authorize to /ui/v2/login, which no longer had a
backend and returned 503.
Set loginV2.required=false in the provisioning run so the server serves its
own login page. Verified: authorize now redirects to /ui/login/login and the
page renders the "Welcome Back!" login form.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
The api-gw gates /v1/* on a capability named in the access token. ZITADEL
had no such role, so every inference call was 403 once the token came from
ZITADEL instead of Authentik.
- add llm:inference (Authentik spelling, keeps consumer manifests unchanged)
and llm-inference (dash alias) to the project roles
- enable projectRoleAssertion on the project, which is what puts
urn:zitadel:iam:org:project:roles into the access token; without it the
role list never reaches the resource server
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
Commit a02bee5 had both the Ingress rule and the cloudflared tunnel
hostname for authentik.riotpiao.com. Both were lost, so the tunnel
answered 404 for the hostname and the ingress had no matching rule.
The CoreDNS rewrite and the authentik-server Service were still intact;
only the two routing entries were missing. Re-add the Ingress (namespace
iam, service authentik-server port 80, TLS terminated by the wildcard
default-ssl-certificate) and the tunnel hostname.
rock