Every consumer SSO login failed for two independent reasons, and neither was
recorded in git.
1. The OIDC client secrets lived only in the cluster. They are written by
scripts/iam/zitadel-provision.py, whose `apply` phase calls
_generate_client_secret for every app on every run. A rotation therefore
invalidated the stored secret in ArgoCD, Grafana, MinIO, Immich and
Paperless, with no git value to compare against. Declaring the five Secrets
as SOPS manifests makes ArgoCD the owner, so the next rotation is a
reviewable diff instead of silent drift.
2. Forgejo, Paperless and Immich store the identity-provider link inside their
own databases, not in Kubernetes. Those rows still carried the Authentik
subject and provider, which produced three different symptoms from one
cause: Forgejo never left /user/link_account, Paperless showed the social
signup page, and Immich answered 400 "email already linked to different
account". The PostSync Job re-points each row at the ZITADEL `sub`,
matching on email or username rather than the numeric ids this was first
observed on, so a re-run converges.
Verified with a real browser login against a live ZITADEL session: Forgejo,
Paperless, Immich, Grafana and ArgoCD all reach their authenticated landing
page. Vault has no UI by design. MinIO remains open: the operator sidecar
serves its console with no OIDC environment, so it advertises form login only.
Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped
being synced. The new smtp-gotify relay would have sat in git and never been
created.
Add an iam-support Application for that path, and drop the stale
rbac-dashboard-rolebinding reference from its kustomization: the file went away
with Authentik, and a missing resource fails the whole app.
ZITADEL sends verification codes and password resets by SMTP only, and this
cluster has no mail path. Every human user therefore sits at
verified_email = empty and can never finish a first login, at ZITADEL or at
Forgejo.
Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525
and forwards each message to the existing Gotify instance, so codes arrive on
the phone through gotify/igotify instead of email.
Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because
the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app
token in a SOPS-encrypted Secret; values never carry the token.
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
rock
Story Crater Bot