fix(iam): make ZITADEL OIDC secrets and identity links IaC-owned #85

2026-10-31T20:37:23Z

rock commented

2026-10-31 20:37:23 +00:00

Why

Every consumer SSO login was broken for two independent reasons, and neither was recorded in git. This PR makes both IaC-owned.

1. OIDC client secrets lived only in the cluster

scripts/iam/zitadel-provision.py calls _generate_client_secret for every app on every apply run. A rotation therefore invalidated the stored secret in ArgoCD, Grafana, MinIO, Immich and Paperless, with no git value to compare against.

Verified against ZITADEL's token endpoint. A valid client with a bogus code returns Errors.User.Code.Invalid; a bad secret returns invalid_client. Before this change six apps returned invalid secret.

Declaring the five Secrets as SOPS manifests makes ArgoCD the owner, so the next rotation is a reviewable diff instead of silent drift.

2. Identity links are stored inside three application databases

Forgejo, Paperless and Immich keep the IdP link in their own database, not in Kubernetes. Those rows still carried the Authentik subject and provider. One cause, three different symptoms:

App	Symptom
Forgejo	never left `/user/link_account`
Paperless	showed the social signup page
Immich	`400 email already linked to different account`

The PostSync Job re-points each row at the ZITADEL sub, matching on email/username rather than the numeric ids this was first observed on, so a re-run converges and a fresh database works.

3. MinIO shipped with OIDC disabled

identity_openid defaults to off. The tenant had all eight MINIO_IDENTITY_OPENID_* values but not MINIO_IDENTITY_OPENID_ENABLE, so the whole block was inert. Also drops MINIO_IDENTITY_OPENID_CLAIM_PREFIX, which was declared with an empty value.

Verification

Real browser logins against a live ZITADEL session (password + TOTP):

App	Result
ArgoCD	✅ `/applications`
Grafana	✅ dashboards home
Forgejo	✅ "Rock - Dashboard"
Paperless	✅ "Dashboard - Paperless-ngx"
Immich	✅ `/photos`, lands on the admin account
Vault	n/a — no UI by design
MinIO	⚠️ server OIDC now enabled; the console is the operator sidecar, which carries no OIDC env — see below

Known limitation (not fixed here)

The MinIO console at minio.riotpiao.com is the operator sidecar (minio/operator:v5.0.0), which has zero environment variables and does not read MINIO_IDENTITY_OPENID_*. Upstream docs require CONSOLE_IDP_* for a separately-run console, and MinIO removed console SSO from the free build after RELEASE.2025-04-22T22-12-26Z (issues #21311, #21415). Our release is 2025-10-15. Server-side OIDC is enabled and usable via STS; browser SSO needs a separate decision.

Safety

Every SOPS file decrypts.
kubectl kustomize builds; kubectl apply --dry-run=server passes for both k8s/infra/iam and the Tenant.
The repair Job is idempotent and prunes itself via hook-delete-policy: BeforeHookCreation.

## Why Every consumer SSO login was broken for two independent reasons, and neither was recorded in git. This PR makes both IaC-owned. ### 1. OIDC client secrets lived only in the cluster `scripts/iam/zitadel-provision.py` calls `_generate_client_secret` for **every** app on **every** `apply` run. A rotation therefore invalidated the stored secret in ArgoCD, Grafana, MinIO, Immich and Paperless, with no git value to compare against. Verified against ZITADEL's token endpoint. A valid client with a bogus code returns `Errors.User.Code.Invalid`; a bad secret returns `invalid_client`. Before this change six apps returned `invalid secret`. Declaring the five Secrets as SOPS manifests makes ArgoCD the owner, so the next rotation is a reviewable diff instead of silent drift. ### 2. Identity links are stored inside three application databases Forgejo, Paperless and Immich keep the IdP link in their own database, not in Kubernetes. Those rows still carried the **Authentik** subject and provider. One cause, three different symptoms: | App | Symptom | |---|---| | Forgejo | never left `/user/link_account` | | Paperless | showed the social signup page | | Immich | `400 email already linked to different account` | The PostSync Job re-points each row at the ZITADEL `sub`, matching on **email/username** rather than the numeric ids this was first observed on, so a re-run converges and a fresh database works. ### 3. MinIO shipped with OIDC disabled `identity_openid` defaults to `off`. The tenant had all eight `MINIO_IDENTITY_OPENID_*` values but not `MINIO_IDENTITY_OPENID_ENABLE`, so the whole block was inert. Also drops `MINIO_IDENTITY_OPENID_CLAIM_PREFIX`, which was declared with an empty value. ## Verification Real browser logins against a live ZITADEL session (password + TOTP): | App | Result | |---|---| | ArgoCD | ✅ `/applications` | | Grafana | ✅ dashboards home | | Forgejo | ✅ "Rock - Dashboard" | | Paperless | ✅ "Dashboard - Paperless-ngx" | | Immich | ✅ `/photos`, lands on the **admin** account | | Vault | n/a — no UI by design | | MinIO | ⚠️ server OIDC now enabled; the console is the operator sidecar, which carries no OIDC env — see below | ## Known limitation (not fixed here) The MinIO console at `minio.riotpiao.com` is the **operator sidecar** (`minio/operator:v5.0.0`), which has zero environment variables and does not read `MINIO_IDENTITY_OPENID_*`. Upstream docs require `CONSOLE_IDP_*` for a separately-run console, and MinIO removed console SSO from the free build after `RELEASE.2025-04-22T22-12-26Z` (issues #21311, #21415). Our release is `2025-10-15`. Server-side OIDC is enabled and usable via STS; browser SSO needs a separate decision. ## Safety - Every SOPS file decrypts. - `kubectl kustomize` builds; `kubectl apply --dry-run=server` passes for both `k8s/infra/iam` and the Tenant. - The repair Job is idempotent and prunes itself via `hook-delete-policy: BeforeHookCreation`.

rock added 2 commits 2026-10-31 20:37:24 +00:00

fix(iam): make the ZITADEL OIDC secrets and identity links IaC-owned cf1263235b

Every consumer SSO login failed for two independent reasons, and neither was
recorded in git.

1. The OIDC client secrets lived only in the cluster. They are written by
   scripts/iam/zitadel-provision.py, whose `apply` phase calls
   _generate_client_secret for every app on every run. A rotation therefore
   invalidated the stored secret in ArgoCD, Grafana, MinIO, Immich and
   Paperless, with no git value to compare against. Declaring the five Secrets
   as SOPS manifests makes ArgoCD the owner, so the next rotation is a
   reviewable diff instead of silent drift.

2. Forgejo, Paperless and Immich store the identity-provider link inside their
   own databases, not in Kubernetes. Those rows still carried the Authentik
   subject and provider, which produced three different symptoms from one
   cause: Forgejo never left /user/link_account, Paperless showed the social
   signup page, and Immich answered 400 "email already linked to different
   account". The PostSync Job re-points each row at the ZITADEL `sub`,
   matching on email or username rather than the numeric ids this was first
   observed on, so a re-run converges.

Verified with a real browser login against a live ZITADEL session: Forgejo,
Paperless, Immich, Grafana and ArgoCD all reach their authenticated landing
page. Vault has no UI by design. MinIO remains open: the operator sidecar
serves its console with no OIDC environment, so it advertises form login only.

fix(minio): enable the OIDC identity provider on the tenant 506f618b8a

MinIO ships identity_openid disabled by default. Every MINIO_IDENTITY_OPENID_*
value was already correct, but without MINIO_IDENTITY_OPENID_ENABLE=on the
whole block is inert: the console answered /api/v1/login with
{"loginStrategy":"form"} and never rendered an SSO button.

Also drop MINIO_IDENTITY_OPENID_CLAIM_PREFIX, which was declared with an empty
value. An empty prefix makes the claim resolver prepend nothing and then look
for a nested property that cannot exist, so omitting the key is correct.

ArgoCD owns this Tenant with selfHeal, so the change has to land in git to
survive; a direct kubectl apply was reverted within seconds.

rock merged commit 4e84571f0c into main

2026-10-31 20:53:57 +00:00

rock deleted branch fix/zitadel-oidc-iac

2026-10-31 20:54:23 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#85