Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
Three faults stopped the controller from running, each found by running it:
image arch the first build was arm64 and every node is amd64, so the pod died
with 'exec format error'. imagePullPolicy IfNotPresent kept the
cached wrong-arch layer, so it is now Always.
leases leader election needs coordination.k8s.io/leases. Without it the
controller never acquired its lock and never reconciled.
secrets list the manager caches Secrets, so it needs list and watch, not only
get, create, update and patch.
go.mod and go.sum gained the procfs entry the build needs.
Dockerfile: static binary on distroless, no shell, user 65532.
Workflow runs on the golang runner for a push to main that touches the controller.
It builds two tags, the commit sha and 0.1.0, which is the tag the Deployment asks
for, and runs the image with --help so a broken binary fails the job instead of the
cluster.
The runners were dead before this: every task fetch logged 'lookup
forgejo-gitea-http.cicd.svc.cluster.local: no such host'. DNS resolved from a fresh
pod and the Service had a healthy endpoint, so the runner pods held stale network
state. A rollout restart cleared it and all three poll without error.
grafana-oidc and comfyui-oauth2-proxy held nothing but the rotated credential, so
both files are gone. oauth2-credentials survives with the two secrets that were not
rotated (api-gw, poimen) and drops the three that were (grafana, minio, paperless).
Leaving them would be worse than untidy: sops-secrets ran with selfHeal, so a stale
file silently rewrites a rotated secret back to a value ZITADEL no longer accepts.
32 files remain, down from 34.
Eight objects cover grafana, argocd, comfyui, minio, paperless, immich, forgejo and
api-gw. Each names its ZITADEL app, the Secret it writes, the consumer to roll and
whether the consumer reloads hot or needs a restart.
All are suspended: the controller image does not exist yet, and an unsuspended
object would claim an intent that no process acts on.
The README records the retirement target for the 34 .enc.yaml files: six
root-of-trust secrets that cannot rotate themselves survive, everything else
becomes a KeyRotation and leaves git.
Gating /oauth2 with the app redirects /oauth2/start to itself and the browser
loops. ingress-nginx applies an ingress's annotations to the locations that
ingress defines, so the login path belongs on its own object without auth-url.
This split failed once before, and the split was not the reason: the app ingress
carried a configuration-snippet that the controller rejects, which dropped that
whole object and left only the login path rendered.
The controller logs "annotation group ConfigurationSnippet contains risky
annotation" and a rejected annotation group drops the entire ingress from
nginx.conf. comfyui.riotpiao.com therefore answered 404 for every request with no
line in the access log, which is why the host looked routed but served nothing.
ingress-nginx sets the Upgrade and Connection headers itself for a websocket
upgrade, so the snippet was never needed.
Two Ingresses on one host do not merge annotations. nginx applies the annotations
of a single ingress per host, so the separate /oauth2 object silently dropped the
auth_request on the app ingress: every request fell through to the backend and
ComfyUI answered 404 for the site root.
Move /oauth2 and / into the same ingress, so the auth annotations and both paths
belong to one object.
comfyui.riotpiao.com is served with no authentication at all: anyone who can
resolve the host reaches the UI and its queue. ComfyUI has no OIDC support, so the
front door is oauth2-proxy, and nginx asks it for a decision on every request.
Nothing inside the cluster calls this host, so the whole host is gated rather than
carved into open and closed paths. If a client ever needs direct access, add a skip
rule for its exact paths.
The ZITADEL app is created by the provisioning script, and the client secret,
cookie secret and client id live in a SOPS secret written by ksops.
Replace the dead secret-rotation scaffold (an Argo app that could not render, a
CRD that was never installed, and no controller code) with a working design.
One resource describes one credential:
spec.provider zitadel-machine-key | zitadel-oidc-client
spec.strategy overlap | atomic
spec.target Secret to write, consumers to roll, reload hot|restart
spec.verify introspection or token exchange
The state machine is shared by every provider:
Prepare -> Publish -> Verify -> Retire
Order is the safety property. Retire runs only for an overlapping provider and
only after the new credential obtained a token. A failure before that point leaves
the working credential in place, which is exactly what went wrong by hand today
when a regenerated Grafana secret was never re-pointed.
Machine keys overlap in ZITADEL, so that provider rotates with no gap. Client
secrets do not overlap, so that provider is atomic: it publishes, rolls the
consumer, verifies, and restores the previous value from a shadow key if the
introspection check fails.
The controller authenticates with its own machine key, the single bootstrap
credential, read from a mounted Secret. Everything else it creates itself, so no
credential value needs to live in git.
Verified: go build and go vet clean, 12 tests pass covering scheduling, the shadow
key rollback path, provider strategies and assertion signing.
Not yet wired to Argo CD: the image must be built and pushed first, so the app is
not added until a Deployment can actually pull.
The live cluster has no Authentik workload, no DNS record and no tunnel route,
but two declarations still pointed at it.
controlplane.tftpl still set kube-apiserver oidc-issuer-url to
https://authentik.riotpiao.com/application/o/kubernetes/ and pinned that hostname
in extraHostEntries. The live API server carries zero OIDC arguments, so the
template and reality had diverged: the next machine-config render would have
re-added an issuer that does not resolve.
management-service exported KMSVC_AUTHENTIK_ISSUER_URL and
KMSVC_AUTHENTIK_AUDIENCE, both empty. Rename them to KMSVC_ZITADEL_* so the names
describe the identity provider that actually exists.
The CoreDNS Corefile kept one rewrite for authentik.riotpiao.com. The Terraform
source file no longer contains it, so the live ConfigMap was stale; that line is
removed live and the next render converges without it.
Grafana sign-in failed with "Failed to get token from provider". The deployed
client secret no longer matched, because _ensure_oidc_apps called
_generate_client_secret for every app on every run, and ZITADEL stores exactly one
client secret per app: each call replaced the previous value and invalidated every
deployed copy.
Two changes:
1. Generate a secret only when the app is created. For an app that already exists,
reuse the value from state.json, because the API cannot read a secret back.
2. Re-encrypt grafana-oidc-secrets.enc.yaml with the live secret, so Grafana's
token exchange succeeds again.
Verified with the introspection endpoint:
old deployed secret -> {"error":"unauthorized_client"}
new secret -> {"active":false} (credentials accepted)
Grafana gains one contact point, `gotify`, of type webhook. It posts to the
bridge in the notifications namespace because Gotify rejects a Grafana payload
with 400 "Field 'message' is required".
maxAlerts 0 keeps every alert in one request, and the bridge names the extra
count in the message body instead of dropping alerts.
Gotify rejects a Grafana payload with 400 "Field 'message' is required", so a
translation step is mandatory. This adds a small standard-library Python
service: it accepts Grafana's JSON, builds a Gotify message, and posts it with an
app token.
Mapping:
firing -> priority 8, title "[FIRING] <alertname>"
resolved -> priority 3, title "[RESOLVED] <alertname>"
body -> the summary annotation plus severity, namespace, pod, instance, job
A multi-alert notification names the alert count instead of dropping the rest.
The script ships as a ConfigMap so no image build is needed. The app token comes
from a SOPS secret and belongs to Gotify application `grafana`.
The bridge runs server-side, so it cannot run a browser flow and ZITADEL has no
part in it. Give it a Gotify client token instead: GOTIFY_URLS points at the
public host, GOTIFY_CLIENT_TOKENS comes from a SOPS secret.
The token belongs to user `rock`, is named igotify-assist, and has
expiresAfterInactivitySeconds 0 so it never idles out. Delete the client row in
Gotify to revoke it.
Verified with the token: GET /client returns 200 and GET /message returns the
latest message.
The first value was a base32 string, so oauth2-proxy rejected it at startup:
cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 56
bytes.
Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put
oauth2-proxy in front of it and let nginx ask the proxy for every request.
The Gotify REST API keeps its own client-token check. /message, /application,
/client, /user, /version, /stream and /health are skipped by the proxy, so
Forgejo webhooks, the harness, poimen-sender and the Android client keep working
without a browser session. Only the web UI requires SSO.
The /oauth2 path needs its own ingress: the auth_request annotation would
otherwise also guard /oauth2/start, and the login entry point would demand a
login first.
The client id and secret come from project role provisioning:
`zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is
SOPS-encrypted under k8s/argocd/secrets.
Gotify's own login remains behind the proxy. ZITADEL decides who reaches the
service, Gotify decides what that person may do inside it.
The chart declares its ServiceAccount, ConfigMap, Role and RoleBinding as
pre-install/pre-upgrade Helm hooks with hook-delete-policy before-hook-creation.
ArgoCD maps those to PreSync and prunes them when a sync ends, so every sync
deleted the ServiceAccount and ConfigMap the ZITADEL Deployment mounts. New pods
failed with `serviceaccount "zitadel" not found`, leaving the IdP one restart
from an outage.
Add a zatadel-core Application owning hook-free copies, keeping them as ordinary
desired state that survives every sync. Prune is off for this path: deleting a
ServiceAccount here would recreate the outage.
The ConfigMap also carries the notification settings, pointing
DefaultInstance.SMTPConfiguration at the smtp-gotify relay, with
SMTPSenderAddressMatchesInstanceDomain false because the sender
[email protected] differs from the instance domain zatadel.riotpiao.com.
Drop the envVarsSecret approach: ZITADEL created no SMTP provider from those env
vars (no instance.smtp.config.added event).
ZITADEL must send verification codes, and it only speaks SMTP. Route that at
the smtp-gotify relay so codes reach Gotify and the phone.
The settings go in as a ZITADEL_DEFAULTINSTANCE_SMTPCONFIGURATION_* envFrom
secret, not as configmapConfig. Editing configmapConfig makes ArgoCD stop
managing the rendered zitadel-config-yaml ConfigMap, which the Deployment still
mounts; that twice left the IdP one restart away from failing. Env vars override
the same keys and leave that ConfigMap alone.
Validated with helm template 10.4.0: envFrom appears on both ZITADEL containers
and zitadel-config-yaml still renders.
Point DefaultInstance.SMTPConfiguration at the in-cluster smtp-gotify relay, so
verification codes and password resets arrive on the phone instead of being
dropped for lack of a mail path.
Validated with `helm template zitadel/zitadel 10.4.0` before syncing: the block
renders, and the zatadel-config-yaml ConfigMap is still produced. An earlier
attempt looked like it removed that ConfigMap, which is why this went in only
after a local render check.
The relay stages each message in /tmp before pushing. readOnlyRootFilesystem
left it with a read-only /tmp, so it answered '250 queued' and then dropped the
message: no Gotify push, no error line. Mount an emptyDir at /tmp.
The sync removed zitadel-config-yaml while the Deployment still mounts it, so
the next pod restart would have failed. Restore the previous render first; the
relay stays deployed and can be wired once the chart accepts the block.
Two defects in the first cut, both silent from the sender's side because the
relay answers 250 either way:
- GOTIFY_URL needs a trailing slash; without it the push has no path.
- GOTIFY_TITLE_TEMPLATE referenced .Subject, which the relay does not expose,
so template execution failed.
The relay's default title already carries the subject, so drop the template.
Also register the SOPS file in secret-generator.yaml: the sops-secrets app only
renders files listed there, so the Secret was never created and the relay sat in
CreateContainerConfigError.
Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped
being synced. The new smtp-gotify relay would have sat in git and never been
created.
Add an iam-support Application for that path, and drop the stale
rbac-dashboard-rolebinding reference from its kustomization: the file went away
with Authentik, and a missing resource fails the whole app.
ZITADEL sends verification codes and password resets by SMTP only, and this
cluster has no mail path. Every human user therefore sits at
verified_email = empty and can never finish a first login, at ZITADEL or at
Forgejo.
Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525
and forwards each message to the existing Gotify instance, so codes arrive on
the phone through gotify/igotify instead of email.
Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because
the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app
token in a SOPS-encrypted Secret; values never carry the token.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
Commit a02bee5 had both the Ingress rule and the cloudflared tunnel
hostname for authentik.riotpiao.com. Both were lost, so the tunnel
answered 404 for the hostname and the ingress had no matching rule.
The CoreDNS rewrite and the authentik-server Service were still intact;
only the two routing entries were missing. Re-add the Ingress (namespace
iam, service authentik-server port 80, TLS terminated by the wildcard
default-ssl-certificate) and the tunnel hostname.
The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.
- deployment: mount cloudflared-config (config.yml) and the
cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
the ingress certificate (force-ssl-redirect makes plaintext impossible)
Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.
Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).
Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.
Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:
write /prometheus/queries.active: no space left on device
failed to initialize active query tracker
18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
Two defects in the ksops generator:
1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
while every other entry uses four. YAML rejected the list, and kustomize
reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
39`. That failed the whole sops-secrets application, so no Secret in that app
could be created. Verified by running the same kustomize build inside
argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.
2. The masterkey value was generated during the first verification pass and
appeared in that session's output. Rotate it now, before the instance exists:
a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
ZITADEL data exists yet, so rotation costs nothing. After the first sync it
would mean re-encrypting system.encryption_keys.
The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.
Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
rock
pi