Two defects that a real rotation exposed:
merge writeSecret replaced the whole Secret map, so every key a consumer
keeps beside the credential disappeared. paperless lost
SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then
failed to start while the rotation reported success.
backoff a Failed attempt returned 0, so the controller retried in a tight loop.
It now waits two minutes and then retries.
All eight objects carry both identifiers ZITADEL needs: the app resource id for the
secret endpoint and the client id for introspection.
Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui,
argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload
exists for it.
Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
Eight objects cover grafana, argocd, comfyui, minio, paperless, immich, forgejo and
api-gw. Each names its ZITADEL app, the Secret it writes, the consumer to roll and
whether the consumer reloads hot or needs a restart.
All are suspended: the controller image does not exist yet, and an unsuspended
object would claim an intent that no process acts on.
The README records the retirement target for the 34 .enc.yaml files: six
root-of-trust secrets that cannot rotate themselves survive, everything else
becomes a KeyRotation and leaves git.