rock
b8f934806e
comfyui served nothing and its oauth2-proxy could not start: the Secret carried only client-secret while the Deployment asked for client-id, cookie-secret and client-secret. Rather than repair a service nobody used, remove it from every layer. Gone: - k8s/apps/comfyui/* and its ArgoCD Application - the cloudflared tunnel rule for comfyui.riotpiao.com - the CoreDNS rewrite in the terraform Corefile (the tracked source) - the Cloudflare DNS record (terraform tfvars) - the ZITADEL OIDC app and its two rotation tables - the KeyRotation object Not gone by this commit: the comfyui namespace and its 50Gi PVC. ArgoCD does not own either, so a person deletes both by hand after the prune. The local cluster-config/talos-*.yaml copies are gitignored; they carry the same removal and need an apply-config to reach the live CoreDNS.
Credential rotation
One KeyRotation object describes one credential. The controller creates the value,
writes the Secret, proves the new value works, then retires the old one.
Why the state order is the safety property
Prepare -> Publish -> Verify -> Retire
Retire runs only for a provider whose old credential still works after the publish
step, and only after Verify passes. A failure before that point leaves the working
credential untouched. The hand-run rotation that broke Grafana did the opposite: it
regenerated the secret and never re-pointed the consumer.
Two strategies, decided by the provider, not by preference
| Provider | Overlap in ZITADEL | Strategy | Retire |
|---|---|---|---|
zitadel-machine-key |
many keys per service user | overlap |
after the new key obtains a token |
zitadel-oidc-client |
one secret per app | atomic |
nothing to retire |
The single master credential
The controller authenticates with one machine key, mounted from Secret
iam/key-rotation-controller. That key is the only credential the cluster cannot
rotate by itself, so it is the only one that stays encrypted in git.
Every other credential is created by the controller, which means the 34 .enc.yaml
files under k8s/argocd/secrets/ are migration debt, not a design. The retirement
target is one root-of-trust file per thing that cannot rotate itself:
| Survives (root of trust) | Why it cannot rotate |
|---|---|
| ZITADEL instance masterkey | encrypts the whole ZITADEL database |
| Vault unseal keys | unsealing needs a human-held key |
| Talos repair config | recovery path, must work when the cluster does not |
| homelab CA | signs the wildcard certificate |
| controller bootstrap key | chicken and egg: it creates every other key |
| etcd defrag credentials | needed by a job that runs outside the controller |
Everything else — Grafana, Argo CD, Forgejo, MinIO, Paperless, Immich, api-gw, Gotify — is a KeyRotation object, and its secret leaves git.
Status
The CRD is installed. Eight KeyRotation objects exist for the consumers above and
all are suspend: true: the controller image does not exist yet, so no process acts
on them. poimen-memory is excluded from this migration on purpose.