Point DefaultInstance.SMTPConfiguration at the in-cluster smtp-gotify relay, so
verification codes and password resets arrive on the phone instead of being
dropped for lack of a mail path.
Validated with `helm template zitadel/zitadel 10.4.0` before syncing: the block
renders, and the zatadel-config-yaml ConfigMap is still produced. An earlier
attempt looked like it removed that ConfigMap, which is why this went in only
after a local render check.
The relay stages each message in /tmp before pushing. readOnlyRootFilesystem
left it with a read-only /tmp, so it answered '250 queued' and then dropped the
message: no Gotify push, no error line. Mount an emptyDir at /tmp.
The sync removed zitadel-config-yaml while the Deployment still mounts it, so
the next pod restart would have failed. Restore the previous render first; the
relay stays deployed and can be wired once the chart accepts the block.
Two defects in the first cut, both silent from the sender's side because the
relay answers 250 either way:
- GOTIFY_URL needs a trailing slash; without it the push has no path.
- GOTIFY_TITLE_TEMPLATE referenced .Subject, which the relay does not expose,
so template execution failed.
The relay's default title already carries the subject, so drop the template.
Also register the SOPS file in secret-generator.yaml: the sops-secrets app only
renders files listed there, so the Secret was never created and the relay sat in
CreateContainerConfigError.
Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped
being synced. The new smtp-gotify relay would have sat in git and never been
created.
Add an iam-support Application for that path, and drop the stale
rbac-dashboard-rolebinding reference from its kustomization: the file went away
with Authentik, and a missing resource fails the whole app.
ZITADEL sends verification codes and password resets by SMTP only, and this
cluster has no mail path. Every human user therefore sits at
verified_email = empty and can never finish a first login, at ZITADEL or at
Forgejo.
Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525
and forwards each message to the existing Gotify instance, so codes arrive on
the phone through gotify/igotify instead of email.
Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because
the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app
token in a SOPS-encrypted Secret; values never carry the token.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
Commit a02bee5 had both the Ingress rule and the cloudflared tunnel
hostname for authentik.riotpiao.com. Both were lost, so the tunnel
answered 404 for the hostname and the ingress had no matching rule.
The CoreDNS rewrite and the authentik-server Service were still intact;
only the two routing entries were missing. Re-add the Ingress (namespace
iam, service authentik-server port 80, TLS terminated by the wildcard
default-ssl-certificate) and the tunnel hostname.
The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.
- deployment: mount cloudflared-config (config.yml) and the
cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
the ingress certificate (force-ssl-redirect makes plaintext impossible)
Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.
Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).
Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.
Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:
write /prometheus/queries.active: no space left on device
failed to initialize active query tracker
18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
Two defects in the ksops generator:
1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
while every other entry uses four. YAML rejected the list, and kustomize
reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
39`. That failed the whole sops-secrets application, so no Secret in that app
could be created. Verified by running the same kustomize build inside
argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.
2. The masterkey value was generated during the first verification pass and
appeared in that session's output. Rotate it now, before the instance exists:
a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
ZITADEL data exists yet, so rotation costs nothing. After the first sync it
would mean re-encrypting system.encryption_keys.
The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.
Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.
macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.
The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.
Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
The Application pointed at github.com/ToyoGlobal/azhi-infra, branch
feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled:
status stayed Unknown because the source repository could not be read, and the
destination namespace azi-riotpiao was never created. ArgoCD reported it as an
OutOfSync managed resource of homelab-root with nothing behind it.
Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from
that repository is referenced any more, so the allowlist entry is dead weight.
Not touched here, because it is not in git: Secret azhi-infra-github in
namespace argocd holds the read-only GitHub credential for that repository.
Delete it separately once no other Application needs the repo.
The live Application object was deleted by hand. Until this commit reaches
main, the next homelab-root sync recreates it, because main still declares it.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.
Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
disk is gone
The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
The homelab AppProject allowlist had no azhi repository, so an Application
pointing at ToyoGlobal/azhi-infra fails admission with 'repository not
permitted'. Adding the URL is the whole gate.
The Application itself targets the repository default branch
(harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That
overlay renders postgres and the API into namespace azi-riotpiao.
The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR
pull credential and azhi-riotpiao-app-env carries the database credentials,
the JWT secrets and the AES key.
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
cluster-maintenance tracks main (targetRevision: main), so the cluster never
had the CronJob or its Secret
This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
would have reclaimed nothing
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.
The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:
- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).
Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
Exit code 143 proved the previous job killed itself: the script's own cmdline contains the
string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver')
and skipping $$ removes that failure mode. Staggered 2 minutes apart.
Static pods survive mirror-pod deletion, so the first bootstrap job had no effect.
These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself,
staggered by 100s so only one control plane restarts at a time.
The first job could not pull its image; this one deletes the apiserver pods via the
in-cluster API with the mounted service-account token instead of a kubectl image.
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.
Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.
Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
rock
pi