etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
rock
abfdda5f57