fix(etcd): add a working 15-day defrag CronJob under the backend quota
etcd's backend file only grows: the apiserver compacts every 5m, but compaction marks superseded revisions free inside the file, and only defrag returns that space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached 2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets could not post node status (3 nodes NotReady), apiserver writes returned 500 and the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after compaction - the rest was dead pages. Compaction had also stalled while the alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have reclaimed nothing. Compact must come first. An earlier revision of this CronJob never ran, for two independent reasons, both fixed here: - its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] failed with exec: "/bin/sh": no such file or directory - it existed only on a side branch while the ArgoCD application cluster-maintenance tracks main, so the cluster never had it Contents: - CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the day-of-month field fires on the 1st, 16th and 31st, putting two runs one day apart. - Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4 against the 3.6.11 servers. - Compaction uses the MINIMUM revision across members, so a lagging member is never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat ~1.2M entries behind the leader with a frozen raft term and needed a reboot to rejoin. - Defrag skips any member whose applied index lags the leader by 100k+, and runs one member at a time since defrag pauses that member's apply loop. - Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a client certificate signed by the cluster's etcd CA, valid to 2036-10.
This commit is contained in:
@@ -37,3 +37,4 @@ files:
|
||||
- oauth2-credentials.enc.yaml
|
||||
- forgejo-registry-token.enc.yaml
|
||||
- pi-sandbox-secrets.enc.yaml
|
||||
- etcd-defrag-creds.enc.yaml
|
||||
|
||||
Reference in New Issue
Block a user