fix(etcd): deploy the defrag CronJob so etcd cannot fill the backend quota #59

2026-10-12T12:27:18Z

rock commented

2026-10-12 12:27:18 +00:00

Root cause

The etcd maintenance CronJob exists only on an unmerged branch. ArgoCD renders
main, so the cluster never received the CronJob or its Secret. Nothing
compacted or defragged etcd, so the backend file grew past the 2 GiB quota and
etcd rejected every write cluster-wide.

Observed on 2026-10-12:

create namespace returned etcdserver: mvcc: database space exceeded
all three control-plane members held 2.1 GB against a 2 GiB quota
dbSizeInUse equalled dbSize, so a defrag alone reclaimed 36 KB
node Ready heartbeats were 87 hours stale, because the node controller
could not persist a NotReady transition

Change

k8s/infra/cluster-maintenance/etcd-defrag-cronjob.yaml
k8s/argocd/secrets/etcd-defrag-creds.enc.yaml (SOPS, age recipient age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla)
one generator entry in k8s/argocd/secrets/secret-generator.yaml

Verification

The same sequence ran by hand against the live cluster: compact to the minimum
revision, wait 20 seconds, defrag each member, then disarm. Every member fell
from 2.1 GB to 46.8 MB and the NOSPACE alarm cleared. Writes succeeded
afterwards.

## Root cause The etcd maintenance CronJob exists only on an unmerged branch. ArgoCD renders `main`, so the cluster never received the CronJob or its Secret. Nothing compacted or defragged etcd, so the backend file grew past the 2 GiB quota and etcd rejected every write cluster-wide. Observed on 2026-10-12: - `create namespace` returned `etcdserver: mvcc: database space exceeded` - all three control-plane members held 2.1 GB against a 2 GiB quota - `dbSizeInUse` equalled `dbSize`, so a defrag alone reclaimed 36 KB - node `Ready` heartbeats were 87 hours stale, because the node controller could not persist a `NotReady` transition ## Change - `k8s/infra/cluster-maintenance/etcd-defrag-cronjob.yaml` - `k8s/argocd/secrets/etcd-defrag-creds.enc.yaml` (SOPS, age recipient `age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla`) - one generator entry in `k8s/argocd/secrets/secret-generator.yaml` ## Verification The same sequence ran by hand against the live cluster: compact to the minimum revision, wait 20 seconds, defrag each member, then disarm. Every member fell from 2.1 GB to 46.8 MB and the NOSPACE alarm cleared. Writes succeeded afterwards.

rock added 1 commit 2026-10-12 12:27:19 +00:00

fix(etcd): add a working 15-day defrag CronJob under the backend quota abfdda5f57

etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.

An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
  failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
  cluster-maintenance tracks main, so the cluster never had it

Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
  epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
  day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
  apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
  against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
  never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
  ~1.2M entries behind the leader with a frozen raft term and needed a reboot
  to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
  runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
  client certificate signed by the cluster's etcd CA, valid to 2036-10.

rock merged commit e150969070 into main

2026-10-12 12:27:29 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#59