feat(etcd): weekly defrag CronJob to stay under the etcd backend quota

etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.

The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:

- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
  revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
  CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).

Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).

This commit is contained in:

harness

2026-10-07 13:37:45 +08:00

parent cc62f74404

commit 1ce9f86635

3 changed files with 129 additions and 0 deletions

									
										k8s/argocd/secrets/secret-generator.yaml
									
		+1
		
												View File
												
				@@ -34,3 +34,4 @@ files:

				  - forgejo-smtp-secrets.enc.yaml

				  - paperless-ai-secrets.enc.yaml

				    - pi-sandbox-secrets.enc.yaml

				    - etcd-defrag-creds.enc.yaml