rock
abfdda5f57
etcd's backend file only grows: the apiserver compacts every 5m, but compaction marks superseded revisions free inside the file, and only defrag returns that space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached 2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets could not post node status (3 nodes NotReady), apiserver writes returned 500 and the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after compaction - the rest was dead pages. Compaction had also stalled while the alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have reclaimed nothing. Compact must come first. An earlier revision of this CronJob never ran, for two independent reasons, both fixed here: - its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] failed with exec: "/bin/sh": no such file or directory - it existed only on a side branch while the ArgoCD application cluster-maintenance tracks main, so the cluster never had it Contents: - CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the day-of-month field fires on the 1st, 16th and 31st, putting two runs one day apart. - Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4 against the 3.6.11 servers. - Compaction uses the MINIMUM revision across members, so a lagging member is never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat ~1.2M entries behind the leader with a frozen raft term and needed a reboot to rejoin. - Defrag skips any member whose applied index lags the leader by 100k+, and runs one member at a time since defrag pauses that member's apply loop. - Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a client certificate signed by the cluster's etcd CA, valid to 2036-10.
41 lines
1.2 KiB
YAML
41 lines
1.2 KiB
YAML
apiVersion: viaduct.ai/v1
|
|
kind: ksops
|
|
metadata:
|
|
name: sops-secret-generator
|
|
annotations:
|
|
config.kubernetes.io/function: |
|
|
exec:
|
|
path: ksops
|
|
files:
|
|
- agent-pod-models.enc.yaml
|
|
- agent-pod-ssh-key.enc.yaml
|
|
- authentik-secrets.enc.yaml
|
|
- cloudflare-secrets.enc.yaml
|
|
- forgejo-registry-pat.enc.yaml
|
|
- forgejo-registry-pat-api.enc.yaml
|
|
- forgejo-registry-pull.enc.yaml
|
|
- forgejo-runner-token.enc.yaml
|
|
- forgejo-secrets.enc.yaml
|
|
- grafana-oidc-secrets.enc.yaml
|
|
- grafana-secrets.enc.yaml
|
|
- homarr-auth-oidc.enc.yaml
|
|
- homarr-db-encryption.enc.yaml
|
|
- homarr-secrets.enc.yaml
|
|
- homelab-ca-secrets.enc.yaml
|
|
- loki-secrets.enc.yaml
|
|
- minio-secrets.enc.yaml
|
|
- paperless-secrets.enc.yaml
|
|
- vault-secrets.enc.yaml
|
|
- vault-unseal-keys.enc.yaml
|
|
- portfolio-secrets.enc.yaml
|
|
- gotify-admin-secrets.enc.yaml
|
|
- gotify-tokens-secrets.enc.yaml
|
|
- gotify-smtp-secrets.enc.yaml
|
|
- forgejo-smtp-secrets.enc.yaml
|
|
- paperless-ai-secrets.enc.yaml
|
|
- memory-agent-oidc.enc.yaml
|
|
- oauth2-credentials.enc.yaml
|
|
- forgejo-registry-token.enc.yaml
|
|
- pi-sandbox-secrets.enc.yaml
|
|
- etcd-defrag-creds.enc.yaml
|