fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2 #62

2026-10-13T00:09:54Z

rock commented

2026-10-13 00:09:54 +00:00

The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk) failed.

Evidence

Device on talos-cp-2	ioerr_cnt
sda, sdb, sdc, sdd	3
sde	54,199,562, rising ~7,000/minute

Its XFS filesystem hung, so five longhorn replica processes plus longhorn-manager blocked in uninterruptible D state in openat and unlinkat. D state cannot be killed, so they held the node's three replica rebuild slots permanently. No new replica could start on talos-cp-2, and every new volume there failed with FailedAttachVolume ... DeadlineExceeded — which is what stopped the azhi database from starting.

Change

Removed from every declaration so it cannot be reintroduced:

terraform/terraform.tfvars.local.bak — longhorn_disks for talos-cp-2.
k8s/infra/longhorn/longhorn-add-disks-job.yaml — the disk4 patch block.
The same Job's disk-count guard, 5 to 4. Without this the Job fires on every run once the disk is gone.

Still to do, outside this PR

The live cluster has the disk fenced (allowScheduling=false, evictionRequested=true). Once talos-cp-2 is powered back on, the replicas are evicted, then the disk is retired at the controller and the Talos machine config is re-applied.

The disk behind `/var/lib/longhorn-disk4` (sde1, 2TB PERC H710P virtual disk) failed. ## Evidence | Device on talos-cp-2 | ioerr_cnt | |---|---| | sda, sdb, sdc, sdd | 3 | | **sde** | **54,199,562**, rising ~7,000/minute | Its XFS filesystem hung, so five `longhorn` replica processes plus `longhorn-manager` blocked in uninterruptible `D` state in `openat` and `unlinkat`. `D` state cannot be killed, so they held the node's three replica rebuild slots permanently. No new replica could start on talos-cp-2, and every new volume there failed with `FailedAttachVolume ... DeadlineExceeded` — which is what stopped the azhi database from starting. ## Change Removed from every declaration so it cannot be reintroduced: 1. `terraform/terraform.tfvars.local.bak` — `longhorn_disks` for `talos-cp-2`. 2. `k8s/infra/longhorn/longhorn-add-disks-job.yaml` — the disk4 patch block. 3. The same Job's disk-count guard, `5` to `4`. Without this the Job fires on every run once the disk is gone. ## Still to do, outside this PR The live cluster has the disk fenced (`allowScheduling=false`, `evictionRequested=true`). Once `talos-cp-2` is powered back on, the replicas are evicted, then the disk is retired at the controller and the Talos machine config is re-applied.

rock added 5 commits 2026-10-13 00:09:55 +00:00

fix(minio): freeze the community image by digest and alert on pull failure d7b96fb29b

MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.

Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.

Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}

feat(storage): add local-path-provisioner as the fleet's persistent-storage backend cc62f74404

feat(etcd): weekly defrag CronJob to stay under the etcd backend quota 1ce9f86635

etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.

The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:

- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
  revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
  CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).

Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).

fix(etcd): make the defrag CronJob runnable and 15-day e4caa7e841

The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
  exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
  cluster-maintenance tracks main (targetRevision: main), so the cluster never
  had the CronJob or its Secret

This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
  matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
  16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
  compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
  behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
  NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
  would have reclaimed nothing

fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2 d21ee9630f

The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.

Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
  disk is gone

The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.

rock merged commit 7a1213d504 into main

2026-10-13 00:10:02 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#62