fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2

The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.

Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
  disk is gone

The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
This commit is contained in:
rock
2026-10-13 08:09:42 +08:00
parent e4caa7e841
commit d21ee9630f
2 changed files with 21 additions and 11 deletions
+13 -10
View File
@@ -41,13 +41,24 @@ spec:
# - /var/lib/longhorn-disk1 (sdb1, 600GB)
# - /var/lib/longhorn-disk2 (sdc1, 500GB)
# - /var/lib/longhorn-disk3 (sdd1, 160GB)
# - /var/lib/longhorn-disk4 (sde1, 2TB)
#
# /var/lib/longhorn-disk4 (sde1, the 2TB PERC virtual disk) was RETIRED
# on 2026-10-12 and must not be re-added. The device returned SCSI I/O
# errors at ~7,000/minute, the XFS filesystem on it hung, and every
# process that touched it blocked in unkillable D state. Those unkillable
# processes held Longhorn's per-node rebuild slots forever, so no new
# replica could start on talos-cp-2 and every new volume there failed to
# attach. It is removed from the Terraform disk list too, so a re-render
# does not put it back.
echo "Checking talos-cp-2..."
CURRENT_DISKS=$(kubectl -n longhorn-system get nodes.longhorn.io talos-cp-2 -o json | jq -r '.spec.disks | keys | length')
echo " Current disk count: $CURRENT_DISKS"
if [ "$CURRENT_DISKS" -lt 5 ]; then
# 4 = default-disk-080400000000 + disk1..disk3. It was 5 while disk4
# existed; leaving the old threshold would re-apply the patch on every
# run once disk4 is gone.
if [ "$CURRENT_DISKS" -lt 4 ]; then
echo " Adding extra disks to talos-cp-2..."
kubectl -n longhorn-system patch nodes.longhorn.io talos-cp-2 --type merge -p '{
"spec": {
@@ -75,14 +86,6 @@ spec:
"path": "/var/lib/longhorn-disk3",
"storageReserved": 5368709120,
"tags": []
},
"disk4": {
"allowScheduling": true,
"diskType": "filesystem",
"evictionRequested": false,
"path": "/var/lib/longhorn-disk4",
"storageReserved": 21474836480,
"tags": []
}
}
}