fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2

The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.

Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
  disk is gone

The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
This commit is contained in:
rock
2026-10-13 08:09:42 +08:00
parent e4caa7e841
commit d21ee9630f
2 changed files with 21 additions and 11 deletions
+8 -1
View File
@@ -20,6 +20,7 @@ controlplane_configs = {
"talos-cp-1" = {
hostname = "talos-cp-1"
lan_ip = "192.168.1.166"
public_ip = "https://k8s-api.riotpiao.com"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/nvme0n1"
@@ -32,6 +33,7 @@ controlplane_configs = {
"talos-cp-2" = {
hostname = "talos-cp-2"
lan_ip = "192.168.1.214"
public_ip = "https://k8s-api.riotpiao.com"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/disk/by-id/wwn-0x644a842029bd3f002720989b07d7143d"
@@ -39,7 +41,11 @@ controlplane_configs = {
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b60f4375d97062", mountpoint = "/var/lib/longhorn-disk1" },
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b5ffeb8bb8b47f", mountpoint = "/var/lib/longhorn-disk2" },
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b6000c8dabb266", mountpoint = "/var/lib/longhorn-disk3" },
{ device = "/dev/disk/by-id/wwn-0x6b083fe0c5782700321370dc23fd765b", mountpoint = "/var/lib/longhorn-disk4" },
# /var/lib/longhorn-disk4 (wwn-0x6b083fe0c5782700321370dc23fd765b) was
# retired 2026-10-12: the 2TB PERC virtual disk failed with continuous
# SCSI I/O errors, hung its XFS filesystem, and wedged Longhorn replica
# rebuilds on talos-cp-2 with unkillable D-state processes. Do not
# re-add it.
]
zone = "az-b"
allow_scheduling = true
@@ -49,6 +55,7 @@ controlplane_configs = {
"talos-cp-3" = {
hostname = "talos-cp-3"
lan_ip = "192.168.1.162"
public_ip = "https://k8s-api.riotpiao.com"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/nvme0n1"