fix(longhorn): enable worker-1 scheduling, 400Gi reserved

32 degraded + 3 faulted volumes from replica rebuild failures.
cp-1/cp-2 over-committed; worker-1 had 602Gi free but scheduling
disabled. Enable it with 400Gi reserved for LLM weights.

- Add worker-1 to longhorn-nodes.yaml (allowScheduling: true)
- Add worker-1 disk config to add-disks-job (storageReserved: 400Gi,
  evictionRequested: false)
This commit is contained in:
2026-09-28 23:06:22 +08:00
parent bb24418a2e
commit c472a994ce
2 changed files with 33 additions and 25 deletions
@@ -117,6 +117,22 @@ spec:
echo " ✓ talos-cp-3 already has $CP3_DISKS disks configured"
fi
# worker-1: enable scheduling, reserve 400Gi for LLM weights,
# clear eviction flag that was set when scheduling was disabled.
echo "Configuring worker-1 disk..."
kubectl -n longhorn-system patch nodes.longhorn.io worker-1 --type merge -p '{
"spec": {
"disks": {
"default-disk-1030400000000": {
"allowScheduling": true,
"evictionRequested": false,
"storageReserved": 429496729600
}
}
}
}'
echo " ✓ worker-1 disk configured (400Gi reserved)"
echo
echo "Waiting for disks to be ready..."
sleep 10
+17 -25
View File
@@ -1,33 +1,25 @@
# Longhorn Node CRDs for cp-2 and cp-3.
# These nodes have the control-plane taint, so Longhorn doesn't auto-discover them.
# Explicit Node CRDs + the taint-toleration setting enable storage across all 3 nodes.
# Longhorn Node CRDs — enable scheduling on nodes that need explicit opt-in.
#
# `spec.disks` is deliberately absent. Longhorn owns disk identity: it names the
# entry itself (`default-disk-080400000000`, not `default-disk`) and writes
# `storageReserved`, `diskType` and `evictionRequested` into it. Declaring a
# `default-disk` key here never matched the live one, so the Application sat
# OutOfSync and selfHeal kept trying to add a SECOND disk record pointing at the
# same /var/lib/longhorn path — which is worse than the drift it was fixing.
# Control-plane nodes (cp-2, cp-3) have taints so Longhorn doesn't auto-discover.
# worker-1 was manually disabled; re-enabled 2026-09-28 to relieve degraded-volume
# pressure (400Gi reserved for LLM weights via the add-disks-job, ~300Gi available
# for general replica scheduling).
#
# What these objects are actually for is `allowScheduling: true` on tainted
# control-plane nodes. That is all they need to declare.
# `spec.disks` deliberately absent — Longhorn owns disk identity and writes
# storageReserved/diskType/evictionRequested itself. Disk-level config lives in
# longhorn-add-disks-job.yaml. These objects only set `allowScheduling: true`.
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
name: talos-cp-2
namespace: longhorn-system
spec:
name: talos-cp-2
allowScheduling: true
tags: []
metadata: { name: talos-cp-2, namespace: longhorn-system }
spec: { name: talos-cp-2, allowScheduling: true, tags: [] }
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
name: talos-cp-3
namespace: longhorn-system
spec:
name: talos-cp-3
allowScheduling: true
tags: []
metadata: { name: talos-cp-3, namespace: longhorn-system }
spec: { name: talos-cp-3, allowScheduling: true, tags: [] }
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata: { name: worker-1, namespace: longhorn-system }
spec: { name: worker-1, allowScheduling: true, tags: [] }