fix(longhorn): isolate GPU worker from general storage scheduling
- Remove expand-replicas-job (blindly forced all volumes to 3 replicas, ignoring StorageClass settings) - Add diskSelector: 'storage' to longhorn and longhorn-cnpg StorageClasses so replicas only land on CP nodes (cp-1, cp-2, cp-3) - Tag all CP node disks with 'storage' via PostSync job (disk names are runtime-discovered, can't hardcode in Node CRs) - Disable scheduling on worker-1 Node CR — only longhorn-llm-local (diskSelector: 'llm') can use it - worker-1 is GPU-only: llm-models and comfyui use dedicated SCs
This commit is contained in:
@@ -1,16 +1,14 @@
|
||||
# Longhorn Node CRDs for cp-2 and cp-3.
|
||||
# These nodes have the control-plane taint, so Longhorn doesn't auto-discover them.
|
||||
# Explicit Node CRDs + the taint-toleration setting enable storage across all 3 nodes.
|
||||
# Longhorn Node CRDs for cp-2, cp-3, and worker-1.
|
||||
# cp-2/cp-3 have the control-plane taint, so Longhorn doesn't auto-discover them.
|
||||
# Explicit Node CRDs + the taint-toleration setting enable storage across all 3 CP nodes.
|
||||
#
|
||||
# `spec.disks` is deliberately absent. Longhorn owns disk identity: it names the
|
||||
# entry itself (`default-disk-080400000000`, not `default-disk`) and writes
|
||||
# `storageReserved`, `diskType` and `evictionRequested` into it. Declaring a
|
||||
# `default-disk` key here never matched the live one, so the Application sat
|
||||
# OutOfSync and selfHeal kept trying to add a SECOND disk record pointing at the
|
||||
# same /var/lib/longhorn path — which is worse than the drift it was fixing.
|
||||
# `spec.disks` is deliberately absent for CP nodes. Longhorn owns disk identity:
|
||||
# it names the entry itself and writes `storageReserved`, `diskType` and
|
||||
# `evictionRequested` into it. Disk tags are applied via kubectl patch (see
|
||||
# longhorn-tag-disks-job.yaml) since disk names are runtime-discovered.
|
||||
#
|
||||
# What these objects are actually for is `allowScheduling: true` on tainted
|
||||
# control-plane nodes. That is all they need to declare.
|
||||
# worker-1 has scheduling disabled so only StorageClasses with explicit
|
||||
# diskSelector (e.g. `llm` for longhorn-llm-local) can use it.
|
||||
---
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Node
|
||||
@@ -31,3 +29,13 @@ spec:
|
||||
name: talos-cp-3
|
||||
allowScheduling: true
|
||||
tags: []
|
||||
---
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Node
|
||||
metadata:
|
||||
name: worker-1
|
||||
namespace: longhorn-system
|
||||
spec:
|
||||
name: worker-1
|
||||
allowScheduling: false
|
||||
tags: []
|
||||
|
||||
Reference in New Issue
Block a user