Adds expand-replicas-job.yaml: PostSync hook Job that:
- Waits for all 3 Longhorn nodes to be Ready
- Patches every volume with numberOfReplicas < 3 to 3
- Runs idempotently on every longhorn-config sync (BeforeHookCreation
deletes previous job, so re-runs are safe)
This ensures existing 1-replica volumes (created before the HA setup) get
expanded automatically via GitOps, not via manual kubectl patch.
Why PostSync: needs to run AFTER the taint-toleration setting and Node CRDs
are applied, otherwise there aren't 3 nodes available yet and the expansion
would fail (Longhorn can't create replicas on nodes that don't exist).
Changes:
- k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml: new Setting
to tolerate node-role.kubernetes.io/control-plane:NoSchedule taint,
allowing Longhorn DaemonSet to run on cp-2/cp-3 (not just cp-1)
- k8s/infrastructure/longhorn/longhorn-nodes.yaml: explicit Node CRDs for
talos-cp-2 and talos-cp-3 (auto-discovery doesn't work when nodes have
taints; these define /var/lib/longhorn as the storage path)
- k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml: bump
numberOfReplicas from 1→3 (true HA: each volume gets 3 copies across
3 nodes; if one node fails, 2 others still have the data)
- k8s/infrastructure/longhorn/kustomization.yaml: add new resources
Root cause: Longhorn was only running on talos-cp-1 (.213) because cp-2/cp-3
have the control-plane taint and Longhorn DaemonSet had no matching toleration.
Every workload with a PVC was forced to schedule on cp-1 (via nodeSelector or
implicit co-location with the storage), defeating the entire purpose of a 3-node
HA cluster.
With this fix:
- Longhorn manager runs on all 3 nodes
- Storage is replicated 3x (erasure-coded across nodes)
- Pods can schedule on any node without PVC attachment failures
- True HA: lose 1 node, cluster still serves all volumes
Story Crater Bot