feat(longhorn): one-shot destructive disk health scan job #95

2026-11-02T04:28:38Z

rock commented

2026-11-02 04:28:38 +00:00

Adds a one-shot Job that proves a candidate Longhorn disk is fully usable by writing a CRC32C-verified pattern across the whole raw device and reading every block back. SMART is a self-assessment; this is the actual proof.

Deliberately not an ArgoCD hook. It is destructive. A PostSync hook would re-run on every sync and wipe the volume. Applied by hand, once.

Why not Talos. Talos declares disks by mounting them (machine.disks). It has no primitive to execute a command on the host, so a scan cannot live in the machine config. A Kubernetes Job is the IaC-correct home.

Order of operations

Scan the raw device (this job) — disk must be unmounted
Add the disk to terraform.tfvars.local longhorn_disks, make apply-cp
Register with Longhorn via longhorn-add-disks-job.yaml

First target. talos-cp-1 /dev/sdb, a 2 TB ST2000LM003 (wwid naa.50004cf20f5c4c39), no partitions, not mounted, not referenced by Longhorn.

Measured so far. 124 MB/s write on the live drive. Budget ~8-9 h for write plus verify.

Known caveat on this disk. SMART passes with zero reallocated, zero pending, zero CRC errors. But Load_Cycle_Count is 431,856 against a typical ~600k rating, and power-on hours are 21,956. A passing scan does not make this a good home for a continuously-fsyncing Longhorn replica. Treat it as scratch or backup tier.

Adds a one-shot Job that proves a candidate Longhorn disk is fully usable by writing a CRC32C-verified pattern across the whole raw device and reading every block back. SMART is a self-assessment; this is the actual proof. **Deliberately not an ArgoCD hook.** It is destructive. A PostSync hook would re-run on every sync and wipe the volume. Applied by hand, once. **Why not Talos.** Talos declares disks by mounting them (`machine.disks`). It has no primitive to execute a command on the host, so a scan cannot live in the machine config. A Kubernetes Job is the IaC-correct home. **Order of operations** 1. Scan the raw device (this job) — disk must be unmounted 2. Add the disk to `terraform.tfvars.local` `longhorn_disks`, `make apply-cp` 3. Register with Longhorn via `longhorn-add-disks-job.yaml` **First target.** `talos-cp-1 /dev/sdb`, a 2 TB ST2000LM003 (wwid `naa.50004cf20f5c4c39`), no partitions, not mounted, not referenced by Longhorn. **Measured so far.** 124 MB/s write on the live drive. Budget ~8-9 h for write plus verify. **Known caveat on this disk.** SMART passes with zero reallocated, zero pending, zero CRC errors. But `Load_Cycle_Count` is 431,856 against a typical ~600k rating, and power-on hours are 21,956. A passing scan does not make this a good home for a continuously-fsyncing Longhorn replica. Treat it as scratch or backup tier.

rock added 1 commit 2026-11-02 04:28:39 +00:00

feat(longhorn): add one-shot disk health scan job 5fbbe45593

SMART self-assessment is not proof a disk is usable. This job writes a
CRC32C-verified pattern across the whole raw device with fio, then reads
every block back and compares. Any unreadable or mismatched sector fails
the Job.

Deliberately NOT an ArgoCD hook: it is destructive, and a PostSync hook
would re-run on every sync and wipe the volume. Apply by hand, once, on a
disk with no partitions and no mounts.

Talos cannot host this. Talos declares disks by mounting them
(machine.disks); it has no primitive to execute a command on the host, so a
scan cannot live in the machine config. A Kubernetes Job is the IaC-correct
home.

Order of operations:
  1. scan the raw device (this job) - disk must be unmounted
  2. add the disk to terraform.tfvars.local longhorn_disks, make apply-cp
  3. register it with Longhorn via longhorn-add-disks-job.yaml

First target: talos-cp-1 /dev/sdb, a 2 TB ST2000LM003 (wwid
naa.50004cf20f5c4c39), currently with no partitions and not mounted.
Measured 124 MB/s write, so budget ~8-9 h for write plus verify.

rock merged commit 1fd5f25932 into main

2026-11-02 04:32:36 +00:00

rock deleted branch feat/longhorn-disk-scan-job

2026-11-02 04:32:45 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#95