retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:
write /prometheus/queries.active: no space left on device
failed to initialize active query tracker
18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
rock
Story Crater Bot