riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity

fix(monitoring): cap Prometheus retention below the PVC size

Browse Source 
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:

  write /prometheus/queries.active: no space left on device
  failed to initialize active query tracker

18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
This commit is contained in:
rock
2026-10-24 21:43:01 -07:00
parent 6312674058
commit b7fc6ed60b
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/infra/monitoring/prometheus-values.yaml
+4 -1
View File
@@ -55,7 +55,10 @@ alertmanager:
prometheus: prometheus:
prometheusSpec: prometheusSpec:
retention: 30d retention: 30d
retentionSize: "95GB" # Must stay under the PVC size below (25Gi). At 95GB the limit never
# triggered, the volume filled, and Prometheus crashlooped with
# "no space left on device" writing /prometheus/queries.active.
retentionSize: "18GB"
# scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate # scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate
# the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no # the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no