fix(minio): freeze the community image by digest and alert on pull failure
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.
Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.
Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
This commit is contained in:
@@ -27,6 +27,24 @@ spec:
|
||||
annotations:
|
||||
summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
|
||||
|
||||
# ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
|
||||
- alert: PodImagePullFailure
|
||||
expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
|
||||
|
||||
# Pod not ready
|
||||
- alert: PodNotReady
|
||||
expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
|
||||
|
||||
# CrashLoopBackOff
|
||||
- alert: PodCrashLooping
|
||||
expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0
|
||||
|
||||
Reference in New Issue
Block a user