fix(minio): pin frozen community image and alert on image pull failure
MinIO deleted the community edition images upstream. An anonymous pull of
docker.io/minio/minio and quay.io/minio/mc returns HTTP 401, and the Docker Hub
API returns "object not found". talos-cp-1 dropped the cached image during the
DiskPressure event on 2026-09-22. The tenant pod was recreated on 2026-09-23 and
has been in ImagePullBackOff since. MinIO served no traffic for 13 days, which
put vault-0 and loki-0 into CrashLoopBackOff and left the paperless backups
stuck.
Pin the tenant to the last community release by digest, and repoint the two
minio/mc references at the archived Bitnami client image, also by digest. Alert
on ImagePullBackOff and on a pod that stays not ready, because neither case
raised any alert.
Refs: httphub.docker.io/v2/repositories/minio/minio/tags -> {"message":"object not found"}
This commit is contained in:
@@ -27,6 +27,24 @@ spec:
|
||||
annotations:
|
||||
summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
|
||||
|
||||
# ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
|
||||
- alert: PodImagePullFailure
|
||||
expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
|
||||
|
||||
# Pod not ready
|
||||
- alert: PodNotReady
|
||||
expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
|
||||
|
||||
# CrashLoopBackOff
|
||||
- alert: PodCrashLooping
|
||||
expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0
|
||||
|
||||
Reference in New Issue
Block a user