fix(minio): freeze the community image by digest and alert on pull failure

MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.

Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.

Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
This commit is contained in:
2026-10-06 11:42:51 +08:00
parent b6e317b34c
commit d7b96fb29b
4 changed files with 25 additions and 4 deletions
@@ -27,6 +27,24 @@ spec:
annotations:
summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
# ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
- alert: PodImagePullFailure
expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
# Pod not ready
- alert: PodNotReady
expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
# CrashLoopBackOff
- alert: PodCrashLooping
expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0