fix(minio): pin frozen community image and alert on image pull failure

MinIO deleted the community edition images upstream. An anonymous pull of
docker.io/minio/minio and quay.io/minio/mc returns HTTP 401, and the Docker Hub
API returns "object not found". talos-cp-1 dropped the cached image during the
DiskPressure event on 2026-09-22. The tenant pod was recreated on 2026-09-23 and
has been in ImagePullBackOff since. MinIO served no traffic for 13 days, which
put vault-0 and loki-0 into CrashLoopBackOff and left the paperless backups
stuck.

Pin the tenant to the last community release by digest, and repoint the two
minio/mc references at the archived Bitnami client image, also by digest. Alert
on ImagePullBackOff and on a pod that stays not ready, because neither case
raised any alert.

Refs: httphub.docker.io/v2/repositories/minio/minio/tags -> {"message":"object not found"}
This commit is contained in:
2026-10-06 11:42:36 +08:00
parent b6e317b34c
commit 62ab27930c
4 changed files with 25 additions and 4 deletions
@@ -27,6 +27,24 @@ spec:
annotations:
summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
# ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
- alert: PodImagePullFailure
expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
# Pod not ready
- alert: PodNotReady
expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
# CrashLoopBackOff
- alert: PodCrashLooping
expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0