fix(minio): freeze the community image by digest and alert on pull failure #57

2026-10-06T03:43:02Z

rock commented

2026-10-06 03:43:02 +00:00

Follow-up to #53. That PR moved the tenant to coollabsio/minio:RELEASE.2025-10-15T17-29-55Z as a moving tag. This pins the same bytes by digest, so a retag or a deletion on the mirror cannot break the pull again.

Also fixes the two quay.io/minio/mc:latest references. That repository returns HTTP 401 on an anonymous pull, so the paperless backup CronJob and the provisioning Job both fail. They now use the archived Bitnami client image, pinned by digest.

Adds two alert rules: PodImagePullFailure and PodNotReady. The MinIO pod sat in ImagePullBackOff for 13 days and raised nothing.

Verification

kubectl kustomize k8s/infra/minio/ renders with the pinned digests.
kubectl kustomize k8s/apps/paperless/ renders.
Structural gate: 0 error, 1 warning. The warning is pre-existing (untagged postgres:16-alpine in backup-cronjob.yaml:26).
Digests resolved against registry-1.docker.io on 2026-10-06.

Not in this PR

Alertmanager still routes to receiver "null", so no alert leaves the cluster. That needs the Gotify token, which needs the SOPS age key.

Follow-up to #53. That PR moved the tenant to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` as a moving tag. This pins the same bytes by digest, so a retag or a deletion on the mirror cannot break the pull again. Also fixes the two `quay.io/minio/mc:latest` references. That repository returns HTTP 401 on an anonymous pull, so the paperless backup CronJob and the provisioning Job both fail. They now use the archived Bitnami client image, pinned by digest. Adds two alert rules: `PodImagePullFailure` and `PodNotReady`. The MinIO pod sat in ImagePullBackOff for 13 days and raised nothing. ## Verification - `kubectl kustomize k8s/infra/minio/` renders with the pinned digests. - `kubectl kustomize k8s/apps/paperless/` renders. - Structural gate: 0 error, 1 warning. The warning is pre-existing (untagged `postgres:16-alpine` in backup-cronjob.yaml:26). - Digests resolved against registry-1.docker.io on 2026-10-06. ## Not in this PR Alertmanager still routes to receiver `"null"`, so no alert leaves the cluster. That needs the Gotify token, which needs the SOPS age key.

rock added 1 commit 2026-10-06 03:43:03 +00:00

fix(minio): freeze the community image by digest and alert on pull failure d7b96fb29b

MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.

Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.

Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}

rock merged commit 64cd1f2c4e into main

2026-10-06 03:43:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#57