fix(storage): pin minio tenant to a published image release #53

2026-10-06T01:46:59Z

rock commented

2026-10-06 01:46:59 +00:00

Why

minio/minio:RELEASE.2025-07-23T15-54-02Z no longer exists. The Docker Hub repository
minio/minio returns 404 for every tag (checked: the pinned tag and latest), and
MinIO has moved its images behind registry authentication (quay.io/minio/minio → 401,
registry.min.io/minio/minio → auth required, dl.min.io binaries → 410 Gone).

The tenant therefore sat in ImagePullBackOff — 44,229 attempts over 6d23h — and the chain
from there is:

minio tenant pod not ready (1/2)
  -> headless Service minio-cluster-hl has no endpoints
  -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
  -> Vault S3 storage init fails at startup
  -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days

Control checks (so this is not a network story)

check	result
`alpine:3.20` pulled on talos-cp-1	Succeeded — docker.io works
`quay.io/prometheus/node-exporter:v1.8.2` on the same node	Succeeded — public quay works
`minio/minio` (docker.io)	HTTP 404, repo removed
old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent`	not cached on the node

Change

Pin spec.image to coollabsio/minio:RELEASE.2025-10-15T17-29-55Z — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries /usr/bin/docker-entrypoint.sh (the layout the operator
expects, which runs minio server … --console-address :9090) and reports
minio version RELEASE.2025-10-15T17-29-55Z.

Caveat

This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to quay.io/minio/minio. Supply-chain decision, flagged deliberately.

After merge (ArgoCD, not kubectl)

The storage app tracks k8s/infra/minio, so the sync rolls the tenant; then:

kubectl get pods -n storage          # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault  # vault-0 -> Running (storage init succeeds)

Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.

## Why `minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository `minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401, `registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone). The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain from there is: ``` minio tenant pod not ready (1/2) -> headless Service minio-cluster-hl has no endpoints -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host -> Vault S3 storage init fails at startup -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days ``` ## Control checks (so this is not a network story) | check | result | | --- | --- | | `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works | | `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works | | `minio/minio` (docker.io) | HTTP 404, repo removed | | old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node | ## Change Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator expects, which runs `minio server … --console-address :9090`) and reports `minio version RELEASE.2025-10-15T17-29-55Z`. ## Caveat This is a community mirror, not MinIO's official image, because the official ones now require credentials. If MinIO registry credentials are available, add an imagePullSecret and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately. ## After merge (ArgoCD, not kubectl) The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then: ``` kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2 kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds) ``` Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and Longhorn currently fails to create engines for new volume attaches cluster-wide. If the restarted pod cannot attach, that is a second, separate fault.

rock added 1 commit 2026-10-06 01:47:02 +00:00

fix(storage): pin minio tenant to a published image release 8bb8851a1f

rock merged commit f3015522e3 into main

2026-10-06 01:48:58 +00:00

rock deleted branch fix/minio-image

2026-10-06 01:48:59 +00:00

rock referenced this pull request

2026-10-06 02:11:06 +00:00

fix(minio): replace the dead minio/mc image in the provision hook and backup job #56

rock referenced this pull request

2026-10-06 03:43:02 +00:00

fix(minio): freeze the community image by digest and alert on pull failure #57