From f3015522e3b1ed669a4c5349fdef3a6471d062d8 Mon Sep 17 00:00:00 2001 From: rock Date: Tue, 6 Oct 2026 01:48:57 +0000 Subject: [PATCH] fix(storage): pin minio tenant to a published image release (#53) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why `minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository `minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401, `registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone). The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain from there is: ``` minio tenant pod not ready (1/2) -> headless Service minio-cluster-hl has no endpoints -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host -> Vault S3 storage init fails at startup -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days ``` ## Control checks (so this is not a network story) | check | result | | --- | --- | | `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works | | `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works | | `minio/minio` (docker.io) | HTTP 404, repo removed | | old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node | ## Change Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator expects, which runs `minio server … --console-address :9090`) and reports `minio version RELEASE.2025-10-15T17-29-55Z`. ## Caveat This is a community mirror, not MinIO's official image, because the official ones now require credentials. If MinIO registry credentials are available, add an imagePullSecret and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately. ## After merge (ArgoCD, not kubectl) The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then: ``` kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2 kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds) ``` Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and Longhorn currently fails to create engines for new volume attaches cluster-wide. If the restarted pod cannot attach, that is a second, separate fault.Reviewed-on: https://forgejo.riotpiao.com/riotpiao-poimen/homelab/pulls/53 Co-authored-by: rock --- k8s/infra/minio/minio-tenant.yaml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/k8s/infra/minio/minio-tenant.yaml b/k8s/infra/minio/minio-tenant.yaml index e756d30..810821e 100644 --- a/k8s/infra/minio/minio-tenant.yaml +++ b/k8s/infra/minio/minio-tenant.yaml @@ -11,7 +11,12 @@ metadata: prometheus.io/port: "9000" prometheus.io/scrape: "true" spec: - image: minio/minio:RELEASE.2025-07-23T15-54-02Z + # Was minio/minio:RELEASE.2025-07-23T15-54-02Z — that Docker Hub repository no longer + # exists (404 for every tag), so the tenant sat in ImagePullBackOff and Vault's S3 + # backend could not resolve minio-cluster-hl. Official images now need registry auth + # (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream + # release; swap it for the official image once MinIO registry credentials exist. + image: coollabsio/minio:RELEASE.2025-10-15T17-29-55Z # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000. # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert).