riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 2 Actions Packages Projects 1 Releases Wiki Activity

fix(storage): pin minio tenant to a published image release (#53)

Browse Source 
## Why

`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).

The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:

```
minio tenant pod not ready (1/2)
  -> headless Service minio-cluster-hl has no endpoints
  -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
  -> Vault S3 storage init fails at startup
  -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```

## Control checks (so this is not a network story)

| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |

## Change

Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.

## Caveat

This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.

## After merge (ArgoCD, not kubectl)

The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:

```
kubectl get pods -n storage          # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault  # vault-0 -> Running (storage init succeeds)
```

Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53

Co-authored-by: rock <[email protected]>
This commit was merged in pull request #53.
This commit is contained in:
rock
2026-10-06 01:48:57 +00:00
committed by rock
parent d60b520968
commit f3015522e3
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/infra/minio/minio-tenant.yaml
+6 -1
View File
@@ -11,7 +11,12 @@ metadata:
prometheus.io/port: "9000" prometheus.io/port: "9000"
prometheus.io/scrape: "true" prometheus.io/scrape: "true"
spec: spec:
image: minio/minio:RELEASE.2025-07-23T15-54-02Z # Was minio/minio:RELEASE.2025-07-23T15-54-02Z — that Docker Hub repository no longer
# exists (404 for every tag), so the tenant sat in ImagePullBackOff and Vault's S3
# backend could not resolve minio-cluster-hl. Official images now need registry auth
# (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream
# release; swap it for the official image once MinIO registry credentials exist.
image: coollabsio/minio:RELEASE.2025-10-15T17-29-55Z
# Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000. # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000.
# External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert). # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert).