fix(minio): replace the dead minio/mc image in the provision hook and backup job #56

2026-10-06T02:11:06Z

rock commented

2026-10-06 02:11:06 +00:00

Why the MinIO fix (#53) never took effect

minio-tenant is stuck in a Running sync operation:

phase=Running since 2026-10-06T01:45:53Z
message: waiting for completion of hook batch/Job/minio-provision-paperless

That PostSync hook Job pins quay.io/minio/mc:latest, which is no longer pullable (MinIO's
registry now demands auth; minio/mc on Docker Hub is 404). The hook pod sits in
ImagePullBackOff forever, ArgoCD keeps waiting for it, the sync never completes, and so the
tenant's new image from #53 was never applied — MinIO stays on the removed
minio/minio:RELEASE.2025-07-23T15-54-02Z, and Vault (S3 storage on MinIO) stays in
CrashLoopBackOff.

So the dead client image is now the only thing between us and a working MinIO + Vault.

Change

Both places that use it switch to a live image, verified by pulling and running it on
talos-cp-1: bitnamilegacy/minio-client:2025.7.21-debian-12-r3 —

mc version DEVELOPMENT.2025-07-21T05-28-08Z     (at /opt/bitnami/minio-client/bin/mc, on PATH)

k8s/infra/minio/minio-provision-paperless-job.yaml (the blocking PostSync hook)
k8s/apps/paperless/backup-cronjob.yaml

The hook's other image, bitnami/kubectl:latest, still pulls fine (verified: Client
Version v1.37.1), so it is left alone. The mc invocations (alias set, mb,
admin user add, admin policy create/attach) need no change — only the image.

Gate: structural-check.sh -> 0 error. YAML parses; the kubectl apply --dry-run=server
"field is immutable" on the Job is expected — it already exists and ArgoCD recreates hook
Jobs (hook-delete-policy: BeforeHookCreation).

After merging

The stuck operation needs a nudge before ArgoCD will re-run the hook: Sync the
minio-tenant app in the ArgoCD UI, or delete the stuck hook Job
(kubectl -n storage delete job minio-provision-paperless) so the automated policy starts a
fresh sync. Then the chain should complete on its own: hook runs -> skip minio-provision-paperless image applies -> MinIO Ready -> minio-cluster-hl gets endpoints -> Vault storage init succeeds -> vault-0 Running.

Unrelated, still open

paperless-backup has ~22 pods pending for 22 days for a different reason: PVC
paperless-media does not exist in the namespace where those CronJob pods were created.
Worth a separate fix.

## Why the MinIO fix (#53) never took effect `minio-tenant` is stuck in a **Running** sync operation: ``` phase=Running since 2026-10-06T01:45:53Z message: waiting for completion of hook batch/Job/minio-provision-paperless ``` That PostSync hook Job pins `quay.io/minio/mc:latest`, which is no longer pullable (MinIO's registry now demands auth; `minio/mc` on Docker Hub is 404). The hook pod sits in ImagePullBackOff forever, ArgoCD keeps waiting for it, the sync never completes, and so the tenant's new image from #53 was never applied — MinIO stays on the removed `minio/minio:RELEASE.2025-07-23T15-54-02Z`, and Vault (S3 storage on MinIO) stays in CrashLoopBackOff. So the dead client image is now the *only* thing between us and a working MinIO + Vault. ## Change Both places that use it switch to a live image, verified by pulling and running it on talos-cp-1: `bitnamilegacy/minio-client:2025.7.21-debian-12-r3` — ``` mc version DEVELOPMENT.2025-07-21T05-28-08Z (at /opt/bitnami/minio-client/bin/mc, on PATH) ``` - `k8s/infra/minio/minio-provision-paperless-job.yaml` (the blocking PostSync hook) - `k8s/apps/paperless/backup-cronjob.yaml` The hook's other image, `bitnami/kubectl:latest`, still pulls fine (verified: Client Version v1.37.1), so it is left alone. The `mc` invocations (`alias set`, `mb`, `admin user add`, `admin policy create/attach`) need no change — only the image. Gate: `structural-check.sh` -> 0 error. YAML parses; the `kubectl apply --dry-run=server` "field is immutable" on the Job is expected — it already exists and ArgoCD recreates hook Jobs (`hook-delete-policy: BeforeHookCreation`). ## After merging The stuck operation needs a nudge before ArgoCD will re-run the hook: **Sync** the `minio-tenant` app in the ArgoCD UI, or delete the stuck hook Job (`kubectl -n storage delete job minio-provision-paperless`) so the automated policy starts a fresh sync. Then the chain should complete on its own: hook runs -> skip minio-provision-paperless image applies -> MinIO Ready -> `minio-cluster-hl` gets endpoints -> Vault storage init succeeds -> vault-0 Running. ## Unrelated, still open `paperless-backup` has ~22 pods pending for 22 days for a different reason: PVC `paperless-media` does not exist in the namespace where those CronJob pods were created. Worth a separate fix.

rock added 1 commit 2026-10-06 02:11:07 +00:00

fix(minio): replace the dead minio/mc image in the provision hook and backup job 71b3cef2eb

This pull request has changes conflicting with the target branch.

k8s/apps/paperless/backup-cronjob.yaml
k8s/infra/minio/minio-provision-paperless-job.yaml

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/minio-mc-image:fix/minio-mc-image

git checkout fix/minio-mc-image

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#56