1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator)
- minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled)
- minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking
plain-HTTP clients like Vault's S3 backend - this was the real cause of the
Vault S3 hang)
- argocd ingress: moved from namespace cicd -> argocd (service lives in argocd
namespace; ingress in wrong namespace can never route, was returning 503)
- removed duplicate kmsvc ingress (sqs namespace already has management-service
ingress with proper TLS block for same host/backend)
Audit method: cross-checked every ingress backend.service.{name,port} against
actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
Switch Tenant from credsSecret to configuration field (v5 pods read config.env
shell exports); pin image to RELEASE.2025-07-23 (the old 2024-06 tag was pulled
from Docker Hub, ErrImagePull); drop prometheusOperator:true (made operator fail
reconcile hunting Prometheus in ns default). MinIO now serves S3, 4/4 drives OK,
root auth works. Operator's cosmetic 'empty tenant credentials' health-log is
harmless (documented inline).
Old Tenant used unknown v2 fields (pools[].size/storageClass, spec.console/metrics/ingress)
and referenced nonexistent minio.min.io/v1alpha1 Bucket/Policy/User kinds, so the app
never synced. Rewrites to valid v2: single erasure-coded pool (4 vols) pinned to
talos-cp-1/az-a (only schedulable+Longhorn node per 3-CP topology), spec.buckets +
spec.users declarative provisioning, prometheusOperator ServiceMonitor, features.domains.
Drops hand-rolled minio-service (operator owns it), dead multi-site replication job,
and legacy alias. Adds mc-based PostSync job for the ollama scoped policy, and
SOPS-encrypted minio-creds/oidc/user secrets for IaC record.
Add MinIO Policies and Users via CRD alongside Buckets.
Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)
Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
Story Crater Bot