fix(monitoring,minio): prometheus CRD sync loop + stuck minio-policy-setup hook

1. prometheus CRD sync failure (OutOfSync, permanently failing):
   - helm.skipCrds: true on the prometheus Application - stop ArgoCD from
     managing these CRDs through client-side apply (kube-prometheus-stack's
     CRDs are large enough that the kubectl.kubernetes.io/last-applied-
     configuration annotation exceeds etcd's 262144-byte limit on every sync).
   - New prometheus-crds Application: plain git-sourced YAML (extracted via
     helm show crds, committed under k8s/platform/monitoring/crds/), synced
     with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
     because there's no clean way to ask ArgoCD's Helm source for 'render only
     the crds/ directory' - a committed plain-YAML source is unambiguous.
   - ServerSideApply=true can't go on the main prometheus Application: it
     conflicts with managedNamespaceMetadata's forced namespace apply
     ('--force cannot be used with --server-side'), hence the split.

2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
   - minio-policy-setup PostSync hook Job was NAME:
  mc alias set - set a new alias to configuration file

USAGE:
  mc alias set ALIAS URL ACCESSKEY SECRETKEY

FLAGS:
  --path value                     bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
  --api value                      API signature. Valid options are '[S3v4, S3v2]'
  --config-dir value, -C value     path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
  --quiet, -q                      disable progress bar display [$MC_QUIET]
  --disable-pager, --dp            disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
  --no-color                       disable color theme [$MC_NO_COLOR]
  --json                           enable JSON lines formatted output [$MC_JSON]
  --debug                          enable debug output [$MC_DEBUG]
  --resolve value                  resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
  --insecure                       disable SSL certificate verification [$MC_INSECURE]
  --limit-upload value             limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
  --limit-download value           limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
  --custom-header value, -H value  add custom HTTP header to the request. 'key:value' format.
  --help, -h                       show help

EXAMPLES:
  1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
     $ set +o history
     $ mc alias set myminio http://localhost:9000 minio minio123
     $ set -o history
  2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
     turn off bash history momentarily.
     $ set +o history
     $ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
     $ set -o history
  3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
     $ set +o history
     $ mc alias set mys3 https://s3.amazonaws.com \
                 BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
     $ set -o history
  4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
     $ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
     Enter Access Key: BKIKJAA5BMMU2RHO6IBB
     Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
  5. Add Amazon S3 storage service under "mys3" alias using piped keys.
     $ set +o history
     $ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
                 mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
     $ set -o history against
     http://minio.storage.svc.cluster.local:9000 - stale port. The minio
     Service's port now tracks requestAutoCert on the Tenant (443 when
     auto-TLS is on, 80 when off - we set it to false earlier), so 9000
     doesn't exist on that Service anymore and the job hung in its 'waiting
     for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
     (PostSync hooks block the sync from completing until they succeed).
   - Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
     headless per-pod Service, which always listens on 9000 regardless of
     the Tenant's TLS mode, so this can't silently break again the same way.

k8s/argocd/apps/10-storage-observability.yaml

@@ -91,6 +91,7 @@ spec:
       chart: kube-prometheus-stack
       targetRevision: "*"
       helm:
+        skipCrds: true
         valueFiles:
           - $values/k8s/platform/monitoring/prometheus-values.yaml
     - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
@@ -112,7 +113,50 @@ spec:
       - CreateNamespace=true
       # ServerSideApply removed — it conflicts with managedNamespaceMetadata's
       # forced namespace apply ("--force cannot be used with --server-side").
-      # The large monitoring CRDs are installed out-of-band instead.
+      # helm.skipCrds: true above stops ArgoCD from ever managing the CRDs
+      # through this Application (previously it kept re-patching them via
+      # client-side apply and hitting etcd's 262144-byte annotation limit on
+      # kubectl.kubernetes.io/last-applied-configuration, permanently failing
+      # sync). CRDs are applied once via the separate prometheus-crds
+      # Application below, which uses ServerSideApply=true (no namespace-
+      # metadata conflict since CRDs are cluster-scoped).
+---
+# CRDs only, extracted to plain YAML (`helm show crds kube-prometheus-stack`)
+# and committed to git under k8s/platform/monitoring/crds/, applied via Server-
+# Side Apply to avoid the etcd 262144-byte last-applied-configuration
+# annotation limit that client-side apply hits on these very large CRDs
+# (prometheuses, alertmanagers, scrapeconfigs, etc). A plain git path source
+# (not a remote Helm source) is used deliberately so ArgoCD applies exactly
+# these 8 CRD manifests and nothing else — no ambiguity about what "CRDs only"
+# means from a Helm chart. Split out from the main `prometheus` Application
+# (helm.skipCrds: true there) because ServerSideApply conflicts with that
+# app's managedNamespaceMetadata.
+# NOTE: bump k8s/platform/monitoring/crds/kube-prometheus-stack-crds.yaml
+# whenever the kube-prometheus-stack chart version changes materially
+# (`helm show crds prometheus-community/kube-prometheus-stack > ...`).
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: prometheus-crds
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: homelab
+  source:
+    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+    targetRevision: main
+    path: k8s/platform/monitoring/crds
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: monitoring
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ServerSideApply=true
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application

k8s/infrastructure/minio/minio-policy-job.yaml

@@ -37,7 +37,13 @@ spec:
               set -e
 
               # Wait for the tenant S3 endpoint to answer.
-              until mc alias set local "http://minio.storage.svc.cluster.local:9000" \
+              # NOTE: minio's ClusterIP Service port tracks requestAutoCert on
+              # the Tenant - port 443 when auto-TLS is on, port 80 when it's
+              # off (requestAutoCert: false, our current setting - see
+              # minio-tenant.yaml). Use the headless per-pod Service instead,
+              # which always listens on 9000 regardless of TLS mode, so this
+              # job doesn't silently hang again if that setting ever flips.
+              until mc alias set local "http://minio-cluster-hl.storage.svc.cluster.local:9000" \
                 "$ROOT_USER" "$ROOT_PASSWORD" 2>/dev/null; do
                 echo "waiting for minio..."; sleep 5
               done