Story Crater Bot
d2d377a6a2
1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
187 lines
5.8 KiB
YAML
187 lines
5.8 KiB
YAML
# Wave 1 — MinIO (operator + tenant), Longhorn policy, Prometheus stack.
|
|
# Helm charts pull from public repos; values come from the git repo via a
|
|
# second "ref: values" source (ArgoCD multi-source pattern).
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: minio-operator
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
spec:
|
|
project: homelab
|
|
sources:
|
|
- repoURL: https://operator.min.io/
|
|
chart: operator
|
|
targetRevision: "5.0.18"
|
|
helm:
|
|
valueFiles:
|
|
- $values/k8s/infrastructure/minio/minio-operator-values.yaml
|
|
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
ref: values
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: storage
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
---
|
|
# Tenant + buckets + replication are raw CRs (MinIO Tenant CRD from operator).
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: minio-tenant
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
spec:
|
|
project: homelab
|
|
source:
|
|
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
path: k8s/infrastructure/minio
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: storage
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
---
|
|
# Longhorn itself is substrate (bootstrap-installed); this app manages only its
|
|
# ServiceMonitor / policy manifests.
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: longhorn-config
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
spec:
|
|
project: homelab
|
|
source:
|
|
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
path: k8s/infrastructure/longhorn
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: longhorn-system
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
---
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: prometheus
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
spec:
|
|
project: homelab
|
|
sources:
|
|
- repoURL: https://prometheus-community.github.io/helm-charts
|
|
chart: kube-prometheus-stack
|
|
targetRevision: "*"
|
|
helm:
|
|
skipCrds: true
|
|
valueFiles:
|
|
- $values/k8s/platform/monitoring/prometheus-values.yaml
|
|
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
ref: values
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: monitoring
|
|
syncPolicy:
|
|
managedNamespaceMetadata:
|
|
# node-exporter needs hostNetwork/hostPID/hostPath/hostPort; blocked by
|
|
# default baseline PSS (DaemonSet created 0 pods, Prometheus STS stuck).
|
|
labels:
|
|
pod-security.kubernetes.io/enforce: privileged
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
# ServerSideApply removed — it conflicts with managedNamespaceMetadata's
|
|
# forced namespace apply ("--force cannot be used with --server-side").
|
|
# helm.skipCrds: true above stops ArgoCD from ever managing the CRDs
|
|
# through this Application (previously it kept re-patching them via
|
|
# client-side apply and hitting etcd's 262144-byte annotation limit on
|
|
# kubectl.kubernetes.io/last-applied-configuration, permanently failing
|
|
# sync). CRDs are applied once via the separate prometheus-crds
|
|
# Application below, which uses ServerSideApply=true (no namespace-
|
|
# metadata conflict since CRDs are cluster-scoped).
|
|
---
|
|
# CRDs only, extracted to plain YAML (`helm show crds kube-prometheus-stack`)
|
|
# and committed to git under k8s/platform/monitoring/crds/, applied via Server-
|
|
# Side Apply to avoid the etcd 262144-byte last-applied-configuration
|
|
# annotation limit that client-side apply hits on these very large CRDs
|
|
# (prometheuses, alertmanagers, scrapeconfigs, etc). A plain git path source
|
|
# (not a remote Helm source) is used deliberately so ArgoCD applies exactly
|
|
# these 8 CRD manifests and nothing else — no ambiguity about what "CRDs only"
|
|
# means from a Helm chart. Split out from the main `prometheus` Application
|
|
# (helm.skipCrds: true there) because ServerSideApply conflicts with that
|
|
# app's managedNamespaceMetadata.
|
|
# NOTE: bump k8s/platform/monitoring/crds/kube-prometheus-stack-crds.yaml
|
|
# whenever the kube-prometheus-stack chart version changes materially
|
|
# (`helm show crds prometheus-community/kube-prometheus-stack > ...`).
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: prometheus-crds
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "0"
|
|
spec:
|
|
project: homelab
|
|
source:
|
|
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
path: k8s/platform/monitoring/crds
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: monitoring
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
- ServerSideApply=true
|
|
---
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: blackbox-exporter
|
|
namespace: argocd
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "1"
|
|
spec:
|
|
project: homelab
|
|
sources:
|
|
- repoURL: https://prometheus-community.github.io/helm-charts
|
|
chart: prometheus-blackbox-exporter
|
|
targetRevision: "~11"
|
|
helm:
|
|
valueFiles:
|
|
- $values/k8s/platform/monitoring/blackbox-exporter-values.yaml
|
|
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
ref: values
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: monitoring
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|