1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
Our Ingress objects intentionally omit per-rule tls: blocks (single wildcard
cert served via --default-ssl-certificate). nginx-ingress's normal ssl-redirect
setting only forces HTTPS 301 for hosts with an explicit TLS block on their own
Ingress resource - since none of ours do, plain http://*.riotpiao.com requests
were served directly instead of redirected, exposing every client-facing
console (forgejo, authentik, argocd, grafana, vault, etc.) over plaintext HTTP.
force-ssl-redirect forces the redirect globally regardless of per-ingress TLS
block presence. Verified fix works (tested via manual patch then reverted -
confirmed 308 redirects to https:// on forgejo/authentik/argocd) before
committing via GitOps.
Root cause of widespread 'Unknown' sync status / Skipping auto-sync across
almost every Application: CoreDNS rewrites forgejo.riotpiao.com to the nginx
ingress controller service (rewrite name forgejo.riotpiao.com -> ingress-nginx-
controller...), which only listens on 80/443, not 3000. Every git fetch from
argocd-repo-server to the :3000 repoURL was timing out (context deadline
exceeded), so ArgoCD couldn't compare desired vs live state for any app.
Fix: use https://forgejo.riotpiao.com (no port, TLS via nginx + wildcard cert)
consistent with the 'all external endpoints HTTPS' requirement. Verified git
smart-http response 200 on the new URL before committing.
- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator)
- minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled)
- minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking
plain-HTTP clients like Vault's S3 backend - this was the real cause of the
Vault S3 hang)
- argocd ingress: moved from namespace cicd -> argocd (service lives in argocd
namespace; ingress in wrong namespace can never route, was returning 503)
- removed duplicate kmsvc ingress (sqs namespace already has management-service
ingress with proper TLS block for same host/backend)
Audit method: cross-checked every ingress backend.service.{name,port} against
actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
Wildcard cert was left on letsencrypt-staging; staging root is not
browser-trusted so HTTPS to *.riotpiao.com fails cert validation.
Switch issuerRef to letsencrypt-prod to issue a trusted wildcard.
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
Loki keys are under .loki.storage.s3 not .loki.s3 (returned null). Emit a separate
authentik-secrets Secret (not 'authentik', which the Helm chart owns) and merge it
via envFrom on server/worker/migrate. Remove discover fileName (caused MatchRepository
timeouts; app names the plugin explicitly).
Correct CMP setup (prior attempt used unsupported config): repoServer.initContainers
fetches sops v3.9.0 + yq v4.44.3 into a shared volume; repoServer.extraContainers
runs argocd-cmp-server with plugin.yaml from the sops-cmp-plugin ConfigMap, age key
from sops-age Secret. Plugin emits authentik/loki-s3-creds/grafana-admin/grafana-oidc
Secrets from decrypted enc files. sops-secrets Application (wave 0) uses the plugin at
repo root. Unblocks authentik/loki/grafana which were Degraded on missing secrets.
Substrate had no owner since Terraform was deleted (Pure GitOps). Adds 5 wave-0/1
Applications: cert-manager v1.21.0 (installCRDs, CP tolerations), ingress-nginx
4.15.1 (LB 192.168.1.160), reloader 2.2.14 at wave 0; LE ClusterIssuers +
*.riotpiao.com wildcard cert at wave 1 (DNS-01 via Cloudflare). Adds 3 chart
repos to AppProject sourceRepos and SOPS-encrypted cloudflare-api-token secret.
Cert starts on letsencrypt-staging; flip to prod after clean issue.
Switch Tenant from credsSecret to configuration field (v5 pods read config.env
shell exports); pin image to RELEASE.2025-07-23 (the old 2024-06 tag was pulled
from Docker Hub, ErrImagePull); drop prometheusOperator:true (made operator fail
reconcile hunting Prometheus in ns default). MinIO now serves S3, 4/4 drives OK,
root auth works. Operator's cosmetic 'empty tenant credentials' health-log is
harmless (documented inline).
Old Tenant used unknown v2 fields (pools[].size/storageClass, spec.console/metrics/ingress)
and referenced nonexistent minio.min.io/v1alpha1 Bucket/Policy/User kinds, so the app
never synced. Rewrites to valid v2: single erasure-coded pool (4 vols) pinned to
talos-cp-1/az-a (only schedulable+Longhorn node per 3-CP topology), spec.buckets +
spec.users declarative provisioning, prometheusOperator ServiceMonitor, features.domains.
Drops hand-rolled minio-service (operator owns it), dead multi-site replication job,
and legacy alias. Adds mc-based PostSync job for the ollama scoped policy, and
SOPS-encrypted minio-creds/oidc/user secrets for IaC record.
The operator chart moved to https://operator.min.io/; https://charts.min.io/ only
ships the standalone minio chart, causing 'chart operator not found in index'.
Pin to 5.0.18 (v5.x schema matches minio-operator-values.yaml operator.image.tag v5.0.0);
targetRevision '*' was fragile. Unblocks minio-tenant (needs operator CRDs).
Reverts complex CMP plugin setup (helm chart doesn't support repoServer.extraContainers).
Instead: sops-secrets Application uses directory source (no plugin), emits placeholder
README. Manually-created Secrets (grafana-admin) live in target namespaces.
Full CMP plugin work deferred to future stage. Grafana values still wired to
admin.existingSecret (no-op until Secret exists, which it now does).
This unblocks cluster deployment without waiting for ArgoCD CMP plumbing.
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard
Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
Consolidate three separate Forgejo Actions (argocd-sync, security-scan, validate-k8s) into single cluster-ci workflow for cleaner CI/CD pipeline with proper job sequencing and reduced auth overhead.
Add Terraform configuration for Talos cluster machine configs:
- Provider setup for Talos
- Centralized variables for CP and worker configs
- Template-based config generation for controlplane.yaml and worker-*.yaml
- Sensitive data separated in terraform.tfvars (gitignored)
- Local state tracking for infrastructure
Use CI_RUNNER and CI_RUNNER_SECRET for repo clone authentication.
Embed credentials in git clone URL: https://user:token@host/repo.git
Removes dependency on GITHUB_TOKEN (GitHub-specific) and improves Forgejo compatibility.
Recreate ingress-nginx-controller LoadBalancer service that was deleted.
Add to k8s/bootstrap/ingress/ kustomization for ArgoCD management.
LoadBalancer assigned IP: 192.168.1.160 (via MetalLB)
ArgoCD now accessible via: https://192.168.1.160/ (or update DNS)
Add MinIO Policies and Users via CRD alongside Buckets.
Resources now declarative:
- Bucket: riotpiao-models (versioning enabled)
- Policy: policy-ollama (scoped bucket access)
- User: user-ollama (service account for Ollama/LLM)
Access keys can be overridden via SOPS or kustomize overlays.
All MinIO resource creation now git-tracked and version controlled.
- main.tf: remove kubeconfig_path local (no longer used with direct auth)
- providers.tf: wrap file() with try() to handle plan-time on non-pod systems
try() allows terraform plan to work locally; at runtime in pod, files exist and are used.
- providers.tf: use host + token + ca_crt from mounted service account secrets
- workflow: remove kubeconfig generation step (no longer needed)
- variables.tf: remove unused kubeconfig_path variable
This is the standard pattern for running terraform inside k8s pods.
- Link core-cli app to OAuth2 provider (was hardcoded to 0)
- Add core-cli user to authentik_admins for CI access
- Fix terraform init: use 'endpoint' not 'endpoints.s3' for S3 backend
(Terraform 1.9.4 compatibility, matches state.tf config)
Reverting Phase 4 Pod Job implementation in favor of CI runner (Forgejo Actions).
Deleted:
- k8s/argocd/apps/phase4-terraform-0.yaml
- k8s/hooks/phase4/ (terraform-apply-hook.yaml, terraform-rbac.yaml, terraform-s3-secrets.enc.yaml)
Reason: Pod Job approach had limitations (eviction, timeouts, pod security policies).
Next: Implement Forgejo Actions CI workflow for terraform apply.
Story Crater Bot