Our Ingress objects intentionally omit per-rule tls: blocks (single wildcard
cert served via --default-ssl-certificate). nginx-ingress's normal ssl-redirect
setting only forces HTTPS 301 for hosts with an explicit TLS block on their own
Ingress resource - since none of ours do, plain http://*.riotpiao.com requests
were served directly instead of redirected, exposing every client-facing
console (forgejo, authentik, argocd, grafana, vault, etc.) over plaintext HTTP.
force-ssl-redirect forces the redirect globally regardless of per-ingress TLS
block presence. Verified fix works (tested via manual patch then reverted -
confirmed 308 redirects to https:// on forgejo/authentik/argocd) before
committing via GitOps.
Root cause of widespread 'Unknown' sync status / Skipping auto-sync across
almost every Application: CoreDNS rewrites forgejo.riotpiao.com to the nginx
ingress controller service (rewrite name forgejo.riotpiao.com -> ingress-nginx-
controller...), which only listens on 80/443, not 3000. Every git fetch from
argocd-repo-server to the :3000 repoURL was timing out (context deadline
exceeded), so ArgoCD couldn't compare desired vs live state for any app.
Fix: use https://forgejo.riotpiao.com (no port, TLS via nginx + wildcard cert)
consistent with the 'all external endpoints HTTPS' requirement. Verified git
smart-http response 200 on the new URL before committing.
- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator)
- minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled)
- minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking
plain-HTTP clients like Vault's S3 backend - this was the real cause of the
Vault S3 hang)
- argocd ingress: moved from namespace cicd -> argocd (service lives in argocd
namespace; ingress in wrong namespace can never route, was returning 503)
- removed duplicate kmsvc ingress (sqs namespace already has management-service
ingress with proper TLS block for same host/backend)
Audit method: cross-checked every ingress backend.service.{name,port} against
actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
Wildcard cert was left on letsencrypt-staging; staging root is not
browser-trusted so HTTPS to *.riotpiao.com fails cert validation.
Switch issuerRef to letsencrypt-prod to issue a trusted wildcard.
Adds optional cloudflare_talos_sans (machine.certSANs, talos API :50000) and
cloudflare_apiserver_sans (cluster.apiServer.certSANs, kube-apiserver :6443) per
control-plane node. cp-1 gets cp1.homelab + cp1-talos.homelab; cp-2/cp-3 get
their cpN-talos.homelab. Values set in gitignored tfvars.
authentik/temporal DB users+databases were never provisioned (old helmfile hook
gone; db-init-job only made schemas in shared app DB). Adds managed.roles
(authentik/temporal login roles, passwords from basic-auth secrets) + Database CRs
(dedicated DBs owned by each role). Role secrets applied out-of-band (SOPS), not in
kustomize resources so data-schemas app doesn't choke on ciphertext.
Story Crater Bot