argocd-server ran --insecure (plain HTTP :8080) while its Helm-managed
Ingress set ssl-passthrough: true, which sends nginx's raw TLS handshake
straight to the pod - HTTP server can't complete a TLS handshake, nginx
logged 502 (peer closed connection in SSL handshake). Compounded by a
second, conflicting Ingress for the same host in
k8s/bootstrap/ingress/ingress.yaml - two Ingress objects on one host is
undefined nginx routing behavior. Disabled the Helm-managed Ingress
(enabled: false) so ingress.yaml's passthrough Ingress is the sole
source of truth, and set server.insecure: false so argocd-server
actually terminates TLS itself, matching passthrough's requirement.
repoServer CPU limit (500m) was too tight once the SOPS sidecar added real
decrypt work under the liveness probe's 1s timeout — repo-server kept
getting killed mid-sync. Raised to 1000m (node has 23+ idle cores, no
scarcity). Separately, the generate script's doc-separator fix exposed
that several .enc.yaml files (cloudflared, temporal, authentik, loki) are
raw Helm-values snippets, not K8s manifests — ArgoCD hard-failed the whole
batch on the first one missing 'kind:'. Script now skips those, so
correctly-shaped Secrets (grafana-admin included) sync independently.
grafana-admin also needed an admin-user key alongside admin-password —
the chart looks up both from the same existingSecret.
Sidecar container was absent from live repo-server Deployment (never in
helm history), causing sops-secrets Application to fail with cmp-server
socket not found — cascaded CreateContainerConfigError across every app
depending on SOPS-decrypted secrets. Also fixes duplicate version field
in plugin ConfigMap that produced a mismatched socket filename, and adds
an initContainer to fetch the sops binary into a writable emptyDir since
the sidecar runs non-root. grafana-secrets.enc.yaml rewritten from a bare
values file (never valid as a K8s Secret) to a proper Secret manifest so
grafana-admin now actually gets created.
Story Crater Bot