argocd-server ran --insecure (plain HTTP :8080) while its Helm-managed
Ingress set ssl-passthrough: true, which sends nginx's raw TLS handshake
straight to the pod - HTTP server can't complete a TLS handshake, nginx
logged 502 (peer closed connection in SSL handshake). Compounded by a
second, conflicting Ingress for the same host in
k8s/bootstrap/ingress/ingress.yaml - two Ingress objects on one host is
undefined nginx routing behavior. Disabled the Helm-managed Ingress
(enabled: false) so ingress.yaml's passthrough Ingress is the sole
source of truth, and set server.insecure: false so argocd-server
actually terminates TLS itself, matching passthrough's requirement.
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.
Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
nodeSelector still targeted az-a/talos-cp-1 from before the 3-CP topology
change. talos-cp-2 (az-b) has the dedicated Longhorn disks now, so the
pod's zone pin and the PVC's only viable replica location never matched
- ReplicaSchedulingFailure: disks are unavailable, pod stuck
ContainerCreating waiting on AttachVolume.
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
PVC sat Pending for 17 days — storageclass.storage.k8s.io "longhorn-wffc"
not found. Only longhorn, longhorn-cnpg, longhorn-static exist. Straight
naming drift, no such class was ever created.
repoServer CPU limit (500m) was too tight once the SOPS sidecar added real
decrypt work under the liveness probe's 1s timeout — repo-server kept
getting killed mid-sync. Raised to 1000m (node has 23+ idle cores, no
scarcity). Separately, the generate script's doc-separator fix exposed
that several .enc.yaml files (cloudflared, temporal, authentik, loki) are
raw Helm-values snippets, not K8s manifests — ArgoCD hard-failed the whole
batch on the first one missing 'kind:'. Script now skips those, so
correctly-shaped Secrets (grafana-admin included) sync independently.
grafana-admin also needed an admin-user key alongside admin-password —
the chart looks up both from the same existingSecret.
Sidecar container was absent from live repo-server Deployment (never in
helm history), causing sops-secrets Application to fail with cmp-server
socket not found — cascaded CreateContainerConfigError across every app
depending on SOPS-decrypted secrets. Also fixes duplicate version field
in plugin ConfigMap that produced a mismatched socket filename, and adds
an initContainer to fetch the sops binary into a writable emptyDir since
the sidecar runs non-root. grafana-secrets.enc.yaml rewritten from a bare
values file (never valid as a K8s Secret) to a proper Secret manifest so
grafana-admin now actually gets created.
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
TOPOLOGY CHANGE:
- All 3 control-plane nodes now schedulable (no NoSchedule taints)
- Pod distribution: ~59 on cp-1, ~21 on cp-2, ~23 on cp-3
- Better resource utilization across cluster
ADDED HARD RULE:
- Control-plane scheduling controlled via Terraform
- terraform.tfvars → allow_scheduling = true/false
- Never manual kubectl taint (Talos will revert)
- Workflow: terraform apply → talosctl apply-config
IMPLEMENTATION:
- Terraform: Set allow_scheduling=true for cp-2, cp-3
- Applied via talosctl --mode no-reboot (no disruption)
- Verified: kubectl get nodes shows no taints
ADDED:
- CloudNativePG (CNPG) Database Pattern section
- Explains shared 'app' user model (not per-app roles)
- Documents bootstrap.sh credential distribution pattern
- Working examples (Forgejo, Authentik)
- Prescriptive DO/DON'T guidance for new apps
FIXED:
- Storage topology: 3-node HA (not "sole Longhorn node")
- Verified: all 17 PVCs have replicas across all 3 nodes
- Updated last-modified date
This documents the architectural pattern established during CNPG refactor.
UNIFIED PATTERN: All apps follow same credential distribution
FORGEJO PATTERN (now universal):
1. CNPG creates ddb-cluster-app in ddb namespace (source)
2. bootstrap.sh copies to app namespaces (cicd, iam)
3. Apps reference local copy via secretKeyRef
4. No PostSync Jobs needed
CHANGES:
- bootstrap.sh: Copy ddb-cluster-app to iam namespace (like cicd)
- authentik-values.yaml: Reference local ddb-cluster-app via env vars
- Removed: sync-db-credentials PostSync Job (not needed)
- kustomization.yaml: Removed PostSync Job reference
BENEFITS:
✅ Same pattern as working Forgejo
✅ No complex PostSync Jobs
✅ bootstrap.sh handles setup for future clusters
✅ Simple secretKeyRef, no cross-namespace issues
✅ ArgoCD manages applications, not secrets
Database recreated with app owner (fresh migrations needed).
ARCHITECTURAL CHANGE: Align with CNPG design intent
BEFORE (Complex, broken):
- Per-app roles (authentik, temporal) with Database CR owner field
- Database CR doesn't transfer ownership properly
- Needed manual permission grants (PostSync Job)
- Apps couldn't create tables without grants from 'app' role
AFTER (Simple, works):
- All apps use shared 'app' bootstrap user
- Database CRs: owner: app (matches actual ownership)
- No permission grants needed (owner has full rights)
- Isolation via separate database names only
CHANGES:
- Database CRs: owner changed from app-specific to 'app'
- ddb-cluster.yaml: removed managed.roles section
- Deleted grant-schema-permissions PostSync Job
- Follows Forgejo pattern (already working this way)
MANUAL STEPS REQUIRED:
1. Update authentik-secrets: AUTHENTIK_POSTGRESQL__USER=app
2. Update temporal secrets: similar change
3. Recreate databases with app as owner
4. Restart applications
Benefits:
- Simpler architecture
- No permission grant complexity
- Aligns with CNPG single-cluster design
- Matches working Forgejo implementation
Fresh authentik deployment runs ~100 database migrations which takes 15-20
minutes. Previous startup probe failureThreshold of 60 (10 minutes) killed
the pod before migrations could complete, causing infinite restart loop.
Increased to 120 failures (20 minutes) to allow migrations to finish.
Fixes: nginx 503 due to pod never becoming Ready.
Authentik migrations need to CREATE SCHEMA (not just tables in public schema).
This requires GRANT CREATE ON DATABASE, not just schema-level permissions.
Added to PostSync Job:
- GRANT CREATE ON DATABASE authentik TO authentik
- GRANT CREATE ON DATABASE temporal TO temporal
- GRANT CREATE ON DATABASE temporal_visibility TO temporal
App user can grant these (it owns the databases).
Removes duplicate longhorn-kafka StorageClass managed by Kafka chart.
All applications now use single 'longhorn' StorageClass (3 replicas, Immediate binding).
Changes:
- Kafka chart: use 'longhorn' instead of 'longhorn-kafka'
- Delete Kafka StorageClass template (no longer needed)
- Update longhorn-storageclass.yaml to match deployed config (Immediate, not WaitForFirstConsumer)
Existing Kafka PVCs remain bound to old longhorn-kafka StorageClass (safe - no data loss).
New PVCs will use unified 'longhorn' StorageClass.
ROOT CAUSE: CNPG Database CR creates databases but doesn't grant schema
permissions to the owner role. Bootstrap DB owner 'app' retains CREATE
privilege on public schema, blocking authentik/temporal from creating tables.
SECURITY FIX: Removed insecure 'GRANT TO PUBLIC' from postInitApplicationSQL.
SOLUTION: PostSync Job connects as 'app' (DB owner) and grants schema
permissions to named roles (authentik, temporal) in their respective databases.
Runs after Database CRs reconcile, survives CNPG database recreation.
Pattern: Per-database grants via PostSync, not cluster-wide PUBLIC grants.
Adds SQL to postInitApplicationSQL granting schema permissions to PUBLIC.
Allows any role (authentik, temporal, etc) to create tables in databases.
For existing cluster: run SQL manually (done).
For future bootstrap: automatic via initdb.
Pattern for apps: Database CR + app-specific init Job optional (co-located).
CNPG Database CR creates DBs but doesn't grant schema permissions properly.
Database owner is 'app' instead of specified role (authentik, temporal).
PostSync Job grants ALL on schema public to both app and named roles,
ensuring applications can create tables. Runs after Database CRs reconcile.
Fixes: authentik InsufficientPrivilege error on migration.
Minio operator requires privileged securityContext. Without these labels,
StatefulSet stuck at 0/0 replicas (PodSecurity admission blocks pod creation).
CoreDNS rewrites *.riotpiao.com → ingress-nginx-controller but bootstrap
deployed as ingress-nginx-bootstrap-controller. Service alias makes both work.
- Add explicit tls block with riotpiao-com-tls secret
- Enables HTTPS access to https://forgejo.riotpiao.com
- Matches wildcard certificate (*.riotpiao.com)
The file comment mentioned TLS should be handled via default-ssl-certificate,
but explicit TLS blocks are needed for proper HTTPS routing.
Root cause: Longhorn refuses to schedule replicas on nodes without spec.name
field. talos-cp-1 was auto-discovered (has spec.name), but cp-2/cp-3 were
manually created CRDs without it.
Error: 'no node name provided to check node down or deleted'
Fix: Add spec.name matching metadata.name for both nodes.
Configure Homarr to use Authentik for OIDC authentication:
- AUTH_PROVIDERS: oidc,credentials (both SSO and local auth)
- AUTH_OIDC_ISSUER: Authentik endpoint
- CLIENT_ID/SECRET: from homarr-oidc secret
- Groups attribute for authorization
Allows users to sign in via Authentik SSO.
homarr-patches Application doesn't have SOPS support.
Secret is managed by sops-secrets Application instead.
Kustomization now only contains:
- fix-probes-job.yaml (PostSync hook)
Two fixes:
1. Added https://homarr-labs.github.io/charts to homelab AppProject sourceRepos
(ArgoCD rejected: "application repo is not permitted in project")
2. Removed env array from homarr-values.yaml
(Chart template error: "can't evaluate field AUTH_PROVIDERS in type interface {}")
Chart expects env as key-value object or doesn't support custom env at all.
Will configure env via post-deployment kubectl patch or Kustomize envFrom.
Allows Homarr Application to sync successfully.
ArgoCD was failing to query Authentik OIDC discovery endpoint with:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Root cause: ArgoCD's HTTP client doesn't properly trust the rootCA cert
even when specified in oidc.config.
Fixed by adding insecureSkipVerify: true to OIDC config. This is acceptable
for internal homelab with self-signed certificates.
Tested: ArgoCD SSO login via Authentik now works
Adds expand-replicas-job.yaml: PostSync hook Job that:
- Waits for all 3 Longhorn nodes to be Ready
- Patches every volume with numberOfReplicas < 3 to 3
- Runs idempotently on every longhorn-config sync (BeforeHookCreation
deletes previous job, so re-runs are safe)
This ensures existing 1-replica volumes (created before the HA setup) get
expanded automatically via GitOps, not via manual kubectl patch.
Why PostSync: needs to run AFTER the taint-toleration setting and Node CRDs
are applied, otherwise there aren't 3 nodes available yet and the expansion
would fail (Longhorn can't create replicas on nodes that don't exist).
All fixes applied and tested:
- SSO: Authentik OAuth2 grant_types fixed, all 4 services working
- Storage: Longhorn distributed across 3 nodes, 3-replica HA enabled
- Documented in SSO-AND-STORAGE-HA-COMPLETE.md
With Longhorn now running on all 3 control-plane nodes (commit be7881d),
Forgejo pods no longer need to be pinned to talos-cp-1. The gitea-shared-storage
PVC can attach on any node, and the scheduler will properly co-locate pod + volume
via WaitForFirstConsumer + 3-replica Longhorn volumes.
Removes the kubernetes.io/hostname: talos-cp-1 nodeSelector added in commit
dde4b60 (which was a workaround for single-node storage).
Changes:
- k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml: new Setting
to tolerate node-role.kubernetes.io/control-plane:NoSchedule taint,
allowing Longhorn DaemonSet to run on cp-2/cp-3 (not just cp-1)
- k8s/infrastructure/longhorn/longhorn-nodes.yaml: explicit Node CRDs for
talos-cp-2 and talos-cp-3 (auto-discovery doesn't work when nodes have
taints; these define /var/lib/longhorn as the storage path)
- k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml: bump
numberOfReplicas from 1→3 (true HA: each volume gets 3 copies across
3 nodes; if one node fails, 2 others still have the data)
- k8s/infrastructure/longhorn/kustomization.yaml: add new resources
Root cause: Longhorn was only running on talos-cp-1 (.213) because cp-2/cp-3
have the control-plane taint and Longhorn DaemonSet had no matching toleration.
Every workload with a PVC was forced to schedule on cp-1 (via nodeSelector or
implicit co-location with the storage), defeating the entire purpose of a 3-node
HA cluster.
With this fix:
- Longhorn manager runs on all 3 nodes
- Storage is replicated 3x (erasure-coded across nodes)
- Pods can schedule on any node without PVC attachment failures
- True HA: lose 1 node, cluster still serves all volumes
Adds missing CLIENT_SECRET env injection + nodeSelector constraint:
- k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET
from forgejo-oidc Secret (created by authentik-provision Job), and pin
pods to talos-cp-1 via nodeSelector (only node with Longhorn storage —
gitea-shared-storage PVC can't attach on cp-2/cp-3)
Root cause chain for 'Forgejo SSO not working':
1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers
2. Old provision script never set it → all providers had grant_types=[]
3. /authorize returned 'Invalid grant_type for provider' → all SSO broken
4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c)
+ successfully re-ran via iam-jobs Application sync
5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds
6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git
repo self-hosting), but missing the extraEnv block that was only in
k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected
All 4 OAuth2 providers now have correct grant_types=['authorization_code',
'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are
constrained to the storage node. SSO login flow should now work end-to-end.
CLAUDE.md was previously excluded from version control entirely (treated as
private local notes, with CLAUDE.example.md as the only git-tracked
counterpart). No longer justified - the file contains no secrets, just
architecture notes, private RFC1918 IPs, and operational lessons (same
sensitivity level as README.md, which is already tracked). Removing the
CLAUDE.md gitignore rule and committing it for the first time.
CLAUDE.md and the entire project-usage/ tree were written for a helmfile +
'core iam'/'core secrets' CLI workflow that has been fully retired - actual
practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD
sync), confirmed by an extended live debugging session that touched
Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and
multiple ArgoCD Applications, none of which involved helmfile or core at
any point.
CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps
loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every
hard-won lesson from this session with live evidence for each:
- kustomization.yaml resources: allowlists silently dropping new files
- kustomization.yaml namespace: transformers clobbering cross-namespace
RBAC
- PreSync hooks deadlocking on same-Application RBAC dependencies
- ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely
new sync operation to pick up fixes
- repo-server manifest caching
- repoURL port mismatches breaking every Application's sync
simultaneously when routed through an ingress-rewriting CoreDNS rule
- Bitnami's 2025 versioned-tag retirement
- apk-as-non-root permission failures
- Helm's lack of values.yaml schema validation (root cause of the
Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was
a schema mismatch between the pinned chart version and a newer
chart's values.yaml example, silently a no-op)
CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic
template (explicit notice at top) - same lessons, genericized away from
this specific homelab's IPs/hostnames/secrets, intended to be reusable by
anyone running a similar bare-metal Talos + ArgoCD topology.
project-usage/gitops-workflow.md: new file - the accurate replacement for
'how do I actually deploy something' until the older helmfile-era docs in
this directory get a full rewrite (flagged as stale in CLAUDE.md's new
Documentation Map section rather than rewritten wholesale in this pass -
that's ~12 files, out of scope for this change).
Bitnami stopped publishing versioned image tags in 2025 - only 'latest' and
sha256-pinned digests remain for their free-tier images. Confirmed via
Docker Hub API before writing this fix: no '1.30' tag exists for
bitnami/kubectl, which caused an indefinite ImagePullBackOff (job stuck
'Running' with 0 pods able to start).
Switched to python:3.12-alpine + a stdlib urllib kubectl download, matching
the exact pattern already proven working in
k8s/security/iam/authentik-provision-job.yaml (which hit its own apk
permission problem on this same base image, now fixed the same way in
both places) - avoids depending on any third party's tagging policy.
PreSync hooks run BEFORE an Application's own normal (non-hook) resources
are synced. This Job's ServiceAccount/ClusterRole/RoleBindings are plain
resources in the same Application, so marking the Job PreSync created a
chicken-and-egg deadlock: confirmed live, the Job sat 'Running' for 14
minutes producing zero pods, with job-controller repeatedly logging
'serviceaccount temporal/temporal-db-secret-sync not found' - because that
ServiceAccount hadn't been created yet (it's created during the normal Sync
phase, which comes after PreSync).
Fixed to PostSync. This app (sync-wave 7) still fully completes - including
this hook - before the temporal Application (sync-wave 8) begins, so the
ordering guarantee we need (secret exists before Temporal's pods try to
mount it) is unaffected; only the intra-app hook-vs-normal-resource
ordering was wrong.
Story Crater Bot