Root cause: Longhorn refuses to schedule replicas on nodes without spec.name
field. talos-cp-1 was auto-discovered (has spec.name), but cp-2/cp-3 were
manually created CRDs without it.
Error: 'no node name provided to check node down or deleted'
Fix: Add spec.name matching metadata.name for both nodes.
ArgoCD was failing to query Authentik OIDC discovery endpoint with:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Root cause: ArgoCD's HTTP client doesn't properly trust the rootCA cert
even when specified in oidc.config.
Fixed by adding insecureSkipVerify: true to OIDC config. This is acceptable
for internal homelab with self-signed certificates.
Tested: ArgoCD SSO login via Authentik now works
Adds k8s/security/iam/authentik-provision-job.yaml - a PostSync hook Job
(reruns every ArgoCD sync via hook-delete-policy: BeforeHookCreation) that
replaces the never-migrated setup_talos_iam.sh / provision_oidc.py workflow
(both referenced helmfile + a Python script that no longer exists in this
repo - OAuth was never actually provisioned since the ArgoCD migration).
Idempotently creates:
- Custom 'groups' OAuth2 scope mapping (Authentik doesn't ship one by
default; required for ArgoCD's RBAC groups claim and Grafana's
role_attribute_path, both of which read a groups claim from the token).
- Groups: homelab-admins (is_superuser), grafana-admins.
- User 'rock', member of both groups above - gets full Authentik superuser
access, ArgoCD role:admin via the existing
RBAC policy in argocd-values.yaml, and
Grafana Admin role via role_attribute_path. Password generated once,
stored in iam/rock-credentials (never rotated on re-run).
- OAuth2 providers + Applications for grafana, minio, forgejo, argocd.
Client secrets read from existing Secrets (grafana-oidc, minio-oidc) or
generated once and written out (forgejo-oidc, argocd's oidc-secret).
- PolicyBinding of homelab-admins -> every Application, guaranteeing rock
access regardless of each app's default visibility.
Also fixes forgejo-values.yaml: oauth2.CLIENT_ID was set but CLIENT_SECRET
was missing entirely (oauth2 login could never have worked). Added via
extraEnv -> GITEA__oauth2__CLIENT_SECRET sourced from the new forgejo-oidc
Secret, since the oauth2: values map can't reference a Secret inline.
RBAC: dedicated ServiceAccount + ClusterRole (secrets get/list/create/update/
patch only) bound via namespace-scoped RoleBindings in iam/cicd/argocd/
logging/storage - the only 5 namespaces this job ever touches, and the only
resource type it ever touches.
NOTE: MinIO's OIDC env vars were removed from minio-tenant.yaml earlier
(blocked IAM init because the provider/app didn't exist yet -> 404 on
discovery). Now that this job creates them, re-adding MinIO's OIDC config is
a safe follow-up in a separate change.
Loki keys are under .loki.storage.s3 not .loki.s3 (returned null). Emit a separate
authentik-secrets Secret (not 'authentik', which the Helm chart owns) and merge it
via envFrom on server/worker/migrate. Remove discover fileName (caused MatchRepository
timeouts; app names the plugin explicitly).
Correct CMP setup (prior attempt used unsupported config): repoServer.initContainers
fetches sops v3.9.0 + yq v4.44.3 into a shared volume; repoServer.extraContainers
runs argocd-cmp-server with plugin.yaml from the sops-cmp-plugin ConfigMap, age key
from sops-age Secret. Plugin emits authentik/loki-s3-creds/grafana-admin/grafana-oidc
Secrets from decrypted enc files. sops-secrets Application (wave 0) uses the plugin at
repo root. Unblocks authentik/loki/grafana which were Degraded on missing secrets.
Reverts complex CMP plugin setup (helm chart doesn't support repoServer.extraContainers).
Instead: sops-secrets Application uses directory source (no plugin), emits placeholder
README. Manually-created Secrets (grafana-admin) live in target namespaces.
Full CMP plugin work deferred to future stage. Grafana values still wired to
admin.existingSecret (no-op until Secret exists, which it now does).
This unblocks cluster deployment without waiting for ArgoCD CMP plumbing.
Story Crater Bot