Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Client credentials + device code grant, no browser redirect (empty
redirect_uris) - unlike every other SERVICES entry which is
authorization_code web SSO. First real step toward replacing
poimen-memory's static API key with a proper JWT flow.
Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.
Follows the portainer/kmsvc/temporal pattern - group + "permissions"
claim entry only, no Authentik Application/OAuth provider, since
poimen-memory is an internal API-key service, not browser OIDC login.
rock gets it automatically (already in every service_admin_group).
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
rock
Story Crater Bot