The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.
Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).
Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.
Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Move authentik-provision.py to scripts/iam/ (manual-only)
- Remove job/RBAC resources (not needed for local runs)
- Use public URL directly (no sed substitution needed)
- Add app password support via set_key endpoint
- Support both password grant and client_credentials
- Add local-llm OAuth2 provider and application to Authentik provisioning
- Configure JWT-compatible OAuth2 provider (client_id: local-llm)
- Generate client secret on first run, stored in llm-serving/local-llm-jwt Secret
- Bind llm-admins group to local-llm application for admin access
- Add RBAC for provisioning job to create secrets in llm-serving namespace
- Output JWT issuer URL and certificate for local-llm token validation
Client credentials + device code grant, no browser redirect (empty
redirect_uris) - unlike every other SERVICES entry which is
authorization_code web SSO. First real step toward replacing
poimen-memory's static API key with a proper JWT flow.
Confidential client for 'vault login -method=oidc' and the Vault UI's
oidc auth method. homelab-admins gets bound automatically like every
other app in SERVICES. Client secret generated on first provision run
into iam/vault-oidc.
Follows the portainer/kmsvc/temporal pattern - group + "permissions"
claim entry only, no Authentik Application/OAuth provider, since
poimen-memory is an internal API-key service, not browser OIDC login.
rock gets it automatically (already in every service_admin_group).
Self-hosted photo backup (Google Photos replacement) - raw manifests,
no Helm chart, self-contained under k8s/apps/immich including its own
CNPG Postgres. Media PVC shares the cp-3 HDD 2TB/2TB with
paperless-media.
Postgres is pg18, not this repo's usual 16.2: CNPG's official pgvector
extension image (ghcr.io/cloudnative-pg/pgvector) is only published
for pg18, loaded via CNPG's ImageVolume extension mechanism (operator
1.30.0 / k8s 1.36.1 both support it). Immich auto-manages CREATE
EXTENSION itself at startup.
OIDC via a new "immich_role" Authentik scope mapping (homelab-admins/
immich-admins -> "admin" claim, else "user"), consumed by Immich's
OAuth roleClaim setting which re-syncs isAdmin on every login - more
reliable than Immich's racy first-user-is-admin fallback. Config
composed into an immich-oidc Secret and mounted as IMMICH_CONFIG_FILE,
matching the paperless-oidc pattern. k8s RBAC (immich-operator Role +
oidc:immich-admins binding) mirrors paperless/rbac.yaml.
immich namespace pre-created in k8s/infra/databases/namespaces.yaml
(not just immich's own CreateNamespace=true) since the iam PostSync
job's RoleBinding needs it to exist before wave 8.
Adds permissions claim + per-service admin groups in Authentik, scoped
Role/RoleBinding per service, public PKCE kubernetes OAuth2 client, and
kube-apiserver OIDC extraArgs. Also fixes paperless OIDC signup permissions
via adapter override and adds CoreDNS rewrite for authentik.riotpiao.com.
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
Vault's S3 storage backend needs AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
from vault-minio-creds, previously generated by a helmfile presync hook
that no longer exists post-Terraform/helmfile removal. Sourced from the
same MINIO_ROOT_USER/PASSWORD already in .env. vault-unseal-keys still
missing separately — needs a live 'vault operator init' run, deferred.
rock
pi