Retiring Authentik deleted the iam-jobs Application, so k8s/infra/iam stopped
being synced. The new smtp-gotify relay would have sat in git and never been
created.
Add an iam-support Application for that path, and drop the stale
rbac-dashboard-rolebinding reference from its kustomization: the file went away
with Authentik, and a missing resource fails the whole app.
ZITADEL sends verification codes and password resets by SMTP only, and this
cluster has no mail path. Every human user therefore sits at
verified_email = empty and can never finish a first login, at ZITADEL or at
Forgejo.
Add an smtp-gotify relay in the iam namespace: it accepts plain SMTP on :2525
and forwards each message to the existing Gotify instance, so codes arrive on
the phone through gotify/igotify instead of email.
Point DefaultInstance.SMTPConfiguration at that Service with TLS off, because
the relay speaks plain SMTP inside the cluster. The relay holds the Gotify app
token in a SOPS-encrypted Secret; values never carry the token.
Forgejo keeps login sources in its database and exposes no REST endpoint for
them, so neither GitOps nor `tea` (whose admin command only manages users) can
set them. The in-pod CLI is the only supported writer.
This script wraps `gitea admin auth update-oauth` so the rename from the
Authentik-era label to `zitadel` is repeatable instead of a one-off command in
a shell history. Note in the header: the source name is both the button label
and the callback path, so ZITADEL must have both redirect URIs registered
before a rename.
loginV2.required lives in the instance database, not in chart values, so
disabling the hosted Login UI in values is only half the change: the server
kept redirecting /oauth/v2/authorize to /ui/v2/login, which no longer had a
backend and returned 503.
Set loginV2.required=false in the provisioning run so the server serves its
own login page. Verified: authorize now redirects to /ui/login/login and the
page renders the "Welcome Back!" login form.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
The api-gw gates /v1/* on a capability named in the access token. ZITADEL
had no such role, so every inference call was 403 once the token came from
ZITADEL instead of Authentik.
- add llm:inference (Authentik spelling, keeps consumer manifests unchanged)
and llm-inference (dash alias) to the project roles
- enable projectRoleAssertion on the project, which is what puts
urn:zitadel:iam:org:project:roles into the access token; without it the
role list never reaches the resource server
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
Commit a02bee5 had both the Ingress rule and the cloudflared tunnel
hostname for authentik.riotpiao.com. Both were lost, so the tunnel
answered 404 for the hostname and the ingress had no matching rule.
The CoreDNS rewrite and the authentik-server Service were still intact;
only the two routing entries were missing. Re-add the Ingress (namespace
iam, service authentik-server port 80, TLS terminated by the wildcard
default-ssl-certificate) and the tunnel hostname.
The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.
- deployment: mount cloudflared-config (config.yml) and the
cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
the ingress certificate (force-ssl-redirect makes plaintext impossible)
Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.
Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.
Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).
Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.
Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
- remove terraform/terraform.tfvars.local.bak from the index (it carries the
live cluster PKI: CA keys, service-account key, secretbox secret)
- ignore terraform/*.bak and terraform/edge/.secrets/
- generate a fresh WireGuard relay keypair into the git-ignored local tfvars;
the committed example now documents rotation instead of holding key material
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
Adds terraform/edge, a root module with two providers:
- Cloudflare 5.28.1: CNAMEs to the existing tunnel, plus an opt-in resource that
can take the tunnel ingress list over when manage_tunnel_config is true. It is
off by default because that resource REPLACES the whole list, and the tunnel is
remotely managed from the dashboard today.
- Tencent Cloud 1.83.38: VPC, subnet, security group with three rules, key pair,
one CVM relay with cloud-init (WireGuard listener, forwarding sysctls, Caddy),
and DNSPod A records for the .cn hostnames.
The design keeps the homelab inbound-free: the relay terminates TLS in China and
forwards over a WireGuard tunnel the cluster dials outbound. Cloudflare keeps
serving every existing hostname untouched.
Verified: terraform init succeeds for both providers, fmt clean, validate
reports 0 errors, and plan resolves to 10 creates with placeholder variables.
Provider-side name validation caught a real bug during that run: key pair and
security group names allow only letters, digits and underscore, so the derived
names use underscores.
Not applied. Credentials come from CLOUDFLARE_API_TOKEN,
TENCENTCLOUD_SECRET_ID and TENCENTCLOUD_SECRET_KEY; the WireGuard private key is
TF_VAR only. State is local and git-ignored like terraform/zitadel.
Blocked on facts this module cannot invent: the real .cn domain, whether it has
an ICP filing, the CVM image ID, the operator CIDR, and the homelab WireGuard
public key.
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:
write /prometheus/queries.active: no space left on device
failed to initialize active query tracker
18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
Two defects in the ksops generator:
1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
while every other entry uses four. YAML rejected the list, and kustomize
reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
39`. That failed the whole sops-secrets application, so no Secret in that app
could be created. Verified by running the same kustomize build inside
argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.
2. The masterkey value was generated during the first verification pass and
appeared in that session's output. Rotate it now, before the instance exists:
a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
ZITADEL data exists yet, so rotation costs nothing. After the first sync it
would mean re-encrypting system.encryption_keys.
The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.
Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.
macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.
The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.
Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.
The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.
.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.
Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.
- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
into the plain manifests that carry them, write machine keys per agent.
Requires an exactly-one-match regex before it edits a manifest, prints the
diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
while the groups-claim action is disabled.
Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
cannot address one document. sops reads the creation rule from the filename,
so the temp file is passed with --filename-override; without it sops finds no
matching rule and refuses. Ciphertext is staged in the destination directory
and installed with os.replace, so the mode is preserved and EXDEV is
impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
consumer. Writing the new secret into the orphan file would report success
while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
password hash do not appear in ps.
.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.
Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.
Not run against a live ZITADEL: no instance exists yet.
Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open.
It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users
plus keys, one API application for the gateway, project grants, per-agent claim
metadata, and a disabled groups-claim action.
Facts established while writing it:
- Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest
has no client_id field, and provider 3.8.7 rejects a settable application id
(terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a
client_id-preserving re-point; every consumer now also changes client ID.
- Role keys equal the existing group names, so the groups claim can be
byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200
characters with no character-set restriction.
- Machine users get JWT profile keys, not passwords. Per-agent claims move to
user metadata and reach tokens only through the groups-claim action.
- The action stays gated: enable_groups_claim_action defaults to false because
the action runtime API is unverified. Two validations stop a null script from
reaching file().
Checks: terraform fmt -check -recursive clean, terraform validate 0 errors,
plan reaches only the expected missing-credentials error, structural gate
0 error / 0 warning, and one reviewer round: BLOCK on a plan-time
fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed
to SIGN-OFF.
State is local and git-ignored; it carries generated client secrets and machine
keys. Move it to MinIO before a second operator applies.
Not applied: no Zitadel instance is running yet.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
The Application pointed at github.com/ToyoGlobal/azhi-infra, branch
feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled:
status stayed Unknown because the source repository could not be read, and the
destination namespace azi-riotpiao was never created. ArgoCD reported it as an
OutOfSync managed resource of homelab-root with nothing behind it.
Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from
that repository is referenced any more, so the allowlist entry is dead weight.
Not touched here, because it is not in git: Secret azhi-infra-github in
namespace argocd holds the read-only GitHub credential for that repository.
Delete it separately once no other Application needs the repo.
The live Application object was deleted by hand. Until this commit reaches
main, the next homelab-root sync recreates it, because main still declares it.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.
Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
disk is gone
The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
rock
pi