A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).
Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.
Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
- remove terraform/terraform.tfvars.local.bak from the index (it carries the
live cluster PKI: CA keys, service-account key, secretbox secret)
- ignore terraform/*.bak and terraform/edge/.secrets/
- generate a fresh WireGuard relay keypair into the git-ignored local tfvars;
the committed example now documents rotation instead of holding key material
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
Adds terraform/edge, a root module with two providers:
- Cloudflare 5.28.1: CNAMEs to the existing tunnel, plus an opt-in resource that
can take the tunnel ingress list over when manage_tunnel_config is true. It is
off by default because that resource REPLACES the whole list, and the tunnel is
remotely managed from the dashboard today.
- Tencent Cloud 1.83.38: VPC, subnet, security group with three rules, key pair,
one CVM relay with cloud-init (WireGuard listener, forwarding sysctls, Caddy),
and DNSPod A records for the .cn hostnames.
The design keeps the homelab inbound-free: the relay terminates TLS in China and
forwards over a WireGuard tunnel the cluster dials outbound. Cloudflare keeps
serving every existing hostname untouched.
Verified: terraform init succeeds for both providers, fmt clean, validate
reports 0 errors, and plan resolves to 10 creates with placeholder variables.
Provider-side name validation caught a real bug during that run: key pair and
security group names allow only letters, digits and underscore, so the derived
names use underscores.
Not applied. Credentials come from CLOUDFLARE_API_TOKEN,
TENCENTCLOUD_SECRET_ID and TENCENTCLOUD_SECRET_KEY; the WireGuard private key is
TF_VAR only. State is local and git-ignored like terraform/zitadel.
Blocked on facts this module cannot invent: the real .cn domain, whether it has
an ICP filing, the CVM image ID, the operator CIDR, and the homelab WireGuard
public key.
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:
write /prometheus/queries.active: no space left on device
failed to initialize active query tracker
18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
Two defects in the ksops generator:
1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
while every other entry uses four. YAML rejected the list, and kustomize
reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
39`. That failed the whole sops-secrets application, so no Secret in that app
could be created. Verified by running the same kustomize build inside
argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.
2. The masterkey value was generated during the first verification pass and
appeared in that session's output. Rotate it now, before the instance exists:
a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
ZITADEL data exists yet, so rotation costs nothing. After the first sync it
would mean re-encrypting system.encryption_keys.
The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.
Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.
macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.
The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.
Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.
The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.
.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.
Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.
- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
into the plain manifests that carry them, write machine keys per agent.
Requires an exactly-one-match regex before it edits a manifest, prints the
diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
while the groups-claim action is disabled.
Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
cannot address one document. sops reads the creation rule from the filename,
so the temp file is passed with --filename-override; without it sops finds no
matching rule and refuses. Ciphertext is staged in the destination directory
and installed with os.replace, so the mode is preserved and EXDEV is
impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
consumer. Writing the new secret into the orphan file would report success
while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
password hash do not appear in ps.
.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.
Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.
Not run against a live ZITADEL: no instance exists yet.
Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open.
It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users
plus keys, one API application for the gateway, project grants, per-agent claim
metadata, and a disabled groups-claim action.
Facts established while writing it:
- Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest
has no client_id field, and provider 3.8.7 rejects a settable application id
(terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a
client_id-preserving re-point; every consumer now also changes client ID.
- Role keys equal the existing group names, so the groups claim can be
byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200
characters with no character-set restriction.
- Machine users get JWT profile keys, not passwords. Per-agent claims move to
user metadata and reach tokens only through the groups-claim action.
- The action stays gated: enable_groups_claim_action defaults to false because
the action runtime API is unverified. Two validations stop a null script from
reaching file().
Checks: terraform fmt -check -recursive clean, terraform validate 0 errors,
plan reaches only the expected missing-credentials error, structural gate
0 error / 0 warning, and one reviewer round: BLOCK on a plan-time
fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed
to SIGN-OFF.
State is local and git-ignored; it carries generated client secrets and machine
keys. Move it to MinIO before a second operator applies.
Not applied: no Zitadel instance is running yet.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
The Application pointed at github.com/ToyoGlobal/azhi-infra, branch
feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled:
status stayed Unknown because the source repository could not be read, and the
destination namespace azi-riotpiao was never created. ArgoCD reported it as an
OutOfSync managed resource of homelab-root with nothing behind it.
Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from
that repository is referenced any more, so the allowlist entry is dead weight.
Not touched here, because it is not in git: Secret azhi-infra-github in
namespace argocd holds the read-only GitHub credential for that repository.
Delete it separately once no other Application needs the repo.
The live Application object was deleted by hand. Until this commit reaches
main, the next homelab-root sync recreates it, because main still declares it.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.
Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
disk is gone
The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
The homelab AppProject allowlist had no azhi repository, so an Application
pointing at ToyoGlobal/azhi-infra fails admission with 'repository not
permitted'. Adding the URL is the whole gate.
The Application itself targets the repository default branch
(harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That
overlay renders postgres and the API into namespace azi-riotpiao.
The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR
pull credential and azhi-riotpiao-app-env carries the database credentials,
the JWT secrets and the AES key.
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
cluster-maintenance tracks main (targetRevision: main), so the cluster never
had the CronJob or its Secret
This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
would have reclaimed nothing
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.
The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:
- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).
Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
Exit code 143 proved the previous job killed itself: the script's own cmdline contains the
string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver')
and skipping $$ removes that failure mode. Staggered 2 minutes apart.
Static pods survive mirror-pod deletion, so the first bootstrap job had no effect.
These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself,
staggered by 100s so only one control plane restarts at a time.
The first job could not pull its image; this one deletes the apiserver pods via the
in-cluster API with the mounted service-account token instead of a kubectl image.
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.
Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.
Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
## Why
`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).
The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:
```
minio tenant pod not ready (1/2)
-> headless Service minio-cluster-hl has no endpoints
-> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
-> Vault S3 storage init fails at startup
-> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```
## Control checks (so this is not a network story)
| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |
## Change
Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.
## Caveat
This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.
## After merge (ArgoCD, not kubectl)
The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:
```
kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds)
```
Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53
Co-authored-by: rock <[email protected]>
pi
rock