The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.
- deployment: mount cloudflared-config (config.yml) and the
cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
the ingress certificate (force-ssl-redirect makes plaintext impossible)
Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.
Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
Two defects in the ksops generator:
1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
while every other entry uses four. YAML rejected the list, and kustomize
reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
39`. That failed the whole sops-secrets application, so no Secret in that app
could be created. Verified by running the same kustomize build inside
argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.
2. The masterkey value was generated during the first verification pass and
appeared in that session's output. Rotate it now, before the instance exists:
a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
ZITADEL data exists yet, so rotation costs nothing. After the first sync it
would mean re-encrypting system.encryption_keys.
The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.
Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.
The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.
Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
- portfolio-secrets.enc.env: FORGEJO_TOKEN + LLM_API_TOKEN for api.riotpiao.com
- kustomization: secretGenerator for ksops handling at deploy time
- Will be SOPS encrypted with homelab age key before merge
Old root token and unseal keys were dead (lost access to the previous
Vault store). Wiped the S3 backend and vault-0, re-initialized fresh,
rotated these to match the new live unseal keys so ArgoCD's next sync
doesn't clobber them back to the dead ones.
Fixes controlplane.tftpl's install.wipe:true (should be false, live CPs already run false) and syncs coredns Corefile back to what's actually deployed (drops an unrolled-out, stale Kong-era rewrite).
Kong was retired 2026-08-19, replaced by the rock/homelab-frontend Go
gateway (single /v1/chat/completions endpoint, model routed via the
request body's "model" field per API.md). Old per-model baseUrls
(/v1/ornith, /v1/reasoning, /v1/qwen) all 404 against the new gateway.
Also flipping reasoning's supportsTools to true -- confirmed working via
live test now that reasoning runs Qwen3-32B instead of DeepSeek-R1.
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
rock
pi