Compare commits

...
Author SHA1 Message Date
rock 3ec46d26f0 merge: harness/zitadel-iam-wiring 2026-10-25 20:20:54 -07:00
rock 52bca6b1b6 merge: feat/local-path-backend 2026-10-25 20:20:40 -07:00
rock 3e079b1cf7 merge: fix/minio-mc-image (keep digest pin + comment) 2026-10-25 20:20:22 -07:00
rock 227a67a8c1 feat(cloudflared): manage tunnel ingress locally via ConfigMap
The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.

- deployment: mount cloudflared-config (config.yml) and the
  cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
  CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
  the ingress certificate (force-ssl-redirect makes plaintext impossible)

Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
2026-10-25 19:57:37 -07:00
pi d4e9dbf350 refactor(iam): retire terraform/zitadel, provision through the API
The Terraform root module duplicated a writer that the provisioning script could
be, and its state carried generated client secrets. Declare the objects in the
script instead: project, 19 roles, 12 OIDC apps, applied through the management
API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel.

Also add zitadel to the Cloudflare tunnel hostname list, with the import command
and the note that the tunnel's Public Hostname entry is a separate dashboard
step: DNS alone yields a 404 from the tunnel.
2026-10-25 19:35:01 -07:00
pi be44a576d4 fix(iam): own the Zitadel login keypair through ksops
The chart generated the login RSA keypair on every render, so
checksum/secret-login-service-key changed each time and both Zitadel
Deployments stayed permanently OutOfSync.

Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml
(SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point
login.loginServiceKeySecretName at it. The chart then stops generating the
keypair and the Deployment checksums become deterministic.
2026-10-25 19:16:18 -07:00
pi 39081222a3 fix(iam): stop the setup hook running after first boot
A PreSync setup run is not harmless once the instance exists: it replays every
repeatable migration on every sync, and interrupting one leaves a
system.migration.started event with no matching done. ZITADEL then waits
forever on 'migration already started'. That deadlocked this instance twice
(projections.projects4, then repeatable_delete_stale_org_fields).

Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD
sync never runs it. Re-enable deliberately for a schema upgrade.
2026-10-25 19:10:32 -07:00
pi 7e5b793670 fix(iam,edge): freeze first-boot and wire the WireGuard CIDR variable
Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and
initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and
silently rotates the iam-admin key that terraform/zitadel authenticates with.

Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the
variable was declared but the rule still used relay_admin_cidr, which blocks
the homelab peer unless its egress address happens to match the operator CIDR.
2026-10-25 18:48:15 -07:00
pi 354814a531 security: stop tracking Talos secret backup; rotate edge WireGuard key
- remove terraform/terraform.tfvars.local.bak from the index (it carries the
  live cluster PKI: CA keys, service-account key, secretbox secret)
- ignore terraform/*.bak and terraform/edge/.secrets/
- generate a fresh WireGuard relay keypair into the git-ignored local tfvars;
  the committed example now documents rotation instead of holding key material
2026-10-25 18:15:48 -07:00
pi 54fac02783 fix(cloudflared): pin tunnel protocol to http2
QUIC loses long-lived SSE streams when Chinese ISP gateways throttle UDP.
http2 over TCP is the deliberate fallback already in use.
2026-10-25 18:04:53 -07:00
pi eb83aa6290 refactor(edge,iam): apply best practices to relay and Zitadel config
- edge: extract VPC/subnet/bandwidth/disk/DNS/Caddy settings into variables
- edge: add Caddyfile template with SSE-safe flush_interval -1 and security headers
- edge: add WireGuard NAT masquerade so homelab peer accepts forwarded client IPs
- edge: mark user_data sensitive and add prevent_destroy to the relay instance
- edge: validate relay_admin_cidr and Cloudflare hostnames
- edge: parameterise Cloudflare TTL/proxied and enforce proxied tunnel CNAMEs
- iam: fix PDB key to chart-supported pdb and remove unsupported extraVolumes
- iam: keep login hardened; drop unsupported login PDB
2026-10-25 18:03:55 -07:00
pi 6ebecec1c4 fix(zitadel): raise setup job deadline to 20m
The chart default killed zitadel-setup at ~5.5 minutes while it was still
replaying projections.projects4, which failed the whole PreSync and left the
Application OutOfSync/Missing. 1200s leaves room for a cold database without
masking a genuine hang.
2026-10-25 09:47:33 -07:00
rock 091ecdd7bb feat(edge): Terraform for the China-facing relay and Cloudflare DNS
Adds terraform/edge, a root module with two providers:

- Cloudflare 5.28.1: CNAMEs to the existing tunnel, plus an opt-in resource that
  can take the tunnel ingress list over when manage_tunnel_config is true. It is
  off by default because that resource REPLACES the whole list, and the tunnel is
  remotely managed from the dashboard today.
- Tencent Cloud 1.83.38: VPC, subnet, security group with three rules, key pair,
  one CVM relay with cloud-init (WireGuard listener, forwarding sysctls, Caddy),
  and DNSPod A records for the .cn hostnames.

The design keeps the homelab inbound-free: the relay terminates TLS in China and
forwards over a WireGuard tunnel the cluster dials outbound. Cloudflare keeps
serving every existing hostname untouched.

Verified: terraform init succeeds for both providers, fmt clean, validate
reports 0 errors, and plan resolves to 10 creates with placeholder variables.
Provider-side name validation caught a real bug during that run: key pair and
security group names allow only letters, digits and underscore, so the derived
names use underscores.

Not applied. Credentials come from CLOUDFLARE_API_TOKEN,
TENCENTCLOUD_SECRET_ID and TENCENTCLOUD_SECRET_KEY; the WireGuard private key is
TF_VAR only. State is local and git-ignored like terraform/zitadel.

Blocked on facts this module cannot invent: the real .cn domain, whether it has
an ICP filing, the CVM image ID, the operator CIDR, and the homelab WireGuard
public key.
2026-10-24 22:59:54 -07:00
rock 7251032081 Merge pull request 'fix(monitoring): cap Prometheus retention below the PVC size' (#68) from fix/prometheus-retention-size into main 2026-10-25 04:43:21 +00:00
rock b7fc6ed60b fix(monitoring): cap Prometheus retention below the PVC size
retentionSize was 95GB against a 25Gi volume, so the limit never fired. The
volume filled, and Prometheus crashlooped with:

  write /prometheus/queries.active: no space left on device
  failed to initialize active query tracker

18GB keeps roughly 7GiB of headroom for the WAL and block overhead.
2026-10-24 21:43:01 -07:00
rock 6312674058 chore(sms): retire the stuck macOS namespace
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
2026-10-24 20:05:20 -07:00
rock 2fe6d50ffd Merge pull request 'fix(argocd): allow the Zitadel chart repo in the homelab AppProject' (#67) from fix/allow-zitadel-chart-repo into main 2026-10-25 01:26:38 +00:00
rock e2d9717ce1 fix(argocd): allow the Zitadel chart repo in the homelab AppProject
The zitadel Application failed its first sync with InvalidSpecError:
application repo https://charts.zitadel.com is not permitted in project
'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be
listed before any chart from it can be reconciled.
2026-10-24 18:26:21 -07:00
rock 111b19dac2 Merge pull request 'fix(sops): repair the ksops generator list and rotate the Zitadel masterkey' (#66) from fix/sops-generator-indent into main 2026-10-25 01:24:43 +00:00
rock e335694cbb fix(sops): repair the generator list and rotate the Zitadel masterkey
Two defects in the ksops generator:

1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation
   while every other entry uses four. YAML rejected the list, and kustomize
   reported it as `loading generator plugins ... MalformedYAMLError: yaml: line
   39`. That failed the whole sops-secrets application, so no Secret in that app
   could be created. Verified by running the same kustomize build inside
   argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it.

2. The masterkey value was generated during the first verification pass and
   appeared in that session's output. Rotate it now, before the instance exists:
   a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No
   ZITADEL data exists yet, so rotation costs nothing. After the first sync it
   would mean re-encrypting system.encryption_keys.

The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).
2026-10-24 18:24:28 -07:00
rock d1a7d3be6e fix(oidc): remove the API-server restart Jobs from GitOps
The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three
Jobs in restart-apiserver.yaml kill the kube-apiserver process on each
control plane, and bootstrap.yaml carried a fourth Job that deletes the
mirror pods. ArgoCD recreated a deleted Job within two minutes, so the
kill repeated and every API server reached CrashLoopBackOff.

Remove all four. The OIDC authenticator already loaded, so the one-shot
restart is not needed again.
2026-10-24 15:06:32 -07:00
rock 2f664c32d9 fix(cluster): grant queue-operator watches and retire the macOS workload
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.

macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
2026-10-24 04:03:02 -07:00
rock 6c98ca7436 Merge pull request 'fix(sops): drop the orphan forgejo-registry-pat file' (#65) from fix/sops-drop-orphan-forgejo-pat into main 2026-10-24 10:47:14 +00:00
rock 6ee22401fd fix(sops): drop the orphan forgejo-registry-pat file
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.

The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.

Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
2026-10-24 03:46:29 -07:00
rock e9817283e6 Merge pull request 'feat(iam): wire Zitadel as the Authentik replacement' (#63) from harness/zitadel-iam-wiring into main 2026-10-24 10:41:17 +00:00
rock 6e9586aa15 Merge pull request 'chore(argocd): delete the azhi-riotpiao Application' (#64) from chore/remove-azhi-riotpiao-app into main 2026-10-24 10:41:09 +00:00
rock 979a413a5b feat(iam): read the Zitadel bootstrap token from .env
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.

The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.

.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.

Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
2026-10-24 03:38:21 -07:00
rock 2f62351731 feat(iam): one script to provision Zitadel end to end
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.

- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
  at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
  into the plain manifests that carry them, write machine keys per agent.
  Requires an exactly-one-match regex before it edits a manifest, prints the
  diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
  passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
  while the groups-claim action is disabled.

Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
  cannot address one document. sops reads the creation rule from the filename,
  so the temp file is passed with --filename-override; without it sops finds no
  matching rule and refuses. Ciphertext is staged in the destination directory
  and installed with os.replace, so the mode is preserved and EXDEV is
  impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
  no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
  consumer. Writing the new secret into the orphan file would report success
  while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
  password hash do not appear in ps.

.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.

Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.

Not run against a live ZITADEL: no instance exists yet.
2026-10-24 03:38:21 -07:00
rock f81ff0e2fc feat(iam): declare Zitadel identity objects in Terraform
Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open.
It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users
plus keys, one API application for the gateway, project grants, per-agent claim
metadata, and a disabled groups-claim action.

Facts established while writing it:

- Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest
  has no client_id field, and provider 3.8.7 rejects a settable application id
  (terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a
  client_id-preserving re-point; every consumer now also changes client ID.
- Role keys equal the existing group names, so the groups claim can be
  byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200
  characters with no character-set restriction.
- Machine users get JWT profile keys, not passwords. Per-agent claims move to
  user metadata and reach tokens only through the groups-claim action.
- The action stays gated: enable_groups_claim_action defaults to false because
  the action runtime API is unverified. Two validations stop a null script from
  reaching file().

Checks: terraform fmt -check -recursive clean, terraform validate 0 errors,
plan reaches only the expected missing-credentials error, structural gate
0 error / 0 warning, and one reviewer round: BLOCK on a plan-time
fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed
to SIGN-OFF.

State is local and git-ignored; it carries generated client secrets and machine
keys. Move it to MinIO before a second operator applies.

Not applied: no Zitadel instance is running yet.
2026-10-24 03:38:21 -07:00
rock cb1219bf96 feat(iam): wire Zitadel into namespace iam under ArgoCD
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.

- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
  secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
  automated sync deliberately absent for the first sync (chart PreSync hook
  re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
  crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host

Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).

The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
2026-10-24 03:38:21 -07:00
rock 8e9ac37583 chore(argocd): delete the azhi-riotpiao Application
The Application pointed at github.com/ToyoGlobal/azhi-infra, branch
feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled:
status stayed Unknown because the source repository could not be read, and the
destination namespace azi-riotpiao was never created. ArgoCD reported it as an
OutOfSync managed resource of homelab-root with nothing behind it.

Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from
that repository is referenced any more, so the allowlist entry is dead weight.

Not touched here, because it is not in git: Secret azhi-infra-github in
namespace argocd holds the read-only GitHub credential for that repository.
Delete it separately once no other Application needs the repo.

The live Application object was deleted by hand. Until this commit reaches
main, the next homelab-root sync recreates it, because main still declares it.
2026-10-24 03:35:52 -07:00
rock 3c851b16c9 Merge branch 'fix/certmanager-dns01'
Restores wildcard renewal: pin the ACME DNS-01 self-check to the public
resolvers and drop the per-host TLS blocks that competed with the wildcard.
2026-10-24 02:39:38 -07:00
rock e93d4ba86c fix(cert-manager): restore wildcard renewal and drop per-host TLS
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".

Changes:

- cert-manager: pin the self-check to the public resolvers with
  --dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
  Verified against Let's Encrypt staging: the challenge went valid and the
  certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
  cert-manager.io/cluster-issuer annotations. The wildcard covers both
  hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
  with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
  and the zero-downtime renewal path.

Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
2026-10-24 02:30:48 -07:00
rock 7a1213d504 Merge pull request 'fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2' (#62) from fix/retire-longhorn-disk4 into main 2026-10-13 00:10:01 +00:00
rock d21ee9630f fix(longhorn): retire the failed 2TB PERC disk on talos-cp-2
The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.

Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
  disk is gone

The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
2026-10-13 08:09:42 +08:00
rock 40aa23dfdf Merge pull request 'test(argocd): point azhi-riotpiao at the CNPG branch' (#61) from feat/azhi-cnpg-test into main 2026-10-12 13:41:38 +00:00
rock 71fce30506 test(argocd): point azhi-riotpiao at the CNPG branch
The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
2026-10-12 21:24:47 +08:00
rock 0666715a5e Merge pull request 'feat(argocd): reconcile azhi-infra from its private GitHub repo' (#60) from feat/azhi-riotpiao-argocd into main 2026-10-12 12:54:43 +00:00
rock 949ae5cb38 feat(argocd): let ArgoCD reconcile azhi-infra from its private GitHub repo
The homelab AppProject allowlist had no azhi repository, so an Application
pointing at ToyoGlobal/azhi-infra fails admission with 'repository not
permitted'. Adding the URL is the whole gate.

The Application itself targets the repository default branch
(harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That
overlay renders postgres and the API into namespace azi-riotpiao.

The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR
pull credential and azhi-riotpiao-app-env carries the database credentials,
the JWT secrets and the AES key.
2026-10-12 20:54:00 +08:00
rock e150969070 Merge pull request #59 from fix/etcd-defrag-main 2026-10-12 12:27:28 +00:00
rock abfdda5f57 fix(etcd): add a working 15-day defrag CronJob under the backend quota
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.

An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
  failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
  cluster-maintenance tracks main, so the cluster never had it

Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
  epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
  day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
  apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
  against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
  never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
  ~1.2M entries behind the leader with a frozen raft term and needed a reboot
  to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
  runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
  client certificate signed by the cluster's etcd CA, valid to 2036-10.
2026-10-07 17:12:39 +08:00
rock e4caa7e841 fix(etcd): make the defrag CronJob runnable and 15-day
The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
  exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
  cluster-maintenance tracks main (targetRevision: main), so the cluster never
  had the CronJob or its Secret

This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
  matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
  16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
  compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
  behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
  NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
  would have reclaimed nothing
2026-10-07 17:11:35 +08:00
harness 1ce9f86635 feat(etcd): weekly defrag CronJob to stay under the etcd backend quota
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.

The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:

- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
  revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
  CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).

Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
2026-10-07 13:37:45 +08:00
rock 2d9e3abee8 fix(oidc): match /proc/<pid>/comm instead of cmdline when signalling the API server
Exit code 143 proved the previous job killed itself: the script's own cmdline contains the
string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver')
and skipping $$ removes that failure mode. Staggered 2 minutes apart.
2026-10-07 04:07:37 +00:00
rock 9fa8d9cf60 fix(oidc): restart the apiservers by signalling the process, not by deleting the mirror pod
Static pods survive mirror-pod deletion, so the first bootstrap job had no effect.
These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself,
staggered by 100s so only one control plane restarts at a time.
2026-10-07 03:59:22 +00:00
rock 2e4fad3bf7 fix(oidc): use curlimages/curl for the bootstrap job (bitnami/kubectl tag did not exist)
The first job could not pull its image; this one deletes the apiserver pods via the
in-cluster API with the mounted service-account token instead of a kubectl image.
2026-10-07 03:52:10 +00:00
rock 5d22691c3e chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator
k8s/argocd/apps/99-oidc-bootstrap.yaml
2026-10-07 03:46:01 +00:00
rock 4eee161e0f chore(oidc): one-shot bootstrap - restart apiservers to initialise the OIDC authenticator
k8s/infra/oidc-bootstrap/bootstrap.yaml
2026-10-07 03:45:56 +00:00
rock 719fa77521 Merge pull request 'fix(secrets): populate the empty sops placeholders and drop the unused one' (#55) from fix/secrets-placeholders into main 2026-10-06 04:48:51 +00:00
rock ebb1b9b02d feat(storage): add local-path-provisioner as the fleet's persistent-storage backend 2026-10-06 12:46:58 +08:00
rock cc62f74404 feat(storage): add local-path-provisioner as the fleet's persistent-storage backend 2026-10-06 12:45:12 +08:00
rock 64cd1f2c4e fix(minio): freeze the community image by digest and alert on pull failure (#57)
Co-authored-by: rock <[email protected]>
2026-10-06 03:43:12 +00:00
rock d7b96fb29b fix(minio): freeze the community image by digest and alert on pull failure
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.

Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.

Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
2026-10-06 11:42:51 +08:00
rock 71b3cef2eb fix(minio): replace the dead minio/mc image in the provision hook and backup job 2026-10-06 10:10:56 +08:00
rock f780e31f9c Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent) 2026-10-06 10:05:45 +08:00
rock b6e317b34c feat(poimen): pi sandbox workspace volume, encrypted api keys, cluster dns rewrite (#54)
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.

Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
2026-10-06 01:49:12 +00:00
rock f3015522e3 fix(storage): pin minio tenant to a published image release (#53)
## Why

`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).

The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:

```
minio tenant pod not ready (1/2)
  -> headless Service minio-cluster-hl has no endpoints
  -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
  -> Vault S3 storage init fails at startup
  -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```

## Control checks (so this is not a network story)

| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |

## Change

Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.

## Caveat

This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.

## After merge (ArgoCD, not kubectl)

The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:

```
kubectl get pods -n storage          # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault  # vault-0 -> Running (storage init succeeds)
```

Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53

Co-authored-by: rock <[email protected]>
2026-10-06 01:48:57 +00:00
70 changed files with 3155 additions and 463 deletions
+11
View File
@@ -31,6 +31,17 @@ AUTHENTIK_BOOTSTRAP_TOKEN=
# openssl rand -base64 24 # openssl rand -base64 24
AUTHENTIK_PG_PASSWORD= AUTHENTIK_PG_PASSWORD=
# ── ZITADEL (Authentik replacement) ─────────────────────────────────────────────
# Instance URL, same hostname as the OIDC issuer.
ZITADEL_URL="https://zitadel.riotpiao.com"
# PAT of the 'iam-admin' machine user, created by the chart's setup job on the
# first sync. Leave empty until then. Read it with:
# kubectl -n iam get secret iam-admin-pat -o jsonpath='{.data.pat}' | base64 -d
# scripts/iam/zitadel-provision.py reads this variable first, then the cluster
# Secret. The masterkey and the client secrets never live in this file.
ZITADEL_BOOTSTRAP_TOKEN=
# ── MinIO ────────────────────────────────────────────────────────────────────── # ── MinIO ──────────────────────────────────────────────────────────────────────
MINIO_ROOT_USER= MINIO_ROOT_USER=
MINIO_ROOT_PASSWORD= MINIO_ROOT_PASSWORD=
+20
View File
@@ -47,6 +47,15 @@ k8s/storage/test/test
terraform/.terraform/ terraform/.terraform/
terraform/*.tfstate terraform/*.tfstate
terraform/*.tfstate.* terraform/*.tfstate.*
# The Zitadel root module keeps its own state (see terraform/zitadel/providers.tf).
# That state carries generated client secrets and machine keys.
terraform/zitadel/*.tfstate
terraform/zitadel/*.tfstate.*
terraform/zitadel/*.tfvars
terraform/edge/*.tfstate
terraform/edge/*.tfstate.*
terraform/edge/*.tfvars
terraform/edge/terraform.tfvars.local
terraform.tfvars.local terraform.tfvars.local
skills-lock.json skills-lock.json
secrets-plaintext.yaml secrets-plaintext.yaml
@@ -69,3 +78,14 @@ k8s/**/*-secret.yaml
# IAM provisioning scripts contain credential references — never commit # IAM provisioning scripts contain credential references — never commit
scripts/iam/*.py scripts/iam/*.py
# Exception: this one reads every credential from the cluster at run time and
# embeds none. Keep it reviewable in git.
!scripts/iam/zitadel-provision.py
# Runtime credential export from the old Authentik provisioning run. Holds live
# M2M tokens; never commit it.
/authentik-m2m-tokens.env
# Never commit backup copies of plaintext tfvars (they carry cluster PKI)
terraform/*.bak
terraform/edge/.secrets/
@@ -0,0 +1,22 @@
{
"nextId": 2,
"loops": [
{
"id": "1",
"prompt": "Check Longhorn and CNPG recovery progress in the homelab cluster and report terse. Run: kubectl -n longhorn-system get volumes.longhorn.io -o json (count by state/robustness), kubectl -n longhorn-system get replicas.longhorn.io -o json (count running/stopped/failed), kubectl get clusters.postgresql.cnpg.io -A (ready instances per cluster), kubectl -n cicd get pods,pvc | grep forgejo-db. Report: volumes healthy vs degraded, replicas running vs stopped, and each CNPG cluster ready count. If every volume is attached/healthy and every CNPG cluster shows 3/3 ready, say RECOVERED, list any leftover anomalies, and stop this loop with LoopDelete. Otherwise name what is still rebuilding and keep going.",
"trigger": {
"type": "cron",
"schedule": "*/5 * * * *"
},
"status": "expired",
"recurring": true,
"createdAt": 1792898370946,
"updatedAt": 1792901371403,
"expiresAt": 1793503170946,
"source": "tool",
"readOnly": true,
"maxFires": 6,
"fireCount": 6
}
]
}
@@ -0,0 +1,37 @@
{
"schema_version": 1,
"diagram_type": "architecture",
"meta": {
"title": "Poimen Memory — Schema Relations",
"quality_profile": "showcase",
"viewBox": [900, 480]
},
"components": [
{ "id": "node", "type": "database", "label": "memory_node", "sublabel": "Episodes (raw messages)", "pos": [40, 60], "size": [150, 55], "tag": "BIGSERIAL" },
{ "id": "entity", "type": "database", "label": "memory_entity", "sublabel": "Extracted entities", "pos": [260, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "edge", "type": "database", "label": "memory_edge", "sublabel": "Facts / relationships", "pos": [480, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "community", "type": "database", "label": "memory_community", "sublabel": "Entity clusters", "pos": [700, 60], "size": [150, 55], "tag": "UUID" },
{ "id": "pgvector", "type": "cloud", "label": "pgvector HNSW", "sublabel": "768-dim cosine search", "pos": [140, 260], "size": [160, 55], "tag": "nomic-embed-v2" },
{ "id": "tsvector", "type": "cloud", "label": "pg ts_rank", "sublabel": "Full-text search", "pos": [400, 260], "size": [160, 55], "tag": "GIN index" },
{ "id": "tenant", "type": "security", "label": "tenant_id", "sublabel": "Multi-tenant isolation", "pos": [660, 260], "size": [140, 55], "tag": "W7 / mig 014" }
],
"boundaries": [
{ "kind": "region", "label": "Temporal Knowledge Graph (Zep §2)", "wraps": ["node", "entity", "edge", "community"] },
{ "kind": "security-group", "label": "Search & Isolation", "wraps": ["pgvector", "tsvector", "tenant"] }
],
"connections": [
{ "id": "node-entity", "from": "node", "to": "entity", "variant": "emphasis" },
{ "id": "entity-edge", "from": "entity", "to": "edge", "variant": "emphasis" },
{ "id": "entity-community", "from": "community", "to": "entity", "variant": "dashed" },
{ "id": "edge-node", "from": "edge", "to": "node", "variant": "dashed" },
{ "id": "entity-vec", "from": "entity", "to": "pgvector", "variant": "default" },
{ "id": "entity-fts", "from": "entity", "to": "tsvector", "variant": "dashed" },
{ "id": "edge-fts", "from": "edge", "to": "tsvector", "variant": "dashed" },
{ "id": "tenant-all", "from": "tenant", "to": "entity", "variant": "security" }
],
"cards": [
{ "dot": "emerald", "title": "Three-Tier Graph", "items": ["memory_node → episode subgraph", "memory_entity → entity subgraph", "memory_edge → fact relationships", "memory_community → clusters (Phase 4)"] },
{ "dot": "rose", "title": "Bi-Temporal", "items": ["t_valid / t_invalid: event time", "t_created / t_expired: transaction time", "Contradiction: active → candidate → confirmed"] },
{ "dot": "cyan", "title": "Search", "items": ["HNSW: name, summary, fact embeddings", "ts_rank: BM25 fallback (always available)", "tenant_id: per-table isolation (mig 014)"] }
]
}
@@ -0,0 +1,158 @@
{
"schema_version": 1,
"diagram_type": "dataflow",
"meta": {
"title": "Poimen Memory \u2014 Schema Relations",
"viewBox": [
960,
480
],
"quality_profile": "showcase"
},
"stages": [
{
"label": "Episodes"
},
{
"label": "Entities"
},
{
"label": "Relationships"
},
{
"label": "Indexes"
}
],
"nodes": [
{
"id": "node",
"type": "database",
"label": "memory_node",
"sublabel": "Raw messages",
"stage": 0,
"row": 0,
"tag": "BIGSERIAL"
},
{
"id": "entity",
"type": "database",
"label": "memory_entity",
"sublabel": "Named entities",
"stage": 1,
"row": 0,
"tag": "UUID + tenant_id"
},
{
"id": "community",
"type": "database",
"label": "memory_community",
"sublabel": "Clusters (Phase 4)",
"stage": 1,
"row": 1,
"tag": "UUID"
},
{
"id": "edge",
"type": "database",
"label": "memory_edge",
"sublabel": "Facts + bi-temporal",
"stage": 2,
"row": 0,
"tag": "UUID + tenant_id"
},
{
"id": "hnsw",
"type": "cloud",
"label": "HNSW Index",
"sublabel": "768-dim cosine",
"stage": 3,
"row": 0,
"tag": "pgvector"
},
{
"id": "fts",
"type": "cloud",
"label": "GIN Index",
"sublabel": "ts_rank BM25",
"stage": 3,
"row": 1,
"tag": "full-text"
}
],
"flows": [
{
"from": "node",
"to": "entity",
"label": "LLM extraction",
"classification": "ingest",
"variant": "emphasis",
"route": "straight"
},
{
"from": "entity",
"to": "edge",
"label": "source + target FK",
"classification": "foreign key",
"variant": "emphasis",
"route": "straight"
},
{
"from": "entity",
"to": "community",
"label": "community_id",
"classification": "membership",
"variant": "dashed",
"route": "straight",
"labelDy": 30
},
{
"from": "edge",
"to": "hnsw",
"classification": "vector",
"variant": "default",
"route": "straight",
"label": "embedding",
"labelDy": -30
},
{
"from": "edge",
"to": "fts",
"label": "fact text",
"classification": "text",
"variant": "dashed",
"route": "vertical-channel",
"labelDy": 30
}
],
"cards": [
{
"dot": "emerald",
"title": "Three-Tier Graph",
"items": [
"memory_node \u2192 episodes (raw messages)",
"memory_entity \u2192 extracted entities",
"memory_edge \u2192 facts between entities",
"memory_community \u2192 clusters (Phase 4)"
]
},
{
"dot": "rose",
"title": "Bi-Temporal + Tenant",
"items": [
"t_valid/t_invalid: when true in reality",
"t_created/t_expired: system transaction time",
"tenant_id: row-level multi-tenant isolation",
"Contradiction: active \u2192 candidate \u2192 confirmed"
]
},
{
"dot": "cyan",
"title": "Search Indexes",
"items": [
"HNSW: name, summary, fact embeddings",
"GIN: ts_rank on name+summary and fact",
"RRF fusion: sem_weight + lex_weight"
]
}
]
}
@@ -0,0 +1,63 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "Service Acting on Behalf of User",
"viewBox": [1000, 600],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
{ "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC" },
{ "id": "worker", "type": "backend", "label": "Backend Worker", "sublabel": "temporal/queue" },
{ "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256" }
],
"segments": [
{ "from": 150, "to": 260, "label": "User Auth" },
{ "from": 270, "to": 370, "label": "Delegation" },
{ "from": 380, "to": 510, "label": "Downstream" }
],
"messages": [
{ "id": "login", "from": "user", "to": "app", "y": 165, "label": "login (OIDC)", "variant": "default" },
{ "id": "oidc-flow", "from": "app", "to": "authentik", "y": 200, "label": "authorization_code flow", "variant": "emphasis" },
{ "id": "user-jwt", "from": "authentik", "to": "app", "y": 235, "label": "user JWT {groups,roles}", "variant": "return" },
{ "id": "trigger-job", "from": "user", "to": "app", "y": 285, "label": "trigger async job", "variant": "default" },
{ "id": "queue-job", "from": "app", "to": "worker", "y": 320, "label": "enqueue job + user JWT", "variant": "emphasis", "note": "JWT passed as job metadata" },
{ "id": "job-ack", "from": "worker", "to": "app", "y": 355, "label": "job queued", "variant": "return" },
{ "id": "call-memory", "from": "worker", "to": "memory", "y": 400, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Bearer <user JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 435, "label": "GET /jwks", "variant": "dashed" },
{ "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 470, "label": "RSA pubkey", "variant": "return" },
{ "id": "ingest-ok", "from": "memory", "to": "worker", "y": 505, "label": "202 Accepted", "variant": "return", "note": "uses USER's groups/roles" }
],
"activations": [
{ "participant": "user", "from": 160, "to": 291, "type": "external" },
{ "participant": "app", "from": 165, "to": 361, "type": "frontend" },
{ "participant": "authentik", "from": 195, "to": 241, "type": "security" },
{ "participant": "worker", "from": 315, "to": 511, "type": "backend" },
{ "participant": "memory", "from": 395, "to": 511, "type": "backend" },
{ "participant": "jwks", "from": 430, "to": 476, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "Delegation Pattern",
"items": [
"User JWT travels with the job payload",
"Worker uses user's token for downstream",
"Memory sees USER as the principal"
]
},
{
"dot": "rose",
"title": "Security Model",
"items": [
"No service-account escalation",
"User's permissions bound the action",
"Audit trail traces to user.sub"
]
}
]
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"diagram_type": "architecture",
"meta": {
"title": "Memory Service Fine-Grained RBAC",
"quality_profile": "showcase",
"viewBox": [900, 520]
},
"components": [
{ "id": "jwt", "type": "security", "label": "JWT Token", "sublabel": "groups[] + roles[]", "pos": [40, 60], "size": [140, 60], "tag": "Authentik" },
{ "id": "cap-check", "type": "backend", "label": "Capability Check", "sublabel": "has_capability()", "pos": [260, 60], "size": [160, 60], "tag": "Phase 1" },
{ "id": "scope-checker", "type": "backend", "label": "Scope Checker", "sublabel": "filter_resources()", "pos": [500, 60], "size": [160, 60], "tag": "Phase 2" },
{ "id": "decision", "type": "backend", "label": "AccessDecision", "sublabel": "Allow / Deny", "pos": [740, 60], "size": [130, 60] },
{ "id": "project-scope", "type": "database", "label": "ProjectScope", "sublabel": "dev/staging/prod/*", "pos": [40, 200], "size": [140, 55] },
{ "id": "visibility-scope", "type": "database", "label": "VisibilityScope", "sublabel": "public/private", "pos": [220, 200], "size": [140, 55] },
{ "id": "owner-scope", "type": "database", "label": "OwnerScope", "sublabel": "self/any/user", "pos": [400, 200], "size": [140, 55] },
{ "id": "group-scope", "type": "database", "label": "GroupScope", "sublabel": "in_any_group()", "pos": [580, 200], "size": [140, 55] },
{ "id": "entities", "type": "database", "label": "Entities", "sublabel": "memory_entity", "pos": [40, 340], "size": [120, 50] },
{ "id": "edges", "type": "database", "label": "Edges", "sublabel": "memory_edge", "pos": [200, 340], "size": [120, 50] },
{ "id": "projects", "type": "database", "label": "Projects", "sublabel": "project_id", "pos": [360, 340], "size": [120, 50] },
{ "id": "convos", "type": "database", "label": "Conversations", "sublabel": "owner_id", "pos": [520, 340], "size": [120, 50] }
],
"boundaries": [
{ "kind": "security-group", "label": "Access Control Pipeline", "wraps": ["cap-check", "scope-checker", "decision"] },
{ "kind": "region", "label": "Scope Checkers", "wraps": ["project-scope", "visibility-scope", "owner-scope", "group-scope"] },
{ "kind": "region", "label": "Protected Resources", "wraps": ["entities", "edges", "projects", "convos"] }
],
"connections": [
{ "id": "jwt-to-cap", "from": "jwt", "to": "cap-check", "label": "claims", "variant": "emphasis" },
{ "id": "cap-to-scope", "from": "cap-check", "to": "scope-checker", "variant": "emphasis" },
{ "id": "scope-to-decision", "from": "scope-checker", "to": "decision", "variant": "emphasis" },
{ "id": "scope-to-proj", "from": "scope-checker", "to": "project-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-vis", "from": "scope-checker", "to": "visibility-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-owner", "from": "scope-checker", "to": "owner-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
{ "id": "scope-to-group", "from": "scope-checker", "to": "group-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" }
],
"cards": [
{ "dot": "emerald", "title": "Phase 1: Capability", "items": ["has_capability(memory:read)", "Checks roles[] claim", "Binary allow/deny"] },
{ "dot": "rose", "title": "Phase 2: Scope", "items": ["Project: dev/staging/prod", "Visibility: public/private", "Owner: self-owned only", "Group: poimen-admins/devs"] },
{ "dot": "cyan", "title": "JWT Claims", "items": ["groups: user.groups", "roles: attributes.roles", "permissions: group names"] }
]
}
+54
View File
@@ -0,0 +1,54 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "Service-to-Service (M2M) Authentication",
"viewBox": [1080, 520],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "worker", "type": "backend", "label": "Ingest Worker", "sublabel": "temporal/cron" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
{ "id": "memory", "type": "backend", "label": "Memory Service", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
],
"segments": [
{ "from": 150, "to": 230, "label": "Token Mint" },
{ "from": 240, "to": 380, "label": "API + RBAC" }
],
"messages": [
{ "id": "token-req", "from": "worker", "to": "authentik", "y": 165, "label": "POST /token grant_type=password", "variant": "emphasis", "note": "client_id + username + app_password + scope" },
{ "id": "jwt-resp", "from": "authentik", "to": "worker", "y": 205, "label": "JWT {groups,roles,aud,iss}", "variant": "return", "note": "groups=[poimen-admins] roles=[memory:write]" },
{ "id": "api-call", "from": "worker", "to": "memory", "y": 260, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 295, "label": "GET /jwks (cached)", "variant": "dashed" },
{ "id": "jwks-resp", "from": "jwks", "to": "memory", "y": 330, "label": "RSA pubkey (kid)", "variant": "return" },
{ "id": "verify-ok", "from": "memory", "to": "worker", "y": 365, "label": "202 Accepted", "variant": "return", "note": "sig ✓ iss ✓ aud ✓ has_capability ✓" }
],
"activations": [
{ "participant": "worker", "from": 160, "to": 371, "type": "backend" },
{ "participant": "authentik", "from": 165, "to": 211, "type": "security" },
{ "participant": "memory", "from": 255, "to": 371, "type": "backend" },
{ "participant": "jwks", "from": 290, "to": 336, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "M2M Pattern",
"items": [
"grant_type=password (Authentik aliases client_credentials)",
"App-password token authenticates the service account",
"NOT client_secret — that creates nameless SA"
]
},
{
"dot": "rose",
"title": "JWT Claims",
"items": [
"groups: from user.groups (poimen-admins)",
"roles: from user.attributes.roles",
"iss/aud: per-provider issuer"
]
}
]
}
+61
View File
@@ -0,0 +1,61 @@
{
"schema_version": 1,
"diagram_type": "sequence",
"meta": {
"title": "User Direct Access with JWT",
"viewBox": [1000, 640],
"column_fit": "spread",
"quality_profile": "showcase"
},
"participants": [
{ "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
{ "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
{ "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
{ "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
{ "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
],
"segments": [
{ "from": 150, "to": 370, "label": "OIDC Login" },
{ "from": 380, "to": 540, "label": "API + RBAC" }
],
"messages": [
{ "id": "click-login", "from": "user", "to": "app", "y": 165, "label": "click Login", "variant": "default" },
{ "id": "redirect-auth", "from": "app", "to": "authentik", "y": 200, "label": "redirect /authorize", "variant": "emphasis", "note": "scope=openid groups roles" },
{ "id": "show-login", "from": "authentik", "to": "user", "y": 235, "label": "login form", "variant": "default" },
{ "id": "submit-creds", "from": "user", "to": "authentik", "y": 270, "label": "username + password", "variant": "security" },
{ "id": "auth-code", "from": "authentik", "to": "app", "y": 305, "label": "redirect ?code=xxx", "variant": "return" },
{ "id": "exchange-token", "from": "app", "to": "authentik", "y": 340, "label": "POST /token code=xxx", "variant": "emphasis" },
{ "id": "jwt-resp", "from": "authentik", "to": "app", "y": 375, "label": "JWT {groups,roles,sub}", "variant": "return" },
{ "id": "query-api", "from": "app", "to": "memory", "y": 420, "label": "GET /memory/query", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
{ "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 455, "label": "GET /jwks (cached)", "variant": "dashed" },
{ "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 490, "label": "RSA pubkey", "variant": "return" },
{ "id": "query-resp", "from": "memory", "to": "app", "y": 525, "label": "200 JSON results", "variant": "return", "note": "has_capability(memory:read) ✓" }
],
"activations": [
{ "participant": "user", "from": 160, "to": 276, "type": "external" },
{ "participant": "app", "from": 165, "to": 531, "type": "frontend" },
{ "participant": "authentik", "from": 195, "to": 381, "type": "security" },
{ "participant": "memory", "from": 415, "to": 531, "type": "backend" },
{ "participant": "jwks", "from": 450, "to": 496, "type": "security" }
],
"cards": [
{
"dot": "emerald",
"title": "Authorization Code Flow",
"items": [
"User authenticates directly with Authentik",
"App exchanges code for JWT",
"Scopes: openid groups roles permissions"
]
},
{
"dot": "rose",
"title": "JWT Contents",
"items": [
"iss: .../application/o/poimen/",
"aud: poimen (provider client_id)",
"groups: user's group memberships"
]
}
]
}
+58
View File
@@ -0,0 +1,58 @@
# cloudflared ingress configuration.
#
# This list is the tunnel's routing table: for each incoming hostname the
# connector dials the matching origin. The final rule is a catch-all, so a
# hostname absent from this list gets 404 from the connector — DNS alone does
# not route a request into a service.
#
# Management model: this file makes the tunnel LOCALLY managed. cloudflared is
# given a credentials file instead of a token, so Cloudflare stops pushing the
# remotely-managed ingess list and this ConfigMap wins. Revert the Deployment to
# `run --token` to return to the dashboard-managed list.
#
# Origin services self-reference the public hostname. That resolves because
# CoreDNS rewrites each of these names to ingress-nginx
# (terraform/files/coredns/Corefile), so the connector reaches the in-cluster
# ingress instead of looping back out through Cloudflare.
apiVersion: v1
kind: ConfigMap
metadata:
name: cloudflared-config
namespace: cloudflared
data:
config.yml: |
tunnel: 32d0bf93-e1bd-45b8-93a9-71ff9f83de80
credentials-file: /etc/cloudflared/creds/credentials.json
# WARP-to-Tunnel routing. The remotely-managed config had this enabled;
# dropping the key would break WARP access to private addresses. cloudflared
# 2026.x has no `enabled` field — the key's presence is what enables it, and
# `enabled: true` here makes the binary warn about an unused key.
warp-routing: {}
ingress:
- hostname: paperless.riotpiao.com
service: https://paperless.riotpiao.com
- hostname: authentik.riotpiao.com
service: https://authentik.riotpiao.com
- hostname: img.riotpiao.com
service: https://img.riotpiao.com
- hostname: riotpiao.com
service: https://riotpiao.com
- hostname: grafana.riotpiao.com
service: https://grafana.riotpiao.com
- hostname: gotify.riotpiao.com
service: https://gotify.riotpiao.com
- hostname: igotify.riotpiao.com
service: https://igotify.riotpiao.com
- hostname: poimen.riotpiao.com
service: https://poimen.riotpiao.com
- hostname: comfyui.riotpiao.com
service: https://comfyui.riotpiao.com
- hostname: forgejo.riotpiao.com
service: https://forgejo.riotpiao.com
# ZITADEL. Without this rule the connector answers 404 for the hostname
# even though the DNS CNAME points at the tunnel.
- hostname: zitadel.riotpiao.com
service: https://zitadel.riotpiao.com
- service: http_status:404
+49 -7
View File
@@ -13,21 +13,63 @@ spec:
labels: labels:
app: cloudflared app: cloudflared
spec: spec:
# Ingress-nginx forces HTTPS (force-ssl-redirect), so cloudflared must
# dial https://<hostname> and trust homelab-ca or the TLS handshake fails
# with "certificate signed by unknown authority". Same pattern as
# authentik-values.yaml: merge the system bundle with homelab-ca.crt.
initContainers:
- name: merge-ca-certs
image: debian:bookworm
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
command:
- sh
- -c
- (cat /etc/ssl/certs/ca-certificates.crt 2>/dev/null; cat /homelab-ca/homelab-ca.crt) > /merged/ca-bundle.crt
volumeMounts:
- name: homelab-ca
mountPath: /homelab-ca
readOnly: true
- name: merged-ca
mountPath: /merged
volumes:
- name: config
configMap:
name: cloudflared-config
- name: credentials
secret:
secretName: cloudflared-credentials
- name: homelab-ca
configMap:
name: homelab-ca
- name: merged-ca
emptyDir: {}
containers: containers:
- name: cloudflared - name: cloudflared
image: cloudflare/cloudflared:latest image: cloudflare/cloudflared:latest
args: args:
- tunnel - tunnel
- --no-autoupdate - --no-autoupdate
- --protocol
- http2
- --config
- /etc/cloudflared/config.yml
- run - run
- --token
- $(TUNNEL_TOKEN)
env: env:
- name: TUNNEL_TOKEN - name: SSL_CERT_FILE
valueFrom: value: /merged/ca-bundle.crt
secretKeyRef: volumeMounts:
name: cloudflared-token - name: config
key: token mountPath: /etc/cloudflared/config.yml
subPath: config.yml
readOnly: true
- name: credentials
mountPath: /etc/cloudflared/creds
readOnly: true
- name: merged-ca
mountPath: /merged
readOnly: true
resources: resources:
requests: requests:
cpu: 10m cpu: 10m
+1
View File
@@ -3,3 +3,4 @@ kind: Kustomization
namespace: cloudflared namespace: cloudflared
resources: resources:
- deployment.yaml - deployment.yaml
- configmap.yaml
+3 -5
View File
@@ -4,7 +4,9 @@ metadata:
name: comfyui name: comfyui
namespace: comfyui namespace: comfyui
annotations: annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod # No cert-manager annotation and no tls: block. This host is covered by
# the *.riotpiao.com wildcard, which nginx serves as its
# default-ssl-certificate (ingress-nginx/riotpiao-com-tls).
nginx.ingress.kubernetes.io/proxy-read-timeout: "600" nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-body-size: "0"
@@ -16,10 +18,6 @@ metadata:
proxy_set_header Connection "upgrade"; proxy_set_header Connection "upgrade";
spec: spec:
ingressClassName: nginx ingressClassName: nginx
tls:
- secretName: comfyui-tls
hosts:
- comfyui.riotpiao.com
rules: rules:
- host: comfyui.riotpiao.com - host: comfyui.riotpiao.com
http: http:
@@ -12,6 +12,19 @@ rules:
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues"] resources: ["queues"]
verbs: ["get", "list", "watch", "update", "patch"] verbs: ["get", "list", "watch", "update", "patch"]
# The operator watches TemporalWorker objects and creates Deployments for
# them. Without these two rules the controller logs
# "cannot list resource temporalworkers/deployments ... at the cluster scope"
# and never reconciles.
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["kmsvc.io"]
resources: ["temporalworkers/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["kmsvc.io"] - apiGroups: ["kmsvc.io"]
resources: ["queues/status"] resources: ["queues/status"]
verbs: ["get", "update", "patch"] verbs: ["get", "update", "patch"]
+2 -1
View File
@@ -48,7 +48,8 @@ spec:
mountPath: /backup mountPath: /backup
containers: containers:
- name: mc-mirror - name: mc-mirror
image: quay.io/minio/mc:latest # bitnamilegacy/minio-client:2025.7.21-debian-12-r3 replaces the dead quay.io/minio/mc image
image: docker.io/bitnamilegacy/minio-client:2025.7.21-debian-12-r3@sha256:73bd39f7899a0cef12b8dd5df13aa93a3ed1aaa44236542442e9ac76819ac158
env: env:
- name: ACCESS_KEY - name: ACCESS_KEY
valueFrom: valueFrom:
-115
View File
@@ -1,115 +0,0 @@
# macOS VM (Docker-OSX) hosting the BlueBubbles server.
#
# ── Why a VM and not a container ────────────────────────────────────────────
# Containers share the host kernel. macOS binaries are Mach-O and need XNU plus
# Cocoa/IOKit, which a Linux kernel cannot provide, so no macOS container exists
# or can exist. Docker-OSX is QEMU running a macOS guest, packaged in a
# container — a VM in a box, not a macOS container.
#
# ── Why this works on worker-2 ──────────────────────────────────────────────
# Verified on the existing hardware: amd64, `vmx` (Intel VT-x) present, and
# /dev/kvm exists on Talos nodes (KVM is compiled into Talos' kernel, not a
# module). Bare metal, so no nested virtualisation needed.
#
# ── Read this before relying on it ──────────────────────────────────────────
# 1. Setup is INTERACTIVE. First boot runs the macOS installer: connect over
# VNC (:5999), erase the disk in Disk Utility, install, create a user, sign
# into iMessage, THEN install BlueBubbles inside the guest. This manifest
# only provides the machine; it does not provision macOS.
# 2. iMessage activation on non-Apple hardware is a coin flip. BlueBubbles'
# own guidance: "test sending an iMessage to yourself. If it does not
# succeed, it's likely best to restart from the beginning."
# 3. Apple's macOS licence permits virtualisation only on Apple hardware. This
# is a Hackintosh. Use a throwaway Apple ID, not a primary one.
# 4. BlueBubbles labels this path "not for beginners", "no guarantees or
# warranty".
#
# Private API (reactions, typing indicators, edit/unsend) needs SIP disabled
# inside the guest and is NOT required for plain send/receive. Skip it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: macos-bluebubbles
labels:
app.kubernetes.io/name: macos-bluebubbles
app.kubernetes.io/part-of: sms
spec:
replicas: 1
# Recreate: the qcow2 disk is RWO and a second pod must never attach it
# concurrently — two QEMU processes on one image corrupts it.
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: macos-bluebubbles
template:
metadata:
labels:
app.kubernetes.io/name: macos-bluebubbles
app.kubernetes.io/part-of: sms
spec:
# Dedicated node. The taint keeps everything else off worker-2; this
# toleration is what lets the VM on. Both halves are required.
nodeSelector:
workload: imessage
tolerations:
- key: workload
operator: Equal
value: imessage
effect: NoSchedule
containers:
- name: macos
image: sickcodes/docker-osx:latest@sha256:3a3c82c79bc4e73531f819ccdfa4053b3084efd7c1f645678dbf8b4b3a24369c
# QEMU needs /dev/kvm; Talos enforces `baseline` cluster-wide, so this
# only schedules because the sms namespace is labelled privileged.
securityContext:
privileged: true
env:
# Generates a unique serial / board-serial / UUID / MAC and persists
# them to bootdisk.qcow2. This synthetic identity is what iMessage
# activates against — it must stay stable across restarts, which is
# why the PVC matters.
- name: GENERATE_UNIQUE
value: "true"
# Identity is only plausible if it matches a real product line.
- name: DEVICE_MODEL
value: "iMacPro1,1"
- name: RAM
value: "12"
- name: CORES
value: "6"
- name: EXTRA
# Expose the BlueBubbles server port from the guest to the pod.
# Guest :1234 (BlueBubbles default) -> pod :1234.
value: "-device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp::1234-:1234"
ports:
- name: vnc
containerPort: 5999
protocol: TCP
- name: bluebubbles
containerPort: 1234
protocol: TCP
resources:
requests:
cpu: "6"
memory: 14Gi
limits:
cpu: "12"
memory: 20Gi
volumeMounts:
- name: macos-disk
mountPath: /home/arch/OSX-KVM/disk
- name: kvm
mountPath: /dev/kvm
# No readiness probe on purpose. The guest takes many minutes to boot,
# and until macOS + BlueBubbles are installed BY HAND there is nothing
# listening on 1234. A probe here would crash-loop the pod through the
# entire interactive install.
volumes:
- name: macos-disk
persistentVolumeClaim:
claimName: macos-disk
- name: kvm
hostPath:
path: /dev/kvm
type: CharDevice
-10
View File
@@ -1,10 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sms
resources:
- namespace.yaml
- storageclass.yaml
- pvc-macos.yaml
- deployment-macos.yaml
- service.yaml
- networkpolicy.yaml
-19
View File
@@ -1,19 +0,0 @@
# iMessage delivery for the cluster.
#
# BlueBubbles' server is a macOS Electron app paired with an Objective-C helper
# that hooks Messages.app private APIs — it cannot be containerised on Linux,
# because containers share the host kernel and macOS needs XNU + Cocoa. The only
# way to run it on Talos is a full macOS VM under QEMU/KVM (Docker-OSX), which
# needs a privileged pod with /dev/kvm.
#
# Hence privileged PodSecurity: the cluster default from the Talos controlplane
# is `enforce: baseline`, which forbids privileged containers and host devices.
# Scope is limited to this namespace.
apiVersion: v1
kind: Namespace
metadata:
name: sms
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
-22
View File
@@ -1,22 +0,0 @@
# Default-deny. This namespace runs a privileged QEMU VM signed into an Apple
# ID and exposes an unauthenticated VNC console; nothing should reach it except
# opted-in clients.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sms-default-deny
spec:
podSelector:
matchLabels:
app.kubernetes.io/part-of: sms
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
sms-client: "true"
ports:
- protocol: TCP
port: 1234
-25
View File
@@ -1,25 +0,0 @@
# Persistent macOS disk image + generated hardware identity (bootdisk.qcow2).
#
# This volume is NOT disposable: it holds the VM's serial number, board serial,
# UUID and MAC, which together form the identity iMessage was activated against.
# Losing it means re-running activation, which is the least reliable step of the
# whole setup.
#
# Docker-OSX documents 128GB minimum for the guest image; 200Gi leaves room for
# the installer, the base system, and qcow2 growth.
#
# ⚠️ Single replica (see storageclass.yaml — capacity and IO both rule out 3).
# Losing worker-2's disk therefore means losing the activated identity and
# redoing iMessage activation. Once the guest is installed and activated, take
# a Longhorn snapshot/backup of this volume; that is the only redundancy here.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: macos-disk
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn-imessage-local
resources:
requests:
storage: 200Gi
-31
View File
@@ -1,31 +0,0 @@
# VNC is how you drive the interactive macOS install. Deliberately ClusterIP —
# it is an unauthenticated console onto a machine holding a live Apple ID
# session. Reach it with `kubectl port-forward`, never an Ingress.
apiVersion: v1
kind: Service
metadata:
name: macos-vnc
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: macos-bluebubbles
ports:
- name: vnc
port: 5999
targetPort: vnc
---
# The BlueBubbles REST API, once installed inside the guest. This is the stable
# name cluster services use, so callers never depend on the pod IP or on whether
# the backend is this VM or a real Mac mini later.
apiVersion: v1
kind: Service
metadata:
name: bluebubbles
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: macos-bluebubbles
ports:
- name: http
port: 1234
targetPort: bluebubbles
-32
View File
@@ -1,32 +0,0 @@
# Dedicated StorageClass for the macOS VM disk.
#
# The default `longhorn` class does not work here, for two independent reasons:
#
# 1. Replica count. Default is 3, and Longhorn schedules against
# storageMaximum - storageScheduled with over-provisioning at 100%. Free
# space is cp-1 146Gi / cp-2 8Gi / cp-3 146Gi / worker-1 292Gi, so a 200Gi
# volume has only one node that can hold even a single replica — a 3-replica
# volume fails outright with ReplicaSchedulingFailure.
# 2. Binding mode. `Immediate` provisions the volume the moment the PVC is
# created, before any pod is scheduled. Combined with strict-local that
# pins the data to an arbitrary node, not the one the VM runs on.
#
# So: one replica, kept local to the VM, bound only once the pod has a node.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-imessage-local
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
# The pod is pinned to worker-2 by nodeSelector; wait for it to be scheduled so
# the replica is placed on that node and not somewhere else.
volumeBindingMode: WaitForFirstConsumer
parameters:
# A qcow2 backing a live VM is latency-sensitive and rewritten constantly.
# Serving it over the network from another node's disk would be the single
# worst thing for guest responsiveness, so force it local.
numberOfReplicas: "1"
dataLocality: "strict-local"
staleReplicaTimeout: "30"
fsType: "ext4"
@@ -249,3 +249,26 @@ spec:
selfHeal: true selfHeal: true
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: local-path-provisioner
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/local-path
destination:
server: https://kubernetes.default.svc
namespace: local-path-storage
# No CreateNamespace: the Namespace object (with its privileged PSA label,
# which the helper Pods need for their hostPath) is part of the manifest.
syncPolicy:
automated:
prune: true
selfHeal: true
+49
View File
@@ -57,6 +57,55 @@ spec:
prune: true prune: true
selfHeal: true selfHeal: true
--- ---
# ZITADEL — the Authentik replacement. Coexistence, not cutover: Authentik keeps
# serving in the same `iam` namespace until the last consumer moves, then this
# becomes the only IdP. Migration plan (outside git):
# ~/workplace/homelab-doc/ZITADEL_MIGRATION.html
#
# Wave 3 ordering: sops-secrets (wave 0) creates the `zitadel-masterkey` Secret,
# databases (wave 2) creates the `zitadel-db` CNPG cluster and its
# `zitadel-db-app` DSN secret. Both must exist before the init job starts.
#
# automated sync is deliberately absent for the first sync — same reason as
# forgejo. The chart ships `pre-install,pre-upgrade` Helm hooks (zitadel-init,
# zitadel-setup), which ArgoCD maps to PreSync and re-runs on every sync, manual
# or automated. A human reads the first sync's hook output before any repeat can
# run. The init job runs `init zitadel` only (CNPG owns the database and the app
# user), and the setup job creates the first org and the `iam-admin` machine key
# that Terraform consumes. Read the SECOND-SYNC HAZARD block at the top of
# k8s/infra/iam/zitadel-values.yaml before enabling automated sync.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: zitadel
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: homelab
revisionHistoryLimit: 3
sources:
- repoURL: https://charts.zitadel.com
chart: zitadel
# Pinned: the chart trails the upstream app release, and the verified
# masterkey/passwap facts were read at v4.19.x. A floating tag would
# change the app version without a diff anyone reviews.
targetRevision: 10.4.0
helm:
valueFiles:
- $values/k8s/infra/iam/zitadel-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: iam
syncPolicy:
syncOptions:
- CreateNamespace=true
---
# Raw IAM manifests: key-rotation cronjob + authentik migration job. # Raw IAM manifests: key-rotation cronjob + authentik migration job.
apiVersion: argoproj.io/v1alpha1 apiVersion: argoproj.io/v1alpha1
kind: Application kind: Application
-31
View File
@@ -110,37 +110,6 @@ spec:
syncOptions: syncOptions:
- CreateNamespace=true - CreateNamespace=true
--- ---
# iMessage/SMS delivery. Raw manifests: a privileged macOS VM (Docker-OSX)
# running the BlueBubbles server, plus its dedicated local StorageClass.
#
# Pinned to worker-2 via nodeSelector `workload: imessage` + a matching
# toleration for that node's taint. Until worker-2 is provisioned this app
# syncs everything except the pod, which stays Pending — that is expected.
#
# No CreateNamespace: namespace.yaml carries `pod-security: privileged`, which
# the VM needs (/dev/kvm, privileged), and an ArgoCD-created namespace would
# not have those labels.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sms
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/apps/sms
destination:
server: https://kubernetes.default.svc
namespace: sms
syncPolicy:
automated:
prune: true
selfHeal: true
---
# Document management. Raw manifests (no Helm): postgres is the dedicated # Document management. Raw manifests (no Helm): postgres is the dedicated
# paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is # paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
# in-cluster only (no PVC), media lives on the cp-3 USB HDD (see # in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
+20
View File
@@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: talos-repair
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "6"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/talos-repair
destination:
server: https://kubernetes.default.svc
namespace: cicd
syncPolicy:
automated:
prune: true
selfHeal: true
+20
View File
@@ -0,0 +1,20 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: oidc-bootstrap
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
targetRevision: main
path: k8s/infra/oidc-bootstrap
destination:
server: https://kubernetes.default.svc
namespace: kube-system
syncPolicy:
automated:
prune: true
selfHeal: true
+1
View File
@@ -35,6 +35,7 @@ spec:
- https://grafana.github.io/helm-charts - https://grafana.github.io/helm-charts
- https://helm.releases.hashicorp.com - https://helm.releases.hashicorp.com
- https://charts.goauthentik.io - https://charts.goauthentik.io
- https://charts.zitadel.com
- https://strimzi.io/charts/ - https://strimzi.io/charts/
- https://charts.bitnami.com/bitnami - https://charts.bitnami.com/bitnami
- https://homarr-labs.github.io/charts - https://homarr-labs.github.io/charts
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: cloudflared-credentials
namespace: cloudflared
type: Opaque
stringData:
credentials.json: ENC[AES256_GCM,data:LnJVGuxJAOwDYbLgX00OHICDDldZgwgvGafOIwKoJKnjg3FjFMQYThNCOL/H+6tctuKixJNcAckjg70tmCKWxNq9I35XX/YSqob6Qv4fywv32hsNIAe44BU035MGZ5zmc6bZZ8wAgTJZw6Th4TzY7taFBCRdj2fW4T+qVfq7strn68G4ufVMK6UXlOkEDLjR2Vxj32lqSLdbdMma96XB+265hwL6IVfaugG662mcGhPwgHk=,iv:NFmyHN5m2mJUBgn1qgmJZ2YgkFxBZyp7+gU1Ng6B/7M=,tag:/66JtHK5Yu7pRxB0b0eifQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsaDVycUE4RFpoRjdpTlpX
dGpzN3hwRFQ2cmtXVXVzWnZJOXJEZXZRWEhNCkJwbm9QV0lGcGtNRTJqbStrc3l1
Q2M4bE1WbzJpTXZQWjJyTGpFS0JYcVUKLS0tIDBvU2U5aEtVTTZUaE85eEkyYitG
bGtSVkhKNnRuNlZWMWh6Z1M4VXM1WU0KdusvtM85V0xrO+uJFUY+UH0tuwdF0GiW
79J4Rlxk8ne25psKsvwjLYLxZ53xjFHCC4FCHTNkYwTKCDEKjw46Lg==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-26T02:45:04Z"
mac: ENC[AES256_GCM,data:KF7KwXCnNLAg1YI3u0lLI/ho4sITHAq09ac2WO4TK3TCthkuivb89YjhuuJKs0NFhjn6aRLqmZavUqcXH2HujiryqbLNVexOShQM0wSzgCzmWdbi0UzpgJ4KcQA3PufDtsQz4HHLXx396EbZrM3Ie3WSu/PvRm5iWOxzTGCGc9Q=,iv:Eo920oXXD1FLZVJXywZ6C+LRL+qZjmnZXv0+Oa/GMb8=,tag:6LuPVSet4WjvrvGUAxaJlw==,type:str]
version: 3.13.2
@@ -0,0 +1,27 @@
apiVersion: v1
kind: Secret
metadata:
name: etcd-defrag-creds
namespace: kube-system
labels:
app.kubernetes.io/name: etcd-defrag
type: Opaque
stringData:
ca.crt: ENC[AES256_GCM,data:vso/Ouz5QRoRF2CFj9TsGYhoGkX8oGJIeiFCoS92AAQbyyGPTiZ/fPNFM03p+TpeKDb+Z+Fyo/HXuDeT5Fee5kjPnUHXf2hiUBmJLyHnPMp5i57dRzTC8X4VaNeV/1MmP2WpkWwieWOGBxJVIIrgeQhQexxmIzSlQ5Io9dVOH8F6iyakwdf7qfLFGdoer3rfrXadSOJtKRpLneFJDL2OmR1042LZ8KtNuRRXI/qrdpgr2bnyxFuCRc+tk5QY73mpwwWfKGn6FW/rVdjiEftkC+AvTUItx7RkYMrz0PlTYPUOLustEF2gIgWheYZhE5ebZDurVSG//sudjASliYJItwgSk3G5mZihOSDA7Jcus+U8p1iHQ9CQsey3CSAMmuSGRFmq42kI8135kjtBYrzVswHjX1/51qPMyA2TIeWlRJ7AJlHihG317gT1vPeM1ULl3PkuCxaBLCtjotZBzKLJ/6zXr1ig77Baz71YOVB0D0MgFvzIdao+mdxS8WLOZdiXOCwbI5Y2eo7wJHG4BnpKEdx1rjPkv7DI2NNEoXA7e5FvJWinyAxOy6YGH4Fv6dTlEVhI3Nl/1X+CeafL6Hk8YoqA04Hm+/dJCZi7SsF0r6zfPNNsZLivM8DN18h8KYM3eVY3CwtBldE7atz+oprQ6K/yGUbblHRQwuG3sH3AHrCKLPiUrMIi0jWc0IpdPZRZFL7SnMULfV05QgcvGySOJCuQU+/OYzrRs7umOjwpqvi9PyJYVXBliiYfJutvmfpFAhQj,iv:KnW9LYNzxZ6r3Q6LTeOrgSLLPgiJlibVp6YmTEKBkjE=,tag:KTL1nOWdNWDA4dLVaA4//g==,type:str]
client.crt: ENC[AES256_GCM,data:IEmZe505SwkW3U5vmRtFxR6PpuSFGWEHflUZwyhSODzYG1zLM8olVWDtH+yUVI54kLXxTtcpMkKrTBjpAxHG8zUTLPdTCVzPMq6U20jH4PYFMzoldnSgpRl/eiADUmE4I32CBbPdM5nkPvjVEYMmRafEGyJctjG6Shhh/sUO1yE9A+h1R7o3UCRPttWTozSKN4dPi75v1TSAOLyXqvQYl5Ea9iKW47ea7Q+XTcoKfMPXsTJp44kiZnVW0VT/7xNydR7AVWJJzYltY5380j0Ho+7PfNd+QqmdodkaZPS4b0PaoinMiZ40JnChGVhdS91j3IkUlU1g/+G2iVnPR9KB07xqRubqiASxIvpSaSCeYl3rR7+dXYX18VpWcWnJ6XoREoqxJiqmMQWYSROYDTd4G/zs4Usuh5r0wIodmCjud+g89mmXFtH0PbsMCVvZDUswjVfeT2PUBm036TXlxF/1aAy2GDqycbPgJgDLyw9tdiQUZe2iA+Z/Wx80qgQjmmqCaw77cDergLFj9KnByl3Tinwfm+IWFeQiDW+fwSzz+/vrpl55BkSVkVhw7r2Jd6zyg4VtLVXbxYhzdNz695N+fjkZHuPE5tCwr4WCzAg3vSS9hA7IHo8k93WO8s6+KlyjuwTdjPIYZghebO4ktwdqvhvuqxv8Nzwpc5J7Z2H2IChjXg==,iv:l8BwxxEfHfYrm+mM5UE6Yj42DeeU9y23jjIE1KjIiME=,tag:HwRT6/myIsxc84BYDj3K6w==,type:str]
client.key: ENC[AES256_GCM,data:QD/Glf3lFslBEdEoJ3CNrYzlw2HLCvGSeVuJLtxnFne2aJOuw0sk/e0h/kSAxvG4dZ+LIWJLLbqTjKHOAhkvS/ZOA7GSea4ubNkvTkw8+JBRuYwb2D2iB8kZhlER/CpiGkYJukjkgGP1CzHdvBY1M88pW9w6rH74MD3jY47qExIaNduQj5vkUQZ8ajqkd79l0nk98ZTawfPG4AFcEtAIKAPIuubT2GZ3qqNLjOI7agbtRv8xI44hhb2wj5J9xdl1GrbOWwG5EOo+hLNrcAAOCcWBpNTUCKBy7mrk2ID8jX3HsFWN76DDlQzkhcphMhlM7g==,iv:49HGJAEgllEkB3YSiZLmbWs/vlbFbY52kiKesdGVC/Y=,tag:g8YZ9at3V2l6RFRfvO2lAQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z0h0R2dhbDdRcFJydE9o
R0hWeCtwVzluZWZlR0VtOXcxdW9FbklOQXlzCmo3eXNyNlFjejNETkJiVVRVWFhr
a2pORnhIQTI0UUk2N3dWMDZJcEhYU1UKLS0tIEduaHlmMHQza1A0TGVZb0JrbTYz
MWJwQjcrdkkzQXhOTmhycERsM0FtNVEKG+gBRcvUsyfRzuW999EQNzyDh2/5fY6p
8ab2xvTOJPiMUBTBuQZsA+iDImytUO/02ROQsbEZkALijp1nqrsV/g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-07T05:37:26Z"
mac: ENC[AES256_GCM,data:vu6ltV0jYQNGV4eqIcqAp9p2TJ8ULBX5JBELUgm/nA9GX+I6Pv3fyjpcHRSWY0HbLvCwgXd22CG4IYcj4HJFtQGZR7OcQA5WM5plZH3RCPdvPNix/798vkETfpJxorQjlHoq4nlr3dEmUCbzdtE07qV2W1M6GJr4C9xKvB947bQ=,iv:aA9mjZCFAoLEhGqkl6NfyBZ3x3vWGfLYYAEe8jXuA4U=,tag:Drdiji33ckDd8ildpvSEXw==,type:str]
version: 3.13.2
@@ -1,23 +0,0 @@
apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
metadata:
name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
stringData:
REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
lastmodified: "2026-08-23T23:14:53Z"
mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
@@ -0,0 +1,24 @@
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: pi-code-agent-api-keys
namespace: poimen-sandbox
stringData:
ANTHROPIC_API_KEY: ENC[AES256_GCM,data:TBcno/dzj3qykHjeX96UKZekuA/WMQT2E7a1ziPyYo5J6lWLPydgli84KeJZhH62/8CnnFtFOhVIe4FWCGPGjvqKH6M8WbiQHaP9y7FN0qQUpFHKNVAQ2R+s0wInhw0eUmANvd7bRx8aIZ2X,iv:oex19yVtWUevoJ0A3mkMLCfWu85o1gPSNp43x1iV0LE=,tag:BW3zQnwvagPceeNird9CZg==,type:str]
DEEPSEEK_API_KEY: ENC[AES256_GCM,data:vq/5gOsMVjlVMd0oXiBtHFHFEr+IF2YR2iP2OBFWx3ZFNoQ=,iv:QhLU/BRIJb/7QKDCRyw+HWPphZifVEhIOLywYDgOb20=,tag:BZdtXw+WsuK+P4E0caTa0Q==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqNm1LNE5nWktnd3c3STg3
UlVWbWJxWElxUHMwa0JVMk81Z0c4d0JaQm1vCnpLQWJWV3hxOU00NzJIaG51Mmlr
ZWpNTFJWcmdtV0w4YndQeHpvSUx2MTAKLS0tIDVnNkNuVWhPUFZ5V2hFOEt1eitG
blVqRy8yNnYzSU1FNzZGNkxrRElvWVEKL3J77INje75+CvKXaJHC3CvONgHv7d0a
a10MITvfreFr6M4AN3Z/mno49SDMp15Q4oQRagHHxSTS1K54mWHo8g==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-06T01:19:49Z"
mac: ENC[AES256_GCM,data:GpYl8IRefms36GvO3RW41c0US7SOgoOhd4rumxsCdEf5ME9kJ0eOxPx8klMgnlYrkBnwHmad8jj7C+HcA5Eim7mdUaYOZZWPFohvC6N4GLHX85aH78P09CelqG5tdm/Gm+1QfxYu0vANYGCAVjLK7YbBbZjbZmbvVfQvAbs3mOY=,iv:y3lglGu8FhTHQpO1poSIKgGYJFu5J9yjIaOjulda6fY=,tag:5Sgh4TbtNFMw7xOJjYRn0g==,type:str]
version: 3.13.2
+9 -1
View File
@@ -11,7 +11,7 @@ files:
- agent-pod-ssh-key.enc.yaml - agent-pod-ssh-key.enc.yaml
- authentik-secrets.enc.yaml - authentik-secrets.enc.yaml
- cloudflare-secrets.enc.yaml - cloudflare-secrets.enc.yaml
- forgejo-registry-pat.enc.yaml - cloudflared-credentials.enc.yaml
- forgejo-registry-pat-api.enc.yaml - forgejo-registry-pat-api.enc.yaml
- forgejo-registry-pull.enc.yaml - forgejo-registry-pull.enc.yaml
- forgejo-runner-token.enc.yaml - forgejo-runner-token.enc.yaml
@@ -36,3 +36,11 @@ files:
- memory-agent-oidc.enc.yaml - memory-agent-oidc.enc.yaml
- oauth2-credentials.enc.yaml - oauth2-credentials.enc.yaml
- forgejo-registry-token.enc.yaml - forgejo-registry-token.enc.yaml
- pi-sandbox-secrets.enc.yaml
- etcd-defrag-creds.enc.yaml
- zitadel-masterkey.enc.yaml
# Login UI RSA keypair, adopted from the chart-generated Secret so ArgoCD
# owns it. Without this, the chart re-generates the keypair on every render,
# which changes checksum/secret-login-service-key and leaves both Zitadel
# Deployments permanently OutOfSync.
- zitadel-login-service-key.enc.yaml
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: talos-repair-config
namespace: cicd
type: Opaque
stringData:
config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-07T04:27:33Z"
mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
version: 3.13.2
@@ -0,0 +1,24 @@
apiVersion: v1
kind: Secret
metadata:
name: zitadel-login-service-key
namespace: iam
type: kubernetes.io/tls
data:
tls.crt: ENC[AES256_GCM,data:Ad4omL7pD4QhixW6wSVsVor5FtHbWyOHeYbxOQsEBAaSuvb7JIkvnKqo+Xy6zKuMmRViUIbqhlLQDXCaseAVGgpu3PdtShwJ7LRXJmr0XtQV1M6YtRbRexN6UpFA3XdD+JfifMhtvd29rM3WzLPX69uk4k7voYN5m0pNXSN23jVeIeqm+xB1CpxDW/guZ2hA5+b5XBzVOkMsSOSdWFWH2lK7nInv6ZatCa0/fuIKmB5nZpSV9tHpCTiTnBuNc0rW1X4HOWa8zMiSbjQAqtDitm7lrVHJyImXwBjmuS0wM35VwrwxVUUEjS8kgqnyb02JvNRdNjuwOmIcyKjrdgO1B1ft0CrAbRNkk2dYR8Y7SNhR4F4tYYF5L2szQgj+5tcYrnRGEI2QkXbcmTa0Y9V99+D2bKjt5t59Pi/1r8/jt0CNy0GeeZDvsN5scy0/A4WqtJ2sZDyTnTPwtUpAAwVC2WSqsD11gkzhyRJAgmAmEgfPIL7kTZsZBoSQeuuCJ82Ghqx9kLTpui6kLULxxUpP9oefsKw9PjTwExRbNNcPV5p7PPi9dN00TNe9iYuFqoZFP/lp5SMjDqbsV4Ie1ffcfyZsIHMuOv5Ths7n8ExYdR0zIn+aQePXASt+dE6njhFiblxyBZsn0xON3EIUaE9e3WsdcQKjlPOaAW/DvU6PwBqCeaB2nYzM2NFk6W3lN8BahTE9k7xXrILHIrXuIXezotCK6jclb5yZmdm/fTCX1iejGVDktvvcmX6bbv3YXTRZmsviRhLFHA8oClU0alTsY5Q3nn2Xl0UfXU+rW+10AIpY5WFMzP6A8hXvHx0ISC1S4AIqjmlsDmq2yEZKpbiL+Bs52ymW7t2rQ+KGhWH/bcC3VrJwUWnVYP9b80xyHZdKqubwLc5UcDzCs/Lan23VpLxSJkqb8vZP7SHQ1nc+rlwrqFDkzQ+yMNLZmutqbaAmMyXbrR51Rz3VqofraPXjS+lmLAuEcuTEFbrImsct5Zb57MtKvhHi6CUWt/Vj/NOUJlwy09zPy4JAMMJ81ooyBj6CBHY2f1vfR/UUDN5VZdfg4dUWXS0CQdOb3hx19COr4lR4RcpA46qJEjum+4icm5ckziYT5Nm7NB+ydXJ+Eys2Ba0dGRM6CIeQVhghhCDOZ6uoQ2A6DNql6wjrfUml8UwuJNO0nXVqNS28pLwddsOjlbsysJAUEma6EHXb2Vjb90JZErVswGuhQ2+UdhtpuIkgBYX/MMCZV942C+K41A9478BaCiWuC9KbnAcx81ceR+T3syARctZvQaYaiH7WfF8qyo0VOGzZRYjFbqeyHefiOmPmWTS/XbIti3evyhmxHwjUcidK2njrPB0O/3w6R69UA1p7ph9EgFsj18gPYnJthikgX8vcfVaVnPIAwCqY5DJJmIFTNXVkNDBX+2orrW2qj6PEH+go+uc+aUmu646niqaMAcrUl7gGNGd2OWFHdIRKdtogOvoVpQMDB4eWHPRsL29EVt3rnknHfpq7cIjQXuk9XtlB8zjllYdrh50Q8ZNBOvzWgCiMvAyanfIA+kW9KHQlmb1qJGYp7it2Lrg1K7sDEekrlbuoNsf+2rl8YRQz8pVBuCSzG9DPp4+Rj+ddiqX/Rrs3ufWDKFV9zJoN1EpJ8+z6n8rsQJBe/lSFLW1mLVkaPSNNdIQfdJ6MbsHpMlqqFYP8dOKckxv82uOkHoCLHrgaXiiMd1dBsX1fBiv2NvzDiuLyN81d7R2wNAQbkG9b+/+C6wVr6kDg0n6seseoCyeSI++PJIkCrkTXzd7rLs8Of9CHZQEB15wjemlmYGlJnCiFWPFa6h0jJPatPrfTYv+GpqzWV8u1Uh3aNjd+g1wVQok/ojNOZJahUSI4B67HkJI+I7WOF9C+rRCwNutqshIxYzW7CpyVey5P6Kyv+/cucDEgb0ggH032MBtzTxazpJcE4Y6fTrOiXHp9j2KjzyUt2YeCCL8UMwbwJvW9p1I5gx3ArsOuyHm1kA==,iv:PLsjmrTwjM7IUS7ZBSyVvJfAJzCSUtU+UAvHbSwoaVk=,tag:pUQwbHoZgfHW4CpPKoULAw==,type:str]
tls.key: ENC[AES256_GCM,data:rO1NVJZP9TP96C48it4+2lGyRutwqdlX6qiM9R/+0hgYc5ceeeopaCUG/i/LOgltxRQK8qkyed/W3zU2VI6alBpt75XkT3o9ZRy3uxe8mB3ixS7+xQnMc0ci8Mf2bCtGzqwUQj8XPRP4K7Mxw8r4XO5rQLnSmNme0+BxS+XiE6hqifS16UUKfLuFeUtTEoI850sg1GQt0uaHf9kOrpmvsbg3sflO2dQXk8uSFi3B1ghG5DdWImVoc4f5kFK86XG65WVVfYjLaUvm7ZBPT0mwvq3SgPdEXeWq1fp7tMQqCRKuluRZon1e+J5SlWOPUTxwyTq336oK2Nfk3HsiQ3uCG76oCmI5yTe2tCN1f2IXFXBvLNM9QTxc/KFhI4uI6bvHjimOSpXYuRdroH0hc4ohLP44WG0E8Nz7sAb3Ai3PbDTKnqYQemvGEUTdcdbWuMK5/JE4f1IE9DgOOO9RF+s7uXTdzZAmoMcRtVsckQJEJl7AKgqSn7BbK0cjRL/QXclNySQ/qcgSDFiV/pPbGC9JFweJon5F0Qn/AIAnGGzqmVbS6x45UAJieI5oI/30OxjzF+Dov6EGMoE0mys+6l4j8v1EFCtHhhopS7zM4jZRTnNzlZWPtEISU2kopLroGpKESE145VQnB7X66F40KXcOIiXmU1H7Yxw5rJ0SqLAFWQw7gyNdzbnbfCmGASByxnYYwZRy3QDZYrQlthLd5tlGe0cg7j02ZSHmmJNQOx5yOmSQI7z4njDvJlOPncC14T59yDZUIYYLTAYjL8nmCNk2yD55ImVGRIb/f0JGAc2vHMlDN6fkuiSLtbASz11PwWzdln9tS9BTpRx7sfv5+kMhFkzomufftAJbs5XdOxcvGRHkRkdyCn3t8jcRXaNDBpq3o8zWLNh+411I65DMDN1B0c4+7Uw4N17I+KfrVh5Fb8ix0LW9aEL/SUxeIqdEu0ZMTncxGS0QVzZmtJUTdWuT0JjtgfsD65jqpg9l8RdIKX4i0ox2+GeQuXKCQiCaXS47yDxUcICYZB7UDnfwBcXlnGWw1frvkBBqwvwUVa7G7Al1A1a35ob1qb9gpyJZGqmEyV4Qe3PmfUHkp+/bAWarLXnk8P4l2faFjCEnPDIejGMQgDdtkodq7lfqwtHwg02bXNl9X3FbXjyp51/IJshE9dhHLkniWyHwZvHZJ4jjqndf08682gxTnW0bscfv7d6lKznE7rgd/el5ygdlL0PSr0pbxJtcd7MYrACMfZZxRuY3PWZayr1n4iaqh161S8ldeEcLCiHFvDeGHslUK+Sl383AHhs46NHEVgMbhuKJ1thHckQSgpqm9vGE1gBIo0d7z40EulgD+KS4Qm/vdydCgqtzyNmFDg17L+SVCQiyckGxgl/j1YzQKz0o1EWXNssk121m7Nh/e/bYNN8kC6doJPBQBkdhU/+DVZa4+/je+qpqvqnjpheX42k49ubk638QhOkcMmx6ZNnEbL9Ra0mNJHZO27t6++LgN2XkPg6t8VO94yx1c/VFyoF9Qg21O1+BadK3RF9l9s1eUB7n93YwUDhxZpimr8j8oLCn+PvksKwzWEIiAFXUWvuROwh6yC31GghO6veGx7UqXGPM2O8+mkOxXZObbnYAAhteRhXO/k15HL3qf94pudZ6+tETmeej4FKdhZzF9iOsTu/69izQFhl57E+n5hDjRzeVANKQOdq2x7qDG7pGfpJzrDVR95Z1q7Y/d+7FJjBML+YvzGvbxZno+PbPmLGrgxSebAWgO73fUDkBHvosMDjeHjg0NKBXYfR3kImceo4iJwF0bOeFQA09j+6OKW1EId9r1EzYwPGR0DUDt+A9f4DkU/27dbKUmJovBPIAXtYHErwZ7EW0ITd8ebNXyOiArz8JJietDsR++lwetArzrDszsPsiumoSu1/FajtbBfUQqllxJBeoIqNEPHQED2ysSWzqTlHe8tfmY3Vp/PlZh8GbpA/EV5uikWzDSaZbADqXGny/xd+g+8mm4J3DxgPOEEgFqeg+q4C1U4yEy3gYkknT6AcIO8MGOa3daRowEv2zfhVlrNhv2ezLjzsPBQJSK/ph9KZbYiREZVCBQioJJlvfG/dkkycjkVqvTNtn3swhipxUqYc0vKmXFG+R9+fUOP5u9ctqmdIf7Jclew4OBO1ML+sH+l6NojgfX+T+OqQ8EyNMY2Aa7i24ppwS8ySDwyYEk9kjopVZ8L3MSw3CQnWBNMg734qUawCUoFMtGPZ0jC+o6DiV18xpkB1Ffrmjs+72AoKk6wR9zIYyVQG7wr7O4PLWeXEX2NgBDeKaWi0gHInVAb2puJKgGgaZbImuecSaKaqA0y1fXu3+4teozv9eLkBxhYUJRwIBQbA7E3lQ4ocziE3ZMxdAIGAOEJHgDs6UreB4ZMLgdKxqhYzR3GxrmZmspOw/TVBGGjClrefGZh0/gLkXSDZ0uugW1rNM3TkA9t2U0eBCNyRh+8sScUKzja4Ei2ukkfoCNEf0alJkG3udwBtjDBcTRhQJARFD+UWMut5Xqw/dAxs777/VmY00/wD9FaCq0aVc/CJI++oGYgunPDd7ACSTyuY0UST1AEBrCrFBB+T+eKrFcU/4gBXEGiAY4x8lrp7fNZ/nsGFvBvs5sxq9NvLFXGA8Tv3LHU8x8m8oGHVnT0AaFJIKM8vGf2bkdD89EmQkKG9pfnKPCTbyFszqOJlgORDLSn9bNvDYlVC3dS5HQbqdv1a1fn1WrLDEXTNuD1Nj6dJXUBLAZfqzRHaPRi04YNpw9MY+c6f3XmAvNmwrFgdnxHja47ObX4XGigltlCWUSImRboBinCeZ9k9GdQ82Pw36dO433Z3IjtjuFLTWFsUurpAvmgiemzJ3xnhra6pOEr2jy2dw6fUL4gMQtPpSwqB4Qd9TY6MOUpwBqK0MpuRMBisa3zOBphJyvnXOIG9RCYzivzOQM+PKzLmznjTyeql1Ipj/9LwEgQ==,iv:PbRgjgm5ERWlAkkWc7As8eIx8e0rjTUkQ7XlDBrEUlU=,tag:yh2/JuHu5I+dWlxVAmkkXA==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOGNaakZoV3lhemhMSVVF
RklKbEsxeVFMWWdwQ3ZqR0lhRHlPM1dWT3h3Cnk2RnZUNnplVEwvNW9qMExsSUlt
VVVXejMrcDRRTEdtT1krdThqWThLWWcKLS0tIEVZZEVBWFZyVi84S1puRHdtMWR1
cDBHc1J5aHFpQkxSSEx2VDRjZ1RsdXcKG5MgnUBpFmlNGSkxBijXp7m492ujyqsC
TbTfU8SWJwXhio7V+zDbXG+JW4R51wr9F9Sj47DCw7b64nbsD3N5TQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-26T02:15:51Z"
mac: ENC[AES256_GCM,data:+2eGERwwzeK5JE9H6qhsu+tskR3Y9c7DecVGopfG8Ggq5SHQBfYyvz//IU+MGCOb6nIyP3JPf9+9IDS2DthVcw+RLtviQkN1TeWuUTDbxBxHdCoUv9zepwU7+d0qUMrdUK0N9QQSRZ4TpiwceViDeMuhPF/vTmADP9QWwid/qwk=,iv:nuJo37Dh8MAf+qI/Ty2jsW0IheNhgmZ4l0Va8pJG6+4=,tag:vCpifXvYF1/QU7XExc6raQ==,type:str]
version: 3.13.2
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: zitadel-masterkey
namespace: iam
type: Opaque
stringData:
masterkey: ENC[AES256_GCM,data:Da543Jt48o6p3f6/AbZRsjChz3NexPJoi/mC/LB6jUI=,iv:irS0n76tZ6QzoQMm0dWrvbw+SPfXz4h13jQb/Bjfgwg=,tag:tfbgaAdxmPIjleB1OXLn0Q==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZld2VG9NaFhIVWpBcGpO
SmVwNm9FT2wrQmFKb0RGQmY4TCtITjFEZjBjCmNuNXNJYTdsZE44cDNzR2hiNTdy
QWlVdzBtdjd0cSsxUjQxMzNyblhWTkkKLS0tIHBLT2x1MDBGdjE1eHFOdHVGa1NO
ZGM1K2lDTksraFdPeFdSZ1JyU1VtcjAKgTjjhr7u4YRuQVZQ7kAa1clJ0Rzy10hQ
Umcd4/YStyXHZjsJJpFSpOFdP5cZscGo3xBgpXtxAscKTS1Nw4derg==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-25T01:24:21Z"
mac: ENC[AES256_GCM,data:QFVk7KLsMLCAMH/PHVGCRyfU3k2JiO8AE3jJXmhnlUB4C8MYCTyOWQbnTFXozCMOr0kp+hAt3dYn/e23SWhyvJ7BgVLRb5QF1py704Dr46atVoLUvwHpFI7dyGK/LDTxqiM40RXAnDtWUJlTkVCzKdSFiV8CokIoByKWwxFPayM=,iv:t7HWMcpQl/DJ8Rp51Du7b5gURZO+C+FodsZSJuy5jBc=,tag:qEXSS4S0Y5cvn7T6EjakhA==,type:str]
version: 3.13.2
@@ -8,6 +8,19 @@
# depend on them existing first. # depend on them existing first.
installCRDs: true installCRDs: true
# Point the ACME DNS-01 self-check at the public resolvers directly.
#
# Without these flags cert-manager resolves through the pod resolver
# (CoreDNS at 10.96.0.10). The self-check then never saw the TXT record and
# reported "Waiting for DNS-01 challenge propagation" forever. The wildcard
# certificate renewal stalled on 2026-09-23 and the cert expired on
# 2026-10-23, which broke every Cloudflare Tunnel origin with
# "x509: certificate has expired".
#
extraArgs:
- --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
- --dns01-recursive-nameservers-only
# Tolerate the control-plane taint (talos-cp-1 is the only schedulable node). # Tolerate the control-plane taint (talos-cp-1 is the only schedulable node).
tolerations: tolerations:
- key: node-role.kubernetes.io/control-plane - key: node-role.kubernetes.io/control-plane
@@ -271,3 +271,40 @@ data:
NDKi7/Gxha790QN3TD/U7p9GcEtfXKv8KISKfxnfD9o9Yx6pWOXcgvGefQkpXuRm NDKi7/Gxha790QN3TD/U7p9GcEtfXKv8KISKfxnfD9o9Yx6pWOXcgvGefQkpXuRm
kGzDM+aY/VKs5Et5/RyKavBDihuOcumLsmoRg/bFoB/xpeURTyiehUdu kGzDM+aY/VKs5Et5/RyKavBDihuOcumLsmoRg/bFoB/xpeURTyiehUdu
-----END CERTIFICATE----- -----END CERTIFICATE-----
---
apiVersion: v1
kind: ConfigMap
metadata:
name: homelab-ca
namespace: cloudflared
data:
ca.crt: |
-----BEGIN CERTIFICATE-----
MIIFBjCCAu6gAwIBAgIJAKoBPdKPDL5AMA0GCSqGSIb3DQEBCwUAMDAxEzARBgNV
BAMMCmhvbWVsYWItY2ExGTAXBgNVBAoMEHJpb3RwaWFvLmhvbWVsYWIwHhcNMjYw
ODEyMjA1NDQzWhcNMzYwODA5MjA1NDQzWjAwMRMwEQYDVQQDDApob21lbGFiLWNh
MRkwFwYDVQQKDBByaW90cGlhby5ob21lbGFiMIICIjANBgkqhkiG9w0BAQEFAAOC
Ag8AMIICCgKCAgEAzw796vZRFOBc/orUu9J6rv91A3SNLJsqz/VuetwAXCaVSlK5
kL/RaBKKkcXToCC5avwix6W21FnwLUW4R8QlQdin8+XEHpx2LY5R4GmSvaCbU7wm
YwYgVGQVv8KlgcMbz/pF09vXU29cHUHwlkN+BWVGAxVGWxam94j1pBzV0dOhhRw2
q5sZYUUrPO5l775od2FMMnMzoRFugxDvJof0p3fAChNAIiFhP95lrcb2YsTwK+a2
6mk3AZc0gMtmgVsH+zryGFcAmWLixq41ye/H23ktj2bjEHCr9mIDpuSOKLyLBzrg
js/HqdxvwkZkkxwXft0Vi7GN0/LGHMdPrDE499pJelAHbxUpBoOk8sfXSNnfESI8
v20LybATGrKXFlOhMKshY1a3JLAoZTaW0xkG4PM4HT7lK/ovr/NLF6kdP7HxJFFQ
+4WjySiwbjusNobpTaQWVwCjD3Imf+2feAqJeX3z+P3EhxfrmvEojLOdUiiPcjMn
CFl9u2EpSlax53umSkcBQJGfFKM17fZzhtakjqhCTzAfXXlxZy4hsHDtfzRcfaoB
Zz/PiJ0TtJSC7TvxZZoPpGexQdoeT7kJEksTvGi5OSfJD7HaKe5RQSZGgpz7ATB+
YMBKbjhMXVu+TAdcO1qZRN0UCncrliN61ki9F0tE8kNgpS9biZ6kheoJMZUCAwEA
AaMjMCEwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcN
AQELBQADggIBAFN1At11d/+Ue6cEjCklnRQpLuD5agXaE+pi5oHNfnggSlO8UjUj
lDzfnmxi3IAI6HruzDARM9J95qJboUxWOAEmK9SNnZNS/1b0GyikP3YcF1gIFqtW
6SgmVfC1cVUFKGt/+QQzZxhOLuoydv9Ozu12nxsTW/WY2t+cpeL+1wrsM22RA6W/
xV/bzYbwzCUT9v4tMZLZHk5CkXztVFXSJyW/8w9rVZdUGZM+mwwn81PLSE0v3s/2
wJH8eUqGl73i3Xgn7CVanN4cJYMGYVVlRD6Qr6muBN4Vh0ETY1PrjzsZ6XsV6vk9
glMbQnErYMZHZIDtnVdaBt60MecUD9O4eLeip0SE6bVRAucKBXA9xNMilMQEL8/X
FBMAxXl10rowBlbX2oaFSOM80eLTla4a3gSL7+0vua3C4Dtv2bLo8ZjCyqXwOJeb
N85cqcVZStz3qlgQN1nG2qGE3BS6scOs7+mUGe9CmS8acJV++PKm8CtmLH6/ibjV
tAc/6+Vg/keo0kty7H7L6oQd+gKpuaJ2I+wKYe4l9UwHi9zEb8YjzGB4n2AtsoWI
NDKi7/Gxha790QN3TD/U7p9GcEtfXKv8KISKfxnfD9o9Yx6pWOXcgvGefQkpXuRm
kGzDM+aY/VKs5Et5/RyKavBDihuOcumLsmoRg/bFoB/xpeURTyiehUdu
-----END CERTIFICATE-----
+15 -7
View File
@@ -1,13 +1,21 @@
# k8s/bootstrap/ingress/riotpiao-com-cert.yaml # k8s/bootstrap/ingress/riotpiao-com-cert.yaml
# Public wildcard cert for *.riotpiao.com, signed by Let's Encrypt (DNS-01 via # Public wildcard cert for *.riotpiao.com, signed by Let's Encrypt (DNS-01 via
# Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml). Distinct from # Cloudflare, see ../cert-manager/letsencrypt-issuer.yaml).
# homelab-tls (self-signed homelab-ca, internal *.riotpiao.com) —
# reference riotpiao-com-tls explicitly on ingresses that expose riotpiao.com
# hosts to the internet; it is not nginx's default-ssl-certificate.
# #
# issuerRef starts on letsencrypt-staging. Switch to letsencrypt-prod once the # nginx serves this secret as its default-ssl-certificate
# staging cert issues cleanly, then delete this Certificate's Secret to force # (../ingress/nginx-values.yaml), so every Ingress that omits a tls: block uses
# a re-issue against prod (staging certs aren't browser-trusted). # it. Do not add per-host tls: blocks for *.riotpiao.com hosts: they create a
# second, single-host certificate and split the wildcard coverage.
#
# Renewal is automatic and does not drop traffic:
# - cert-manager starts a new order 30 days before expiry (renewBefore).
# - cert-manager writes the renewed pair into the same secret.
# - ingress-nginx watches the secret and hot-reloads. No pod restart.
# - The old pair stays valid until the new one is written, so no gap exists.
#
# The DNS-01 self-check must reach the public resolvers directly. Without the
# --dns01-recursive-nameservers flags in ../../cert-manager/cert-manager-values.yaml
# the self-check never sees the TXT record and renewal stalls until expiry.
apiVersion: cert-manager.io/v1 apiVersion: cert-manager.io/v1
kind: Certificate kind: Certificate
metadata: metadata:
@@ -107,7 +107,9 @@ ingress:
enabled: true enabled: true
className: nginx className: nginx
annotations: annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod # No cert-manager annotation. forgejo.riotpiao.com is covered by the
# *.riotpiao.com wildcard, served as nginx's default-ssl-certificate.
#
# This chart Ingress and the hand-written one in # This chart Ingress and the hand-written one in
# k8s/bootstrap/ingress/ingress.yaml both claim forgejo.riotpiao.com. # k8s/bootstrap/ingress/ingress.yaml both claim forgejo.riotpiao.com.
# ingress-nginx breaks the tie by oldest creationTimestamp, and the chart's # ingress-nginx breaks the tie by oldest creationTimestamp, and the chart's
@@ -125,10 +127,9 @@ ingress:
paths: paths:
- path: / - path: /
pathType: Prefix pathType: Prefix
tls: # tls: intentionally omitted. The chart would create a second, single-host
- secretName: forgejo-tls # certificate (forgejo-tls) that expired on 2026-10-23 and competed with the
hosts: # wildcard. The wildcard covers this host.
- forgejo.riotpiao.com
# Resources # Resources
resources: resources:
@@ -0,0 +1,177 @@
# etcd backend maintenance: compact, defrag, disarm NOSPACE.
#
# Why this exists
# ---------------
# etcd's backend file only grows. The apiserver compacts every 5m, but
# compaction marks superseded revisions free *inside* the file; only `defrag`
# returns that space to the filesystem. Nothing ran defrag, so the file grew
# until it crossed the 2 GiB quota:
#
# 2026-10-07 backend 2,148,843,520 B (2.1 GB) on talos-cp-1 and talos-cp-3
# -> etcd raised NOSPACE on those two members
# -> etcd rejected ALL writes cluster-wide
# -> kubelets could not post node status: 3 nodes NotReady
# -> apiserver writes returned 500, the tunnel API path 502
#
# The live data was tiny: 15,943 keys, 69 MB after compaction. The rest was
# dead pages. Recovery was `compact <current-revision>` then `defrag` per
# member then `alarm disarm`. Compaction had also stalled while the alarm was
# up, so `dbSizeInUse` read 100% of the file - defrag alone would NOT have
# worked. Order matters: compact first, then defrag.
#
# The earlier revision of this file never ran, for two independent reasons:
# 1. it existed only on branch fix/etcd-defrag-cron; the ArgoCD application
# `cluster-maintenance` tracks `main` (targetRevision: main), so the
# cluster never had the CronJob or the Secret.
# 2. the image was quay.io/coreos/etcd, which has NO /bin/sh (verified:
# `exec: "/bin/sh": no such file or directory`). Any container that runs
# a shell script needs an image that ships one.
# This revision fixes both: image bitnamilegacy/etcd (ships bash + etcdctl
# 3.6.4, matching the 3.6.11 servers) pinned by digest.
#
# Cadence: every 15 days. Measured growth is ~29 MB/day, so a 15-day cycle
# peaks near 500 MB - comfortably under the 2 GiB quota. The schedule is daily
# and the script exits unless the UTC epoch day is a multiple of 15. That gives
# an exact 15-day interval; `*/15` in the day-of-month field does not (it fires
# on the 1st, 16th and 31st, which puts two runs one day apart).
#
# Safety
# ------
# 1. Compaction uses the MINIMUM revision across members. Compacting to a
# revision a member has not reached wedges that member - measured on
# 2026-10-07, when talos-cp-2 froze ~1.2M entries behind the leader and
# needed a reboot to rejoin.
# 2. A member whose raft applied index lags the leader by more than 100000 is
# skipped and reported. Defragging a frozen member reclaims nothing.
# 3. Members are defragged one at a time. A defrag pauses the member's apply
# loop for the duration of the file copy.
#
# Credentials: Secret etcd-defrag-creds in kube-system, SOPS-managed in
# k8s/argocd/secrets/etcd-defrag-creds.enc.yaml (ksops generator 00-secrets).
# Its client certificate is signed by the cluster's etcd CA and expires
# 2036-10; rotate it before then by re-running the mint and re-encrypting.
#
# Companion alerts (monitoring): etcd_mvcc_db_total_size_in_bytes /
# etcd_server_quota_backend_bytes > 0.8, and an etcd NOSPACE alarm alert.
apiVersion: batch/v1
kind: CronJob
metadata:
name: etcd-defrag
namespace: kube-system
labels:
app.kubernetes.io/name: etcd-defrag
app.kubernetes.io/part-of: cluster-maintenance
spec:
schedule: "0 3 * * *" # daily; the script acts only every 15th day
concurrencyPolicy: Forbid
startingDeadlineSeconds: 600
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
restartPolicy: OnFailure
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: defrag
image: bitnamilegacy/etcd:3.6.4@sha256:c86c06f1ce6a08b6df21eb7a9655b1818778b64c85b5cbb8413ebf7534789730
env:
- name: ETCDCTL_API
value: "3"
- name: ETCDCTL_CACERT
value: /etc/etcd/ca.crt
- name: ETCDCTL_CERT
value: /etc/etcd/client.crt
- name: ETCDCTL_KEY
value: /etc/etcd/client.key
- name: EPS
value: "https://192.168.1.166:2379 https://192.168.1.214:2379 https://192.168.1.162:2379"
command: ["/bin/bash", "-c"]
args:
- |
set -u
DAY=$(( $(date -u +%s) / 86400 ))
if [ $(( DAY % 15 )) -ne 0 ]; then
echo "skip: UTC epoch day $DAY is not a 15-day boundary"
exit 0
fi
echo "== before =="
for ep in $EPS; do
etcdctl --endpoints="$ep" endpoint status -w table || true
done
MINREV=""
MAXAPPLIED=0
for ep in $EPS; do
json=$(etcdctl --endpoints="$ep" endpoint status -w json 2>/dev/null) || continue
rev=$(printf '%s' "$json" | sed -n 's/.*"revision":\([0-9]*\).*/\1/p' | head -1)
app=$(printf '%s' "$json" | sed -n 's/.*"raftAppliedIndex":\([0-9]*\).*/\1/p' | head -1)
echo "member $ep revision=${rev:-?} raftAppliedIndex=${app:-?}"
if [ -n "$rev" ] && { [ -z "$MINREV" ] || [ "$rev" -lt "$MINREV" ]; }; then
MINREV="$rev"
fi
if [ -n "$app" ] && [ "$app" -gt "$MAXAPPLIED" ]; then
MAXAPPLIED="$app"
fi
done
if [ -z "$MINREV" ]; then
echo "FATAL: no member returned a revision; nothing attempted"
exit 1
fi
echo "== compact to revision $MINREV =="
etcdctl --endpoints="$(echo "$EPS" | awk '{print $1}')" compact "$MINREV" || echo "COMPACT-FAILED (continuing: defrag still reclaims free pages)"
# Physical deletion of superseded keys runs in the background
# after the compaction commit; give it a moment to drain.
sleep 20
echo "== defrag each member (sequential, one at a time) =="
rc=0
for ep in $EPS; do
json=$(etcdctl --endpoints="$ep" endpoint status -w json 2>/dev/null) || { echo "SKIP $ep (unreachable)"; rc=1; continue; }
app=$(printf '%s' "$json" | sed -n 's/.*"raftAppliedIndex":\([0-9]*\).*/\1/p' | head -1)
if [ -n "$app" ] && [ "$app" -lt $(( MAXAPPLIED - 100000 )) ]; then
echo "SKIP $ep (applied index $app lags leader $MAXAPPLIED; defrag would reclaim nothing)"
rc=1
continue
fi
echo "-- $ep"
if ! etcdctl --endpoints="$ep" defrag; then
echo "DEFRAG-FAILED $ep"
rc=1
fi
done
echo "== disarm NOSPACE =="
for ep in $EPS; do
etcdctl --endpoints="$ep" alarm list 2>/dev/null || true
etcdctl --endpoints="$ep" alarm disarm 2>/dev/null || true
done
echo "== after =="
for ep in $EPS; do
etcdctl --endpoints="$ep" endpoint status -w table || true
done
etcdctl --endpoints="$(echo "$EPS" | awk '{print $1}')" alarm list || true
exit $rc
resources:
requests: { cpu: 10m, memory: 32Mi }
limits: { cpu: 200m, memory: 128Mi }
volumeMounts:
- name: creds
mountPath: /etc/etcd
readOnly: true
volumes:
- name: creds
secret:
secretName: etcd-defrag-creds
defaultMode: 0444
+1
View File
@@ -7,5 +7,6 @@ kind: Kustomization
resources: resources:
- namespaces.yaml - namespaces.yaml
- authentik-db.yaml - authentik-db.yaml
- zitadel-db.yaml
- temporal-db.yaml - temporal-db.yaml
- paperless-db.yaml - paperless-db.yaml
+42
View File
@@ -0,0 +1,42 @@
# Dedicated CNPG Postgres for ZITADEL (GitOps, wave 2 — before the zitadel
# chart at wave 3). Same shape as authentik-db.yaml: CNPG generates the secret
# `zitadel-db-app` and the service `zitadel-db-rw` in ns iam, and
# k8s/infra/iam/zitadel-values.yaml reads the DSN from that secret locally (no
# cross-namespace secret copy).
#
# ZITADEL requires PostgreSQL 14+; the repo convention image is 16.2.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: zitadel-db
namespace: iam
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
bootstrap:
initdb:
database: zitadel
owner: app
encoding: UTF8
localeCollate: C
localeCType: C
enableSuperuserAccess: false
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "2Gi", cpu: "1" }
storage:
size: 10Gi
storageClass: longhorn-cnpg
monitoring:
enablePodMonitor: true
affinity:
# preferred + tolerations: HA across nodes without deadlocking a partly-
# schedulable 3-CP (same as authentik-db and forgejo-db).
podAntiAffinityType: preferred
topologyKey: kubernetes.io/hostname
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
+226
View File
@@ -0,0 +1,226 @@
# k8s/infra/iam/zitadel-values.yaml
# ZITADEL — Authentik replacement. Coexistence is deliberate: Authentik keeps
# running, read-only, in the same `iam` namespace until the last consumer moves.
# Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
#
# Chart: zitadel/zitadel 10.4.0, app v4.19.2. The plan cites v4.19.4; the chart
# has not published it yet. Pin the chart, never `*`, until the v4.19.4 chart
# exists — a silent app bump invalidates the verified masterkey/passwap facts.
#
# URL: https://zitadel.riotpiao.com. TLS terminates at ingress-nginx, which
# serves the wildcard riotpiao-com-tls as its default-ssl-certificate, so
# ExternalSecure=true and TLS.Enabled=false (no per-rule tls: block needed).
#
# Database: CNPG cluster zitadel-db (wave 2). The DSN comes from CNPG's
# generated `zitadel-db-app` secret. When ZITADEL_DATABASE_POSTGRES_DSN is set,
# ZITADEL ignores the discrete Database.Postgres.* fields, so neither a
# password nor a host is written into this file.
#
# Masterkey: Secret `zitadel-masterkey` (k8s/argocd/secrets/
# zitadel-masterkey.enc.yaml), key `masterkey`, exactly 32 bytes. It wraps the
# rows in system.encryption_keys, which encrypt every client secret and private
# key. Losing it makes those values unreadable: keep the offline sealed copy.
#
# Ingress uses the chart's own objects (controller: nginx adds the gRPC backend
# annotation ZITADEL needs for its HTTP/2 API). Do not hand-write a second
# Ingress for this host.
zitadel:
configmapConfig:
ExternalSecure: true
ExternalDomain: zitadel.riotpiao.com
TLS:
Enabled: false
# First boot is COMPLETE (2026-10-26): the instance, the system org and the
# `iam-admin` machine key all exist. Skip stops the setup hook from running
# the first-instance step again. Without it, every sync re-runs the
# machinekeyWriter, which silently rotates the iam-admin key Terraform
# reads. ZITADEL objects are declared by scripts/iam/zitadel-provision.py.
# Never set this back to false.
FirstInstance:
Skip: true
# Rendered into a Secret by the chart; read by pods and init jobs alike.
masterkeySecretName: zitadel-masterkey
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "2Gi", cpu: "1" }
# Best practice: run as non-root, drop all capabilities, and refuse
# privilege escalation. The chart's image already supports a non-root UID.
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
# FirstInstance defaults stay for the FIRST sync: the chart creates org
# `zitadel` plus the `iam-admin` machine user and its machine-key Secret.
# Terraform consumes that key to declare projects, roles, OIDC apps and the
# groups action.
#
# SECOND-SYNC HAZARD — read before enabling automated sync.
# The chart marks eight objects as `pre-install,pre-upgrade` hooks (two
# ServiceAccounts, two ConfigMaps, Role, RoleBinding, zitadel-init,
# zitadel-setup), and one cleanup Job as `post-delete`. ArgoCD maps the first
# eight to PreSync with `before-hook-creation`: it deletes and re-creates them
# on EVERY sync, not only on a chart change.
# - zitadel-init (`init zitadel`) — schema step
# - zitadel-setup (`setup`) — first instance + iam-admin machine key
# On an initialized instance the setup step can fail, which aborts the sync
# before the Deployment is applied. If it succeeds instead, its
# machinekeyWriter container re-applies the iam-admin key Secret that
# Terraform reads, which silently rotates a credential in use.
# After the first sync reports healthy:
# 1. set configmapConfig.FirstInstance.Skip: true here — permanently: the
# setup steps are schema-only once the instance exists, and Skip alone
# stops the machinekeyWriter rotation
# 2. set initJob.enabled: false below
# 3. commit, sync once, then enable automated+selfHeal on the Application
# For a later schema upgrade, re-enable initJob.enabled only — never set
# FirstInstance.Skip back to false.
# ArgoCD never prunes a hook Job, so a Job that failed before step 2 stays and
# keeps ZitadelSetupJobFailed firing until a human deletes it.
# DSN mode (chart renders this through the zitadel.dbEnv helper). CNPG's `uri`
# key is a postgresql:// URL, which ZITADEL accepts as a DSN.
env:
- name: ZITADEL_DATABASE_POSTGRES_DSN
valueFrom:
secretKeyRef:
name: zitadel-db-app
key: uri
# Bundled Bitnami Postgres is disabled: the CNPG cluster above is the system of
# record. Enabling it would create a second, unmanaged database.
postgresql:
enabled: false
# CNPG already created the database, the `app` user and its grants, so the init
# job runs only the ZITADEL-internal step. Running the full default command
# would try to create a database and a superuser it does not own.
#
# enabled: false after the first successful boot (2026-10-26). The schema is in
# place; the init hook has nothing left to do and re-running it on every sync
# only adds a PreSync failure surface. Re-enable it for a deliberate schema
# upgrade, then set it back to false.
initJob:
enabled: false
command: "zitadel"
activeDeadlineSeconds: 600
# The setup job runs the full first-instance creation: schema migrations, the
# system org, the iam-admin machine key, and 13 projection replays. 20 minutes
# leaves room for a cold database without masking a genuine hang.
#
# DONE, 2026-10-26 — first boot completed. The annotations are moved from
# pre-install/pre-upgrade to post-delete because a PreSync setup run is not
# harmless after boot:
# - it re-runs every repeatable migration on every sync;
# - interrupting one (a sync aborted, or a human deleting the Job) leaves a
# `system.migration.started` event with no matching `done`, and ZITADEL then
# waits forever on `migration already started`. That is exactly how this
# instance deadlocked on projections.projects4 and later on
# repeatable_delete_stale_org_fields.
# A post-delete hook is never executed by ArgoCD sync, so setup runs only on
# first install. Re-enable it deliberately (and only with initJob too) for a
# schema upgrade, then revert.
setupJob:
activeDeadlineSeconds: 1200
annotations:
helm.sh/hook: post-delete
helm.sh/hook-delete-policy: hook-succeeded
helm.sh/hook-weight: "2"
# Two API replicas: the read path survives one node drain, and both share the
# same masterkey and database. The login UI stays at one replica — a session
# cookie is not shared, so a second pod buys nothing.
replicaCount: 2
revisionHistoryLimit: 3
# Best practice: a PDB guarantees at least one API pod stays available during
# voluntary disruptions (node drain, rolling upgrade). Without it a drain can
# take both replicas down at once. minAvailable 1 is correct for 2 replicas;
# minAvailable 2 would block every drain forever.
pdb:
enabled: true
minAvailable: 1
# Best practice: spread replicas across nodes so a single node failure cannot
# take both API pods. Anti-affinity alone only prefers separation; topology
# spread enforces the failure-domain budget.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: zitadel
ingress:
enabled: true
className: nginx
controller: nginx
login:
enabled: true
# ksops owns this Secret (k8s/argocd/secrets/zitadel-login-service-key.enc.yaml,
# rendered by the sops-secrets Application). Pointing the chart at it stops the
# chart from auto-generating an RSA keypair every render, which changed
# checksum/secret-login-service-key and kept both Deployments permanently
# OutOfSync. The keypair must stay RSA: the login container signs JWTs (RS256).
loginServiceKeySecretName: zitadel-login-service-key
replicaCount: 1
revisionHistoryLimit: 3
ingress:
enabled: true
className: nginx
# Best practice: the login UI is the public attack surface. Harden the pod
# even at one replica. The chart has no login PDB key; a single replica has
# nothing to protect with minAvailable.
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
# Chart-owned ServiceMonitor. The cluster's Prometheus runs with
# serviceMonitorSelectorNilUsesHelmValues=false, so the release label is not
# required for discovery. Endpoint is /debug/metrics on port http2-server.
metrics:
enabled: true
serviceMonitor:
enabled: true
scrapeInterval: 30s
scrapeTimeout: 10s
# The 3-CP / 2-worker cluster mixes control-plane taints with worker capacity.
# Without the toleration a rolling upgrade can leave both replicas Pending.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: zitadel
+7
View File
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# No commonLabels on purpose: this is a vendored third-party manifest and
# kustomize would also rewrite the provisioner Deployment's selector, which
# is immutable on re-apply.
resources:
- local-path-storage.yaml
@@ -0,0 +1,200 @@
# Vendored from https://github.com/rancher/local-path-provisioner
# deploy/local-path-storage.yaml @ v0.0.37 (release 2026-08-05)
#
# Local modifications (the only three; re-apply them on any re-vendor):
# 1. the namespace carries pod-security.kubernetes.io/enforce=privileged, because the
# provisioner's helper Pods mount a hostPath and this cluster defaults to the
# baseline policy, which forbids that;
# 2. config.json nodePathMap is /var/lib/local-path-provisioner — Talos mounts the root
# filesystem read-only, so upstream's /opt path cannot be created;
# 3. the helper Pod's busybox image is pinned to a tag instead of a floating name.
apiVersion: v1
kind: Namespace
metadata:
name: local-path-storage
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: local-path-provisioner-role
namespace: local-path-storage
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: local-path-provisioner-role
rules:
- apiGroups: [""]
resources: ["nodes", "persistentvolumeclaims", "configmaps", "pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: local-path-provisioner-bind
namespace: local-path-storage
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: local-path-provisioner-role
subjects:
- kind: ServiceAccount
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-path-provisioner-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: local-path-provisioner-role
subjects:
- kind: ServiceAccount
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: local-path-provisioner
namespace: local-path-storage
spec:
replicas: 1
selector:
matchLabels:
app: local-path-provisioner
template:
metadata:
labels:
app: local-path-provisioner
spec:
serviceAccountName: local-path-provisioner-service-account
containers:
- name: local-path-provisioner
image: docker.io/rancher/local-path-provisioner:v0.0.37
imagePullPolicy: IfNotPresent
command:
- local-path-provisioner
- --debug
- start
- --config
- /etc/config/config.json
ports:
- name: health
containerPort: 8080
protocol: TCP
startupProbe:
httpGet:
path: /health
port: health
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 12
livenessProbe:
httpGet:
path: /health
port: health
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: health
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: config-volume
mountPath: /etc/config/
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CONFIG_MOUNT_PATH
value: /etc/config/
- name: HEALTH_PORT
value: "8080"
volumes:
- name: config-volume
configMap:
name: local-path-config
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
kind: ConfigMap
apiVersion: v1
metadata:
name: local-path-config
namespace: local-path-storage
data:
config.json: |-
{
"nodePathMap":[
{
"node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
"paths":["/var/lib/local-path-provisioner"]
}
]
}
setup: |-
#!/bin/sh
set -eu
mkdir -m 0777 -p "$VOL_DIR"
teardown: |-
#!/bin/sh
set -eu
rm -rf "$VOL_DIR"
helperPod.yaml: |-
apiVersion: v1
kind: Pod
metadata:
name: helper-pod
spec:
priorityClassName: system-node-critical
tolerations:
- key: node.kubernetes.io/disk-pressure
operator: Exists
effect: NoSchedule
containers:
- name: helper-pod
image: docker.io/library/busybox:1.36
imagePullPolicy: IfNotPresent
+13 -10
View File
@@ -41,13 +41,24 @@ spec:
# - /var/lib/longhorn-disk1 (sdb1, 600GB) # - /var/lib/longhorn-disk1 (sdb1, 600GB)
# - /var/lib/longhorn-disk2 (sdc1, 500GB) # - /var/lib/longhorn-disk2 (sdc1, 500GB)
# - /var/lib/longhorn-disk3 (sdd1, 160GB) # - /var/lib/longhorn-disk3 (sdd1, 160GB)
# - /var/lib/longhorn-disk4 (sde1, 2TB) #
# /var/lib/longhorn-disk4 (sde1, the 2TB PERC virtual disk) was RETIRED
# on 2026-10-12 and must not be re-added. The device returned SCSI I/O
# errors at ~7,000/minute, the XFS filesystem on it hung, and every
# process that touched it blocked in unkillable D state. Those unkillable
# processes held Longhorn's per-node rebuild slots forever, so no new
# replica could start on talos-cp-2 and every new volume there failed to
# attach. It is removed from the Terraform disk list too, so a re-render
# does not put it back.
echo "Checking talos-cp-2..." echo "Checking talos-cp-2..."
CURRENT_DISKS=$(kubectl -n longhorn-system get nodes.longhorn.io talos-cp-2 -o json | jq -r '.spec.disks | keys | length') CURRENT_DISKS=$(kubectl -n longhorn-system get nodes.longhorn.io talos-cp-2 -o json | jq -r '.spec.disks | keys | length')
echo " Current disk count: $CURRENT_DISKS" echo " Current disk count: $CURRENT_DISKS"
if [ "$CURRENT_DISKS" -lt 5 ]; then # 4 = default-disk-080400000000 + disk1..disk3. It was 5 while disk4
# existed; leaving the old threshold would re-apply the patch on every
# run once disk4 is gone.
if [ "$CURRENT_DISKS" -lt 4 ]; then
echo " Adding extra disks to talos-cp-2..." echo " Adding extra disks to talos-cp-2..."
kubectl -n longhorn-system patch nodes.longhorn.io talos-cp-2 --type merge -p '{ kubectl -n longhorn-system patch nodes.longhorn.io talos-cp-2 --type merge -p '{
"spec": { "spec": {
@@ -75,14 +86,6 @@ spec:
"path": "/var/lib/longhorn-disk3", "path": "/var/lib/longhorn-disk3",
"storageReserved": 5368709120, "storageReserved": 5368709120,
"tags": [] "tags": []
},
"disk4": {
"allowScheduling": true,
"diskType": "filesystem",
"evictionRequested": false,
"path": "/var/lib/longhorn-disk4",
"storageReserved": 21474836480,
"tags": []
} }
} }
} }
@@ -52,14 +52,15 @@ spec:
restartPolicy: Never restartPolicy: Never
initContainers: initContainers:
- name: kubectl-copy - name: kubectl-copy
image: bitnami/kubectl:latest image: bitnami/kubectl:latest@sha256:f7f9e4f64d9e114650c115a4ac6fd383394b3d494061a8f17b1a0f6c8d55bc25
command: ["sh", "-c", "cp $(which kubectl) /shared/kubectl"] command: ["sh", "-c", "cp $(which kubectl) /shared/kubectl"]
volumeMounts: volumeMounts:
- name: shared - name: shared
mountPath: /shared mountPath: /shared
containers: containers:
- name: provision - name: provision
image: quay.io/minio/mc:latest # bitnamilegacy/minio-client:2025.7.21-debian-12-r3 replaces the dead quay.io/minio/mc image
image: docker.io/bitnamilegacy/minio-client:2025.7.21-debian-12-r3@sha256:73bd39f7899a0cef12b8dd5df13aa93a3ed1aaa44236542442e9ac76819ac158
volumeMounts: volumeMounts:
- name: shared - name: shared
mountPath: /shared mountPath: /shared
+9 -1
View File
@@ -11,7 +11,15 @@ metadata:
prometheus.io/port: "9000" prometheus.io/port: "9000"
prometheus.io/scrape: "true" prometheus.io/scrape: "true"
spec: spec:
image: minio/minio:RELEASE.2025-07-23T15-54-02Z # Was minio/minio:RELEASE.2025-07-23T15-54-02Z — that Docker Hub repository no longer
# exists (404 for every tag), so the tenant sat in ImagePullBackOff and Vault's S3
# backend could not resolve minio-cluster-hl. Official images now need registry auth
# (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream
# release; swap it for the official image once MinIO registry credentials exist.
# The digest freezes those bytes, so a retag or a deletion on the mirror cannot break
# the pull again. The same digest already sits in the containerd cache on talos-cp-1,
# so the first start needs no registry access.
image: docker.io/coollabsio/minio:RELEASE.2025-10-15T17-29-55Z@sha256:69b55a1c1c5dc285ce04db96689f5b2102317fc77a50680a1874ca6efd1c87f9
# Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000. # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000.
# External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert). # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert).
@@ -27,6 +27,24 @@ spec:
annotations: annotations:
summary: "{{ $value }} pod(s) stuck in Pending state for >10m" summary: "{{ $value }} pod(s) stuck in Pending state for >10m"
# ImagePullBackOff on a vendor-deleted image produced no alert for 13 days
- alert: PodImagePullFailure
expr: sum(kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) by (namespace, pod) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image"
# Pod not ready
- alert: PodNotReady
expr: sum(kube_pod_status_ready{condition="false"}) by (namespace, pod) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} is not ready"
# CrashLoopBackOff # CrashLoopBackOff
- alert: PodCrashLooping - alert: PodCrashLooping
expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0 expr: sum(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}) by (namespace, pod) > 0
@@ -0,0 +1,68 @@
# ZITADEL alert parity with the Authentik rules it will replace.
#
# Only two signals are ported at this point. The metrics endpoint
# (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
# no request or error counters that this cluster has verified, so no error-rate
# rule is written here. Writing one against a guessed series name would produce
# a permanently-firing or permanently-silent alert — both worse than none.
# AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
# carries zero providers and is deleted with the migration, not migrated.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: zitadel-rules
namespace: iam
spec:
groups:
- name: zitadel.rules
interval: 30s
rules:
- alert: ZitadelDown
# absent() is required: equality alone cannot fire when the series is
# gone (ServiceMonitor deleted, Service selector changed, pods scaled
# to zero). Those are exactly the outages a target-down rule exists for.
expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
for: 10m
labels:
severity: critical
annotations:
summary: "ZITADEL API target is down"
description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
- alert: ZitadelLoginUIDown
expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
for: 10m
labels:
severity: warning
annotations:
summary: "ZITADEL login UI has no available replica"
description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
- alert: ZitadelRestarting
# pod!~"zitadel-db-.*" matters: the CNPG database pods share the
# `zitadel` name prefix, and a database restart must not page as an
# IdP crash loop.
expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
for: 5m
labels:
severity: warning
annotations:
summary: "ZITADEL pod restarts more than 3 times per hour"
description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
# The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
# restartPolicy: Never, so their restart counter stays 0. Without this
# rule a failed schema init or first-instance setup reaches nobody but
# the person who runs `kubectl -n iam get job`.
# No absent() guard here on purpose: the chart ships only
# `before-hook-creation`, so a successful hook Job is replaced at the
# next PreSync. Once the jobs are retired, absent() would fire on a
# healthy cluster.
- alert: ZitadelSetupJobFailed
expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
for: 5m
labels:
severity: critical
annotations:
summary: "ZITADEL init/setup job failed"
description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"
+1
View File
@@ -8,6 +8,7 @@ resources:
- alerts/ingress-alerts.yaml - alerts/ingress-alerts.yaml
- alerts/svc-argocd-rules.yaml - alerts/svc-argocd-rules.yaml
- alerts/svc-authentik-rules.yaml - alerts/svc-authentik-rules.yaml
- alerts/svc-zitadel-rules.yaml
- alerts/svc-forgejo-rules.yaml - alerts/svc-forgejo-rules.yaml
- alerts/svc-grafana-rules.yaml - alerts/svc-grafana-rules.yaml
- alerts/svc-minio-rules.yaml - alerts/svc-minio-rules.yaml
+4 -1
View File
@@ -55,7 +55,10 @@ alertmanager:
prometheus: prometheus:
prometheusSpec: prometheusSpec:
retention: 30d retention: 30d
retentionSize: "95GB" # Must stay under the PVC size below (25Gi). At 95GB the limit never
# triggered, the volume filled, and Prometheus crashlooped with
# "no space left on device" writing /prometheus/queries.active.
retentionSize: "18GB"
# scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate # scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate
# the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no # the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no
+33
View File
@@ -0,0 +1,33 @@
# One-shot bootstrap: restart the control-plane apiservers ONCE so they pick up the
# OIDC authenticator (they last started before the Authentik provider existed, so OIDC
# tokens were never accepted -> every login got 401). Delete this whole directory after.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: oidc-bootstrap
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: oidc-bootstrap
namespace: kube-system
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: oidc-bootstrap
namespace: kube-system
subjects:
- kind: ServiceAccount
name: oidc-bootstrap
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: oidc-bootstrap
+1
View File
@@ -9,3 +9,4 @@ resources:
- poimen-sandbox-namespace.yaml - poimen-sandbox-namespace.yaml
- poimen-sandbox-quota.yaml - poimen-sandbox-quota.yaml
- pi-session-netpol.yaml - pi-session-netpol.yaml
- pi-agent-workspace-pv.yaml
@@ -0,0 +1,57 @@
apiVersion: v1
kind: PersistentVolume
metadata:
finalizers:
- kubernetes.io/pv-protection
labels:
app: pi-coding-agent-sandbox
part-of: poimen-toolkit
name: pi-agent-workspace-pv
annotations:
argocd.argoproj.io/sync-wave: '-1'
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pi-agent-workspace
namespace: poimen-sandbox
hostPath:
path: /var/lib/pi-agent-sandbox
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- talos-cp-2
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: 'yes'
argocd.argoproj.io/sync-wave: '-1'
finalizers:
- kubernetes.io/pvc-protection
labels:
app: pi-coding-agent-sandbox
part-of: poimen-toolkit
name: pi-agent-workspace
namespace: poimen-sandbox
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: ''
volumeMode: Filesystem
volumeName: pi-agent-workspace-pv
+12
View File
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Stable secret name: the job mounts talos-repair-config by name, so the generated
# Secret must not get a hash suffix (same convention as k8s/argocd/secrets).
generatorOptions:
disableNameSuffixHash: true
# ksops decrypts talosconfig-secret.enc.yaml into the Secret the job mounts.
# Kept local to this app on purpose: no edits to the shared secrets generator.
generators:
- secret-generator.yaml
resources:
- restart-kubelet.yaml
@@ -0,0 +1,57 @@
# One-shot repair: the kubelets on cp-1, cp-2 and worker-1 stopped posting node status at
# 04:11-04:14 UTC and never reconnected, so those nodes are NotReady/unreachable and every
# pod on them (Cilium included) shows NotReady. This job runs on the one healthy node (cp-3),
# talks to the Talos API on the LAN, and uses Talos' endpoint->node proxying to reach the
# other three: it reports the kubelet service state, then restarts it.
apiVersion: batch/v1
kind: Job
metadata:
name: talos-restart-stalled-kubelets
namespace: cicd
spec:
backoffLimit: 1
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: talos-cp-3
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: talosctl
image: docker.io/curlimages/curl:8.22.0
command: ["/bin/sh", "-c"]
args:
- |
set -u
echo "fetching talosctl v1.13.3 (matches the node version)"
curl -sSL -o /tmp/talosctl \
https://github.com/siderolabs/talos/releases/download/v1.13.3/talosctl-linux-amd64 \
|| { echo "DOWNLOAD FAILED"; exit 1; }
chmod +x /tmp/talosctl
T="/tmp/talosctl --talosconfig /talos/config -e 192.168.1.162"
NODES="192.168.1.166 192.168.1.214 192.168.1.223"
echo "=== BEFORE: kubelet service state ==="
for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
echo "=== kernel log tail (last 15 lines per node) ==="
for n in $NODES; do echo "--- $n"; $T -n $n dmesg --tail 15 2>&1 | tail -15 || true; done
echo "=== ACTION: restart kubelet ==="
for n in $NODES; do echo "--- restarting $n"; $T -n $n restart kubelet 2>&1 || true; sleep 20; done
sleep 60
echo "=== AFTER ==="
for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
volumeMounts:
- name: talos
mountPath: /talos
readOnly: true
volumes:
- name: talos
secret:
secretName: talos-repair-config
defaultMode: 0444
items:
- key: config
path: config
@@ -0,0 +1,10 @@
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: talos-repair-secret-generator
annotations:
config.kubernetes.io/function: |
exec:
path: ksops
files:
- talosconfig-secret.enc.yaml
@@ -0,0 +1,23 @@
apiVersion: v1
kind: Secret
metadata:
name: talos-repair-config
namespace: cicd
type: Opaque
stringData:
config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
encrypted_regex: ^(data|stringData)$
lastmodified: "2026-10-07T04:27:33Z"
mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
version: 3.13.2
+634
View File
@@ -0,0 +1,634 @@
#!/usr/bin/env python3
"""Provision ZITADEL end to end: machine key, Terraform, SOPS secrets, users.
One command replaces the manual sequence the Authentik script
(scripts/iam/provision-rbac.py) used to do in a single pass. Phases run in a
fixed order and every phase is safe to re-run:
keys extract the iam-admin machine key and PAT from namespace iam
apply create project, roles and OIDC apps through the ZITADEL API
secrets write generated client ids/secrets back into the SOPS files
users import one human user, converting the Django password hash
verify check discovery, client count and the groups-claim gate
`all` runs keys, apply, secrets, verify. It never runs users: the password hash
must be supplied by a human, one user at a time.
Nothing here talks to Authentik. Authentik keeps serving until a consumer is
moved deliberately.
Usage:
scripts/iam/zitadel-provision.py keys
scripts/iam/zitadel-provision.py apply --org-id <id>
scripts/iam/zitadel-provision.py secrets --dry-run
scripts/iam/zitadel-provision.py secrets --apply-config
scripts/iam/zitadel-provision.py users --django-hash '<pbkdf2_sha256$...>'
scripts/iam/zitadel-provision.py verify
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import re
import signal
import stat
import subprocess
import sys
import tempfile
from pathlib import Path
import yaml
REPO = Path(__file__).resolve().parents[2]
SOPS_DIR = REPO / "k8s" / "argocd" / "secrets"
SOPS_CONFIG = REPO / ".sops.yaml"
CONF_DIR = Path(os.environ.get("ZITADEL_CONF_DIR", Path.home() / ".config" / "zitadel"))
STATE_FILE = CONF_DIR / "state.json"
NAMESPACE = os.environ.get("ZITADEL_NAMESPACE", "iam")
BASE_URL = os.environ.get("ZITADEL_BASE_URL", "https://zitadel.riotpiao.com")
# ── desired ZITADEL state ──────────────────────────────────────────────────
# Declared here, applied through the API by the `apply` phase. This replaces the
# retired terraform/zitadel root module: same objects, no state file to drift and
# no second writer.
PROJECT_NAME = os.environ.get("ZITADEL_PROJECT_NAME", "homelab")
PROJECT_GROUP = "homelab"
# Role keys EQUAL the Authentik group names, so the `groups` claim keeps working
# for every consumer that maps on it (kube-apiserver RBAC, MinIO policies,
# Grafana roles).
ROLES = {
"homelab-admins": "Cluster administrators with full access",
"minio-admins": "MinIO administrators",
"minio-photos": "Photos bucket (Immich) access",
"minio-documents": "Documents bucket (Paperless) access",
"minio-backups": "Backups bucket read-only access",
"poimen-admins": "Poimen memory administrators",
"poimen-devs": "Dev and staging projects access",
"poimen-prod-readonly": "Production projects read-only access",
"paperless-admins": "Paperless administrators",
"paperless-finance": "Finance documents",
"paperless-legal": "Legal documents",
"paperless-hr": "HR documents",
"grafana-admins": "Grafana administrators",
"grafana-editors": "Grafana dashboard editors",
"grafana-viewers": "Grafana dashboard viewers",
"sqs-users": "SQS/Temporal queue read access",
"sqs-writers": "SQS/Temporal queue read/write access",
"s3-users": "S3 read access",
"s3-writers": "S3 read/write access",
}
_WEB = {"app_type_id": "OIDC_APP_TYPE_WEB", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_BASIC"}
_NATIVE = {"app_type_id": "OIDC_APP_TYPE_NATIVE", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_NONE"}
_PKCE = {"app_type_id": "OIDC_APP_TYPE_USER_AGENT", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_NONE"}
OIDC_APPS = {
"api-gw": {**_WEB, "redirect_uris": [
"http://localhost:3000/callback", "https://api.riotpiao.com/callback"]},
"argocd": {**_WEB, "redirect_uris": [
"http://localhost:8080/auth/callback",
"https://argocd.riotpiao.com/auth/callback"]},
"forgejo": {**_WEB, "redirect_uris": [
"http://localhost:3000/user/oauth2/authentik/callback",
"https://forgejo.riotpiao.com/user/oauth2/authentik/callback"]},
"grafana": {**_WEB, "redirect_uris": [
"http://localhost:3000/login/generic_oauth",
"https://grafana.riotpiao.com/login/generic_oauth"]},
"homarr": {**_WEB, "redirect_uris": [
"http://localhost:7575/auth/callback",
"https://homarr.riotpiao.com/auth/callback"]},
"immich": {**_WEB, "app_type_id": "OIDC_APP_TYPE_NATIVE", "redirect_uris": [
"app.immich:///oauth-callback", "https://img.riotpiao.com/auth/login",
"https://img.riotpiao.com/user/oauth2/callback"]},
"kubernetes": {**_PKCE, "dev_mode": True, "redirect_uris": [
"http://localhost:8000", "http://localhost:18000"]},
"minio": {**_WEB, "redirect_uris": [
"http://localhost:9000/auth/sso/oauth2/code",
"https://minio.riotpiao.com/auth/sso/oauth2/code"]},
"paperless": {**_WEB, "redirect_uris": [
"http://localhost:8000/accounts/oidc/authentik/login/callback/",
"https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"]},
"poimen": {**_WEB, "redirect_uris": [
"http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"]},
"queue-sqs": {**_WEB, "redirect_uris": [
"http://localhost:8080/callback", "https://queue.riotpiao.com/callback"]},
"vault": {**_WEB, "redirect_uris": [
"http://localhost:8200/ui/vault/auth/oidc/oidc/callback",
"https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback"]},
}
_TEMP_FILES: set[str] = set()
def _cleanup_temps(*_a) -> None:
for path in list(_TEMP_FILES):
try:
os.unlink(path)
except OSError:
pass
_TEMP_FILES.discard(path)
for _sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP):
signal.signal(_sig, lambda s, f: (_cleanup_temps(), sys.exit(130)))
def die(msg: str) -> None:
_cleanup_temps()
print(f"FATAL: {msg}", file=sys.stderr)
sys.exit(1)
def run(cmd: list[str], **kw) -> subprocess.CompletedProcess:
return subprocess.run(cmd, check=False, text=True, capture_output=True, **kw)
def need(binary: str) -> None:
if subprocess.run(["sh", "-c", f"command -v {binary}"], capture_output=True).returncode != 0:
die(f"{binary} is not on PATH")
def write_private(path: Path, data: bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
os.fchmod(fd, 0o600) # open() mode applies at creation only
with os.fdopen(fd, "wb") as fh:
fh.write(data)
finally:
pass
def secret_value(name: str, key: str) -> str:
cp = run(["kubectl", "-n", NAMESPACE, "get", "secret", name, "-o", f"jsonpath={{{key}}}"])
if cp.returncode != 0:
die(f"cannot read secret {NAMESPACE}/{name}: {cp.stderr.strip()}")
raw = cp.stdout.strip()
if not raw:
die(f"secret {NAMESPACE}/{name} has no key {key}")
return base64.b64decode(raw).decode()
def load_env_file(path: Path = REPO / ".env") -> None:
"""Load KEY=VALUE lines from .env without overriding the real environment.
The file is git-ignored and holds the bootstrap token a human pastes in.
Sourcing it in the shell would work too, but then every phase depends on the
caller remembering to do it.
"""
if not path.exists():
return
for line in path.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
key = key.strip()
value = value.strip().strip('"').strip("'")
if key and key not in os.environ:
os.environ[key] = value
def bootstrap_token() -> str | None:
"""Resolve the iam-admin PAT: environment, then cached file, then cluster.
Order matters. ZITADEL_BOOTSTRAP_TOKEN in .env is filled by a human after the
first sync and wins. The cached file under ~/.config/zitadel keeps a run
working when the API server is unreachable. The cluster Secret is the last
resort and the only source that always reflects a rotation.
"""
token = os.environ.get("ZITADEL_BOOTSTRAP_TOKEN", "").strip()
if token:
return token
cached = CONF_DIR / "iam-admin.pat"
if cached.exists():
return cached.read_text().strip()
try:
return secret_value("iam-admin-pat", ".data.pat").strip()
except SystemExit:
return None
def curl_json(method: str, url: str, token: str | None = None, body: dict | None = None,
timeout: int = 20) -> tuple[int, str]:
"""Call the API without putting a secret in the process argv.
`curl --config -` reads the options from stdin, so neither the bearer token
nor the password hash of a user import shows up in `ps`.
"""
def esc(value: str) -> str:
return value.replace("\\", "\\\\").replace('"', '\\"')
cfg = [f'url = "{esc(url)}"', f'request = "{method}"', "silent", "show-error",
f"max-time = {timeout}", 'write-out = "\\n%{http_code}"',
'header = "Content-Type: application/json"']
# Reaching the instance by name needs either public DNS through the tunnel
# or a pinned address. ZITADEL picks the instance from the Host header, so a
# plain port-forward fails with "Instance not found". Set this to bypass DNS:
# ZITADEL_CURL_RESOLVE=zitadel.riotpiao.com:443:192.168.1.160
pinned = os.environ.get("ZITADEL_CURL_RESOLVE", "").strip()
if pinned:
cfg.append(f'resolve = "{esc(pinned)}"')
if token:
cfg.append(f'header = "Authorization: Bearer {esc(token)}"')
if body is not None:
cfg.append(f'data = "{esc(json.dumps(body))}"')
cp = run(["curl", "--config", "-"], input="\n".join(cfg) + "\n")
if cp.returncode != 0:
die(f"{method} {url} failed: {cp.stderr.strip()}")
payload, _, code = cp.stdout.rpartition("\n")
return int(code.strip() or 0), payload
# ── phase: keys ────────────────────────────────────────────────────────────
def phase_keys(_args: argparse.Namespace) -> int:
write_private(CONF_DIR / "iam-admin.json", secret_value("iam-admin", ".data.iam-admin\\.json").encode())
write_private(CONF_DIR / "iam-admin.pat", secret_value("iam-admin-pat", ".data.pat").strip().encode())
print(f"machine key -> {CONF_DIR / 'iam-admin.json'} (0600)")
print(f"PAT -> {CONF_DIR / 'iam-admin.pat'} (0600)")
print("both files stay outside the repo; never commit them")
return 0
# ── phase: apply ───────────────────────────────────────────────────────────
def _api(method: str, path: str, token: str, body: dict | None = None,
timeout: int = 30) -> tuple[int, dict]:
code, payload = curl_json(method, f"{BASE_URL}{path}", token=token, body=body, timeout=timeout)
try:
return code, (json.loads(payload) if payload.strip() else {})
except json.JSONDecodeError:
return code, {"_raw": payload}
def _ensure_project(token: str, name: str) -> str:
code, d = _api("POST", "/management/v1/projects/_search", token, {})
for row in d.get("result") or []:
if row.get("name") == name:
return row["id"]
code, d = _api("POST", "/management/v1/projects", token, {"name": name})
if code not in (200, 201) or not d.get("id"):
die(f"create project {name} failed: {code} {d}")
return d["id"]
def _ensure_roles(token: str, project_id: str, roles: dict[str, str]) -> None:
code, d = _api("POST", f"/management/v1/projects/{project_id}/roles/_search", token, {})
have = {r.get("roleKey") for r in (d.get("result") or [])}
for key, display in roles.items():
if key in have:
continue
code, d = _api("POST", f"/management/v1/projects/{project_id}/roles", token,
{"roleKey": key, "displayName": display, "group": PROJECT_GROUP})
if code not in (200, 201):
die(f"create role {key} failed: {code} {d}")
def _ensure_oidc_apps(token: str, project_id: str, apps: dict[str, dict]) -> tuple[dict, dict]:
code, d = _api("POST", f"/management/v1/projects/{project_id}/apps/_search", token, {})
existing = {a.get("name"): a for a in (d.get("result") or [])}
ids: dict[str, str] = {}
secrets: dict[str, str] = {}
for name, spec in apps.items():
row = existing.get(name)
if row:
ids[name] = (row.get("oidcConfig") or {}).get("clientId", "")
app_id = row.get("id")
else:
body = {
"name": name,
"redirectUris": spec["redirect_uris"],
"responseTypes": ["OIDC_RESPONSE_TYPE_CODE"],
"grantTypes": ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE",
"OIDC_GRANT_TYPE_REFRESH_TOKEN"],
"appType": spec.get("app_type_id", "OIDC_APP_TYPE_WEB"),
"authMethodType": spec.get("auth_method_id", "OIDC_AUTH_METHOD_TYPE_BASIC"),
"devMode": bool(spec.get("dev_mode")),
"accessTokenType": "OIDC_TOKEN_TYPE_BEARER",
"version": "OIDC_VERSION_1_0",
"idTokenRoleAssertion": True,
}
code, d = _api("POST", f"/management/v1/projects/{project_id}/apps/oidc", token, body)
if code not in (200, 201) or not d.get("appId"):
die(f"create app {name} failed: {code} {d}")
app_id = d["appId"]
ids[name] = d.get("clientId", "")
if spec.get("auth_method_id") == "OIDC_AUTH_METHOD_TYPE_NONE":
continue
code, d = _api("POST",
f"/management/v1/projects/{project_id}/apps/{app_id}"
"/oidc_config/_generate_client_secret", token, {})
if code in (200, 201) and d.get("clientSecret"):
secrets[name] = d["clientSecret"]
return ids, secrets
def phase_apply(_args: argparse.Namespace) -> int:
"""Declare the ZITADEL objects through the API. No Terraform, no state drift."""
token = bootstrap_token()
if not token:
die("no iam-admin PAT — run the keys phase first")
project_id = _ensure_project(token, PROJECT_NAME)
print(f"project : {PROJECT_NAME} ({project_id})")
_ensure_roles(token, project_id, ROLES)
print(f"roles : {len(ROLES)} ensured")
ids, secrets = _ensure_oidc_apps(token, project_id, OIDC_APPS)
print(f"oidc apps : {len(ids)} ensured, {len(secrets)} secrets issued")
state = {
"project_id": {"value": project_id},
"client_ids": {"value": ids},
"client_secrets": {"value": secrets},
"machine_keys": {"value": {}},
"groups_claim_enabled": {"value": False},
}
write_private(STATE_FILE, json.dumps(state, indent=2, sort_keys=True).encode())
print(f"state -> {STATE_FILE} (0600)")
return 0
# ── phase: secrets ─────────────────────────────────────────────────────────
# client -> [(sops file, Secret name inside the file, key, kind)]
# Only secrets with a git-tracked SOPS file appear here. Every other client is
# printed as manual: guessing a path writes a credential into the wrong Secret.
SECRET_MAP = {
"grafana": [("grafana-oidc-secrets.enc.yaml", "grafana-oidc",
"GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET", "secret")],
"minio": [("minio-secrets.enc.yaml", "minio-oidc",
"MINIO_IDENTITY_OPENID_CLIENT_SECRET", "secret")],
"paperless-ai-agent": [("paperless-ai-secrets.enc.yaml", "paperless-ai-config",
"LLM_AUTH_CLIENT_SECRET", "secret")],
}
# Clients whose client_id lives in a plain (unencrypted) manifest. They need the
# id update as much as the secret: ZITADEL generates a new client_id for every
# application, so a secret-only re-point yields invalid_client.
CLIENT_ID_MAP = {
"grafana": ("k8s/infra/logging/grafana-values.yaml",
r"(?m)^(\s*client_id:\s*)\S+\s*$", r"\g<1>{value}"),
"minio": ("k8s/infra/minio/minio-tenant.yaml",
r"(?m)^(\s*value:\s*)\"minio\"\s*$", r'\g<1>"{value}"'),
"paperless-ai-agent": ("k8s/apps/paperless/paperless-ai.yaml",
r"client_id=paperless-ai-agent", "client_id={value}"),
}
# Secrets that a live workload mounts but no SOPS file in git defines. The phase
# cannot re-point them safely, so it names them.
MANUAL_ONLY = {
"homarr": "Secret homarr-oidc (keys client-id, client-secret) has no SOPS file in git — "
"adopt it first, then add it to SECRET_MAP. The repo also carries an orphan "
"homarr-auth-oidc.enc.yaml that nothing consumes.",
}
def state_outputs() -> dict:
"""Read the object map written by the apply phase."""
if not STATE_FILE.exists():
die(f"{STATE_FILE} missing — run the apply phase first")
return json.loads(STATE_FILE.read_text())
def sops_env() -> dict:
env = dict(os.environ)
env.setdefault("SOPS_AGE_KEY_FILE", str(Path.home() / ".sops" / "key.txt"))
return env
def write_secret(rel_path: str, secret_name: str, key: str, value: str,
dry_run: bool) -> None:
path = SOPS_DIR / rel_path
if not path.exists():
die(f"{path} does not exist — the mapping and the repo disagree")
env = sops_env()
dec = run(["sops", "--config", str(SOPS_CONFIG), "decrypt", str(path)], env=env)
if dec.returncode != 0:
die(f"cannot decrypt {rel_path}: {dec.stderr.strip()}")
docs = [d for d in yaml.safe_load_all(dec.stdout) if d]
matching = [d for d in docs if (d.get("metadata") or {}).get("name") == secret_name]
if len(matching) != 1:
die(f"{rel_path}: expected exactly one document named {secret_name}, found {len(matching)}")
current = (matching[0].get("stringData") or {}).get(key)
if current == value:
print(f" unchanged {rel_path}:{secret_name}:{key}")
return
if dry_run:
print(f" would set {rel_path}:{secret_name}:{key} (len {len(value)})")
return
if len(docs) == 1:
cp = run(["sops", "--config", str(SOPS_CONFIG), "set", "--value-stdin", "--idempotent",
str(path), f'["stringData"]["{key}"]'], env=env, input=json.dumps(value))
if cp.returncode != 0:
die(f"sops set failed on {rel_path}: {cp.stderr.strip()}")
print(f" wrote {rel_path}:{secret_name}:{key}")
return
# Multi-document file. `sops set` cannot address one document, so the whole
# file is rewritten. sops encrypt reads the creation rule from the FILENAME,
# so the temp path is overridden with the repo-relative path; without that
# override sops finds no matching rule and refuses.
matching[0].setdefault("stringData", {})[key] = value
plain = "---\n".join(yaml.safe_dump(d, sort_keys=False) for d in docs)
fd, tmp = tempfile.mkstemp(suffix=".yaml", dir=tempfile.gettempdir())
_TEMP_FILES.add(tmp)
try:
with os.fdopen(fd, "w") as fh:
fh.write(plain)
enc = run(["sops", "--config", str(SOPS_CONFIG), "encrypt",
"--filename-override", str(path), tmp], env=env)
if enc.returncode != 0:
die(f"sops encrypt failed for {rel_path}: {enc.stderr.strip()}")
mode = stat.S_IMODE(path.stat().st_mode)
fd2, staged = tempfile.mkstemp(suffix=".enc.yaml", dir=str(path.parent))
try:
with os.fdopen(fd2, "w") as fh:
fh.write(enc.stdout)
os.chmod(staged, mode)
os.replace(staged, path) # same directory: atomic, never EXDEV
finally:
if os.path.exists(staged):
os.unlink(staged)
print(f" wrote {rel_path}:{secret_name}:{key} (multi-document rewrite)")
finally:
if os.path.exists(tmp):
os.unlink(tmp)
_TEMP_FILES.discard(tmp)
def update_client_id(client: str, value: str, dry_run: bool, apply_config: bool) -> None:
spec = CLIENT_ID_MAP.get(client)
if not spec:
return
rel_path, pattern, repl = spec
path = REPO / rel_path
if not path.exists():
die(f"{path} does not exist — the client-id mapping is stale")
text = path.read_text()
matches = list(re.finditer(pattern, text))
if len(matches) != 1:
die(f"{rel_path}: expected 1 match for the {client} client id, found {len(matches)}")
new_line = matches[0].expand(repl.format(value=value))
if matches[0].group(0) == new_line:
print(f" unchanged {rel_path}: {client} client id")
return
if dry_run or not apply_config:
print(f" would change {rel_path}: {matches[0].group(0).strip()} -> {new_line.strip()}")
if not apply_config:
print(" re-run with --apply-config to write this plaintext manifest edit")
return
path.write_text(text[:matches[0].start()] + new_line + text[matches[0].end():])
print(f" wrote {rel_path}: {client} client id")
def phase_secrets(args: argparse.Namespace) -> int:
need("sops")
out = state_outputs()
ids = out.get("client_ids", {}).get("value", {})
secrets = out.get("client_secrets", {}).get("value", {})
machine_keys = out.get("machine_keys", {}).get("value", {})
if not ids and not machine_keys:
die("no ZITADEL objects recorded yet — run the apply phase first")
print("[1/3] client secrets into SOPS")
for client, entries in SECRET_MAP.items():
for rel_path, secret_name, key, kind in entries:
value = ids.get(client) if kind == "id" else secrets.get(client)
if value:
write_secret(rel_path, secret_name, key, value, args.dry_run)
print("[2/3] client ids in plain manifests")
for client in list(CLIENT_ID_MAP) + list(SECRET_MAP):
if ids.get(client):
update_client_id(client, ids[client], args.dry_run, args.apply_config)
print("[3/3] machine keys")
key_dir = CONF_DIR / "machine-keys"
for name, key_json in machine_keys.items():
if args.dry_run:
print(f" would write {key_dir / (name + '.json')}")
else:
write_private(key_dir / f"{name}.json", key_json.encode())
print(f" wrote {key_dir / (name + '.json')} (0600)")
unhandled = sorted(set(ids) - set(SECRET_MAP) - set(CLIENT_ID_MAP) - set(MANUAL_ONLY))
if MANUAL_ONLY or unhandled:
print("\n MANUAL — no safe write path exists for these clients:")
for client, why in MANUAL_ONLY.items():
print(f" {client}: {why}")
for client in unhandled:
print(f" {client}: set its client_id and secret in its own manifest or Secret file")
return 0
# ── phase: users ───────────────────────────────────────────────────────────
HASH_RE = re.compile(r"^pbkdf2_sha256\$(?P<rounds>\d+)\$(?P<salt>[^$]+)\$(?P<hash>[^$]+)$")
def convert_hash(django_hash: str) -> str:
"""Django pbkdf2_sha256 -> passwap MCF $pbkdf2-sha256$...
ZITADEL imports one MCF string via hashedPassword.hash (max 200 chars) and
pins passwap, whose pbkdf2 format is $pbkdf2-sha256$<rounds>$<salt>$<hash>
with alt-base64 (standard base64, + -> ., no padding). Rounds, salt and hash
bytes are unchanged, so the password still verifies.
"""
m = HASH_RE.match(django_hash.strip())
if not m:
die("unsupported hash format — only pbkdf2_sha256 is convertible")
alt = m.group("hash").rstrip("=").replace("+", ".")
mcf = f"$pbkdf2-sha256${m.group('rounds')}${m.group('salt')}${alt}"
if len(mcf) > 200:
die(f"converted hash is {len(mcf)} chars, ZITADEL accepts 200")
return mcf
def phase_users(args: argparse.Namespace) -> int:
token = bootstrap_token()
if not token:
die("no bootstrap token — set ZITADEL_BOOTSTRAP_TOKEN in .env, or run the keys phase")
if not args.django_hash:
die("pass --django-hash with the pbkdf2_sha256 value exported from Authentik")
body = {
"username": args.username,
"profile": {"givenName": args.given_name, "familyName": args.family_name},
"email": {"email": args.email, "isVerified": True},
"hashedPassword": {"hash": convert_hash(args.django_hash)},
"passwordChangeRequired": False,
}
code, payload = curl_json("POST", f"{BASE_URL}/v2/users/human", token=token, body=body)
if code in (200, 201):
print(f" {args.username}: created")
return 0
if code == 409:
print(f" {args.username}: already exists — the password was NOT updated")
return 1
die(f"create {args.username} -> {code} {payload[:300]}")
# ── phase: verify ──────────────────────────────────────────────────────────
def phase_verify(_args: argparse.Namespace) -> int:
code, _ = curl_json("GET", f"{BASE_URL}/.well-known/openid-configuration", timeout=10)
print(f"discovery : {code}")
if code != 200:
die("discovery endpoint is not 200 — the instance or the ingress is not ready")
out = state_outputs()
ids = out.get("client_ids", {}).get("value", {})
print(f"client apps : {len(ids)}")
enabled = bool(out.get("groups_claim_enabled", {}).get("value"))
print(f"groups action : {enabled}")
if not enabled:
print("BLOCKED for cutover: the groups claim action is disabled. Every consumer")
print("that reads `groups` (kube-apiserver, MinIO, Grafana) sees no groups.")
print("Enable it only after the live gate passes, with the script path set.")
return 0
PHASES = {"keys": phase_keys, "apply": phase_apply, "secrets": phase_secrets,
"users": phase_users, "verify": phase_verify}
def main() -> int:
parser = argparse.ArgumentParser(description="Provision ZITADEL end to end.")
parser.add_argument("phase", choices=[*PHASES, "all"])
parser.add_argument("--org-id", help="ZITADEL org id (required for apply)")
parser.add_argument("--dry-run", action="store_true", help="print actions, write nothing")
parser.add_argument("--apply-config", action="store_true",
help="also rewrite client ids inside plain manifests")
parser.add_argument("--username", default="rock")
parser.add_argument("--email", default="[email protected]")
parser.add_argument("--given-name", default="Rock")
parser.add_argument("--family-name", default="Liang")
parser.add_argument("--django-hash", help="pbkdf2_sha256 value exported from Authentik")
args = parser.parse_args()
load_env_file()
need("kubectl")
if args.phase == "all":
if not args.org_id:
die("all requires --org-id")
for name in ("keys", "apply"):
rc = PHASES[name](args)
if rc:
return rc
rc = phase_secrets(args)
if rc:
return rc
return phase_verify(args)
return PHASES[args.phase](args)
if __name__ == "__main__":
sys.exit(main())
+51
View File
@@ -0,0 +1,51 @@
# Cloudflare side of the edge.
#
# The tunnel itself is remotely managed today: its token lives in
# k8s/apps/cloudflared and the public hostnames are configured in the Zero
# Trust dashboard. Terraform can own the DNS records without owning the tunnel,
# which is the safe half: a record change is reversible, a tunnel config rewrite
# removes every hostname the plan does not list.
locals {
# CNAME target of every remotely-managed tunnel.
tunnel_target = "${var.tunnel_id}.cfargotunnel.com"
}
resource "cloudflare_dns_record" "tunnel" {
for_each = toset(var.cloudflare_tunnel_hostnames)
zone_id = var.cloudflare_zone_id
name = each.value
type = "CNAME"
content = local.tunnel_target
proxied = var.cloudflare_dns_proxied
ttl = var.cloudflare_dns_ttl
comment = "cloudflared tunnel; managed by terraform/edge"
# A proxied CNAME must point at the tunnel target. A CNAME to any other
# hostname breaks the tunnel mapping.
lifecycle {
precondition {
condition = var.cloudflare_dns_proxied
error_message = "Tunnel CNAMEs must be proxied; an unproxied record bypasses the tunnel."
}
}
}
# Optional: take the tunnel configuration into Terraform. Off by default.
# Applying it REPLACES the whole ingress list. Import first:
# terraform import cloudflare_zero_trust_tunnel_cloudflared_config.this \
# <account_id>/<tunnel_id>
resource "cloudflare_zero_trust_tunnel_cloudflared_config" "this" {
count = var.manage_tunnel_config ? 1 : 0
account_id = var.cloudflare_account_id
tunnel_id = var.tunnel_id
config = {
ingress = concat(
[for host, service in var.tunnel_ingress : { hostname = host, service = service }],
[{ service = "http_status:404" }],
)
}
}
+40
View File
@@ -0,0 +1,40 @@
output "cloudflare_records" {
description = "Hostnames now managed in the Cloudflare zone."
value = sort(keys(cloudflare_dns_record.tunnel))
}
output "tunnel_config_managed" {
description = "Whether Terraform owns the tunnel ingress list. False means the dashboard still does."
value = var.manage_tunnel_config
}
output "relay_public_ip" {
description = "Public address of the Tencent relay. Point the .cn DNS at it, or let the module do it."
value = tencentcloud_instance.relay.public_ip
}
output "china_hostnames" {
description = "Fully qualified China hostnames served by the relay."
value = [for host in var.china_hostnames : "${host}.${var.china_domain}"]
}
output "wireguard_peer" {
description = "Peer stanza for the homelab side. Add it to the cluster's wg0.conf; the relay already trusts this public key."
value = <<-EOT
[Peer]
PublicKey = <relay public key; see /etc/wireguard/wg0.conf on the relay>
Endpoint = ${tencentcloud_instance.relay.public_ip}:${var.wireguard_port}
AllowedIPs = ${var.wireguard_relay_address}
PersistentKeepalive = 25
EOT
}
output "next_manual_steps" {
description = "What this module cannot do for you."
value = [
"Confirm the ICP filing for ${var.china_domain} if the relay is in a mainland region.",
"Issue a certificate for the China hostnames on the relay (Caddy ACME needs the DNS-01 challenge for a .cn domain).",
"Run a WireGuard peer in the cluster that dials ${tencentcloud_instance.relay.public_ip}:${var.wireguard_port}.",
"Add the .cn origins to the OIDC client redirect URIs before any China user logs in.",
]
}
+47
View File
@@ -0,0 +1,47 @@
# terraform/edge — the China-facing edge, plus DNS ownership for the existing zone
#
# Purpose: give users in mainland China a short, stable path to the homelab
# without touching the Cloudflare path that serves everyone else.
#
# China user -> <service>.<china_domain> -> Tencent CVM (this module)
# -> WireGuard tunnel -> ingress-nginx (192.168.1.160) -> service
#
# Everyone else -> Cloudflare -> cloudflared tunnel -> same ingress (unchanged)
#
# The homelab still opens no inbound port: the tunnel is established outbound
# from the cluster to the CVM.
#
# Credentials come from the environment, never from a file in git:
# export CLOUDFLARE_API_TOKEN=...
# export TENCENTCLOUD_SECRET_ID=...
# export TENCENTCLOUD_SECRET_KEY=...
# terraform -chdir=terraform/edge init
# terraform -chdir=terraform/edge plan -var china_domain=example.cn ...
#
# State is local and git-ignored. It carries no
# credential today, but it will once InstanceData or certificates are added, so
# treat it as a secret and move it to MinIO before a second operator applies.
terraform {
required_version = ">= 1.6.0"
required_providers {
cloudflare = {
source = "cloudflare/cloudflare"
version = "5.28.1"
}
tencentcloud = {
source = "tencentcloudstack/tencentcloud"
version = "1.83.38"
}
}
}
provider "cloudflare" {
api_token = var.cloudflare_api_token
}
provider "tencentcloud" {
secret_id = var.tencent_secret_id
secret_key = var.tencent_secret_key
region = var.tencent_region
}
+22
View File
@@ -0,0 +1,22 @@
# Rendered by terraform/edge. Do not edit on the instance.
#
# Caddy terminates TLS for the .cn hostnames and forwards over the WireGuard
# tunnel to ingress-nginx. flush_interval -1 disables response buffering so
# Server-Sent Events reach the client chunk-by-chunk.
{
email ${acme_email}
admin off
}
${hosts} {
reverse_proxy ${ingress_upstream} {
flush_interval -1
}
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains"
X-Content-Type-Options "nosniff"
X-Frame-Options "DENY"
Referrer-Policy "strict-origin-when-cross-origin"
}
}
+169
View File
@@ -0,0 +1,169 @@
# The relay: one small CVM in Tencent Cloud that terminates TLS for the .cn
# domain and forwards over WireGuard to the homelab ingress.
#
# Why a relay and not Cloudflare: mainland clients reach a Tencent edge far
# faster than a Cloudflare edge, and Cloudflare's China network is an
# Enterprise product. The homelab keeps zero inbound exposure because the
# tunnel is dialled outbound from the cluster.
#
# ICP FILING: a mainland resource serving HTTP/HTTPS for a domain requires an
# ICP filing naming that domain. Without it this instance accepts the
# connection and the mainland network resets the HTTP layer. Put the relay in a
# non-mainland region (ap-hongkong, ap-singapore) when no filing exists.
resource "tencentcloud_vpc" "edge" {
name = "${replace(var.china_domain, ".", "_")}_edge"
cidr_block = var.edge_vpc_cidr
}
resource "tencentcloud_subnet" "edge" {
name = "${replace(var.china_domain, ".", "_")}_edge"
vpc_id = tencentcloud_vpc.edge.id
availability_zone = var.tencent_availability_zone
cidr_block = var.edge_subnet_cidr
}
resource "tencentcloud_key_pair" "edge" {
key_name = "${replace(var.china_domain, ".", "_")}_edge"
public_key = var.relay_ssh_public_key
}
resource "tencentcloud_security_group" "edge" {
name = "${replace(var.china_domain, ".", "_")}_edge"
description = "TLS + WireGuard for the ${var.china_domain} relay; SSH from the operator only"
}
resource "tencentcloud_security_group_rule" "ssh" {
security_group_id = tencentcloud_security_group.edge.id
type = "ingress"
ip_protocol = "tcp"
port_range = "22"
cidr_ip = var.relay_admin_cidr
policy = "accept"
}
resource "tencentcloud_security_group_rule" "tls" {
security_group_id = tencentcloud_security_group.edge.id
type = "ingress"
ip_protocol = "tcp"
port_range = "80,443"
cidr_ip = "0.0.0.0/0"
policy = "accept"
}
resource "tencentcloud_security_group_rule" "wireguard" {
security_group_id = tencentcloud_security_group.edge.id
type = "ingress"
ip_protocol = "udp"
port_range = tostring(var.wireguard_port)
# The homelab dials outbound and has no static public address, so this is
# 0.0.0.0/0 by default. WireGuard authenticates every packet by public key,
# and an unauthenticated packet is dropped before it reaches the tunnel.
# Narrow this to the homelab egress CIDR when that address is stable.
cidr_ip = var.wireguard_allowed_cidr
policy = "accept"
}
# cloud-init: WireGuard listener, forwarding sysctls, and Caddy for TLS.
# The relay never holds application credentials; it forwards bytes.
#
# NOTE ON THE PRIVATE KEY: user_data is visible in the instance metadata and in
# Terraform state. That is a known limitation of bootstrapping WireGuard this
# way. The argument is wrapped in sensitive() to keep it out of plan/apply
# output, but anyone with console or state access can read it. Rotate the key
# if the state or the instance metadata is ever exposed.
locals {
china_fqdns = [for host in var.china_hostnames : "${host}.${var.china_domain}"]
caddyfile = templatefile("${path.module}/templates/Caddyfile.tpl", {
acme_email = var.acme_email
hosts = join(", ", local.china_fqdns)
ingress_upstream = var.ingress_upstream
})
relay_user_data = <<-EOT
#cloud-config
package_update: true
package_upgrade: false
manage_etc_hosts: true
timezone: Asia/Shanghai
packages: [wireguard, wireguard-tools, caddy]
write_files:
- path: /etc/wireguard/wg0.conf
permissions: "0600"
content: |
[Interface]
Address = ${var.wireguard_relay_address}
ListenPort = ${var.wireguard_port}
PrivateKey = ${var.wireguard_relay_private_key}
# Masquerade forwarded traffic so the homelab peer only has to accept
# packets sourced from the relay's tunnel address. Without this the
# homelab peer's AllowedIPs would drop the original client source IP.
PostUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE
[Peer]
PublicKey = ${var.wireguard_homelab_public_key}
AllowedIPs = ${var.wireguard_homelab_allowed_ips}
PersistentKeepalive = 25
- path: /etc/sysctl.d/99-wireguard.conf
content: |
net.ipv4.ip_forward = 1
- path: /etc/caddy/Caddyfile
permissions: "0644"
content: ${jsonencode(local.caddyfile)}
runcmd:
- sysctl --system
- systemctl enable --now wg-quick@wg0
- caddy validate --config /etc/caddy/Caddyfile
- systemctl enable --now caddy
- systemctl restart caddy
EOT
}
resource "tencentcloud_instance" "relay" {
instance_name = "${replace(var.china_domain, ".", "_")}_edge"
availability_zone = var.tencent_availability_zone
image_id = var.relay_image_id
instance_type = var.relay_instance_type
vpc_id = tencentcloud_vpc.edge.id
subnet_id = tencentcloud_subnet.edge.id
security_groups = [tencentcloud_security_group.edge.id]
key_ids = [tencentcloud_key_pair.edge.id]
allocate_public_ip = true
# Outbound only needs a few Mbps; inbound TLS is the same order.
internet_max_bandwidth_out = var.relay_internet_max_bandwidth_out
system_disk_type = var.relay_system_disk_type
system_disk_size = var.relay_system_disk_size
# The private key is embedded in this bootstrap script. sensitive() keeps it
# out of CLI output; it does not remove it from state or instance metadata.
user_data = sensitive(local.relay_user_data)
tags = {
role = "china-edge"
domain = var.china_domain
}
# Accidental `terraform destroy` would take the only China-facing edge down.
# Remove this block deliberately when a replacement path exists.
lifecycle {
prevent_destroy = true
}
}
# DNS for the .cn domain lives at DNSPod (Tencent). Each subdomain resolves to
# the relay's public address, and the relay proxies to the homelab over the
# tunnel. The "中国境内" line is the default line; no separate line records.
resource "tencentcloud_dnspod_record" "china" {
for_each = toset(var.china_hostnames)
domain = var.china_domain
sub_domain = each.value
record_type = "A"
record_line = var.china_dns_record_line
value = tencentcloud_instance.relay.public_ip
ttl = var.china_dns_ttl
remark = "China edge relay; origin is the homelab ingress over WireGuard"
}
+45
View File
@@ -0,0 +1,45 @@
# Copy to terraform.tfvars.local (git-ignored) and fill in.
# Secrets never belong in a committed file: pass them as environment variables
# instead — TF_VAR_wireguard_relay_private_key, CLOUDFLARE_API_TOKEN,
# TENCENTCLOUD_SECRET_ID, TENCENTCLOUD_SECRET_KEY.
# ── Cloudflare (existing zone) ───────────────────────────────────────────────
cloudflare_zone_id = "REPLACE_WITH_ZONE_ID"
tunnel_id = "REPLACE_WITH_TUNNEL_UUID"
# Adopt existing records with `terraform import` before the first apply, or the
# create call collides with the hand-made record of the same name:
# terraform import 'cloudflare_dns_record.tunnel["zitadel"]' \
# <zone_id>/<record_id>
#
# A tunnel CNAME is only half of the path. The tunnel's Public Hostname list
# (Zero Trust -> Networks -> Tunnels -> Public Hostname) must also map the name
# to https://<name>.riotpiao.com, or Cloudflare answers 404 from the tunnel even
# though DNS resolves.
cloudflare_tunnel_hostnames = ["paperless", "img", "poimen", "zitadel"]
# Leave false until the tunnel config import is understood. True makes
# Terraform replace the entire ingress list with tunnel_ingress.
manage_tunnel_config = false
tunnel_ingress = {}
# ── China edge (Tencent Cloud) ───────────────────────────────────────────────
china_domain = "REPLACE.example.cn"
china_hostnames = ["paperless", "img"]
# Mainland regions require an ICP filing for the domain. Use ap-hongkong or
# ap-singapore when no filing exists.
tencent_region = "ap-guangzhou"
tencent_availability_zone = "ap-guangzhou-6"
relay_instance_type = "S5.SMALL2"
relay_image_id = "REPLACE_WITH_IMAGE_ID"
relay_ssh_public_key = "ssh-ed25519 AAAA... operator@laptop"
relay_admin_cidr = "REPLACE_WITH_OPERATOR_CIDR"
# ── WireGuard between the relay and the cluster ──────────────────────────────
# The relay private key comes from `wg genkey`. Keep it in the environment, or
# in a SOPS file mounted for the run. It is not written by this module.
wireguard_port = 51820
wireguard_homelab_public_key = "REPLACE_WITH_HOMELAB_WG_PUBKEY"
wireguard_relay_address = "10.88.0.1/24"
wireguard_homelab_address = "10.88.0.2/24"
+227
View File
@@ -0,0 +1,227 @@
variable "cloudflare_api_token" {
description = "Cloudflare API token with Zone:DNS:Edit and, for tunnel config, Account:Cloudflare Tunnel:Edit."
type = string
sensitive = true
default = null
}
variable "cloudflare_zone_id" {
description = "Zone ID of the existing public zone (riotpiao.com)."
type = string
}
variable "cloudflare_account_id" {
description = "Account ID. Required only when manage_tunnel_config is true."
type = string
default = null
}
variable "tunnel_id" {
description = "UUID of the cloudflared tunnel that the DNS records point at."
type = string
}
# The dashboard remains the source of truth for the remotely-managed tunnel
# today. Setting this true hands that config to Terraform, which then rewrites
# the whole ingress list: every hostname not listed in tunnel_ingress is
# removed from the tunnel. Import first and read the plan before applying.
variable "manage_tunnel_config" {
description = "Take ownership of the remotely-managed tunnel configuration."
type = bool
default = false
}
variable "tunnel_ingress" {
description = "hostname => origin service, used only when manage_tunnel_config is true."
type = map(string)
default = {}
}
# ── DNS records managed in the Cloudflare zone ───────────────────────────────
# These names already exist, created by hand in the dashboard. Adopt them with
# `terraform import` before the first apply, or Terraform will try to create a
# second record with the same name and Cloudflare will reject it.
variable "cloudflare_tunnel_hostnames" {
description = "Hostnames in the Cloudflare zone that must CNAME to the tunnel."
type = list(string)
default = []
validation {
condition = alltrue([for h in var.cloudflare_tunnel_hostnames : can(regex("^[a-zA-Z0-9._-]+$", h))])
error_message = "cloudflare_tunnel_hostnames entries must be DNS labels or relative names (letters, digits, dot, underscore, hyphen)."
}
}
variable "cloudflare_dns_ttl" {
description = "TTL for tunnel CNAME records. 1 means automatic, which is required when proxied is true."
type = number
default = 1
}
variable "cloudflare_dns_proxied" {
description = "Whether tunnel CNAME records are proxied through Cloudflare. Must stay true for a tunnel origin."
type = bool
default = true
}
# ── China edge ───────────────────────────────────────────────────────────────
variable "china_domain" {
description = "The .cn domain served from Tencent Cloud, without a trailing dot."
type = string
}
variable "china_hostnames" {
description = "Subdomains of china_domain that point at the Tencent relay."
type = list(string)
default = ["paperless", "img"]
}
variable "tencent_secret_id" {
description = "Tencent Cloud SecretId. Prefer TENCENTCLOUD_SECRET_ID in the environment."
type = string
sensitive = true
default = null
}
variable "tencent_secret_key" {
description = "Tencent Cloud SecretKey. Prefer TENCENTCLOUD_SECRET_KEY in the environment."
type = string
sensitive = true
default = null
}
variable "tencent_region" {
description = "Tencent Cloud region for the relay. Mainland regions need an ICP filing for the domain."
type = string
default = "ap-guangzhou"
}
variable "tencent_availability_zone" {
description = "Availability zone inside tencent_region."
type = string
default = "ap-guangzhou-6"
}
variable "relay_instance_type" {
description = "CVM spec for the relay. A relay only terminates TLS and forwards bytes."
type = string
default = "S5.SMALL2"
}
variable "relay_image_id" {
description = "CVM image ID. Find one with: tencentcloud images --image-name 'Ubuntu 24.04' (or the console)."
type = string
}
variable "relay_ssh_public_key" {
description = "Public key injected into the relay for break-glass SSH. Never the private key."
type = string
}
variable "relay_admin_cidr" {
description = "CIDR allowed to reach SSH and WireGuard on the relay. Keep it to the operator, not 0.0.0.0/0."
type = string
validation {
condition = var.relay_admin_cidr != "0.0.0.0/0" && var.relay_admin_cidr != "::/0"
error_message = "relay_admin_cidr must not be the whole internet. Set the operator's egress CIDR."
}
}
variable "wireguard_port" {
description = "UDP port of the WireGuard listener on the relay."
type = number
default = 51820
}
variable "wireguard_allowed_cidr" {
description = "CIDR allowed to reach the WireGuard UDP port. The homelab dials outbound, so this is often 0.0.0.0/0; WireGuard authenticates by key. Restrict it to the homelab egress CIDR when that address is static."
type = string
default = "0.0.0.0/0"
}
variable "wireguard_relay_private_key" {
description = "WireGuard private key of the relay side. Generate with `wg genkey`; pass via TF_VAR_wireguard_relay_private_key."
type = string
sensitive = true
}
variable "wireguard_homelab_public_key" {
description = "WireGuard public key of the homelab peer that dials the relay."
type = string
}
variable "wireguard_relay_address" {
description = "Tunnel address of the relay side."
type = string
default = "10.88.0.1/24"
}
variable "wireguard_homelab_address" {
description = "Tunnel address of the homelab side."
type = string
default = "10.88.0.2/24"
}
# ── Relay networking / sizing ───────────────────────────────────────────────
variable "edge_vpc_cidr" {
description = "CIDR of the relay VPC."
type = string
default = "10.66.0.0/16"
}
variable "edge_subnet_cidr" {
description = "CIDR of the relay subnet."
type = string
default = "10.66.1.0/24"
}
variable "relay_internet_max_bandwidth_out" {
description = "Peak outbound bandwidth in Mbps. The relay only terminates TLS and forwards bytes."
type = number
default = 20
}
variable "relay_system_disk_type" {
description = "System disk type for the relay."
type = string
default = "CLOUD_PREMIUM"
}
variable "relay_system_disk_size" {
description = "System disk size in GB for the relay."
type = number
default = 50
}
# ── DNS records ──────────────────────────────────────────────────────────────
variable "china_dns_ttl" {
description = "TTL in seconds for the .cn A records."
type = number
default = 600
}
variable "china_dns_record_line" {
description = "DNSPod line for the .cn A records. Use 默认 for the default line."
type = string
default = "默认"
}
# ── Caddy reverse proxy ──────────────────────────────────────────────────────
variable "wireguard_homelab_allowed_ips" {
description = "AllowedIPs for the homelab peer. Must include the homelab LAN so Caddy can reach ingress-nginx over the tunnel."
type = string
default = "10.88.0.2/32,192.168.1.0/24"
}
variable "ingress_upstream" {
description = "Upstream URL Caddy proxies to over WireGuard (the homelab ingress-nginx)."
type = string
default = "http://192.168.1.160:80"
}
variable "acme_email" {
description = "Contact email for Caddy ACME (Let's Encrypt)."
type = string
default = "[email protected]"
}
+2
View File
@@ -23,6 +23,7 @@
# via nginx (not the backend service) is critical: a direct rewrite bypasses # via nginx (not the backend service) is critical: a direct rewrite bypasses
# nginx TLS and exposes each app's own self-signed cert, trusted by nothing. # nginx TLS and exposes each app's own self-signed cert, trusted by nothing.
rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name zitadel.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name minio-api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name minio-api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
@@ -39,6 +40,7 @@
rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name pi.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name igotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name igotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
-90
View File
@@ -1,90 +0,0 @@
# RECOVERED from live cluster state via talosctl get machineconfig.
# Regenerated after the original tfvars.local was lost/corrupted.
cluster_id = "Rtc4g2av9EP0mOdxA4M0-QQitzHLWICz-rLBNfrOjgw="
cluster_secret = "8E0CAeylAPKmmUEQmIGmHQAhQf+8c7NUf43CdrQZ+vg="
bootstrap_token = "b2q7lh.9w7hwgrulrd65gr3"
machine_token = "hq9wlf.96l9z46efd79jtr2"
machine_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQekNCOHFBREFnRUNBaEVBMkUwRWZqbG41L1J3eTVjMVJmNkNSakFGQmdNclpYQXdFREVPTUF3R0ExVUUKQ2hNRmRHRnNiM013SGhjTk1qWXdOekE1TURVd056VTRXaGNOTXpZd056QTJNRFV3TnpVNFdqQVFNUTR3REFZRApWUVFLRXdWMFlXeHZjekFxTUFVR0F5dGxjQU1oQUNwR3NQZkVHdEU4anVmNG9JTkNHNnlCQmN6aURFaEpLbDV3CnJib0hSR0tUbzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkdmaExZUUdaOGYzd3lZZAo0SFI3KzlONnZKQXJNQVVHQXl0bGNBTkJBQ1ZQVlAwcGYxMkdUakYvSHJMZmcwZHBLemdBMlE1OGR5Y0paY0ZvClQvNDdPQk8ra29VZm11dXNjVVNNQnBXS0VuWHNYMFNocmNab3hQd1FHM3diblFFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
machine_ca_key = "LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0KTUM0Q0FRQXdCUVlESzJWd0JDSUVJSlR4MXRqZEVOTTg1cGNRRFR0WWRtMFd0QXNBd0tzL1VESlZLN0ZqYU5uUgotLS0tLUVORCBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K"
kubernetes_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpVENDQVRDZ0F3SUJBZ0lSQUtwT1ZaK29CRzljNHFycEpwaUc4TkV3Q2dZSUtvWkl6ajBFQXdJd0ZURVQKTUJFR0ExVUVDaE1LYTNWaVpYSnVaWFJsY3pBZUZ3MHlOakEzTURrd05UQTNOVGhhRncwek5qQTNNRFl3TlRBMwpOVGhhTUJVeEV6QVJCZ05WQkFvVENtdDFZbVZ5Ym1WMFpYTXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVQ1VjJvNkliMUpRZkcza3lCTTBCeTRkd2FLbnlGY2tVdjNVem9yOG1Ba3RaN2JrT2ZKZDlmL2VmcVkKYXBzUFpLV1RHQnYxM3JZbFdvOGtuU3ZnNm83Um8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXdIUVlEVlIwbApCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPCkJCWUVGQ1dPZVJUVHdnN2tnNVNWMG9raFQwY0VDNGwzTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSURURCtNYXUKb2JXdkp6UkNMVEdJaHJHc1daalFnalVmRWl2MnhCTXB6RnNVQWlCQzNNWkYwMjVqVHk4Uno2YjI5czVJQmpkNgpZQTVWd0kvNVFieXlwSGFXQlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
kubernetes_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUrSEdPcUJJYXpJMzdqNmJJUlA1emVxeXEwZzhBU2xZeGplZUlJcEpIcXBvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFK1ZkcU9pRzlTVUh4dDVNZ1ROQWN1SGNHaXA4aFhKRkw5MU02Sy9KZ0pMV2UyNURueVhmWAovM242bUdxYkQyU2xreGdiOWQ2MkpWcVBKSjByNE9xTzBRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
etcd_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJmVENDQVNTZ0F3SUJBZ0lSQVAyaHkwdUhvTm5vSzQyZE9LTk1nU2t3Q2dZSUtvWkl6ajBFQXdJd0R6RU4KTUFzR0ExVUVDaE1FWlhSalpEQWVGdzB5TmpBM01Ea3dOVEEzTlRoYUZ3MHpOakEzTURZd05UQTNOVGhhTUE4eApEVEFMQmdOVkJBb1RCR1YwWTJRd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFRVGlKYUJpSEJPCmJha3JuL0dVdkUxVFd5czRVUkVzVGtVNEM4OG1EdWZYQURjV0NnN2RTRzc0QjkzOGFwSWgybGc4UDhKRDFFRUoKN0RkU1lQVG5zYWh1bzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSApBd0VHQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkliYTBLaEhmM3hhClBQNk1hb3dESUNWVXBLdDVNQW9HQ0NxR1NNNDlCQU1DQTBjQU1FUUNJRzZMYUJGeGgvcys2WXpGdVlwaUxUWlYKS2prOGh5UVNvMmVTZTJTUy81MG5BaUI0a0RNWnR4b1UzTitHc3BEOHVEbXFrNlhxbzZoeE0vbG0yU21OMzlPNAovdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
etcd_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUwZ3JiMk0yblA4c1hxWjVhd3NzNThwbUdvb1FlSWg3a3RoWVlxNzhZZThvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFRTRpV2dZaHdUbTJwSzUveGxMeE5VMXNyT0ZFUkxFNUZPQXZQSmc3bjF3QTNGZ29PM1VodQorQWZkL0dxU0lkcFlQRC9DUTlSQkNldzNVbUQwNTdHb2JnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
aggregator_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJZVENDQVFhZ0F3SUJBZ0lSQUpxeFI3alBzcmhzRUp2cmtEWndYR3N3Q2dZSUtvWkl6ajBFQXdJd0FEQWUKRncweU5qQTNNRGt3TlRBM05UaGFGdzB6TmpBM01EWXdOVEEzTlRoYU1BQXdXVEFUQmdjcWhrak9QUUlCQmdncQpoa2pPUFFNQkJ3TkNBQVFBWWlieHY1ZDVFRGdTbWhlRHlhYjJLZGh4ZFZnZ3lQc2pNcjBET0JMVmxtQmpMcXFqClpNSkk2d1ZmV2hkUjBRVGxVdEFLZDJvREJZLy9TeDBzQi9qY28yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXcKSFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4dwpIUVlEVlIwT0JCWUVGSTkrWm1EYVBybDhrTnhXUUxOT3ErTmVzeEh0TUFvR0NDcUdTTTQ5QkFNQ0Ewa0FNRVlDCklRRC9DRUZ2SUVHMW9qcmRZRVUzUldmTklLV1FwVXlZQWJ1ZGE0T0ZWQS9yOUFJaEFPS1VOZFF6bUk2WVZOdzgKeklDejlLblRxazQrT3dvV3RjNVl5SmlKR29zUgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
aggregator_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUFoOEwycXFLbUxoYkpmczJ2M3ZRWXBFZHF6Ri9hTGZhSmoraEZRazdPY2NvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQUdJbThiK1hlUkE0RXBvWGc4bW05aW5ZY1hWWUlNajdJeks5QXpnUzFaWmdZeTZxbzJUQwpTT3NGWDFvWFVkRUU1VkxRQ25kcUF3V1AvMHNkTEFmNDNBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
service_account_key = "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKSndJQkFBS0NBZ0VBd3NNZU40eE1YQkZ0VVE2dVZ2NEZFU1dNT2ZLc1RKdmxHa0ptVG1URjJIazg1SlJvCkhuQU1nMEkvMEJiMGFUQTJ3MjlkSEUrdUN6dFd5eVVqTzV5eWdJSDQ0Q2RtVnNHVzVua1Z0TmFGNzNpcFllbjYKaG11bmRoWDY1VndqOHpVUmFYZUxtUEc0Y2d4dk5SOXdGYUZXS0ZZdzQzdGtlWjg5S0F5QTNibjFXN3JHTloyOQovcXRYWEcrQnFSSlVuUzhXQmt4dkErUktyRE1OVHF4bjVGOEl1VkpTWkw5bDVEd3lSUnNqMjVMZVdRZUFYenl6CllWcmFKZmVubSt6L3hLZjhLdHFZMSs5U1pwbkZsS2N6TGlWVzg4WkJ5TnR0KytYZjB4N3FSL2lQOHhRQXV0ckIKV1pTbytoa01FRjV5YU0wQWdxcG5mbDBsUmhsM0I0UUx3NHkyTUpoTEpyQ1ltQjl1QllLWU5oRE5NOXk0Z1lrZgpacmdEdm9NdGpsSUJUUHRyVjMyemxYN05Od3d5UjdXOXp5ZlBSVjEvbjhpMk9KR0szeittdGNCR2Y0TS8veDFsClJRNEhzWWhDUVJRaE8vQWp2U2QvQlhFRnB3dEhsb1VYRU5nQ2lPSW00MnFrZTNWb3lLbjJ2Q3g2QlhodlJVZDAKUHgvK0JpM0d4RmRoa2Uxa0doelJLQTdySGhEMVBkVGsyM04wUXV3WHNNVDZRWXVrRDdPZG5KdFZtZVN4TkxqcwptdVpybmpwRzZsaHRDSmdvMGdVeXBYQ2RlWGVnZ0dIc0JCMks4NjIrdThVa0dwMk9IQUhGeldsdDF5L2VXTW9PCmhneWx1SEVLcjkzUU5uNDZsTGtZTDViWjlVTUZ0NXBMRFVrSFQxNkV5dTh0V21ET2ZoQ3Z3M1lmRWY4Q0F3RUEKQVFLQ0FnQUdSbUlSV1JoV3VRc0VHd3g3NmdoQXdxeHZhNFdvbkRjMzd0Njc5Tnc0K3NMKy9GY1ViL2kvTytHeApjeVBoeGJkbCtZOE82L1JJRVZ2ZEJLL0xhbU9INTJnYzFMZ2o0RzNidEJnQ2NRejBwN2NSWEFnQno3TWdCMXBECmpJSHVBbzR5anpMMHRRa0R4Nm5IbE9FNEdUQWM4WlgycGxHWTU0d0JYOUhCRXc0NEs5N1orR0NZTlc0Rm9PUVYKRGUyaStOTGxWZzRYbW9IYlpYT3V6cmcwTCttb2l1SHp0QVQwNHdtZGwxL0M0Y3IvSkZJNi8wb3FQMUthK1kweApaV1BpTXFWWnZoeEJqTWpqWEYzMHlhUkkvdFA3MjYzZjZrM3pXVGNxWnFzV3NZZjF4WFcyajNpK1NaOWVHM043CmpZZHpIL085d2Y2K29BS2s3UW9jT0dGbXBnQnlwdm9JbytlSFdjZlpZNFNXNloyeTRacUl0M2xTSWFHMEtmQjEKUnVrSVBtMlYzNDNlc3Azdy9TV2liRVU2elhvd01sWjlZc0dPalY1MTZDQmJZK0lQcmY2RXY0UFhDWjlnUERxcgpnUFFIZ0lFS294aUdYK2dENG9pYWdUMWVVSk1iYWZkaEkzSHJMWk9YbUpBZU40RHpOZXR1SGRsTk13YnpZZHN5CnpyMllSMkhqNTZydk9hcWZZUFJKV1NmS3ozRlhQdDNDMHJRVmNRekVqV1ZZbm9MZG9yR3FkaU1uMyt1NzF2RW4KSnluZksvN3VRSDY5VTU3NGNKK3FjOFNtNTlPeHBnN1RoODljMUZTeSszTk03bTE2czlJcjQvcG94Q2pWaHNWeQpKRmN2cHE4UTMwek9CQk0waUxyblNKcVRXaXJDRU8xNHlRY0VLbFBVd2VGUWdJK05pUUtDQVFFQTMwMWNrTnZuCmpaRHduRXFQaWp4SlBDSUNEeHcxYWtnVTZUNUpMVkt5SUFnK3BYNHlPZDZtendvV2hpWE5oVUQ5eWxPL21lTEsKSDRPUXVzeWdJUDdkOVNKZi9ZVTV5a1RZaVZLMUdzM1loZXovRmhRQXdBQUg4UmdBdW8rZU9UTUcyT0IxM2loLworQlFYL1lrOG9VR1M0YlpsZWNGWG5pSTZ5S0g1MUJUQ3g0ZTZZcStvdGMrMFc2RzBRVHVBT3JWcjdXWklGY1htClVDdElReXRJN2s2UVd6SXU0K3dnVnU3OUdrUEJTMHNBaXhzU0ppYXRTNzk1TjlhYjYwRzNDeVFWZDZWajJaVmEKWGpCR1oyQXREalpEWS9MMDlZQTlRVUxDbWVqT2hYVFlGNkJuNnRkYmVjeVlZTHdNS2MzWEhqNEt2U2ZaQ01HTwpvUXROdnFoUFFlSWpxUUtDQVFFQTMwZnFCSThDK2QwdXJlM1JhNFFRdUlzUC9xVTMyejIzMUdzbTZOOXlQd0hUClM4ZzlrS0ZDYzhMVGlmWFdnajhIRWxTVjVLSXJseERwZEVWQ1pEM05qdy9GVDdHcHV5SVplN1E3VlhiNld3MnYKY1I0M1FkdUQvOEc4ZkdlZGkrZ1BnKzlzeURCQXFIVzFGSHl1RmVaWGNsVTF1cndOV3V3TlFKUnZHczFoK1NuQQpXVmJxd1ZUL01GMmYxTjlBaFN0alhkM0Nscm1rb0o4ek05YW0zVWg5VUprQ2xSZi9mZVRONXZndE1odS9NNGtGCmVQYUlBMTVqT3h1cEFMN3ptVVVxZ0h1NE9yMm5mYTVNNHMzWTR6TVRYYm9VZGNVT09oRTRzU0J2bGlaak5KL3QKV2pSTmJmUWhxbWRBTTNZZjFGUC8vRW9CYTBPejBxMHNXci92cEhDUlp3S0NBUUFxMlkySnZxa1FZVi9LbmdRdApZcVFyQmR1ZlNxcDFXcCtvb21zb1oxWUhENDMxOCtGdmVXcEpFSWFCOTM4WXN3QUFjMUd4RmZQeldDdk5yTGFOCm5scTVUMzljQnRTd0c4WHhsQTFzdDFOMVg2VVRkNE10Vk5ReFQ0blVRdnI1dnZEeGJTRXhJRlJ1Sm16MEdnR28KY0F6ZmcwQzF2SVF6dEIzVG9rRnVrUTFQZkp3bms4MnNGYzltUmdGeEF4bjRLaGdyMWhTL0dOcTVSNVQyVHJnUQpBc053dkpDQzdDeklnZFBQMW5DaElpTllqamxOV042b1NuWFlZVFpLVHJIeFVWdE5PaytPMFRvbUdOMXB1T3JzCmJ6MC9VTC93M0VyazJ3cTh2Zy9qVENpclgveVE5QUo1dk9rQXB4VXVjSEYzUERDVFc3SXFHL3Bpck9pZVRXM28KRnAwQkFvSUJBRjBxa3JsSU8wT3JTUmtHRE1aQ0d3QUY5cXlZb0EvNVZzVnAySmgrOUJyYVZpSmU4V0Z5Q0ZwcApSdjlmOXh2dDFMT1BXK1JFenMrQUhRbUpCTVR6RE56UEJkUFZIQytiY09xdkw3cmZwR050K0hESTNPRzhDUDRsCkJ0TWFJU0VKdWIraG5kQ0NZZGhwRlIveFRtcVE3SmdtZWY3ckROK05jNUlvM1p0ZmE2d2VBY2JGZjdzZ0RrTk8KTGEwVFlzYXViZzN5eElsRCtTK1VmamI1TUROUlZnalZiOEJxZlE4NDg3bVdnTFZSNHB4TVpsNHM4R0FIZUh4bgpkRU45YWdQZ1duVzJLZzlJcDZUSG9BbGJQMDYrTnl4NndxTEprTUFtQTNQVlJ2cHVGaU1WUUdMTlJDbkhIbTBPCkhEbmM1amNndmNXMTA1WEFjRDVPU0IydHpQN2VnYTBDZ2dFQUZhTHJxMDJ6M2tXaW1iVzdoNi9Pby9YRHdjZ2YKSmdXMDYyYWdWUlpEMjM3Mm4valZSL25kMnFybmwxTFdWaXpzWW91b3hvY2N6NUwvQ2h3MktCOC9Rc1Zxai9KOApOUWh1ZDJ2ZUxjQlUvVzNseVhENnJpajJZMythNzRvM1A4b1psOGdDQmJEck1jajNsMHRZMXRWbm9nOWo0eHQ3Ckp3UXA3djNXb3ArSFVuRWVjbGpscTdlN1VuTGNwZlRDOE5PUmdxbjBGSUtCbXdFNlpJQnQxUzB6NmxSKytJdVMKQk1oTTZTSkZmeUFOVVdjTCt2cHJoRzBjWUZmcUYzajlPZ1dXRnBGTHFWQmdSTlZBVDRoM0U1Ymh4L0lsMTVHMAo2USs2aWlkRElqMEhNNFAzcjhja2lRUlVPdFI1R2NVS1RYZGh1TjkwMjZGRTdEckRvSURVZ0F5dGVnPT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K"
secretbox_encryption_secret = "RLkR4RmeaLmH/abyK3eYf6q22umJ/byLhheVCnh/yrA="
controlplane_configs = {
"talos-cp-1" = {
hostname = "talos-cp-1"
lan_ip = "192.168.1.166"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/nvme0n1"
longhorn_disks = []
zone = "az-a"
allow_scheduling = true
cloudflare_talos_sans = []
cloudflare_apiserver_sans = []
},
"talos-cp-2" = {
hostname = "talos-cp-2"
lan_ip = "192.168.1.214"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/disk/by-id/wwn-0x644a842029bd3f002720989b07d7143d"
longhorn_disks = [
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b60f4375d97062", mountpoint = "/var/lib/longhorn-disk1" },
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b5ffeb8bb8b47f", mountpoint = "/var/lib/longhorn-disk2" },
{ device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b6000c8dabb266", mountpoint = "/var/lib/longhorn-disk3" },
{ device = "/dev/disk/by-id/wwn-0x6b083fe0c5782700321370dc23fd765b", mountpoint = "/var/lib/longhorn-disk4" },
]
zone = "az-b"
allow_scheduling = true
cloudflare_talos_sans = []
cloudflare_apiserver_sans = []
},
"talos-cp-3" = {
hostname = "talos-cp-3"
lan_ip = "192.168.1.162"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/nvme0n1"
longhorn_disks = [
{ device = "/dev/disk/by-id/usb-Seagate_One_Touch_w_PW_00000000NABV3H34-0:0", mountpoint = "/var/lib/longhorn-paperless-media" },
]
zone = "az-c"
allow_scheduling = true
cloudflare_talos_sans = []
cloudflare_apiserver_sans = []
},
}
worker_configs = {
"worker-1" = {
hostname = "worker-1"
lan_ip = "192.168.1.223"
lan_subnet = "192.168.1.0/24"
lan_gateway = "192.168.1.254"
install_disk = "/dev/nvme0n1"
network_interface = "enp28s0f0np0"
zone = "az-a"
gpu_count = 4
node_labels = {}
node_taints = []
factory_image = "factory.talos.dev/metal-installer/0a2153a6dc099a371bf2f63d6c3c22d275c876bf6302dd154c5813072924cb3f:v1.13.3"
swap_size = "64GiB"
ephemeral_max_size = "700GiB"
extra_disks = []
}
}
cluster_config = {
controlplane_ip = "192.168.1.166"
pod_subnets = ["10.244.0.0/16"]
service_subnets = ["10.96.0.0/12"]
dns_servers = ["8.8.8.8", "1.1.1.1"]
dns_domain = "cluster.local"
}