feat(cloudflared): manage tunnel ingress locally via ConfigMap

The tunnel was remotely managed from the Zero Trust dashboard: the Deployment ran 'run --token' and ignored the ConfigMap that already existed in the tree. Switch to a locally-managed tunnel so the ingress routing table lives in git. - deployment: mount cloudflared-config (config.yml) and the cloudflared-credentials Secret; drop TUNNEL_TOKEN env - configmap.yaml: per-hostname ingress rules to ingress-nginx via the CoreDNS rewrite, plus a 404 catch-all - secret-generator: register the SOPS-encrypted credentials file - homelab-ca: add the cloudflared namespace copy so the connector trusts the ingress certificate (force-ssl-redirect makes plaintext impossible) Credentials were extracted from the live cloudflared-token Secret and re-encrypted with SOPS; no plaintext secret is committed.
refactor(iam): retire terraform/zitadel, provision through the API
2026-10-25 19:57:37 -07:00 · 2026-10-25 19:35:01 -07:00 · 2026-10-25 19:16:18 -07:00 · 2026-10-25 19:10:32 -07:00 · 2026-10-25 18:48:15 -07:00 · 2026-10-25 18:15:48 -07:00
54 changed files with 2476 additions and 623 deletions
@@ -31,6 +31,17 @@ AUTHENTIK_BOOTSTRAP_TOKEN=
 #   openssl rand -base64 24
 AUTHENTIK_PG_PASSWORD=
 # ── ZITADEL (Authentik replacement) ─────────────────────────────────────────────
 # Instance URL, same hostname as the OIDC issuer.
 ZITADEL_URL="https://zitadel.riotpiao.com"
 # PAT of the 'iam-admin' machine user, created by the chart's setup job on the
 # first sync. Leave empty until then. Read it with:
 #   kubectl -n iam get secret iam-admin-pat -o jsonpath='{.data.pat}' | base64 -d
 # scripts/iam/zitadel-provision.py reads this variable first, then the cluster
 # Secret. The masterkey and the client secrets never live in this file.
 ZITADEL_BOOTSTRAP_TOKEN=
 # ── MinIO ──────────────────────────────────────────────────────────────────────
 MINIO_ROOT_USER=
 MINIO_ROOT_PASSWORD=
@@ -47,6 +47,15 @@ k8s/storage/test/test
 terraform/.terraform/
 terraform/*.tfstate
 terraform/*.tfstate.*
 # The Zitadel root module keeps its own state (see terraform/zitadel/providers.tf).
 # That state carries generated client secrets and machine keys.
 terraform/zitadel/*.tfstate
 terraform/zitadel/*.tfstate.*
 terraform/zitadel/*.tfvars
 terraform/edge/*.tfstate
 terraform/edge/*.tfstate.*
 terraform/edge/*.tfvars
 terraform/edge/terraform.tfvars.local
 terraform.tfvars.local
 skills-lock.json
 secrets-plaintext.yaml
@@ -69,3 +78,14 @@ k8s/**/*-secret.yaml
 # IAM provisioning scripts contain credential references — never commit
 scripts/iam/*.py
 # Exception: this one reads every credential from the cluster at run time and
 # embeds none. Keep it reviewable in git.
 !scripts/iam/zitadel-provision.py
 # Runtime credential export from the old Authentik provisioning run. Holds live
 # M2M tokens; never commit it.
 /authentik-m2m-tokens.env
 # Never commit backup copies of plaintext tfvars (they carry cluster PKI)
 terraform/*.bak
 terraform/edge/.secrets/
@@ -0,0 +1,22 @@
 {
  "nextId": 2,
  "loops": [
    {
      "id": "1",
      "prompt": "Check Longhorn and CNPG recovery progress in the homelab cluster and report terse. Run: kubectl -n longhorn-system get volumes.longhorn.io -o json (count by state/robustness), kubectl -n longhorn-system get replicas.longhorn.io -o json (count running/stopped/failed), kubectl get clusters.postgresql.cnpg.io -A (ready instances per cluster), kubectl -n cicd get pods,pvc | grep forgejo-db. Report: volumes healthy vs degraded, replicas running vs stopped, and each CNPG cluster ready count. If every volume is attached/healthy and every CNPG cluster shows 3/3 ready, say RECOVERED, list any leftover anomalies, and stop this loop with LoopDelete. Otherwise name what is still rebuilding and keep going.",
      "trigger": {
        "type": "cron",
        "schedule": "*/5 * * * *"
      },
      "status": "expired",
      "recurring": true,
      "createdAt": 1792898370946,
      "updatedAt": 1792901371403,
      "expiresAt": 1793503170946,
      "source": "tool",
      "readOnly": true,
      "maxFires": 6,
      "fireCount": 6
    }
  ]
 }
@@ -0,0 +1,37 @@
 {
  "schema_version": 1,
  "diagram_type": "architecture",
  "meta": {
    "title": "Poimen Memory — Schema Relations",
    "quality_profile": "showcase",
    "viewBox": [900, 480]
  },
  "components": [
    { "id": "node", "type": "database", "label": "memory_node", "sublabel": "Episodes (raw messages)", "pos": [40, 60], "size": [150, 55], "tag": "BIGSERIAL" },
    { "id": "entity", "type": "database", "label": "memory_entity", "sublabel": "Extracted entities", "pos": [260, 60], "size": [150, 55], "tag": "UUID" },
    { "id": "edge", "type": "database", "label": "memory_edge", "sublabel": "Facts / relationships", "pos": [480, 60], "size": [150, 55], "tag": "UUID" },
    { "id": "community", "type": "database", "label": "memory_community", "sublabel": "Entity clusters", "pos": [700, 60], "size": [150, 55], "tag": "UUID" },
    { "id": "pgvector", "type": "cloud", "label": "pgvector HNSW", "sublabel": "768-dim cosine search", "pos": [140, 260], "size": [160, 55], "tag": "nomic-embed-v2" },
    { "id": "tsvector", "type": "cloud", "label": "pg ts_rank", "sublabel": "Full-text search", "pos": [400, 260], "size": [160, 55], "tag": "GIN index" },
    { "id": "tenant", "type": "security", "label": "tenant_id", "sublabel": "Multi-tenant isolation", "pos": [660, 260], "size": [140, 55], "tag": "W7 / mig 014" }
  ],
  "boundaries": [
    { "kind": "region", "label": "Temporal Knowledge Graph (Zep §2)", "wraps": ["node", "entity", "edge", "community"] },
    { "kind": "security-group", "label": "Search & Isolation", "wraps": ["pgvector", "tsvector", "tenant"] }
  ],
  "connections": [
    { "id": "node-entity", "from": "node", "to": "entity", "variant": "emphasis" },
    { "id": "entity-edge", "from": "entity", "to": "edge", "variant": "emphasis" },
    { "id": "entity-community", "from": "community", "to": "entity", "variant": "dashed" },
    { "id": "edge-node", "from": "edge", "to": "node", "variant": "dashed" },
    { "id": "entity-vec", "from": "entity", "to": "pgvector", "variant": "default" },
    { "id": "entity-fts", "from": "entity", "to": "tsvector", "variant": "dashed" },
    { "id": "edge-fts", "from": "edge", "to": "tsvector", "variant": "dashed" },
    { "id": "tenant-all", "from": "tenant", "to": "entity", "variant": "security" }
  ],
  "cards": [
    { "dot": "emerald", "title": "Three-Tier Graph", "items": ["memory_node → episode subgraph", "memory_entity → entity subgraph", "memory_edge → fact relationships", "memory_community → clusters (Phase 4)"] },
    { "dot": "rose", "title": "Bi-Temporal", "items": ["t_valid / t_invalid: event time", "t_created / t_expired: transaction time", "Contradiction: active → candidate → confirmed"] },
    { "dot": "cyan", "title": "Search", "items": ["HNSW: name, summary, fact embeddings", "ts_rank: BM25 fallback (always available)", "tenant_id: per-table isolation (mig 014)"] }
  ]
 }
@@ -0,0 +1,158 @@
 {
  "schema_version": 1,
  "diagram_type": "dataflow",
  "meta": {
    "title": "Poimen Memory \u2014 Schema Relations",
    "viewBox": [
      960,
      480
    ],
    "quality_profile": "showcase"
  },
  "stages": [
    {
      "label": "Episodes"
    },
    {
      "label": "Entities"
    },
    {
      "label": "Relationships"
    },
    {
      "label": "Indexes"
    }
  ],
  "nodes": [
    {
      "id": "node",
      "type": "database",
      "label": "memory_node",
      "sublabel": "Raw messages",
      "stage": 0,
      "row": 0,
      "tag": "BIGSERIAL"
    },
    {
      "id": "entity",
      "type": "database",
      "label": "memory_entity",
      "sublabel": "Named entities",
      "stage": 1,
      "row": 0,
      "tag": "UUID + tenant_id"
    },
    {
      "id": "community",
      "type": "database",
      "label": "memory_community",
      "sublabel": "Clusters (Phase 4)",
      "stage": 1,
      "row": 1,
      "tag": "UUID"
    },
    {
      "id": "edge",
      "type": "database",
      "label": "memory_edge",
      "sublabel": "Facts + bi-temporal",
      "stage": 2,
      "row": 0,
      "tag": "UUID + tenant_id"
    },
    {
      "id": "hnsw",
      "type": "cloud",
      "label": "HNSW Index",
      "sublabel": "768-dim cosine",
      "stage": 3,
      "row": 0,
      "tag": "pgvector"
    },
    {
      "id": "fts",
      "type": "cloud",
      "label": "GIN Index",
      "sublabel": "ts_rank BM25",
      "stage": 3,
      "row": 1,
      "tag": "full-text"
    }
  ],
  "flows": [
    {
      "from": "node",
      "to": "entity",
      "label": "LLM extraction",
      "classification": "ingest",
      "variant": "emphasis",
      "route": "straight"
    },
    {
      "from": "entity",
      "to": "edge",
      "label": "source + target FK",
      "classification": "foreign key",
      "variant": "emphasis",
      "route": "straight"
    },
    {
      "from": "entity",
      "to": "community",
      "label": "community_id",
      "classification": "membership",
      "variant": "dashed",
      "route": "straight",
      "labelDy": 30
    },
    {
      "from": "edge",
      "to": "hnsw",
      "classification": "vector",
      "variant": "default",
      "route": "straight",
      "label": "embedding",
      "labelDy": -30
    },
    {
      "from": "edge",
      "to": "fts",
      "label": "fact text",
      "classification": "text",
      "variant": "dashed",
      "route": "vertical-channel",
      "labelDy": 30
    }
  ],
  "cards": [
    {
      "dot": "emerald",
      "title": "Three-Tier Graph",
      "items": [
        "memory_node \u2192 episodes (raw messages)",
        "memory_entity \u2192 extracted entities",
        "memory_edge \u2192 facts between entities",
        "memory_community \u2192 clusters (Phase 4)"
      ]
    },
    {
      "dot": "rose",
      "title": "Bi-Temporal + Tenant",
      "items": [
        "t_valid/t_invalid: when true in reality",
        "t_created/t_expired: system transaction time",
        "tenant_id: row-level multi-tenant isolation",
        "Contradiction: active \u2192 candidate \u2192 confirmed"
      ]
    },
    {
      "dot": "cyan",
      "title": "Search Indexes",
      "items": [
        "HNSW: name, summary, fact embeddings",
        "GIN: ts_rank on name+summary and fact",
        "RRF fusion: sem_weight + lex_weight"
      ]
    }
  ]
 }
@@ -0,0 +1,63 @@
 {
  "schema_version": 1,
  "diagram_type": "sequence",
  "meta": {
    "title": "Service Acting on Behalf of User",
    "viewBox": [1000, 600],
    "column_fit": "spread",
    "quality_profile": "showcase"
  },
  "participants": [
    { "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
    { "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC" },
    { "id": "worker", "type": "backend", "label": "Backend Worker", "sublabel": "temporal/queue" },
    { "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256" }
  ],
  "segments": [
    { "from": 150, "to": 260, "label": "User Auth" },
    { "from": 270, "to": 370, "label": "Delegation" },
    { "from": 380, "to": 510, "label": "Downstream" }
  ],
  "messages": [
    { "id": "login", "from": "user", "to": "app", "y": 165, "label": "login (OIDC)", "variant": "default" },
    { "id": "oidc-flow", "from": "app", "to": "authentik", "y": 200, "label": "authorization_code flow", "variant": "emphasis" },
    { "id": "user-jwt", "from": "authentik", "to": "app", "y": 235, "label": "user JWT {groups,roles}", "variant": "return" },
    { "id": "trigger-job", "from": "user", "to": "app", "y": 285, "label": "trigger async job", "variant": "default" },
    { "id": "queue-job", "from": "app", "to": "worker", "y": 320, "label": "enqueue job + user JWT", "variant": "emphasis", "note": "JWT passed as job metadata" },
    { "id": "job-ack", "from": "worker", "to": "app", "y": 355, "label": "job queued", "variant": "return" },
    { "id": "call-memory", "from": "worker", "to": "memory", "y": 400, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Bearer <user JWT>" },
    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 435, "label": "GET /jwks", "variant": "dashed" },
    { "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 470, "label": "RSA pubkey", "variant": "return" },
    { "id": "ingest-ok", "from": "memory", "to": "worker", "y": 505, "label": "202 Accepted", "variant": "return", "note": "uses USER's groups/roles" }
  ],
  "activations": [
    { "participant": "user", "from": 160, "to": 291, "type": "external" },
    { "participant": "app", "from": 165, "to": 361, "type": "frontend" },
    { "participant": "authentik", "from": 195, "to": 241, "type": "security" },
    { "participant": "worker", "from": 315, "to": 511, "type": "backend" },
    { "participant": "memory", "from": 395, "to": 511, "type": "backend" },
    { "participant": "jwks", "from": 430, "to": 476, "type": "security" }
  ],
  "cards": [
    {
      "dot": "emerald",
      "title": "Delegation Pattern",
      "items": [
        "User JWT travels with the job payload",
        "Worker uses user's token for downstream",
        "Memory sees USER as the principal"
      ]
    },
    {
      "dot": "rose",
      "title": "Security Model",
      "items": [
        "No service-account escalation",
        "User's permissions bound the action",
        "Audit trail traces to user.sub"
      ]
    }
  ]
 }
@@ -0,0 +1,42 @@
 {
  "schema_version": 1,
  "diagram_type": "architecture",
  "meta": {
    "title": "Memory Service Fine-Grained RBAC",
    "quality_profile": "showcase",
    "viewBox": [900, 520]
  },
  "components": [
    { "id": "jwt", "type": "security", "label": "JWT Token", "sublabel": "groups[] + roles[]", "pos": [40, 60], "size": [140, 60], "tag": "Authentik" },
    { "id": "cap-check", "type": "backend", "label": "Capability Check", "sublabel": "has_capability()", "pos": [260, 60], "size": [160, 60], "tag": "Phase 1" },
    { "id": "scope-checker", "type": "backend", "label": "Scope Checker", "sublabel": "filter_resources()", "pos": [500, 60], "size": [160, 60], "tag": "Phase 2" },
    { "id": "decision", "type": "backend", "label": "AccessDecision", "sublabel": "Allow / Deny", "pos": [740, 60], "size": [130, 60] },
    { "id": "project-scope", "type": "database", "label": "ProjectScope", "sublabel": "dev/staging/prod/*", "pos": [40, 200], "size": [140, 55] },
    { "id": "visibility-scope", "type": "database", "label": "VisibilityScope", "sublabel": "public/private", "pos": [220, 200], "size": [140, 55] },
    { "id": "owner-scope", "type": "database", "label": "OwnerScope", "sublabel": "self/any/user", "pos": [400, 200], "size": [140, 55] },
    { "id": "group-scope", "type": "database", "label": "GroupScope", "sublabel": "in_any_group()", "pos": [580, 200], "size": [140, 55] },
    { "id": "entities", "type": "database", "label": "Entities", "sublabel": "memory_entity", "pos": [40, 340], "size": [120, 50] },
    { "id": "edges", "type": "database", "label": "Edges", "sublabel": "memory_edge", "pos": [200, 340], "size": [120, 50] },
    { "id": "projects", "type": "database", "label": "Projects", "sublabel": "project_id", "pos": [360, 340], "size": [120, 50] },
    { "id": "convos", "type": "database", "label": "Conversations", "sublabel": "owner_id", "pos": [520, 340], "size": [120, 50] }
  ],
  "boundaries": [
    { "kind": "security-group", "label": "Access Control Pipeline", "wraps": ["cap-check", "scope-checker", "decision"] },
    { "kind": "region", "label": "Scope Checkers", "wraps": ["project-scope", "visibility-scope", "owner-scope", "group-scope"] },
    { "kind": "region", "label": "Protected Resources", "wraps": ["entities", "edges", "projects", "convos"] }
  ],
  "connections": [
    { "id": "jwt-to-cap", "from": "jwt", "to": "cap-check", "label": "claims", "variant": "emphasis" },
    { "id": "cap-to-scope", "from": "cap-check", "to": "scope-checker", "variant": "emphasis" },
    { "id": "scope-to-decision", "from": "scope-checker", "to": "decision", "variant": "emphasis" },
    { "id": "scope-to-proj", "from": "scope-checker", "to": "project-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
    { "id": "scope-to-vis", "from": "scope-checker", "to": "visibility-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
    { "id": "scope-to-owner", "from": "scope-checker", "to": "owner-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" },
    { "id": "scope-to-group", "from": "scope-checker", "to": "group-scope", "variant": "dashed", "fromSide": "bottom", "toSide": "top" }
  ],
  "cards": [
    { "dot": "emerald", "title": "Phase 1: Capability", "items": ["has_capability(memory:read)", "Checks roles[] claim", "Binary allow/deny"] },
    { "dot": "rose", "title": "Phase 2: Scope", "items": ["Project: dev/staging/prod", "Visibility: public/private", "Owner: self-owned only", "Group: poimen-admins/devs"] },
    { "dot": "cyan", "title": "JWT Claims", "items": ["groups: user.groups", "roles: attributes.roles", "permissions: group names"] }
  ]
 }
@@ -0,0 +1,54 @@
 {
  "schema_version": 1,
  "diagram_type": "sequence",
  "meta": {
    "title": "Service-to-Service (M2M) Authentication",
    "viewBox": [1080, 520],
    "column_fit": "spread",
    "quality_profile": "showcase"
  },
  "participants": [
    { "id": "worker", "type": "backend", "label": "Ingest Worker", "sublabel": "temporal/cron" },
    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
    { "id": "memory", "type": "backend", "label": "Memory Service", "sublabel": "poimen-memory" },
    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
  ],
  "segments": [
    { "from": 150, "to": 230, "label": "Token Mint" },
    { "from": 240, "to": 380, "label": "API + RBAC" }
  ],
  "messages": [
    { "id": "token-req", "from": "worker", "to": "authentik", "y": 165, "label": "POST /token grant_type=password", "variant": "emphasis", "note": "client_id + username + app_password + scope" },
    { "id": "jwt-resp", "from": "authentik", "to": "worker", "y": 205, "label": "JWT {groups,roles,aud,iss}", "variant": "return", "note": "groups=[poimen-admins] roles=[memory:write]" },
    { "id": "api-call", "from": "worker", "to": "memory", "y": 260, "label": "POST /memory/ingest", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 295, "label": "GET /jwks (cached)", "variant": "dashed" },
    { "id": "jwks-resp", "from": "jwks", "to": "memory", "y": 330, "label": "RSA pubkey (kid)", "variant": "return" },
    { "id": "verify-ok", "from": "memory", "to": "worker", "y": 365, "label": "202 Accepted", "variant": "return", "note": "sig ✓ iss ✓ aud ✓ has_capability ✓" }
  ],
  "activations": [
    { "participant": "worker", "from": 160, "to": 371, "type": "backend" },
    { "participant": "authentik", "from": 165, "to": 211, "type": "security" },
    { "participant": "memory", "from": 255, "to": 371, "type": "backend" },
    { "participant": "jwks", "from": 290, "to": 336, "type": "security" }
  ],
  "cards": [
    {
      "dot": "emerald",
      "title": "M2M Pattern",
      "items": [
        "grant_type=password (Authentik aliases client_credentials)",
        "App-password token authenticates the service account",
        "NOT client_secret — that creates nameless SA"
      ]
    },
    {
      "dot": "rose",
      "title": "JWT Claims",
      "items": [
        "groups: from user.groups (poimen-admins)",
        "roles: from user.attributes.roles",
        "iss/aud: per-provider issuer"
      ]
    }
  ]
 }
@@ -0,0 +1,61 @@
 {
  "schema_version": 1,
  "diagram_type": "sequence",
  "meta": {
    "title": "User Direct Access with JWT",
    "viewBox": [1000, 640],
    "column_fit": "spread",
    "quality_profile": "showcase"
  },
  "participants": [
    { "id": "user", "type": "external", "label": "User", "sublabel": "browser" },
    { "id": "app", "type": "frontend", "label": "Web App", "sublabel": "poimen UI" },
    { "id": "authentik", "type": "security", "label": "Authentik", "sublabel": "OIDC provider" },
    { "id": "memory", "type": "backend", "label": "Memory API", "sublabel": "poimen-memory" },
    { "id": "jwks", "type": "security", "label": "JWKS", "sublabel": "RS256 keys" }
  ],
  "segments": [
    { "from": 150, "to": 370, "label": "OIDC Login" },
    { "from": 380, "to": 540, "label": "API + RBAC" }
  ],
  "messages": [
    { "id": "click-login", "from": "user", "to": "app", "y": 165, "label": "click Login", "variant": "default" },
    { "id": "redirect-auth", "from": "app", "to": "authentik", "y": 200, "label": "redirect /authorize", "variant": "emphasis", "note": "scope=openid groups roles" },
    { "id": "show-login", "from": "authentik", "to": "user", "y": 235, "label": "login form", "variant": "default" },
    { "id": "submit-creds", "from": "user", "to": "authentik", "y": 270, "label": "username + password", "variant": "security" },
    { "id": "auth-code", "from": "authentik", "to": "app", "y": 305, "label": "redirect ?code=xxx", "variant": "return" },
    { "id": "exchange-token", "from": "app", "to": "authentik", "y": 340, "label": "POST /token code=xxx", "variant": "emphasis" },
    { "id": "jwt-resp", "from": "authentik", "to": "app", "y": 375, "label": "JWT {groups,roles,sub}", "variant": "return" },
    { "id": "query-api", "from": "app", "to": "memory", "y": 420, "label": "GET /memory/query", "variant": "emphasis", "note": "Authorization: Bearer <JWT>" },
    { "id": "fetch-jwks", "from": "memory", "to": "jwks", "y": 455, "label": "GET /jwks (cached)", "variant": "dashed" },
    { "id": "jwks-keys", "from": "jwks", "to": "memory", "y": 490, "label": "RSA pubkey", "variant": "return" },
    { "id": "query-resp", "from": "memory", "to": "app", "y": 525, "label": "200 JSON results", "variant": "return", "note": "has_capability(memory:read) ✓" }
  ],
  "activations": [
    { "participant": "user", "from": 160, "to": 276, "type": "external" },
    { "participant": "app", "from": 165, "to": 531, "type": "frontend" },
    { "participant": "authentik", "from": 195, "to": 381, "type": "security" },
    { "participant": "memory", "from": 415, "to": 531, "type": "backend" },
    { "participant": "jwks", "from": 450, "to": 496, "type": "security" }
  ],
  "cards": [
    {
      "dot": "emerald",
      "title": "Authorization Code Flow",
      "items": [
        "User authenticates directly with Authentik",
        "App exchanges code for JWT",
        "Scopes: openid groups roles permissions"
      ]
    },
    {
      "dot": "rose",
      "title": "JWT Contents",
      "items": [
        "iss: .../application/o/poimen/",
        "aud: poimen (provider client_id)",
        "groups: user's group memberships"
      ]
    }
  ]
 }
@@ -0,0 +1,58 @@
 # cloudflared ingress configuration.
 #
 # This list is the tunnel's routing table: for each incoming hostname the
 # connector dials the matching origin. The final rule is a catch-all, so a
 # hostname absent from this list gets 404 from the connector — DNS alone does
 # not route a request into a service.
 #
 # Management model: this file makes the tunnel LOCALLY managed. cloudflared is
 # given a credentials file instead of a token, so Cloudflare stops pushing the
 # remotely-managed ingess list and this ConfigMap wins. Revert the Deployment to
 # `run --token` to return to the dashboard-managed list.
 #
 # Origin services self-reference the public hostname. That resolves because
 # CoreDNS rewrites each of these names to ingress-nginx
 # (terraform/files/coredns/Corefile), so the connector reaches the in-cluster
 # ingress instead of looping back out through Cloudflare.
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: cloudflared-config
  namespace: cloudflared
 data:
  config.yml: |
    tunnel: 32d0bf93-e1bd-45b8-93a9-71ff9f83de80
    credentials-file: /etc/cloudflared/creds/credentials.json
    # WARP-to-Tunnel routing. The remotely-managed config had this enabled;
    # dropping the key would break WARP access to private addresses. cloudflared
    # 2026.x has no `enabled` field — the key's presence is what enables it, and
    # `enabled: true` here makes the binary warn about an unused key.
    warp-routing: {}
    ingress:
      - hostname: paperless.riotpiao.com
        service: https://paperless.riotpiao.com
      - hostname: authentik.riotpiao.com
        service: https://authentik.riotpiao.com
      - hostname: img.riotpiao.com
        service: https://img.riotpiao.com
      - hostname: riotpiao.com
        service: https://riotpiao.com
      - hostname: grafana.riotpiao.com
        service: https://grafana.riotpiao.com
      - hostname: gotify.riotpiao.com
        service: https://gotify.riotpiao.com
      - hostname: igotify.riotpiao.com
        service: https://igotify.riotpiao.com
      - hostname: poimen.riotpiao.com
        service: https://poimen.riotpiao.com
      - hostname: comfyui.riotpiao.com
        service: https://comfyui.riotpiao.com
      - hostname: forgejo.riotpiao.com
        service: https://forgejo.riotpiao.com
      # ZITADEL. Without this rule the connector answers 404 for the hostname
      # even though the DNS CNAME points at the tunnel.
      - hostname: zitadel.riotpiao.com
        service: https://zitadel.riotpiao.com
      - service: http_status:404
@@ -13,21 +13,63 @@ spec:
      labels:
        app: cloudflared
    spec:
      # Ingress-nginx forces HTTPS (force-ssl-redirect), so cloudflared must
      # dial https://<hostname> and trust homelab-ca or the TLS handshake fails
      # with "certificate signed by unknown authority". Same pattern as
      # authentik-values.yaml: merge the system bundle with homelab-ca.crt.
      initContainers:
        - name: merge-ca-certs
          image: debian:bookworm
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 0
          command:
            - sh
            - -c
            - (cat /etc/ssl/certs/ca-certificates.crt 2>/dev/null; cat /homelab-ca/homelab-ca.crt) > /merged/ca-bundle.crt
          volumeMounts:
            - name: homelab-ca
              mountPath: /homelab-ca
              readOnly: true
            - name: merged-ca
              mountPath: /merged
      volumes:
        - name: config
          configMap:
            name: cloudflared-config
        - name: credentials
          secret:
            secretName: cloudflared-credentials
        - name: homelab-ca
          configMap:
            name: homelab-ca
        - name: merged-ca
          emptyDir: {}
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args:
            - tunnel
            - --no-autoupdate
            - --protocol
            - http2
            - --config
            - /etc/cloudflared/config.yml
            - run
            - --token
            - $(TUNNEL_TOKEN)
          env:
-            - name: TUNNEL_TOKEN
+            - name: SSL_CERT_FILE
-              valueFrom:
+              value: /merged/ca-bundle.crt
-                secretKeyRef:
+          volumeMounts:
-                  name: cloudflared-token
+            - name: config
-                  key: token
+              mountPath: /etc/cloudflared/config.yml
              subPath: config.yml
              readOnly: true
            - name: credentials
              mountPath: /etc/cloudflared/creds
              readOnly: true
            - name: merged-ca
              mountPath: /merged
              readOnly: true
          resources:
            requests:
              cpu: 10m
@@ -3,3 +3,4 @@ kind: Kustomization
 namespace: cloudflared
 resources:
  - deployment.yaml
  - configmap.yaml
@@ -12,6 +12,19 @@ rules:
  - apiGroups: ["kmsvc.io"]
    resources: ["queues"]
    verbs: ["get", "list", "watch", "update", "patch"]
  # The operator watches TemporalWorker objects and creates Deployments for
  # them. Without these two rules the controller logs
  # "cannot list resource temporalworkers/deployments ... at the cluster scope"
  # and never reconciles.
  - apiGroups: ["kmsvc.io"]
    resources: ["temporalworkers"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["kmsvc.io"]
    resources: ["temporalworkers/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["kmsvc.io"]
    resources: ["queues/status"]
    verbs: ["get", "update", "patch"]
@@ -1,115 +0,0 @@
 # macOS VM (Docker-OSX) hosting the BlueBubbles server.
 #
 # ── Why a VM and not a container ────────────────────────────────────────────
 # Containers share the host kernel. macOS binaries are Mach-O and need XNU plus
 # Cocoa/IOKit, which a Linux kernel cannot provide, so no macOS container exists
 # or can exist. Docker-OSX is QEMU running a macOS guest, packaged in a
 # container — a VM in a box, not a macOS container.
 #
 # ── Why this works on worker-2 ──────────────────────────────────────────────
 # Verified on the existing hardware: amd64, `vmx` (Intel VT-x) present, and
 # /dev/kvm exists on Talos nodes (KVM is compiled into Talos' kernel, not a
 # module). Bare metal, so no nested virtualisation needed.
 #
 # ── Read this before relying on it ──────────────────────────────────────────
 # 1. Setup is INTERACTIVE. First boot runs the macOS installer: connect over
 #    VNC (:5999), erase the disk in Disk Utility, install, create a user, sign
 #    into iMessage, THEN install BlueBubbles inside the guest. This manifest
 #    only provides the machine; it does not provision macOS.
 # 2. iMessage activation on non-Apple hardware is a coin flip. BlueBubbles'
 #    own guidance: "test sending an iMessage to yourself. If it does not
 #    succeed, it's likely best to restart from the beginning."
 # 3. Apple's macOS licence permits virtualisation only on Apple hardware. This
 #    is a Hackintosh. Use a throwaway Apple ID, not a primary one.
 # 4. BlueBubbles labels this path "not for beginners", "no guarantees or
 #    warranty".
 #
 # Private API (reactions, typing indicators, edit/unsend) needs SIP disabled
 # inside the guest and is NOT required for plain send/receive. Skip it.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: macos-bluebubbles
  labels:
    app.kubernetes.io/name: macos-bluebubbles
    app.kubernetes.io/part-of: sms
 spec:
  replicas: 1
  # Recreate: the qcow2 disk is RWO and a second pod must never attach it
  # concurrently — two QEMU processes on one image corrupts it.
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: macos-bluebubbles
  template:
    metadata:
      labels:
        app.kubernetes.io/name: macos-bluebubbles
        app.kubernetes.io/part-of: sms
    spec:
      # Dedicated node. The taint keeps everything else off worker-2; this
      # toleration is what lets the VM on. Both halves are required.
      nodeSelector:
        workload: imessage
      tolerations:
        - key: workload
          operator: Equal
          value: imessage
          effect: NoSchedule
      containers:
        - name: macos
          image: sickcodes/docker-osx:latest@sha256:3a3c82c79bc4e73531f819ccdfa4053b3084efd7c1f645678dbf8b4b3a24369c
          # QEMU needs /dev/kvm; Talos enforces `baseline` cluster-wide, so this
          # only schedules because the sms namespace is labelled privileged.
          securityContext:
            privileged: true
          env:
            # Generates a unique serial / board-serial / UUID / MAC and persists
            # them to bootdisk.qcow2. This synthetic identity is what iMessage
            # activates against — it must stay stable across restarts, which is
            # why the PVC matters.
            - name: GENERATE_UNIQUE
              value: "true"
            # Identity is only plausible if it matches a real product line.
            - name: DEVICE_MODEL
              value: "iMacPro1,1"
            - name: RAM
              value: "12"
            - name: CORES
              value: "6"
            - name: EXTRA
              # Expose the BlueBubbles server port from the guest to the pod.
              # Guest :1234 (BlueBubbles default) -> pod :1234.
              value: "-device virtio-net-pci,netdev=net0 -netdev user,id=net0,hostfwd=tcp::1234-:1234"
          ports:
            - name: vnc
              containerPort: 5999
              protocol: TCP
            - name: bluebubbles
              containerPort: 1234
              protocol: TCP
          resources:
            requests:
              cpu: "6"
              memory: 14Gi
            limits:
              cpu: "12"
              memory: 20Gi
          volumeMounts:
            - name: macos-disk
              mountPath: /home/arch/OSX-KVM/disk
            - name: kvm
              mountPath: /dev/kvm
          # No readiness probe on purpose. The guest takes many minutes to boot,
          # and until macOS + BlueBubbles are installed BY HAND there is nothing
          # listening on 1234. A probe here would crash-loop the pod through the
          # entire interactive install.
      volumes:
        - name: macos-disk
          persistentVolumeClaim:
            claimName: macos-disk
        - name: kvm
          hostPath:
            path: /dev/kvm
            type: CharDevice
@@ -1,10 +0,0 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: sms
 resources:
  - namespace.yaml
  - storageclass.yaml
  - pvc-macos.yaml
  - deployment-macos.yaml
  - service.yaml
  - networkpolicy.yaml
@@ -1,19 +0,0 @@
 # iMessage delivery for the cluster.
 #
 # BlueBubbles' server is a macOS Electron app paired with an Objective-C helper
 # that hooks Messages.app private APIs — it cannot be containerised on Linux,
 # because containers share the host kernel and macOS needs XNU + Cocoa. The only
 # way to run it on Talos is a full macOS VM under QEMU/KVM (Docker-OSX), which
 # needs a privileged pod with /dev/kvm.
 #
 # Hence privileged PodSecurity: the cluster default from the Talos controlplane
 # is `enforce: baseline`, which forbids privileged containers and host devices.
 # Scope is limited to this namespace.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: sms
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
@@ -1,22 +0,0 @@
 # Default-deny. This namespace runs a privileged QEMU VM signed into an Apple
 # ID and exposes an unauthenticated VNC console; nothing should reach it except
 # opted-in clients.
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: sms-default-deny
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: sms
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              sms-client: "true"
      ports:
        - protocol: TCP
          port: 1234
@@ -1,25 +0,0 @@
 # Persistent macOS disk image + generated hardware identity (bootdisk.qcow2).
 #
 # This volume is NOT disposable: it holds the VM's serial number, board serial,
 # UUID and MAC, which together form the identity iMessage was activated against.
 # Losing it means re-running activation, which is the least reliable step of the
 # whole setup.
 #
 # Docker-OSX documents 128GB minimum for the guest image; 200Gi leaves room for
 # the installer, the base system, and qcow2 growth.
 #
 # ⚠️ Single replica (see storageclass.yaml — capacity and IO both rule out 3).
 # Losing worker-2's disk therefore means losing the activated identity and
 # redoing iMessage activation. Once the guest is installed and activated, take
 # a Longhorn snapshot/backup of this volume; that is the only redundancy here.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: macos-disk
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-imessage-local
  resources:
    requests:
      storage: 200Gi
@@ -1,31 +0,0 @@
 # VNC is how you drive the interactive macOS install. Deliberately ClusterIP —
 # it is an unauthenticated console onto a machine holding a live Apple ID
 # session. Reach it with `kubectl port-forward`, never an Ingress.
 apiVersion: v1
 kind: Service
 metadata:
  name: macos-vnc
 spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: macos-bluebubbles
  ports:
    - name: vnc
      port: 5999
      targetPort: vnc
 ---
 # The BlueBubbles REST API, once installed inside the guest. This is the stable
 # name cluster services use, so callers never depend on the pod IP or on whether
 # the backend is this VM or a real Mac mini later.
 apiVersion: v1
 kind: Service
 metadata:
  name: bluebubbles
 spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: macos-bluebubbles
  ports:
    - name: http
      port: 1234
      targetPort: bluebubbles
@@ -1,32 +0,0 @@
 # Dedicated StorageClass for the macOS VM disk.
 #
 # The default `longhorn` class does not work here, for two independent reasons:
 #
 # 1. Replica count. Default is 3, and Longhorn schedules against
 #    storageMaximum - storageScheduled with over-provisioning at 100%. Free
 #    space is cp-1 146Gi / cp-2 8Gi / cp-3 146Gi / worker-1 292Gi, so a 200Gi
 #    volume has only one node that can hold even a single replica — a 3-replica
 #    volume fails outright with ReplicaSchedulingFailure.
 # 2. Binding mode. `Immediate` provisions the volume the moment the PVC is
 #    created, before any pod is scheduled. Combined with strict-local that
 #    pins the data to an arbitrary node, not the one the VM runs on.
 #
 # So: one replica, kept local to the VM, bound only once the pod has a node.
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
  name: longhorn-imessage-local
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 reclaimPolicy: Delete
 # The pod is pinned to worker-2 by nodeSelector; wait for it to be scheduled so
 # the replica is placed on that node and not somewhere else.
 volumeBindingMode: WaitForFirstConsumer
 parameters:
  # A qcow2 backing a live VM is latency-sensitive and rewritten constantly.
  # Serving it over the network from another node's disk would be the single
  # worst thing for guest responsiveness, so force it local.
  numberOfReplicas: "1"
  dataLocality: "strict-local"
  staleReplicaTimeout: "30"
  fsType: "ext4"
@@ -57,6 +57,55 @@ spec:
      prune: true
      selfHeal: true
 ---
 # ZITADEL — the Authentik replacement. Coexistence, not cutover: Authentik keeps
 # serving in the same `iam` namespace until the last consumer moves, then this
 # becomes the only IdP. Migration plan (outside git):
 # ~/workplace/homelab-doc/ZITADEL_MIGRATION.html
 #
 # Wave 3 ordering: sops-secrets (wave 0) creates the `zitadel-masterkey` Secret,
 # databases (wave 2) creates the `zitadel-db` CNPG cluster and its
 # `zitadel-db-app` DSN secret. Both must exist before the init job starts.
 #
 # automated sync is deliberately absent for the first sync — same reason as
 # forgejo. The chart ships `pre-install,pre-upgrade` Helm hooks (zitadel-init,
 # zitadel-setup), which ArgoCD maps to PreSync and re-runs on every sync, manual
 # or automated. A human reads the first sync's hook output before any repeat can
 # run. The init job runs `init zitadel` only (CNPG owns the database and the app
 # user), and the setup job creates the first org and the `iam-admin` machine key
 # that Terraform consumes. Read the SECOND-SYNC HAZARD block at the top of
 # k8s/infra/iam/zitadel-values.yaml before enabling automated sync.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: zitadel
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "3"
 spec:
  project: homelab
  revisionHistoryLimit: 3
  sources:
    - repoURL: https://charts.zitadel.com
      chart: zitadel
      # Pinned: the chart trails the upstream app release, and the verified
      # masterkey/passwap facts were read at v4.19.x. A floating tag would
      # change the app version without a diff anyone reviews.
      targetRevision: 10.4.0
      helm:
        valueFiles:
          - $values/k8s/infra/iam/zitadel-values.yaml
    - repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: iam
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
 ---
 # Raw IAM manifests: key-rotation cronjob + authentik migration job.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
@@ -110,37 +110,6 @@ spec:
    syncOptions:
      - CreateNamespace=true
 ---
 # iMessage/SMS delivery. Raw manifests: a privileged macOS VM (Docker-OSX)
 # running the BlueBubbles server, plus its dedicated local StorageClass.
 #
 # Pinned to worker-2 via nodeSelector `workload: imessage` + a matching
 # toleration for that node's taint. Until worker-2 is provisioned this app
 # syncs everything except the pod, which stays Pending — that is expected.
 #
 # No CreateNamespace: namespace.yaml carries `pod-security: privileged`, which
 # the VM needs (/dev/kvm, privileged), and an ArgoCD-created namespace would
 # not have those labels.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: sms
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/apps/sms
  destination:
    server: https://kubernetes.default.svc
    namespace: sms
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
 ---
 # Document management. Raw manifests (no Helm): postgres is the dedicated
 # paperless-db CNPG cluster in k8s/infra/databases (wave 2), redis is
 # in-cluster only (no PVC), media lives on the cp-3 USB HDD (see
@@ -1,36 +0,0 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: azhi-riotpiao
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: '8'
 spec:
  project: homelab
  source:
    # Private GitHub. The read-only credential is Secret azhi-infra-github in
    # namespace argocd, created out-of-band.
    repoURL: https://github.com/ToyoGlobal/azhi-infra.git
    # The repository's default branch. It is not `main`: this repo's work
    # happens on the staging slice branch, and `main` is behind it.
    targetRevision: feat/azi-riotpiao-cnpg
    # One directory, because ArgoCD reads a single path per source. That
    # directory is a root kustomization over foundation, postgres and
    # azhi-server.
    path: k8s/overlays/azi-riotpiao
  destination:
    server: https://kubernetes.default.svc
    namespace: azi-riotpiao
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    # The two Secrets this namespace needs are credentials, so they are created
    # by hand and NOT by this Application:
    #   acr-pull                 — the private Alibaba ACR pull credential
    #   azhi-riotpiao-app-env    — DB credentials, JWT secrets, AES key
    # A missing imagePullSecret fails the pod outright and a missing app-env
    # Secret leaves it in CreateContainerConfigError, which is the intended
    # loud failure rather than a pod that runs against unknown values.
@@ -0,0 +1,20 @@
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: talos-repair
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "6"
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    targetRevision: main
    path: k8s/infra/talos-repair
  destination:
    server: https://kubernetes.default.svc
    namespace: cicd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
@@ -16,12 +16,6 @@ spec:
    - https://github.com/Riotpiaole/Poimen-memory.git
    - https://github.com/Riotpiaole/Poimen-workflows.git
    - https://github.com/Riotpiaole/poimen*.git
    # ToyoGlobal (GitHub, private). The azhi-infra Application reads its
    # manifests from here, so the URL must be listed or ArgoCD rejects the
    # Application with "repository not permitted". A read-only credential for
    # this repo lives in Secret azhi-infra-github in namespace argocd; it is
    # created out-of-band and is not committed, because it is a credential.
    - https://github.com/ToyoGlobal/azhi-infra.git
    # In-cluster Forgejo repos — explicit allowlist (no wildcard)
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab.git
    - https://forgejo.riotpiao.com/riotpiao-poimen/homelab-frontend.git
@@ -41,6 +35,7 @@ spec:
    - https://grafana.github.io/helm-charts
    - https://helm.releases.hashicorp.com
    - https://charts.goauthentik.io
    - https://charts.zitadel.com
    - https://strimzi.io/charts/
    - https://charts.bitnami.com/bitnami
    - https://homarr-labs.github.io/charts
@@ -0,0 +1,23 @@
 apiVersion: v1
 kind: Secret
 metadata:
    name: cloudflared-credentials
    namespace: cloudflared
 type: Opaque
 stringData:
    credentials.json: ENC[AES256_GCM,data:LnJVGuxJAOwDYbLgX00OHICDDldZgwgvGafOIwKoJKnjg3FjFMQYThNCOL/H+6tctuKixJNcAckjg70tmCKWxNq9I35XX/YSqob6Qv4fywv32hsNIAe44BU035MGZ5zmc6bZZ8wAgTJZw6Th4TzY7taFBCRdj2fW4T+qVfq7strn68G4ufVMK6UXlOkEDLjR2Vxj32lqSLdbdMma96XB+265hwL6IVfaugG662mcGhPwgHk=,iv:NFmyHN5m2mJUBgn1qgmJZ2YgkFxBZyp7+gU1Ng6B/7M=,tag:/66JtHK5Yu7pRxB0b0eifQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsaDVycUE4RFpoRjdpTlpX
            dGpzN3hwRFQ2cmtXVXVzWnZJOXJEZXZRWEhNCkJwbm9QV0lGcGtNRTJqbStrc3l1
            Q2M4bE1WbzJpTXZQWjJyTGpFS0JYcVUKLS0tIDBvU2U5aEtVTTZUaE85eEkyYitG
            bGtSVkhKNnRuNlZWMWh6Z1M4VXM1WU0KdusvtM85V0xrO+uJFUY+UH0tuwdF0GiW
            79J4Rlxk8ne25psKsvwjLYLxZ53xjFHCC4FCHTNkYwTKCDEKjw46Lg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-10-26T02:45:04Z"
    mac: ENC[AES256_GCM,data:KF7KwXCnNLAg1YI3u0lLI/ho4sITHAq09ac2WO4TK3TCthkuivb89YjhuuJKs0NFhjn6aRLqmZavUqcXH2HujiryqbLNVexOShQM0wSzgCzmWdbi0UzpgJ4KcQA3PufDtsQz4HHLXx396EbZrM3Ie3WSu/PvRm5iWOxzTGCGc9Q=,iv:Eo920oXXD1FLZVJXywZ6C+LRL+qZjmnZXv0+Oa/GMb8=,tag:6LuPVSet4WjvrvGUAxaJlw==,type:str]
    version: 3.13.2
@@ -1,23 +0,0 @@
 apiVersion: ENC[AES256_GCM,data:ECE=,iv:bISz4HovH++X7DW1Qj8Cw0L6+EvPB0+68hh7tfyW5C0=,tag:w+SjD7MsfeIuSf62n+Zl7Q==,type:str]
 kind: ENC[AES256_GCM,data:AxXaV3Mb,iv:xfJb354Rrjz3zLctW2i6hl40yh9EsfIqKxDvmn8jqnU=,tag:1xQ/RR3KhPLKCTWCYjebJg==,type:str]
 metadata:
    name: ENC[AES256_GCM,data:gdQ4OoMYKUPEbsUeA5OI4i1xnh8=,iv:IA2kaCyqQmkYpLolvcTF4aleh+yd/ImXJMhRMvpGCgo=,tag:8qyYYw4EhBKKPzEmpepAeQ==,type:str]
    namespace: ENC[AES256_GCM,data:/mOSyXWmRg==,iv:cpeHUTlMqlJzJttGtuR3DoiMtvVqFmDS0/5Tl7K7c2M=,tag:q4GxWgWn2wQJxJHqnq4WQA==,type:str]
 type: ENC[AES256_GCM,data:92OcrsuV,iv:Z1XBy6iZ6unGrK4/SSdDa58pbPL3022gU/f1AOM4uvc=,tag:cd4wWRq/ohm6BJ/Eo6HOIw==,type:str]
 stringData:
    REGISTRY_PAT: ENC[AES256_GCM,data:zDKruOXhzsIcFZTTp4r4r9rwHih9uf1cCp06KZ6eIXSIZDvR6aQY4A==,iv:5Vch5Z7xn1KkRxrgOs2p/M58nd6WhtcuUhoZDCi8LSY=,tag:14SAZIWGOxJMqLBtOorjYQ==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaG5QVUYyNjhvaENPcUFy
            K3RSSVM2N2hTeFhNK0M5YWZmODhzZU1HaHlFCmNGZWdFcFEvanVwcXpGSmVvRHVx
            aG45c2lBY1RuSTYwbDZTVk1QeHNWOTQKLS0tIDg4TnVMNjNtaU1VQk5zQjUvU2hM
            R245ZVdqc0ZWWVhJb3dOZ3lpU3JjUlUKawSg09ZPq8FKx5tvOVZZ+K4yh7eTQsUp
            be8mWUpS0+eEmNqh35BwU3HrETMQFA6a1kjVp30JOMtqa5rbYlzF7w==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    lastmodified: "2026-08-23T23:14:53Z"
    mac: ENC[AES256_GCM,data:yzi6woUaUCi6w9pG/eKnU7k/VfZgoXg/tW8p9joG8p7iabLWMdlXlx23m/CItw/NE0zeGWA5iZPPFZXOit2vN36VzK3kQNoFW6QbhvYLZi+78C/RWIdcpZkxB/tPIRG0vq8Q1SA+rwGjG/0xeAFh+R7k+YBTMd2XWBH3P7EI5T4=,iv:kf6MDaAVrtDvPIEjHMMLxXDSDRC3I1GpsPeJFUYppiw=,tag:NukVbGLa9EoMYmRsa4nBtA==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
@@ -11,7 +11,7 @@ files:
    - agent-pod-ssh-key.enc.yaml
    - authentik-secrets.enc.yaml
    - cloudflare-secrets.enc.yaml
-    - forgejo-registry-pat.enc.yaml
+    - cloudflared-credentials.enc.yaml
    - forgejo-registry-pat-api.enc.yaml
    - forgejo-registry-pull.enc.yaml
    - forgejo-runner-token.enc.yaml
@@ -38,3 +38,9 @@ files:
    - forgejo-registry-token.enc.yaml
    - pi-sandbox-secrets.enc.yaml
    - etcd-defrag-creds.enc.yaml
    - zitadel-masterkey.enc.yaml
    # Login UI RSA keypair, adopted from the chart-generated Secret so ArgoCD
    # owns it. Without this, the chart re-generates the keypair on every render,
    # which changes checksum/secret-login-service-key and leaves both Zitadel
    # Deployments permanently OutOfSync.
    - zitadel-login-service-key.enc.yaml
@@ -0,0 +1,23 @@
 apiVersion: v1
 kind: Secret
 metadata:
    name: talos-repair-config
    namespace: cicd
 type: Opaque
 stringData:
    config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
            N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
            eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
            aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
            7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-10-07T04:27:33Z"
    mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
    version: 3.13.2
@@ -0,0 +1,24 @@
 apiVersion: v1
 kind: Secret
 metadata:
    name: zitadel-login-service-key
    namespace: iam
 type: kubernetes.io/tls
 data:
    tls.crt: ENC[AES256_GCM,data:Ad4omL7pD4QhixW6wSVsVor5FtHbWyOHeYbxOQsEBAaSuvb7JIkvnKqo+Xy6zKuMmRViUIbqhlLQDXCaseAVGgpu3PdtShwJ7LRXJmr0XtQV1M6YtRbRexN6UpFA3XdD+JfifMhtvd29rM3WzLPX69uk4k7voYN5m0pNXSN23jVeIeqm+xB1CpxDW/guZ2hA5+b5XBzVOkMsSOSdWFWH2lK7nInv6ZatCa0/fuIKmB5nZpSV9tHpCTiTnBuNc0rW1X4HOWa8zMiSbjQAqtDitm7lrVHJyImXwBjmuS0wM35VwrwxVUUEjS8kgqnyb02JvNRdNjuwOmIcyKjrdgO1B1ft0CrAbRNkk2dYR8Y7SNhR4F4tYYF5L2szQgj+5tcYrnRGEI2QkXbcmTa0Y9V99+D2bKjt5t59Pi/1r8/jt0CNy0GeeZDvsN5scy0/A4WqtJ2sZDyTnTPwtUpAAwVC2WSqsD11gkzhyRJAgmAmEgfPIL7kTZsZBoSQeuuCJ82Ghqx9kLTpui6kLULxxUpP9oefsKw9PjTwExRbNNcPV5p7PPi9dN00TNe9iYuFqoZFP/lp5SMjDqbsV4Ie1ffcfyZsIHMuOv5Ths7n8ExYdR0zIn+aQePXASt+dE6njhFiblxyBZsn0xON3EIUaE9e3WsdcQKjlPOaAW/DvU6PwBqCeaB2nYzM2NFk6W3lN8BahTE9k7xXrILHIrXuIXezotCK6jclb5yZmdm/fTCX1iejGVDktvvcmX6bbv3YXTRZmsviRhLFHA8oClU0alTsY5Q3nn2Xl0UfXU+rW+10AIpY5WFMzP6A8hXvHx0ISC1S4AIqjmlsDmq2yEZKpbiL+Bs52ymW7t2rQ+KGhWH/bcC3VrJwUWnVYP9b80xyHZdKqubwLc5UcDzCs/Lan23VpLxSJkqb8vZP7SHQ1nc+rlwrqFDkzQ+yMNLZmutqbaAmMyXbrR51Rz3VqofraPXjS+lmLAuEcuTEFbrImsct5Zb57MtKvhHi6CUWt/Vj/NOUJlwy09zPy4JAMMJ81ooyBj6CBHY2f1vfR/UUDN5VZdfg4dUWXS0CQdOb3hx19COr4lR4RcpA46qJEjum+4icm5ckziYT5Nm7NB+ydXJ+Eys2Ba0dGRM6CIeQVhghhCDOZ6uoQ2A6DNql6wjrfUml8UwuJNO0nXVqNS28pLwddsOjlbsysJAUEma6EHXb2Vjb90JZErVswGuhQ2+UdhtpuIkgBYX/MMCZV942C+K41A9478BaCiWuC9KbnAcx81ceR+T3syARctZvQaYaiH7WfF8qyo0VOGzZRYjFbqeyHefiOmPmWTS/XbIti3evyhmxHwjUcidK2njrPB0O/3w6R69UA1p7ph9EgFsj18gPYnJthikgX8vcfVaVnPIAwCqY5DJJmIFTNXVkNDBX+2orrW2qj6PEH+go+uc+aUmu646niqaMAcrUl7gGNGd2OWFHdIRKdtogOvoVpQMDB4eWHPRsL29EVt3rnknHfpq7cIjQXuk9XtlB8zjllYdrh50Q8ZNBOvzWgCiMvAyanfIA+kW9KHQlmb1qJGYp7it2Lrg1K7sDEekrlbuoNsf+2rl8YRQz8pVBuCSzG9DPp4+Rj+ddiqX/Rrs3ufWDKFV9zJoN1EpJ8+z6n8rsQJBe/lSFLW1mLVkaPSNNdIQfdJ6MbsHpMlqqFYP8dOKckxv82uOkHoCLHrgaXiiMd1dBsX1fBiv2NvzDiuLyN81d7R2wNAQbkG9b+/+C6wVr6kDg0n6seseoCyeSI++PJIkCrkTXzd7rLs8Of9CHZQEB15wjemlmYGlJnCiFWPFa6h0jJPatPrfTYv+GpqzWV8u1Uh3aNjd+g1wVQok/ojNOZJahUSI4B67HkJI+I7WOF9C+rRCwNutqshIxYzW7CpyVey5P6Kyv+/cucDEgb0ggH032MBtzTxazpJcE4Y6fTrOiXHp9j2KjzyUt2YeCCL8UMwbwJvW9p1I5gx3ArsOuyHm1kA==,iv:PLsjmrTwjM7IUS7ZBSyVvJfAJzCSUtU+UAvHbSwoaVk=,tag:pUQwbHoZgfHW4CpPKoULAw==,type:str]
    tls.key: ENC[AES256_GCM,data:rO1NVJZP9TP96C48it4+2lGyRutwqdlX6qiM9R/+0hgYc5ceeeopaCUG/i/LOgltxRQK8qkyed/W3zU2VI6alBpt75XkT3o9ZRy3uxe8mB3ixS7+xQnMc0ci8Mf2bCtGzqwUQj8XPRP4K7Mxw8r4XO5rQLnSmNme0+BxS+XiE6hqifS16UUKfLuFeUtTEoI850sg1GQt0uaHf9kOrpmvsbg3sflO2dQXk8uSFi3B1ghG5DdWImVoc4f5kFK86XG65WVVfYjLaUvm7ZBPT0mwvq3SgPdEXeWq1fp7tMQqCRKuluRZon1e+J5SlWOPUTxwyTq336oK2Nfk3HsiQ3uCG76oCmI5yTe2tCN1f2IXFXBvLNM9QTxc/KFhI4uI6bvHjimOSpXYuRdroH0hc4ohLP44WG0E8Nz7sAb3Ai3PbDTKnqYQemvGEUTdcdbWuMK5/JE4f1IE9DgOOO9RF+s7uXTdzZAmoMcRtVsckQJEJl7AKgqSn7BbK0cjRL/QXclNySQ/qcgSDFiV/pPbGC9JFweJon5F0Qn/AIAnGGzqmVbS6x45UAJieI5oI/30OxjzF+Dov6EGMoE0mys+6l4j8v1EFCtHhhopS7zM4jZRTnNzlZWPtEISU2kopLroGpKESE145VQnB7X66F40KXcOIiXmU1H7Yxw5rJ0SqLAFWQw7gyNdzbnbfCmGASByxnYYwZRy3QDZYrQlthLd5tlGe0cg7j02ZSHmmJNQOx5yOmSQI7z4njDvJlOPncC14T59yDZUIYYLTAYjL8nmCNk2yD55ImVGRIb/f0JGAc2vHMlDN6fkuiSLtbASz11PwWzdln9tS9BTpRx7sfv5+kMhFkzomufftAJbs5XdOxcvGRHkRkdyCn3t8jcRXaNDBpq3o8zWLNh+411I65DMDN1B0c4+7Uw4N17I+KfrVh5Fb8ix0LW9aEL/SUxeIqdEu0ZMTncxGS0QVzZmtJUTdWuT0JjtgfsD65jqpg9l8RdIKX4i0ox2+GeQuXKCQiCaXS47yDxUcICYZB7UDnfwBcXlnGWw1frvkBBqwvwUVa7G7Al1A1a35ob1qb9gpyJZGqmEyV4Qe3PmfUHkp+/bAWarLXnk8P4l2faFjCEnPDIejGMQgDdtkodq7lfqwtHwg02bXNl9X3FbXjyp51/IJshE9dhHLkniWyHwZvHZJ4jjqndf08682gxTnW0bscfv7d6lKznE7rgd/el5ygdlL0PSr0pbxJtcd7MYrACMfZZxRuY3PWZayr1n4iaqh161S8ldeEcLCiHFvDeGHslUK+Sl383AHhs46NHEVgMbhuKJ1thHckQSgpqm9vGE1gBIo0d7z40EulgD+KS4Qm/vdydCgqtzyNmFDg17L+SVCQiyckGxgl/j1YzQKz0o1EWXNssk121m7Nh/e/bYNN8kC6doJPBQBkdhU/+DVZa4+/je+qpqvqnjpheX42k49ubk638QhOkcMmx6ZNnEbL9Ra0mNJHZO27t6++LgN2XkPg6t8VO94yx1c/VFyoF9Qg21O1+BadK3RF9l9s1eUB7n93YwUDhxZpimr8j8oLCn+PvksKwzWEIiAFXUWvuROwh6yC31GghO6veGx7UqXGPM2O8+mkOxXZObbnYAAhteRhXO/k15HL3qf94pudZ6+tETmeej4FKdhZzF9iOsTu/69izQFhl57E+n5hDjRzeVANKQOdq2x7qDG7pGfpJzrDVR95Z1q7Y/d+7FJjBML+YvzGvbxZno+PbPmLGrgxSebAWgO73fUDkBHvosMDjeHjg0NKBXYfR3kImceo4iJwF0bOeFQA09j+6OKW1EId9r1EzYwPGR0DUDt+A9f4DkU/27dbKUmJovBPIAXtYHErwZ7EW0ITd8ebNXyOiArz8JJietDsR++lwetArzrDszsPsiumoSu1/FajtbBfUQqllxJBeoIqNEPHQED2ysSWzqTlHe8tfmY3Vp/PlZh8GbpA/EV5uikWzDSaZbADqXGny/xd+g+8mm4J3DxgPOEEgFqeg+q4C1U4yEy3gYkknT6AcIO8MGOa3daRowEv2zfhVlrNhv2ezLjzsPBQJSK/ph9KZbYiREZVCBQioJJlvfG/dkkycjkVqvTNtn3swhipxUqYc0vKmXFG+R9+fUOP5u9ctqmdIf7Jclew4OBO1ML+sH+l6NojgfX+T+OqQ8EyNMY2Aa7i24ppwS8ySDwyYEk9kjopVZ8L3MSw3CQnWBNMg734qUawCUoFMtGPZ0jC+o6DiV18xpkB1Ffrmjs+72AoKk6wR9zIYyVQG7wr7O4PLWeXEX2NgBDeKaWi0gHInVAb2puJKgGgaZbImuecSaKaqA0y1fXu3+4teozv9eLkBxhYUJRwIBQbA7E3lQ4ocziE3ZMxdAIGAOEJHgDs6UreB4ZMLgdKxqhYzR3GxrmZmspOw/TVBGGjClrefGZh0/gLkXSDZ0uugW1rNM3TkA9t2U0eBCNyRh+8sScUKzja4Ei2ukkfoCNEf0alJkG3udwBtjDBcTRhQJARFD+UWMut5Xqw/dAxs777/VmY00/wD9FaCq0aVc/CJI++oGYgunPDd7ACSTyuY0UST1AEBrCrFBB+T+eKrFcU/4gBXEGiAY4x8lrp7fNZ/nsGFvBvs5sxq9NvLFXGA8Tv3LHU8x8m8oGHVnT0AaFJIKM8vGf2bkdD89EmQkKG9pfnKPCTbyFszqOJlgORDLSn9bNvDYlVC3dS5HQbqdv1a1fn1WrLDEXTNuD1Nj6dJXUBLAZfqzRHaPRi04YNpw9MY+c6f3XmAvNmwrFgdnxHja47ObX4XGigltlCWUSImRboBinCeZ9k9GdQ82Pw36dO433Z3IjtjuFLTWFsUurpAvmgiemzJ3xnhra6pOEr2jy2dw6fUL4gMQtPpSwqB4Qd9TY6MOUpwBqK0MpuRMBisa3zOBphJyvnXOIG9RCYzivzOQM+PKzLmznjTyeql1Ipj/9LwEgQ==,iv:PbRgjgm5ERWlAkkWc7As8eIx8e0rjTUkQ7XlDBrEUlU=,tag:yh2/JuHu5I+dWlxVAmkkXA==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOGNaakZoV3lhemhMSVVF
            RklKbEsxeVFMWWdwQ3ZqR0lhRHlPM1dWT3h3Cnk2RnZUNnplVEwvNW9qMExsSUlt
            VVVXejMrcDRRTEdtT1krdThqWThLWWcKLS0tIEVZZEVBWFZyVi84S1puRHdtMWR1
            cDBHc1J5aHFpQkxSSEx2VDRjZ1RsdXcKG5MgnUBpFmlNGSkxBijXp7m492ujyqsC
            TbTfU8SWJwXhio7V+zDbXG+JW4R51wr9F9Sj47DCw7b64nbsD3N5TQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-10-26T02:15:51Z"
    mac: ENC[AES256_GCM,data:+2eGERwwzeK5JE9H6qhsu+tskR3Y9c7DecVGopfG8Ggq5SHQBfYyvz//IU+MGCOb6nIyP3JPf9+9IDS2DthVcw+RLtviQkN1TeWuUTDbxBxHdCoUv9zepwU7+d0qUMrdUK0N9QQSRZ4TpiwceViDeMuhPF/vTmADP9QWwid/qwk=,iv:nuJo37Dh8MAf+qI/Ty2jsW0IheNhgmZ4l0Va8pJG6+4=,tag:vCpifXvYF1/QU7XExc6raQ==,type:str]
    version: 3.13.2
@@ -0,0 +1,23 @@
 apiVersion: v1
 kind: Secret
 metadata:
    name: zitadel-masterkey
    namespace: iam
 type: Opaque
 stringData:
    masterkey: ENC[AES256_GCM,data:Da543Jt48o6p3f6/AbZRsjChz3NexPJoi/mC/LB6jUI=,iv:irS0n76tZ6QzoQMm0dWrvbw+SPfXz4h13jQb/Bjfgwg=,tag:tfbgaAdxmPIjleB1OXLn0Q==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZld2VG9NaFhIVWpBcGpO
            SmVwNm9FT2wrQmFKb0RGQmY4TCtITjFEZjBjCmNuNXNJYTdsZE44cDNzR2hiNTdy
            QWlVdzBtdjd0cSsxUjQxMzNyblhWTkkKLS0tIHBLT2x1MDBGdjE1eHFOdHVGa1NO
            ZGM1K2lDTksraFdPeFdSZ1JyU1VtcjAKgTjjhr7u4YRuQVZQ7kAa1clJ0Rzy10hQ
            Umcd4/YStyXHZjsJJpFSpOFdP5cZscGo3xBgpXtxAscKTS1Nw4derg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-10-25T01:24:21Z"
    mac: ENC[AES256_GCM,data:QFVk7KLsMLCAMH/PHVGCRyfU3k2JiO8AE3jJXmhnlUB4C8MYCTyOWQbnTFXozCMOr0kp+hAt3dYn/e23SWhyvJ7BgVLRb5QF1py704Dr46atVoLUvwHpFI7dyGK/LDTxqiM40RXAnDtWUJlTkVCzKdSFiV8CokIoByKWwxFPayM=,iv:t7HWMcpQl/DJ8Rp51Du7b5gURZO+C+FodsZSJuy5jBc=,tag:qEXSS4S0Y5cvn7T6EjakhA==,type:str]
    version: 3.13.2
@@ -271,3 +271,40 @@ data:
    NDKi7/Gxha790QN3TD/U7p9GcEtfXKv8KISKfxnfD9o9Yx6pWOXcgvGefQkpXuRm
    kGzDM+aY/VKs5Et5/RyKavBDihuOcumLsmoRg/bFoB/xpeURTyiehUdu
    -----END CERTIFICATE-----
 ---
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: homelab-ca
  namespace: cloudflared
 data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIFBjCCAu6gAwIBAgIJAKoBPdKPDL5AMA0GCSqGSIb3DQEBCwUAMDAxEzARBgNV
    BAMMCmhvbWVsYWItY2ExGTAXBgNVBAoMEHJpb3RwaWFvLmhvbWVsYWIwHhcNMjYw
    ODEyMjA1NDQzWhcNMzYwODA5MjA1NDQzWjAwMRMwEQYDVQQDDApob21lbGFiLWNh
    MRkwFwYDVQQKDBByaW90cGlhby5ob21lbGFiMIICIjANBgkqhkiG9w0BAQEFAAOC
    Ag8AMIICCgKCAgEAzw796vZRFOBc/orUu9J6rv91A3SNLJsqz/VuetwAXCaVSlK5
    kL/RaBKKkcXToCC5avwix6W21FnwLUW4R8QlQdin8+XEHpx2LY5R4GmSvaCbU7wm
    YwYgVGQVv8KlgcMbz/pF09vXU29cHUHwlkN+BWVGAxVGWxam94j1pBzV0dOhhRw2
    q5sZYUUrPO5l775od2FMMnMzoRFugxDvJof0p3fAChNAIiFhP95lrcb2YsTwK+a2
    6mk3AZc0gMtmgVsH+zryGFcAmWLixq41ye/H23ktj2bjEHCr9mIDpuSOKLyLBzrg
    js/HqdxvwkZkkxwXft0Vi7GN0/LGHMdPrDE499pJelAHbxUpBoOk8sfXSNnfESI8
    v20LybATGrKXFlOhMKshY1a3JLAoZTaW0xkG4PM4HT7lK/ovr/NLF6kdP7HxJFFQ
    +4WjySiwbjusNobpTaQWVwCjD3Imf+2feAqJeX3z+P3EhxfrmvEojLOdUiiPcjMn
    CFl9u2EpSlax53umSkcBQJGfFKM17fZzhtakjqhCTzAfXXlxZy4hsHDtfzRcfaoB
    Zz/PiJ0TtJSC7TvxZZoPpGexQdoeT7kJEksTvGi5OSfJD7HaKe5RQSZGgpz7ATB+
    YMBKbjhMXVu+TAdcO1qZRN0UCncrliN61ki9F0tE8kNgpS9biZ6kheoJMZUCAwEA
    AaMjMCEwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwDQYJKoZIhvcN
    AQELBQADggIBAFN1At11d/+Ue6cEjCklnRQpLuD5agXaE+pi5oHNfnggSlO8UjUj
    lDzfnmxi3IAI6HruzDARM9J95qJboUxWOAEmK9SNnZNS/1b0GyikP3YcF1gIFqtW
    6SgmVfC1cVUFKGt/+QQzZxhOLuoydv9Ozu12nxsTW/WY2t+cpeL+1wrsM22RA6W/
    xV/bzYbwzCUT9v4tMZLZHk5CkXztVFXSJyW/8w9rVZdUGZM+mwwn81PLSE0v3s/2
    wJH8eUqGl73i3Xgn7CVanN4cJYMGYVVlRD6Qr6muBN4Vh0ETY1PrjzsZ6XsV6vk9
    glMbQnErYMZHZIDtnVdaBt60MecUD9O4eLeip0SE6bVRAucKBXA9xNMilMQEL8/X
    FBMAxXl10rowBlbX2oaFSOM80eLTla4a3gSL7+0vua3C4Dtv2bLo8ZjCyqXwOJeb
    N85cqcVZStz3qlgQN1nG2qGE3BS6scOs7+mUGe9CmS8acJV++PKm8CtmLH6/ibjV
    tAc/6+Vg/keo0kty7H7L6oQd+gKpuaJ2I+wKYe4l9UwHi9zEb8YjzGB4n2AtsoWI
    NDKi7/Gxha790QN3TD/U7p9GcEtfXKv8KISKfxnfD9o9Yx6pWOXcgvGefQkpXuRm
    kGzDM+aY/VKs5Et5/RyKavBDihuOcumLsmoRg/bFoB/xpeURTyiehUdu
    -----END CERTIFICATE-----
@@ -7,5 +7,6 @@ kind: Kustomization
 resources:
  - namespaces.yaml
  - authentik-db.yaml
  - zitadel-db.yaml
  - temporal-db.yaml
  - paperless-db.yaml
@@ -0,0 +1,42 @@
 # Dedicated CNPG Postgres for ZITADEL (GitOps, wave 2 — before the zitadel
 # chart at wave 3). Same shape as authentik-db.yaml: CNPG generates the secret
 # `zitadel-db-app` and the service `zitadel-db-rw` in ns iam, and
 # k8s/infra/iam/zitadel-values.yaml reads the DSN from that secret locally (no
 # cross-namespace secret copy).
 #
 # ZITADEL requires PostgreSQL 14+; the repo convention image is 16.2.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: zitadel-db
  namespace: iam
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
 spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: zitadel
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
  enableSuperuserAccess: false
  resources:
    requests: { memory: "512Mi", cpu: "250m" }
    limits: { memory: "2Gi", cpu: "1" }
  storage:
    size: 10Gi
    storageClass: longhorn-cnpg
  monitoring:
    enablePodMonitor: true
  affinity:
    # preferred + tolerations: HA across nodes without deadlocking a partly-
    # schedulable 3-CP (same as authentik-db and forgejo-db).
    podAntiAffinityType: preferred
    topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
@@ -0,0 +1,226 @@
 # k8s/infra/iam/zitadel-values.yaml
 # ZITADEL — Authentik replacement. Coexistence is deliberate: Authentik keeps
 # running, read-only, in the same `iam` namespace until the last consumer moves.
 # Migration plan: ~/workplace/homelab-doc/ZITADEL_MIGRATION.html (outside git).
 #
 # Chart: zitadel/zitadel 10.4.0, app v4.19.2. The plan cites v4.19.4; the chart
 # has not published it yet. Pin the chart, never `*`, until the v4.19.4 chart
 # exists — a silent app bump invalidates the verified masterkey/passwap facts.
 #
 # URL: https://zitadel.riotpiao.com. TLS terminates at ingress-nginx, which
 # serves the wildcard riotpiao-com-tls as its default-ssl-certificate, so
 # ExternalSecure=true and TLS.Enabled=false (no per-rule tls: block needed).
 #
 # Database: CNPG cluster zitadel-db (wave 2). The DSN comes from CNPG's
 # generated `zitadel-db-app` secret. When ZITADEL_DATABASE_POSTGRES_DSN is set,
 # ZITADEL ignores the discrete Database.Postgres.* fields, so neither a
 # password nor a host is written into this file.
 #
 # Masterkey: Secret `zitadel-masterkey` (k8s/argocd/secrets/
 # zitadel-masterkey.enc.yaml), key `masterkey`, exactly 32 bytes. It wraps the
 # rows in system.encryption_keys, which encrypt every client secret and private
 # key. Losing it makes those values unreadable: keep the offline sealed copy.
 #
 # Ingress uses the chart's own objects (controller: nginx adds the gRPC backend
 # annotation ZITADEL needs for its HTTP/2 API). Do not hand-write a second
 # Ingress for this host.
 zitadel:
  configmapConfig:
    ExternalSecure: true
    ExternalDomain: zitadel.riotpiao.com
    TLS:
      Enabled: false
    # First boot is COMPLETE (2026-10-26): the instance, the system org and the
    # `iam-admin` machine key all exist. Skip stops the setup hook from running
    # the first-instance step again. Without it, every sync re-runs the
    # machinekeyWriter, which silently rotates the iam-admin key Terraform
    # reads. ZITADEL objects are declared by scripts/iam/zitadel-provision.py.
    # Never set this back to false.
    FirstInstance:
      Skip: true
  # Rendered into a Secret by the chart; read by pods and init jobs alike.
  masterkeySecretName: zitadel-masterkey
  resources:
    requests: { memory: "512Mi", cpu: "250m" }
    limits: { memory: "2Gi", cpu: "1" }
  # Best practice: run as non-root, drop all capabilities, and refuse
  # privilege escalation. The chart's image already supports a non-root UID.
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault
  # FirstInstance defaults stay for the FIRST sync: the chart creates org
  # `zitadel` plus the `iam-admin` machine user and its machine-key Secret.
  # Terraform consumes that key to declare projects, roles, OIDC apps and the
  # groups action.
  #
  # SECOND-SYNC HAZARD — read before enabling automated sync.
  # The chart marks eight objects as `pre-install,pre-upgrade` hooks (two
  # ServiceAccounts, two ConfigMaps, Role, RoleBinding, zitadel-init,
  # zitadel-setup), and one cleanup Job as `post-delete`. ArgoCD maps the first
  # eight to PreSync with `before-hook-creation`: it deletes and re-creates them
  # on EVERY sync, not only on a chart change.
  #   - zitadel-init  (`init zitadel`) — schema step
  #   - zitadel-setup (`setup`)        — first instance + iam-admin machine key
  # On an initialized instance the setup step can fail, which aborts the sync
  # before the Deployment is applied. If it succeeds instead, its
  # machinekeyWriter container re-applies the iam-admin key Secret that
  # Terraform reads, which silently rotates a credential in use.
  # After the first sync reports healthy:
  #   1. set configmapConfig.FirstInstance.Skip: true here — permanently: the
  #      setup steps are schema-only once the instance exists, and Skip alone
  #      stops the machinekeyWriter rotation
  #   2. set initJob.enabled: false below
  #   3. commit, sync once, then enable automated+selfHeal on the Application
  # For a later schema upgrade, re-enable initJob.enabled only — never set
  # FirstInstance.Skip back to false.
  # ArgoCD never prunes a hook Job, so a Job that failed before step 2 stays and
  # keeps ZitadelSetupJobFailed firing until a human deletes it.
 # DSN mode (chart renders this through the zitadel.dbEnv helper). CNPG's `uri`
 # key is a postgresql:// URL, which ZITADEL accepts as a DSN.
 env:
  - name: ZITADEL_DATABASE_POSTGRES_DSN
    valueFrom:
      secretKeyRef:
        name: zitadel-db-app
        key: uri
 # Bundled Bitnami Postgres is disabled: the CNPG cluster above is the system of
 # record. Enabling it would create a second, unmanaged database.
 postgresql:
  enabled: false
 # CNPG already created the database, the `app` user and its grants, so the init
 # job runs only the ZITADEL-internal step. Running the full default command
 # would try to create a database and a superuser it does not own.
 #
 # enabled: false after the first successful boot (2026-10-26). The schema is in
 # place; the init hook has nothing left to do and re-running it on every sync
 # only adds a PreSync failure surface. Re-enable it for a deliberate schema
 # upgrade, then set it back to false.
 initJob:
  enabled: false
  command: "zitadel"
  activeDeadlineSeconds: 600
 # The setup job runs the full first-instance creation: schema migrations, the
 # system org, the iam-admin machine key, and 13 projection replays. 20 minutes
 # leaves room for a cold database without masking a genuine hang.
 #
 # DONE, 2026-10-26 — first boot completed. The annotations are moved from
 # pre-install/pre-upgrade to post-delete because a PreSync setup run is not
 # harmless after boot:
 #  - it re-runs every repeatable migration on every sync;
 #  - interrupting one (a sync aborted, or a human deleting the Job) leaves a
 #    `system.migration.started` event with no matching `done`, and ZITADEL then
 #    waits forever on `migration already started`. That is exactly how this
 #    instance deadlocked on projections.projects4 and later on
 #    repeatable_delete_stale_org_fields.
 # A post-delete hook is never executed by ArgoCD sync, so setup runs only on
 # first install. Re-enable it deliberately (and only with initJob too) for a
 # schema upgrade, then revert.
 setupJob:
  activeDeadlineSeconds: 1200
  annotations:
    helm.sh/hook: post-delete
    helm.sh/hook-delete-policy: hook-succeeded
    helm.sh/hook-weight: "2"
 # Two API replicas: the read path survives one node drain, and both share the
 # same masterkey and database. The login UI stays at one replica — a session
 # cookie is not shared, so a second pod buys nothing.
 replicaCount: 2
 revisionHistoryLimit: 3
 # Best practice: a PDB guarantees at least one API pod stays available during
 # voluntary disruptions (node drain, rolling upgrade). Without it a drain can
 # take both replicas down at once. minAvailable 1 is correct for 2 replicas;
 # minAvailable 2 would block every drain forever.
 pdb:
  enabled: true
  minAvailable: 1
 # Best practice: spread replicas across nodes so a single node failure cannot
 # take both API pods. Anti-affinity alone only prefers separation; topology
 # spread enforces the failure-domain budget.
 topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: zitadel
 ingress:
  enabled: true
  className: nginx
  controller: nginx
 login:
  enabled: true
  # ksops owns this Secret (k8s/argocd/secrets/zitadel-login-service-key.enc.yaml,
  # rendered by the sops-secrets Application). Pointing the chart at it stops the
  # chart from auto-generating an RSA keypair every render, which changed
  # checksum/secret-login-service-key and kept both Deployments permanently
  # OutOfSync. The keypair must stay RSA: the login container signs JWTs (RS256).
  loginServiceKeySecretName: zitadel-login-service-key
  replicaCount: 1
  revisionHistoryLimit: 3
  ingress:
    enabled: true
    className: nginx
  # Best practice: the login UI is the public attack surface. Harden the pod
  # even at one replica. The chart has no login PDB key; a single replica has
  # nothing to protect with minAvailable.
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault
 # Chart-owned ServiceMonitor. The cluster's Prometheus runs with
 # serviceMonitorSelectorNilUsesHelmValues=false, so the release label is not
 # required for discovery. Endpoint is /debug/metrics on port http2-server.
 metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    scrapeInterval: 30s
    scrapeTimeout: 10s
 # The 3-CP / 2-worker cluster mixes control-plane taints with worker capacity.
 # Without the toleration a rolling upgrade can leave both replicas Pending.
 tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
 affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: zitadel
@@ -0,0 +1,68 @@
 # ZITADEL alert parity with the Authentik rules it will replace.
 #
 # Only two signals are ported at this point. The metrics endpoint
 # (/debug/metrics, scraped by the chart's own ServiceMonitor in ns iam) exposes
 # no request or error counters that this cluster has verified, so no error-rate
 # rule is written here. Writing one against a guessed series name would produce
 # a permanently-firing or permanently-silent alert — both worse than none.
 # AuthentikOutpostDown is deliberately not ported: it monitors an outpost that
 # carries zero providers and is deleted with the migration, not migrated.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: zitadel-rules
  namespace: iam
 spec:
  groups:
    - name: zitadel.rules
      interval: 30s
      rules:
        - alert: ZitadelDown
          # absent() is required: equality alone cannot fire when the series is
          # gone (ServiceMonitor deleted, Service selector changed, pods scaled
          # to zero). Those are exactly the outages a target-down rule exists for.
          expr: absent(up{job="zitadel"}) or up{job="zitadel"} == 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "ZITADEL API target is down"
            description: "Prometheus cannot scrape the ZITADEL API for 10 minutes. Token issuance and every OIDC login depend on it. Check the zitadel pods and the zitadel-db cluster in namespace iam. Value: {{ $value }}"
        - alert: ZitadelLoginUIDown
          expr: absent(kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"}) or kube_deployment_status_replicas_available{namespace="iam",deployment="zitadel-login"} == 0
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "ZITADEL login UI has no available replica"
            description: "The login UI serves /ui/v2/login. With zero available replicas a browser login completes at the IdP but cannot render. Value: {{ $value }}"
        - alert: ZitadelRestarting
          # pod!~"zitadel-db-.*" matters: the CNPG database pods share the
          # `zitadel` name prefix, and a database restart must not page as an
          # IdP crash loop.
          expr: absent(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}) or increase(kube_pod_container_status_restarts_total{namespace="iam",pod=~"zitadel.*",pod!~"zitadel-db-.*"}[1h]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "ZITADEL pod restarts more than 3 times per hour"
            description: "A crash loop on the IdP is a cluster-wide login outage. Check the pod logs and the masterkey/database secret references. Value: {{ $value }}"
        # The chart's zitadel-init and zitadel-setup Jobs are PreSync hooks with
        # restartPolicy: Never, so their restart counter stays 0. Without this
        # rule a failed schema init or first-instance setup reaches nobody but
        # the person who runs `kubectl -n iam get job`.
        # No absent() guard here on purpose: the chart ships only
        # `before-hook-creation`, so a successful hook Job is replaced at the
        # next PreSync. Once the jobs are retired, absent() would fire on a
        # healthy cluster.
        - alert: ZitadelSetupJobFailed
          expr: kube_job_status_failed{namespace="iam",job_name=~"zitadel-(init|setup).*"} > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "ZITADEL init/setup job failed"
            description: "A PreSync job failed, so ArgoCD aborted the sync before the Deployment was applied. Read the job pod logs, then delete the Job object to let the next sync retry. Value: {{ $value }}"
@@ -8,6 +8,7 @@ resources:
  - alerts/ingress-alerts.yaml
  - alerts/svc-argocd-rules.yaml
  - alerts/svc-authentik-rules.yaml
  - alerts/svc-zitadel-rules.yaml
  - alerts/svc-forgejo-rules.yaml
  - alerts/svc-grafana-rules.yaml
  - alerts/svc-minio-rules.yaml
@@ -55,7 +55,10 @@ alertmanager:
 prometheus:
  prometheusSpec:
    retention: 30d
-    retentionSize: "95GB"
+    # Must stay under the PVC size below (25Gi). At 95GB the limit never
    # triggered, the volume filled, and Prometheus crashlooped with
    # "no space left on device" writing /prometheus/queries.active.
    retentionSize: "18GB"
    # scrapeTimeout MUST be <= scrapeInterval or the operator refuses to generate
    # the Prometheus config ("scrapeTimeout greater than scrapeInterval") and no
@@ -31,29 +31,3 @@ roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: oidc-bootstrap
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: oidc-bootstrap-restart-apiserver-v2
  namespace: kube-system
 spec:
  backoffLimit: 4
  ttlSecondsAfterFinished: 7200
  template:
    spec:
      serviceAccountName: oidc-bootstrap
      restartPolicy: Never
      containers:
        - name: restart
          image: docker.io/curlimages/curl:8.22.0
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -u
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              for pod in kube-apiserver-talos-cp-1 kube-apiserver-talos-cp-2 kube-apiserver-talos-cp-3; do
                curl -sS --max-time 15 --cacert "$CA" -H "Authorization: Bearer $TOKEN"                   -X DELETE "https://kubernetes.default.svc/api/v1/namespaces/kube-system/pods/$pod?gracePeriodSeconds=0"                   -o /dev/null -w "$pod -> HTTP %{http_code}\n" || echo "$pod -> request failed"
              done
              echo "deletes requested; kubelet recreates each static pod in ~20-40s"
@@ -1,141 +0,0 @@
 # One-shot: restart the control-plane API servers so their OIDC authenticator initialises.
 #
 # Two earlier mechanisms failed and are documented so nobody repeats them:
 #   1. deleting the mirror pod does NOT restart a static pod (deletes returned 200, no restart);
 #   2. matching /proc/*/cmdline for 'kube-apiserver' also matched this script's own cmdline,
 #      so every pod SIGTERMed itself (exit 143) before reaching the real process.
 # This version matches /proc/<pid>/comm (the process name) and skips \$\$, so self-kill is
 # impossible. hostPID + privileged are permitted in the 'cicd' namespace. Delays stagger the
 # three control planes by 2 minutes each, so only one API server is down at a time.
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: oidc-restart-apiserver-cp1-v2
  namespace: cicd
 spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      hostPID: true
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: talos-cp-1
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: kill
          image: docker.io/library/busybox:1.36
          securityContext:
            privileged: true
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "sleeping 10s before restarting the API server on talos-cp-1"
              sleep 10
              self=$$
              killed=0
              for d in /proc/[0-9]*; do
                pid=${d#/proc/}
                [ "$pid" = "$self" ] && continue
                comm=$(cat "$d/comm" 2>/dev/null)
                [ "$comm" = "kube-apiserver" ] || continue
                echo "SIGTERM -> pid $pid"
                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
              done
              echo "signalled $killed API server process(es); kubelet recreates the static pod"
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: oidc-restart-apiserver-cp2-v2
  namespace: cicd
 spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      hostPID: true
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: talos-cp-2
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: kill
          image: docker.io/library/busybox:1.36
          securityContext:
            privileged: true
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "sleeping 120s before restarting the API server on talos-cp-2"
              sleep 120
              self=$$
              killed=0
              for d in /proc/[0-9]*; do
                pid=${d#/proc/}
                [ "$pid" = "$self" ] && continue
                comm=$(cat "$d/comm" 2>/dev/null)
                [ "$comm" = "kube-apiserver" ] || continue
                echo "SIGTERM -> pid $pid"
                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
              done
              echo "signalled $killed API server process(es); kubelet recreates the static pod"
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: oidc-restart-apiserver-cp3-v2
  namespace: cicd
 spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      hostPID: true
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
        - name: kill
          image: docker.io/library/busybox:1.36
          securityContext:
            privileged: true
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "sleeping 240s before restarting the API server on talos-cp-3"
              sleep 240
              self=$$
              killed=0
              for d in /proc/[0-9]*; do
                pid=${d#/proc/}
                [ "$pid" = "$self" ] && continue
                comm=$(cat "$d/comm" 2>/dev/null)
                [ "$comm" = "kube-apiserver" ] || continue
                echo "SIGTERM -> pid $pid"
                kill -TERM "$pid" 2>/dev/null && killed=$((killed+1))
              done
              echo "signalled $killed API server process(es); kubelet recreates the static pod"
@@ -0,0 +1,12 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # Stable secret name: the job mounts talos-repair-config by name, so the generated
 # Secret must not get a hash suffix (same convention as k8s/argocd/secrets).
 generatorOptions:
  disableNameSuffixHash: true
 # ksops decrypts talosconfig-secret.enc.yaml into the Secret the job mounts.
 # Kept local to this app on purpose: no edits to the shared secrets generator.
 generators:
  - secret-generator.yaml
 resources:
  - restart-kubelet.yaml
@@ -0,0 +1,57 @@
 # One-shot repair: the kubelets on cp-1, cp-2 and worker-1 stopped posting node status at
 # 04:11-04:14 UTC and never reconnected, so those nodes are NotReady/unreachable and every
 # pod on them (Cilium included) shows NotReady. This job runs on the one healthy node (cp-3),
 # talks to the Talos API on the LAN, and uses Talos' endpoint->node proxying to reach the
 # other three: it reports the kubelet service state, then restarts it.
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: talos-restart-stalled-kubelets
  namespace: cicd
 spec:
  backoffLimit: 1
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: talos-cp-3
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: talosctl
          image: docker.io/curlimages/curl:8.22.0
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -u
              echo "fetching talosctl v1.13.3 (matches the node version)"
              curl -sSL -o /tmp/talosctl \
                https://github.com/siderolabs/talos/releases/download/v1.13.3/talosctl-linux-amd64 \
                || { echo "DOWNLOAD FAILED"; exit 1; }
              chmod +x /tmp/talosctl
              T="/tmp/talosctl --talosconfig /talos/config -e 192.168.1.162"
              NODES="192.168.1.166 192.168.1.214 192.168.1.223"
              echo "=== BEFORE: kubelet service state ==="
              for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
              echo "=== kernel log tail (last 15 lines per node) ==="
              for n in $NODES; do echo "--- $n"; $T -n $n dmesg --tail 15 2>&1 | tail -15 || true; done
              echo "=== ACTION: restart kubelet ==="
              for n in $NODES; do echo "--- restarting $n"; $T -n $n restart kubelet 2>&1 || true; sleep 20; done
              sleep 60
              echo "=== AFTER ==="
              for n in $NODES; do echo "--- $n"; $T -n $n service kubelet 2>&1 || true; done
          volumeMounts:
            - name: talos
              mountPath: /talos
              readOnly: true
      volumes:
        - name: talos
          secret:
            secretName: talos-repair-config
            defaultMode: 0444
            items:
              - key: config
                path: config
@@ -0,0 +1,10 @@
 apiVersion: viaduct.ai/v1
 kind: ksops
 metadata:
  name: talos-repair-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
 files:
  - talosconfig-secret.enc.yaml
@@ -0,0 +1,23 @@
 apiVersion: v1
 kind: Secret
 metadata:
    name: talos-repair-config
    namespace: cicd
 type: Opaque
 stringData:
    config: ENC[AES256_GCM,data:WO+jinh/2Rq5VeGLoDDJOQL4zfYJINSTYZCwGWndbzfIHw4XRwxDB7BixjCdpPuLOkNIBAXHQpKPBhlQpvRTso70L26njgVdVENUGeNhxvb6/cwl3VYRsfQQxKoQW9Qu6DyCD7ItuyazXMdIbENKBoih7GA2IjKdGeEtpB+a6PO+6rpI0AQRIjM9h65yJ6gvWeLZmb6U0qCFW61SelHmVSsfehxH/R9EKQROKwD0Z/TgYnoS28OuDtixAQVfnrI3gtgUAeXgnPQTJf1Vb9iwbRBvaSWkPSIqa7hOqWEIM2qyrvhTsEb8XDRUHTmp4BCxYktIxoiSWZEbeh0y0HKRzekZIj4rHA+c/JYSPqS0dvn1Ga+8PTc1LNdXDWIcKI58YQ2jzA0oE2vm0YzmOjcDTzTGLDBk49gIql+AMfwoxCop7t1PB7BrJw6M4bj9M5aHIq8ZlGXDvtWZcSEBj0ChZ0q20cB4v4bvMkbuJR/PS25uNkeG+UBCPidBQqCUw37aNf8SDwOaAC+6PeHdLmcJ/dkKntGFlU5MQzWQXfqs/fp95ze3mr7f3et7SqVl7fAlYSdEt6dA2xxnFnqgNwAm3UacV/We6SRnjrFlBAHZ3Q5hIuKkxmszsS4Bh/VuNtZI5YJ4ldnoIZC3aWKDt+rVB6P+Qxp1T1XqZj8GY715DZRP1q+Z7iQ9RcXiJBTD5qWVpnkESOvbTEYv/W3oXTufVRrJ8Ps0on/xWRF5NU04+tqvToC8+MIQDz5rb+eXYCEWjZG9ZhInY+M2HRTLg+c49dRlyqVnnqfNWBejSS/hoQbf5HmkUbtzsCftvu2NwS2nOMhm55xcCsAunB+0uY+Bj4jqyqRvWUjA99yHIdQH5JTJXE4FQPxdBTmBFp8xVdYWvlnCojK4qfv5EurLUunBugdSdYidi2TKYEoW2mX19RdVnkQgBjW9caaZUX9BIi+zNhObKenoWxRORCb0dHw72mwXMaSQM4IDygcuwi/79OtkYik3juECVmI58bu/GFRo2NtmgkGdMD29SMV6rerDtYjGjetgFvWkDy6hrB/Vz4dI37eGt/EH/7G0z/IT9Q2sM8nzJDtCYdicepLQ1WaM0P4BEn2rrqhv81VvDH7M9e7wtz4xly5RYQUr5wkzvoSEIglFekPvPmhIxEsNdaBj6MlVmLTbEbv85oBLwm0ZLRJcgUsrBbZghGKQaI5udAk1dgwOng9v0Ai+ilJ2RSt2VI6WG1rwRnDG+Hv+H1cKT/NwA8ocmM/kxQlUE3BxESrxhTnKq6W0rzPInIj+xeS8v1X4sloGJIiweCCI8YrqUnmYKKQ6Rtd5B4jS5fAa8jp274/R4xvLaR2r70wKp890Iwfb948PwqjNpH/XlOUcfIxPu3jHO65OEyX3OG/+qm+g91AmxlrwZRDYeSioHztz55Du+JKX2Twnb5ttEwVm6I7eQWGbzw50Qgg6I0cHM1YRrXv/h35C8pkXxV7vMlyR0g5ELAoRWYkuveniruKNIIPBw6SY81RtjIIoPvKzNAFssNdiNxfYEsHiDlfK9kUvTLs8qREmLt1rzgD/vHtanmKh7WMtHN42tSUp+NhoHPlEyaaJLUMkFS7suh2S9xPmhisw61npROXrc2PBxLR52i8t0cJYFa1XuZ40KMCZ7kw0UmKiUxeFDTlZkaOTocVqw3qLY906argYnVdkpaoJMUXsnE3JmXch9HG1qpP+hY8LJSOl5ejAiQADHYDm8P/+9VVlNL6eo2O6qFUs9lC+I8hVeSOvJd8+aLeWisFeboKjarhw/52eXwZdTEpPHa7yMDsIQgciqN82yagJYrEL6BevvZ/kGHP5TszYYfcKcN28k9y/R2JYND2TvYeIKb2Nmj9rhwuXBOFnlep4Exw1QS0If80gJG1BCub+u6GKk9FaFHUSH/x3MIRHwVzCHRnZxqbwc1uGZ2I3rRxVq5MHo5p7htS/KYi7+cQjSdRvfdlCofo6YKP38qQwh9y9yAUqMEOABtjUaEJSeWlGq2yh0onm+Uqm54fcCSr7S5MvBTZ1oblLqVUrZBDTIkY5cdbuiyyoj7GPLHrxUo25yiGI+FXHKy136E0Rv7p5N8o3U5O8ycPdk5vOVYgvM3pdLS1zHHsVYIYY/Y/svtDUnMhJONl5AwQ2sDZ2GiWx1nWJ4VumLuwb/Yow/BHtnW6SOeWFSwRkgJdNL5mzrNoexytbrDaSWR0IpuLUbvK/Gi4MJaEkm1acQB1CMOjIymVu1pcih0H1Zl4O+fhQM+bVke6NKrBSUKErHWca9iNH16Du7iXBbaK0Dz73nIhALJKCivr1rWviXXxeQuzTFMRXdK/SalJySHM7ov6qOikQTG/Q97zQ+UYiUnLo3UxlWngXrnzGy8fAb58v9bHhEtnvCrAhiJTer0x4GYQJGwYjQDbN9ucpu00PDnQU0GqYJHqCf4Vm9Y3SYTs92T6Lftvu3nw/ZH6PblcznC6sueEPON5QzzQppepezdolRNyx4g8XYsI+XJlCNrExWxDxErLBTOm3f7aB19RYCS1tG0G/aKEJTXh9RE5VyWSyBzPNMuLZF9HIIHp3yGpqGM7xsYKsMtngBiyXC+i/C/geIgUFJym/sqWn4iPDWtd0YeELg46B5krcolTg87KEeiqDAebieqL5ZiftnVi10kV2L2XPs4ofJjpVI1Q53RsgT89zWXYv97DTZeDwSE8FY2vbADnUpGk/UZ2WMQlX8iwFOTo1woE6AHyqrq6kzGamUGsyoAarqmJk77o2ktzO2M4OqsVqJErSWTcCLZOlxsDWUn/OsQX5+9vaAhHwDHMmrqWo6cJuAtvaLYOHVB/BUOiI/MJLSYeDIrKUML3Ew0GbX89jkR45SPjRdsXQ/oVCJDxm7WjF9jd6bly2wPTmWCzoqeTJU48jPk47tDHWA7QjLEah6PajxbAs/xSN4c2uK9ltR3K/urNpOTYwsFMLgRfzontPe8RMB9A6aZcwQnIhLTpgX8fw/j+aqwqJwVcIk57/KvcOksBjN4rwPQ78kCmpLJ/KEjYh1z8EwIdVX3UDvq8c8YSZnzwsTAVfhWxigvLGzQDTXrr8aNMDXwB2iUVwhNdGw/E1UCSvsKbvw+p9YRaCAzPnnKa8AVAJySAc1Ys/DAYskdKYaTPUwdCXyZfLzfOsFvEzBxqrRgxxK77ij5UQXKINjQpAIP6pkDze+JTwWQsPfcMZ5xtb2tpiUF9FvCVqR1dfblppq3/f3ZHSl7f4s/sOM2/ZRkiI4m8yxhWDMtauxyrnkgiLsOBUcK4MclD5E3H6KtaJODFiCyH+NPtAMmaZ1iduRM51qxmrHlcI5B4ErJum/iZg+zIokXqIi/U6B+tbjj0EKugN+5V03JaoTvdaGQKNvJHnyP0odFN05ZgZnYCKGGh41EGyCZ8/lmrNbAfZRPjRtHw5ypBUiW8Yz+SkumgWA637ASf1AklCNs9MBBuOPmGckDWy8zTvUCgWCwYehIjKSQ+1hYoWpraLNOZy7tHlTQZ7G5zIFR3cJljO4bfmpAGDsSeDQtBsAI90gD4P3ipKFXVQTfl7c3CD1kB8TiOpG0xVGlYQ4dP15z1wTUfHRz82eFxfwn2fc/R/AKmINq9P5PEL31jhLM+gTxctb2BiGn8cCRuorLCg9FtkwC4jy2AluVj6uZZOGvDCYyGqFsdObgwb+FdIoCYDdeZXo+1dxlxhL1258L6h5+yJBgSy6QDN26N6MMNnK2iD7QTQwtB9CfaX3Bk6T0nZ0TT6zSxu7ix7UG+E5AKwoMVcD0EuLlmJ0IhLI+4ZZ8z/QUIzeTZ9ve+r8uyHKmWr2oR8ec5m95PGG5JzWmw30sDelFFl6S3KxsHwzlJsXeXbE5I/Z0RWZQBa8xPbmY54qYXPzp3O+9WYys6A697UzL6fcCORYokB/TR5exBS2saW4OhkwX5ss9aUJGZiMFSgfF1paxqXkGIx3IuDH5xP3QxhGxNMB+T+Yn+meApf2A00dEUng1HRtgBU5Mk/TM19g8l3nVtwTrmAZDm50jIn+G51GpeoBeYPONhGfb1Ngao383TPkvShaNujE3zsJLfJCrNda76rIechIhAC25wWauUMGUU8daL8cdtJnAEeidnolR+lVf5WFKUxvO36X84FCSaR4PqzvlNk7hOvd607h7IGOEVx7/XRLpWDDrrT6hpY653lNf3RCsxdQ7kS9e5hi/RpU1mf9mdTOsf+AtHkkog336jKdyl1/plDV/AlRS8/QjlPfeyvGmA6gEC8cZmBkvyZPbs81tYmif8q2Amd9La3wldh3evYOOkC/b1QQ3fLryoeUl5/9oVprW/aEfmcTxG9CytaFfsx6cAejAb7g6fz/iaWE57fXfth7K7ATi827ZwAs0XX9lLkyGn60ePtqg771isK5/v1jCsbDQDwHwNLehH+BX9zZWWjaQxK85/o09Oaiy0oGY1nivv5kHelmGwj8qqd,iv:a4NHhErZolhOuzNCUU51GQiCfCT8GzZ3GQ24s7fkuyU=,tag:w+GrXmaYWADTrBxMko3xtg==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmk5aE1GN2dpVlREZU1s
            N3V5ZUQweWxMaVo3Y1VjdzZTYW96eFBYZGxvCnNJcjVXKzI4VnliTWFXQy9iSy9k
            eEZmWlVPQzhZK09NZGowZUVXc1BoZTQKLS0tIHIrMEpTM0lMU0w0eG02aUxJY2VZ
            aUQ0cmkvK0UwdzhqcDJwM1RDcnpEU0UKQRiTa9/yNrqlkIdlNeaIDzIKa/xvx5nW
            7wTNfjq2M3go0DMnaVjJA3e15JZBOpOs0QP9z9ydKF/FS4PeCIEFyQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
    encrypted_regex: ^(data|stringData)$
    lastmodified: "2026-10-07T04:27:33Z"
    mac: ENC[AES256_GCM,data:CxZdlGJdIFFTwaeeS9LW2XDDgvUmSsBeBKtFYyDM1JliS8BHtgEv48yuCWxP5sR3SaYtcgNSIQ55zUa89jM6dQHqvjK+1jSYu1xzFWPOlxUu8gVSlU0QLS4GU1RChXuPpTUOjxzn4ymZukGgclZk9iuMvI0/U+nGD8nvs8k0zpM=,iv:MGZ7YsOsjelJPjLYWtALOrM6r8Vd1HgbU/PKL4c3r3k=,tag:dIE38TCobESY3j56NVEvIw==,type:str]
    version: 3.13.2
@@ -0,0 +1,634 @@
 #!/usr/bin/env python3
 """Provision ZITADEL end to end: machine key, Terraform, SOPS secrets, users.
 One command replaces the manual sequence the Authentik script
 (scripts/iam/provision-rbac.py) used to do in a single pass. Phases run in a
 fixed order and every phase is safe to re-run:
  keys         extract the iam-admin machine key and PAT from namespace iam
  apply        create project, roles and OIDC apps through the ZITADEL API
  secrets      write generated client ids/secrets back into the SOPS files
  users        import one human user, converting the Django password hash
  verify       check discovery, client count and the groups-claim gate
 `all` runs keys, apply, secrets, verify. It never runs users: the password hash
 must be supplied by a human, one user at a time.
 Nothing here talks to Authentik. Authentik keeps serving until a consumer is
 moved deliberately.
 Usage:
  scripts/iam/zitadel-provision.py keys
  scripts/iam/zitadel-provision.py apply --org-id <id>
  scripts/iam/zitadel-provision.py secrets --dry-run
  scripts/iam/zitadel-provision.py secrets --apply-config
  scripts/iam/zitadel-provision.py users --django-hash '<pbkdf2_sha256$...>'
  scripts/iam/zitadel-provision.py verify
 """
 from __future__ import annotations
 import argparse
 import base64
 import json
 import os
 import re
 import signal
 import stat
 import subprocess
 import sys
 import tempfile
 from pathlib import Path
 import yaml
 REPO = Path(__file__).resolve().parents[2]
 SOPS_DIR = REPO / "k8s" / "argocd" / "secrets"
 SOPS_CONFIG = REPO / ".sops.yaml"
 CONF_DIR = Path(os.environ.get("ZITADEL_CONF_DIR", Path.home() / ".config" / "zitadel"))
 STATE_FILE = CONF_DIR / "state.json"
 NAMESPACE = os.environ.get("ZITADEL_NAMESPACE", "iam")
 BASE_URL = os.environ.get("ZITADEL_BASE_URL", "https://zitadel.riotpiao.com")
 # ── desired ZITADEL state ──────────────────────────────────────────────────
 # Declared here, applied through the API by the `apply` phase. This replaces the
 # retired terraform/zitadel root module: same objects, no state file to drift and
 # no second writer.
 PROJECT_NAME = os.environ.get("ZITADEL_PROJECT_NAME", "homelab")
 PROJECT_GROUP = "homelab"
 # Role keys EQUAL the Authentik group names, so the `groups` claim keeps working
 # for every consumer that maps on it (kube-apiserver RBAC, MinIO policies,
 # Grafana roles).
 ROLES = {
    "homelab-admins": "Cluster administrators with full access",
    "minio-admins": "MinIO administrators",
    "minio-photos": "Photos bucket (Immich) access",
    "minio-documents": "Documents bucket (Paperless) access",
    "minio-backups": "Backups bucket read-only access",
    "poimen-admins": "Poimen memory administrators",
    "poimen-devs": "Dev and staging projects access",
    "poimen-prod-readonly": "Production projects read-only access",
    "paperless-admins": "Paperless administrators",
    "paperless-finance": "Finance documents",
    "paperless-legal": "Legal documents",
    "paperless-hr": "HR documents",
    "grafana-admins": "Grafana administrators",
    "grafana-editors": "Grafana dashboard editors",
    "grafana-viewers": "Grafana dashboard viewers",
    "sqs-users": "SQS/Temporal queue read access",
    "sqs-writers": "SQS/Temporal queue read/write access",
    "s3-users": "S3 read access",
    "s3-writers": "S3 read/write access",
 }
 _WEB = {"app_type_id": "OIDC_APP_TYPE_WEB", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_BASIC"}
 _NATIVE = {"app_type_id": "OIDC_APP_TYPE_NATIVE", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_NONE"}
 _PKCE = {"app_type_id": "OIDC_APP_TYPE_USER_AGENT", "auth_method_id": "OIDC_AUTH_METHOD_TYPE_NONE"}
 OIDC_APPS = {
    "api-gw": {**_WEB, "redirect_uris": [
        "http://localhost:3000/callback", "https://api.riotpiao.com/callback"]},
    "argocd": {**_WEB, "redirect_uris": [
        "http://localhost:8080/auth/callback",
        "https://argocd.riotpiao.com/auth/callback"]},
    "forgejo": {**_WEB, "redirect_uris": [
        "http://localhost:3000/user/oauth2/authentik/callback",
        "https://forgejo.riotpiao.com/user/oauth2/authentik/callback"]},
    "grafana": {**_WEB, "redirect_uris": [
        "http://localhost:3000/login/generic_oauth",
        "https://grafana.riotpiao.com/login/generic_oauth"]},
    "homarr": {**_WEB, "redirect_uris": [
        "http://localhost:7575/auth/callback",
        "https://homarr.riotpiao.com/auth/callback"]},
    "immich": {**_WEB, "app_type_id": "OIDC_APP_TYPE_NATIVE", "redirect_uris": [
        "app.immich:///oauth-callback", "https://img.riotpiao.com/auth/login",
        "https://img.riotpiao.com/user/oauth2/callback"]},
    "kubernetes": {**_PKCE, "dev_mode": True, "redirect_uris": [
        "http://localhost:8000", "http://localhost:18000"]},
    "minio": {**_WEB, "redirect_uris": [
        "http://localhost:9000/auth/sso/oauth2/code",
        "https://minio.riotpiao.com/auth/sso/oauth2/code"]},
    "paperless": {**_WEB, "redirect_uris": [
        "http://localhost:8000/accounts/oidc/authentik/login/callback/",
        "https://paperless.riotpiao.com/accounts/oidc/authentik/login/callback/"]},
    "poimen": {**_WEB, "redirect_uris": [
        "http://localhost:3000/callback", "https://poimen.riotpiao.com/callback"]},
    "queue-sqs": {**_WEB, "redirect_uris": [
        "http://localhost:8080/callback", "https://queue.riotpiao.com/callback"]},
    "vault": {**_WEB, "redirect_uris": [
        "http://localhost:8200/ui/vault/auth/oidc/oidc/callback",
        "https://vault.riotpiao.com/ui/vault/auth/oidc/oidc/callback"]},
 }
 _TEMP_FILES: set[str] = set()
 def _cleanup_temps(*_a) -> None:
    for path in list(_TEMP_FILES):
        try:
            os.unlink(path)
        except OSError:
            pass
        _TEMP_FILES.discard(path)
 for _sig in (signal.SIGTERM, signal.SIGINT, signal.SIGHUP):
    signal.signal(_sig, lambda s, f: (_cleanup_temps(), sys.exit(130)))
 def die(msg: str) -> None:
    _cleanup_temps()
    print(f"FATAL: {msg}", file=sys.stderr)
    sys.exit(1)
 def run(cmd: list[str], **kw) -> subprocess.CompletedProcess:
    return subprocess.run(cmd, check=False, text=True, capture_output=True, **kw)
 def need(binary: str) -> None:
    if subprocess.run(["sh", "-c", f"command -v {binary}"], capture_output=True).returncode != 0:
        die(f"{binary} is not on PATH")
 def write_private(path: Path, data: bytes) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
    try:
        os.fchmod(fd, 0o600)  # open() mode applies at creation only
        with os.fdopen(fd, "wb") as fh:
            fh.write(data)
    finally:
        pass
 def secret_value(name: str, key: str) -> str:
    cp = run(["kubectl", "-n", NAMESPACE, "get", "secret", name, "-o", f"jsonpath={{{key}}}"])
    if cp.returncode != 0:
        die(f"cannot read secret {NAMESPACE}/{name}: {cp.stderr.strip()}")
    raw = cp.stdout.strip()
    if not raw:
        die(f"secret {NAMESPACE}/{name} has no key {key}")
    return base64.b64decode(raw).decode()
 def load_env_file(path: Path = REPO / ".env") -> None:
    """Load KEY=VALUE lines from .env without overriding the real environment.
    The file is git-ignored and holds the bootstrap token a human pastes in.
    Sourcing it in the shell would work too, but then every phase depends on the
    caller remembering to do it.
    """
    if not path.exists():
        return
    for line in path.read_text().splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        key = key.strip()
        value = value.strip().strip('"').strip("'")
        if key and key not in os.environ:
            os.environ[key] = value
 def bootstrap_token() -> str | None:
    """Resolve the iam-admin PAT: environment, then cached file, then cluster.
    Order matters. ZITADEL_BOOTSTRAP_TOKEN in .env is filled by a human after the
    first sync and wins. The cached file under ~/.config/zitadel keeps a run
    working when the API server is unreachable. The cluster Secret is the last
    resort and the only source that always reflects a rotation.
    """
    token = os.environ.get("ZITADEL_BOOTSTRAP_TOKEN", "").strip()
    if token:
        return token
    cached = CONF_DIR / "iam-admin.pat"
    if cached.exists():
        return cached.read_text().strip()
    try:
        return secret_value("iam-admin-pat", ".data.pat").strip()
    except SystemExit:
        return None
 def curl_json(method: str, url: str, token: str | None = None, body: dict | None = None,
              timeout: int = 20) -> tuple[int, str]:
    """Call the API without putting a secret in the process argv.
    `curl --config -` reads the options from stdin, so neither the bearer token
    nor the password hash of a user import shows up in `ps`.
    """
    def esc(value: str) -> str:
        return value.replace("\\", "\\\\").replace('"', '\\"')
    cfg = [f'url = "{esc(url)}"', f'request = "{method}"', "silent", "show-error",
           f"max-time = {timeout}", 'write-out = "\\n%{http_code}"',
           'header = "Content-Type: application/json"']
    # Reaching the instance by name needs either public DNS through the tunnel
    # or a pinned address. ZITADEL picks the instance from the Host header, so a
    # plain port-forward fails with "Instance not found". Set this to bypass DNS:
    #   ZITADEL_CURL_RESOLVE=zitadel.riotpiao.com:443:192.168.1.160
    pinned = os.environ.get("ZITADEL_CURL_RESOLVE", "").strip()
    if pinned:
        cfg.append(f'resolve = "{esc(pinned)}"')
    if token:
        cfg.append(f'header = "Authorization: Bearer {esc(token)}"')
    if body is not None:
        cfg.append(f'data = "{esc(json.dumps(body))}"')
    cp = run(["curl", "--config", "-"], input="\n".join(cfg) + "\n")
    if cp.returncode != 0:
        die(f"{method} {url} failed: {cp.stderr.strip()}")
    payload, _, code = cp.stdout.rpartition("\n")
    return int(code.strip() or 0), payload
 # ── phase: keys ────────────────────────────────────────────────────────────
 def phase_keys(_args: argparse.Namespace) -> int:
    write_private(CONF_DIR / "iam-admin.json", secret_value("iam-admin", ".data.iam-admin\\.json").encode())
    write_private(CONF_DIR / "iam-admin.pat", secret_value("iam-admin-pat", ".data.pat").strip().encode())
    print(f"machine key  -> {CONF_DIR / 'iam-admin.json'} (0600)")
    print(f"PAT          -> {CONF_DIR / 'iam-admin.pat'} (0600)")
    print("both files stay outside the repo; never commit them")
    return 0
 # ── phase: apply ───────────────────────────────────────────────────────────
 def _api(method: str, path: str, token: str, body: dict | None = None,
         timeout: int = 30) -> tuple[int, dict]:
    code, payload = curl_json(method, f"{BASE_URL}{path}", token=token, body=body, timeout=timeout)
    try:
        return code, (json.loads(payload) if payload.strip() else {})
    except json.JSONDecodeError:
        return code, {"_raw": payload}
 def _ensure_project(token: str, name: str) -> str:
    code, d = _api("POST", "/management/v1/projects/_search", token, {})
    for row in d.get("result") or []:
        if row.get("name") == name:
            return row["id"]
    code, d = _api("POST", "/management/v1/projects", token, {"name": name})
    if code not in (200, 201) or not d.get("id"):
        die(f"create project {name} failed: {code} {d}")
    return d["id"]
 def _ensure_roles(token: str, project_id: str, roles: dict[str, str]) -> None:
    code, d = _api("POST", f"/management/v1/projects/{project_id}/roles/_search", token, {})
    have = {r.get("roleKey") for r in (d.get("result") or [])}
    for key, display in roles.items():
        if key in have:
            continue
        code, d = _api("POST", f"/management/v1/projects/{project_id}/roles", token,
                       {"roleKey": key, "displayName": display, "group": PROJECT_GROUP})
        if code not in (200, 201):
            die(f"create role {key} failed: {code} {d}")
 def _ensure_oidc_apps(token: str, project_id: str, apps: dict[str, dict]) -> tuple[dict, dict]:
    code, d = _api("POST", f"/management/v1/projects/{project_id}/apps/_search", token, {})
    existing = {a.get("name"): a for a in (d.get("result") or [])}
    ids: dict[str, str] = {}
    secrets: dict[str, str] = {}
    for name, spec in apps.items():
        row = existing.get(name)
        if row:
            ids[name] = (row.get("oidcConfig") or {}).get("clientId", "")
            app_id = row.get("id")
        else:
            body = {
                "name": name,
                "redirectUris": spec["redirect_uris"],
                "responseTypes": ["OIDC_RESPONSE_TYPE_CODE"],
                "grantTypes": ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE",
                               "OIDC_GRANT_TYPE_REFRESH_TOKEN"],
                "appType": spec.get("app_type_id", "OIDC_APP_TYPE_WEB"),
                "authMethodType": spec.get("auth_method_id", "OIDC_AUTH_METHOD_TYPE_BASIC"),
                "devMode": bool(spec.get("dev_mode")),
                "accessTokenType": "OIDC_TOKEN_TYPE_BEARER",
                "version": "OIDC_VERSION_1_0",
                "idTokenRoleAssertion": True,
            }
            code, d = _api("POST", f"/management/v1/projects/{project_id}/apps/oidc", token, body)
            if code not in (200, 201) or not d.get("appId"):
                die(f"create app {name} failed: {code} {d}")
            app_id = d["appId"]
            ids[name] = d.get("clientId", "")
        if spec.get("auth_method_id") == "OIDC_AUTH_METHOD_TYPE_NONE":
            continue
        code, d = _api("POST",
                       f"/management/v1/projects/{project_id}/apps/{app_id}"
                       "/oidc_config/_generate_client_secret", token, {})
        if code in (200, 201) and d.get("clientSecret"):
            secrets[name] = d["clientSecret"]
    return ids, secrets
 def phase_apply(_args: argparse.Namespace) -> int:
    """Declare the ZITADEL objects through the API. No Terraform, no state drift."""
    token = bootstrap_token()
    if not token:
        die("no iam-admin PAT — run the keys phase first")
    project_id = _ensure_project(token, PROJECT_NAME)
    print(f"project   : {PROJECT_NAME} ({project_id})")
    _ensure_roles(token, project_id, ROLES)
    print(f"roles     : {len(ROLES)} ensured")
    ids, secrets = _ensure_oidc_apps(token, project_id, OIDC_APPS)
    print(f"oidc apps : {len(ids)} ensured, {len(secrets)} secrets issued")
    state = {
        "project_id": {"value": project_id},
        "client_ids": {"value": ids},
        "client_secrets": {"value": secrets},
        "machine_keys": {"value": {}},
        "groups_claim_enabled": {"value": False},
    }
    write_private(STATE_FILE, json.dumps(state, indent=2, sort_keys=True).encode())
    print(f"state     -> {STATE_FILE} (0600)")
    return 0
 # ── phase: secrets ─────────────────────────────────────────────────────────
 # client -> [(sops file, Secret name inside the file, key, kind)]
 # Only secrets with a git-tracked SOPS file appear here. Every other client is
 # printed as manual: guessing a path writes a credential into the wrong Secret.
 SECRET_MAP = {
    "grafana": [("grafana-oidc-secrets.enc.yaml", "grafana-oidc",
                 "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET", "secret")],
    "minio": [("minio-secrets.enc.yaml", "minio-oidc",
               "MINIO_IDENTITY_OPENID_CLIENT_SECRET", "secret")],
    "paperless-ai-agent": [("paperless-ai-secrets.enc.yaml", "paperless-ai-config",
                            "LLM_AUTH_CLIENT_SECRET", "secret")],
 }
 # Clients whose client_id lives in a plain (unencrypted) manifest. They need the
 # id update as much as the secret: ZITADEL generates a new client_id for every
 # application, so a secret-only re-point yields invalid_client.
 CLIENT_ID_MAP = {
    "grafana": ("k8s/infra/logging/grafana-values.yaml",
                r"(?m)^(\s*client_id:\s*)\S+\s*$", r"\g<1>{value}"),
    "minio": ("k8s/infra/minio/minio-tenant.yaml",
              r"(?m)^(\s*value:\s*)\"minio\"\s*$", r'\g<1>"{value}"'),
    "paperless-ai-agent": ("k8s/apps/paperless/paperless-ai.yaml",
                           r"client_id=paperless-ai-agent", "client_id={value}"),
 }
 # Secrets that a live workload mounts but no SOPS file in git defines. The phase
 # cannot re-point them safely, so it names them.
 MANUAL_ONLY = {
    "homarr": "Secret homarr-oidc (keys client-id, client-secret) has no SOPS file in git — "
              "adopt it first, then add it to SECRET_MAP. The repo also carries an orphan "
              "homarr-auth-oidc.enc.yaml that nothing consumes.",
 }
 def state_outputs() -> dict:
    """Read the object map written by the apply phase."""
    if not STATE_FILE.exists():
        die(f"{STATE_FILE} missing — run the apply phase first")
    return json.loads(STATE_FILE.read_text())
 def sops_env() -> dict:
    env = dict(os.environ)
    env.setdefault("SOPS_AGE_KEY_FILE", str(Path.home() / ".sops" / "key.txt"))
    return env
 def write_secret(rel_path: str, secret_name: str, key: str, value: str,
                 dry_run: bool) -> None:
    path = SOPS_DIR / rel_path
    if not path.exists():
        die(f"{path} does not exist — the mapping and the repo disagree")
    env = sops_env()
    dec = run(["sops", "--config", str(SOPS_CONFIG), "decrypt", str(path)], env=env)
    if dec.returncode != 0:
        die(f"cannot decrypt {rel_path}: {dec.stderr.strip()}")
    docs = [d for d in yaml.safe_load_all(dec.stdout) if d]
    matching = [d for d in docs if (d.get("metadata") or {}).get("name") == secret_name]
    if len(matching) != 1:
        die(f"{rel_path}: expected exactly one document named {secret_name}, found {len(matching)}")
    current = (matching[0].get("stringData") or {}).get(key)
    if current == value:
        print(f"  unchanged {rel_path}:{secret_name}:{key}")
        return
    if dry_run:
        print(f"  would set {rel_path}:{secret_name}:{key} (len {len(value)})")
        return
    if len(docs) == 1:
        cp = run(["sops", "--config", str(SOPS_CONFIG), "set", "--value-stdin", "--idempotent",
                  str(path), f'["stringData"]["{key}"]'], env=env, input=json.dumps(value))
        if cp.returncode != 0:
            die(f"sops set failed on {rel_path}: {cp.stderr.strip()}")
        print(f"  wrote {rel_path}:{secret_name}:{key}")
        return
    # Multi-document file. `sops set` cannot address one document, so the whole
    # file is rewritten. sops encrypt reads the creation rule from the FILENAME,
    # so the temp path is overridden with the repo-relative path; without that
    # override sops finds no matching rule and refuses.
    matching[0].setdefault("stringData", {})[key] = value
    plain = "---\n".join(yaml.safe_dump(d, sort_keys=False) for d in docs)
    fd, tmp = tempfile.mkstemp(suffix=".yaml", dir=tempfile.gettempdir())
    _TEMP_FILES.add(tmp)
    try:
        with os.fdopen(fd, "w") as fh:
            fh.write(plain)
        enc = run(["sops", "--config", str(SOPS_CONFIG), "encrypt",
                   "--filename-override", str(path), tmp], env=env)
        if enc.returncode != 0:
            die(f"sops encrypt failed for {rel_path}: {enc.stderr.strip()}")
        mode = stat.S_IMODE(path.stat().st_mode)
        fd2, staged = tempfile.mkstemp(suffix=".enc.yaml", dir=str(path.parent))
        try:
            with os.fdopen(fd2, "w") as fh:
                fh.write(enc.stdout)
            os.chmod(staged, mode)
            os.replace(staged, path)  # same directory: atomic, never EXDEV
        finally:
            if os.path.exists(staged):
                os.unlink(staged)
        print(f"  wrote {rel_path}:{secret_name}:{key} (multi-document rewrite)")
    finally:
        if os.path.exists(tmp):
            os.unlink(tmp)
        _TEMP_FILES.discard(tmp)
 def update_client_id(client: str, value: str, dry_run: bool, apply_config: bool) -> None:
    spec = CLIENT_ID_MAP.get(client)
    if not spec:
        return
    rel_path, pattern, repl = spec
    path = REPO / rel_path
    if not path.exists():
        die(f"{path} does not exist — the client-id mapping is stale")
    text = path.read_text()
    matches = list(re.finditer(pattern, text))
    if len(matches) != 1:
        die(f"{rel_path}: expected 1 match for the {client} client id, found {len(matches)}")
    new_line = matches[0].expand(repl.format(value=value))
    if matches[0].group(0) == new_line:
        print(f"  unchanged {rel_path}: {client} client id")
        return
    if dry_run or not apply_config:
        print(f"  would change {rel_path}: {matches[0].group(0).strip()} -> {new_line.strip()}")
        if not apply_config:
            print("    re-run with --apply-config to write this plaintext manifest edit")
        return
    path.write_text(text[:matches[0].start()] + new_line + text[matches[0].end():])
    print(f"  wrote {rel_path}: {client} client id")
 def phase_secrets(args: argparse.Namespace) -> int:
    need("sops")
    out = state_outputs()
    ids = out.get("client_ids", {}).get("value", {})
    secrets = out.get("client_secrets", {}).get("value", {})
    machine_keys = out.get("machine_keys", {}).get("value", {})
    if not ids and not machine_keys:
        die("no ZITADEL objects recorded yet — run the apply phase first")
    print("[1/3] client secrets into SOPS")
    for client, entries in SECRET_MAP.items():
        for rel_path, secret_name, key, kind in entries:
            value = ids.get(client) if kind == "id" else secrets.get(client)
            if value:
                write_secret(rel_path, secret_name, key, value, args.dry_run)
    print("[2/3] client ids in plain manifests")
    for client in list(CLIENT_ID_MAP) + list(SECRET_MAP):
        if ids.get(client):
            update_client_id(client, ids[client], args.dry_run, args.apply_config)
    print("[3/3] machine keys")
    key_dir = CONF_DIR / "machine-keys"
    for name, key_json in machine_keys.items():
        if args.dry_run:
            print(f"  would write {key_dir / (name + '.json')}")
        else:
            write_private(key_dir / f"{name}.json", key_json.encode())
            print(f"  wrote {key_dir / (name + '.json')} (0600)")
    unhandled = sorted(set(ids) - set(SECRET_MAP) - set(CLIENT_ID_MAP) - set(MANUAL_ONLY))
    if MANUAL_ONLY or unhandled:
        print("\n  MANUAL — no safe write path exists for these clients:")
        for client, why in MANUAL_ONLY.items():
            print(f"    {client}: {why}")
        for client in unhandled:
            print(f"    {client}: set its client_id and secret in its own manifest or Secret file")
    return 0
 # ── phase: users ───────────────────────────────────────────────────────────
 HASH_RE = re.compile(r"^pbkdf2_sha256\$(?P<rounds>\d+)\$(?P<salt>[^$]+)\$(?P<hash>[^$]+)$")
 def convert_hash(django_hash: str) -> str:
    """Django pbkdf2_sha256 -> passwap MCF $pbkdf2-sha256$...
    ZITADEL imports one MCF string via hashedPassword.hash (max 200 chars) and
    pins passwap, whose pbkdf2 format is $pbkdf2-sha256$<rounds>$<salt>$<hash>
    with alt-base64 (standard base64, + -> ., no padding). Rounds, salt and hash
    bytes are unchanged, so the password still verifies.
    """
    m = HASH_RE.match(django_hash.strip())
    if not m:
        die("unsupported hash format — only pbkdf2_sha256 is convertible")
    alt = m.group("hash").rstrip("=").replace("+", ".")
    mcf = f"$pbkdf2-sha256${m.group('rounds')}${m.group('salt')}${alt}"
    if len(mcf) > 200:
        die(f"converted hash is {len(mcf)} chars, ZITADEL accepts 200")
    return mcf
 def phase_users(args: argparse.Namespace) -> int:
    token = bootstrap_token()
    if not token:
        die("no bootstrap token — set ZITADEL_BOOTSTRAP_TOKEN in .env, or run the keys phase")
    if not args.django_hash:
        die("pass --django-hash with the pbkdf2_sha256 value exported from Authentik")
    body = {
        "username": args.username,
        "profile": {"givenName": args.given_name, "familyName": args.family_name},
        "email": {"email": args.email, "isVerified": True},
        "hashedPassword": {"hash": convert_hash(args.django_hash)},
        "passwordChangeRequired": False,
    }
    code, payload = curl_json("POST", f"{BASE_URL}/v2/users/human", token=token, body=body)
    if code in (200, 201):
        print(f"  {args.username}: created")
        return 0
    if code == 409:
        print(f"  {args.username}: already exists — the password was NOT updated")
        return 1
    die(f"create {args.username} -> {code} {payload[:300]}")
 # ── phase: verify ──────────────────────────────────────────────────────────
 def phase_verify(_args: argparse.Namespace) -> int:
    code, _ = curl_json("GET", f"{BASE_URL}/.well-known/openid-configuration", timeout=10)
    print(f"discovery      : {code}")
    if code != 200:
        die("discovery endpoint is not 200 — the instance or the ingress is not ready")
    out = state_outputs()
    ids = out.get("client_ids", {}).get("value", {})
    print(f"client apps    : {len(ids)}")
    enabled = bool(out.get("groups_claim_enabled", {}).get("value"))
    print(f"groups action  : {enabled}")
    if not enabled:
        print("BLOCKED for cutover: the groups claim action is disabled. Every consumer")
        print("that reads `groups` (kube-apiserver, MinIO, Grafana) sees no groups.")
        print("Enable it only after the live gate passes, with the script path set.")
    return 0
 PHASES = {"keys": phase_keys, "apply": phase_apply, "secrets": phase_secrets,
          "users": phase_users, "verify": phase_verify}
 def main() -> int:
    parser = argparse.ArgumentParser(description="Provision ZITADEL end to end.")
    parser.add_argument("phase", choices=[*PHASES, "all"])
    parser.add_argument("--org-id", help="ZITADEL org id (required for apply)")
    parser.add_argument("--dry-run", action="store_true", help="print actions, write nothing")
    parser.add_argument("--apply-config", action="store_true",
                        help="also rewrite client ids inside plain manifests")
    parser.add_argument("--username", default="rock")
    parser.add_argument("--email", default="[email protected]")
    parser.add_argument("--given-name", default="Rock")
    parser.add_argument("--family-name", default="Liang")
    parser.add_argument("--django-hash", help="pbkdf2_sha256 value exported from Authentik")
    args = parser.parse_args()
    load_env_file()
    need("kubectl")
    if args.phase == "all":
        if not args.org_id:
            die("all requires --org-id")
        for name in ("keys", "apply"):
            rc = PHASES[name](args)
            if rc:
                return rc
        rc = phase_secrets(args)
        if rc:
            return rc
        return phase_verify(args)
    return PHASES[args.phase](args)
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,51 @@
 # Cloudflare side of the edge.
 #
 # The tunnel itself is remotely managed today: its token lives in
 # k8s/apps/cloudflared and the public hostnames are configured in the Zero
 # Trust dashboard. Terraform can own the DNS records without owning the tunnel,
 # which is the safe half: a record change is reversible, a tunnel config rewrite
 # removes every hostname the plan does not list.
 locals {
  # CNAME target of every remotely-managed tunnel.
  tunnel_target = "${var.tunnel_id}.cfargotunnel.com"
 }
 resource "cloudflare_dns_record" "tunnel" {
  for_each = toset(var.cloudflare_tunnel_hostnames)
  zone_id = var.cloudflare_zone_id
  name    = each.value
  type    = "CNAME"
  content = local.tunnel_target
  proxied = var.cloudflare_dns_proxied
  ttl     = var.cloudflare_dns_ttl
  comment = "cloudflared tunnel; managed by terraform/edge"
  # A proxied CNAME must point at the tunnel target. A CNAME to any other
  # hostname breaks the tunnel mapping.
  lifecycle {
    precondition {
      condition     = var.cloudflare_dns_proxied
      error_message = "Tunnel CNAMEs must be proxied; an unproxied record bypasses the tunnel."
    }
  }
 }
 # Optional: take the tunnel configuration into Terraform. Off by default.
 # Applying it REPLACES the whole ingress list. Import first:
 #   terraform import cloudflare_zero_trust_tunnel_cloudflared_config.this \
 #     <account_id>/<tunnel_id>
 resource "cloudflare_zero_trust_tunnel_cloudflared_config" "this" {
  count = var.manage_tunnel_config ? 1 : 0
  account_id = var.cloudflare_account_id
  tunnel_id  = var.tunnel_id
  config = {
    ingress = concat(
      [for host, service in var.tunnel_ingress : { hostname = host, service = service }],
      [{ service = "http_status:404" }],
    )
  }
 }
@@ -0,0 +1,40 @@
 output "cloudflare_records" {
  description = "Hostnames now managed in the Cloudflare zone."
  value       = sort(keys(cloudflare_dns_record.tunnel))
 }
 output "tunnel_config_managed" {
  description = "Whether Terraform owns the tunnel ingress list. False means the dashboard still does."
  value       = var.manage_tunnel_config
 }
 output "relay_public_ip" {
  description = "Public address of the Tencent relay. Point the .cn DNS at it, or let the module do it."
  value       = tencentcloud_instance.relay.public_ip
 }
 output "china_hostnames" {
  description = "Fully qualified China hostnames served by the relay."
  value       = [for host in var.china_hostnames : "${host}.${var.china_domain}"]
 }
 output "wireguard_peer" {
  description = "Peer stanza for the homelab side. Add it to the cluster's wg0.conf; the relay already trusts this public key."
  value       = <<-EOT
    [Peer]
    PublicKey = <relay public key; see /etc/wireguard/wg0.conf on the relay>
    Endpoint = ${tencentcloud_instance.relay.public_ip}:${var.wireguard_port}
    AllowedIPs = ${var.wireguard_relay_address}
    PersistentKeepalive = 25
  EOT
 }
 output "next_manual_steps" {
  description = "What this module cannot do for you."
  value = [
    "Confirm the ICP filing for ${var.china_domain} if the relay is in a mainland region.",
    "Issue a certificate for the China hostnames on the relay (Caddy ACME needs the DNS-01 challenge for a .cn domain).",
    "Run a WireGuard peer in the cluster that dials ${tencentcloud_instance.relay.public_ip}:${var.wireguard_port}.",
    "Add the .cn origins to the OIDC client redirect URIs before any China user logs in.",
  ]
 }
@@ -0,0 +1,47 @@
 # terraform/edge — the China-facing edge, plus DNS ownership for the existing zone
 #
 # Purpose: give users in mainland China a short, stable path to the homelab
 # without touching the Cloudflare path that serves everyone else.
 #
 #   China user -> <service>.<china_domain> -> Tencent CVM (this module)
 #              -> WireGuard tunnel -> ingress-nginx (192.168.1.160) -> service
 #
 #   Everyone else -> Cloudflare -> cloudflared tunnel -> same ingress (unchanged)
 #
 # The homelab still opens no inbound port: the tunnel is established outbound
 # from the cluster to the CVM.
 #
 # Credentials come from the environment, never from a file in git:
 #   export CLOUDFLARE_API_TOKEN=...
 #   export TENCENTCLOUD_SECRET_ID=...
 #   export TENCENTCLOUD_SECRET_KEY=...
 #   terraform -chdir=terraform/edge init
 #   terraform -chdir=terraform/edge plan -var china_domain=example.cn ...
 #
 # State is local and git-ignored. It carries no
 # credential today, but it will once InstanceData or certificates are added, so
 # treat it as a secret and move it to MinIO before a second operator applies.
 terraform {
  required_version = ">= 1.6.0"
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "5.28.1"
    }
    tencentcloud = {
      source  = "tencentcloudstack/tencentcloud"
      version = "1.83.38"
    }
  }
 }
 provider "cloudflare" {
  api_token = var.cloudflare_api_token
 }
 provider "tencentcloud" {
  secret_id  = var.tencent_secret_id
  secret_key = var.tencent_secret_key
  region     = var.tencent_region
 }
@@ -0,0 +1,22 @@
 # Rendered by terraform/edge. Do not edit on the instance.
 #
 # Caddy terminates TLS for the .cn hostnames and forwards over the WireGuard
 # tunnel to ingress-nginx. flush_interval -1 disables response buffering so
 # Server-Sent Events reach the client chunk-by-chunk.
 {
 	email ${acme_email}
 	admin off
 }
 ${hosts} {
 	reverse_proxy ${ingress_upstream} {
 		flush_interval -1
 	}
 	header {
 		Strict-Transport-Security "max-age=31536000; includeSubDomains"
 		X-Content-Type-Options "nosniff"
 		X-Frame-Options "DENY"
 		Referrer-Policy "strict-origin-when-cross-origin"
 	}
 }
@@ -0,0 +1,169 @@
 # The relay: one small CVM in Tencent Cloud that terminates TLS for the .cn
 # domain and forwards over WireGuard to the homelab ingress.
 #
 # Why a relay and not Cloudflare: mainland clients reach a Tencent edge far
 # faster than a Cloudflare edge, and Cloudflare's China network is an
 # Enterprise product. The homelab keeps zero inbound exposure because the
 # tunnel is dialled outbound from the cluster.
 #
 # ICP FILING: a mainland resource serving HTTP/HTTPS for a domain requires an
 # ICP filing naming that domain. Without it this instance accepts the
 # connection and the mainland network resets the HTTP layer. Put the relay in a
 # non-mainland region (ap-hongkong, ap-singapore) when no filing exists.
 resource "tencentcloud_vpc" "edge" {
  name       = "${replace(var.china_domain, ".", "_")}_edge"
  cidr_block = var.edge_vpc_cidr
 }
 resource "tencentcloud_subnet" "edge" {
  name              = "${replace(var.china_domain, ".", "_")}_edge"
  vpc_id            = tencentcloud_vpc.edge.id
  availability_zone = var.tencent_availability_zone
  cidr_block        = var.edge_subnet_cidr
 }
 resource "tencentcloud_key_pair" "edge" {
  key_name   = "${replace(var.china_domain, ".", "_")}_edge"
  public_key = var.relay_ssh_public_key
 }
 resource "tencentcloud_security_group" "edge" {
  name        = "${replace(var.china_domain, ".", "_")}_edge"
  description = "TLS + WireGuard for the ${var.china_domain} relay; SSH from the operator only"
 }
 resource "tencentcloud_security_group_rule" "ssh" {
  security_group_id = tencentcloud_security_group.edge.id
  type              = "ingress"
  ip_protocol       = "tcp"
  port_range        = "22"
  cidr_ip           = var.relay_admin_cidr
  policy            = "accept"
 }
 resource "tencentcloud_security_group_rule" "tls" {
  security_group_id = tencentcloud_security_group.edge.id
  type              = "ingress"
  ip_protocol       = "tcp"
  port_range        = "80,443"
  cidr_ip           = "0.0.0.0/0"
  policy            = "accept"
 }
 resource "tencentcloud_security_group_rule" "wireguard" {
  security_group_id = tencentcloud_security_group.edge.id
  type              = "ingress"
  ip_protocol       = "udp"
  port_range        = tostring(var.wireguard_port)
  # The homelab dials outbound and has no static public address, so this is
  # 0.0.0.0/0 by default. WireGuard authenticates every packet by public key,
  # and an unauthenticated packet is dropped before it reaches the tunnel.
  # Narrow this to the homelab egress CIDR when that address is stable.
  cidr_ip = var.wireguard_allowed_cidr
  policy  = "accept"
 }
 # cloud-init: WireGuard listener, forwarding sysctls, and Caddy for TLS.
 # The relay never holds application credentials; it forwards bytes.
 #
 # NOTE ON THE PRIVATE KEY: user_data is visible in the instance metadata and in
 # Terraform state. That is a known limitation of bootstrapping WireGuard this
 # way. The argument is wrapped in sensitive() to keep it out of plan/apply
 # output, but anyone with console or state access can read it. Rotate the key
 # if the state or the instance metadata is ever exposed.
 locals {
  china_fqdns = [for host in var.china_hostnames : "${host}.${var.china_domain}"]
  caddyfile = templatefile("${path.module}/templates/Caddyfile.tpl", {
    acme_email       = var.acme_email
    hosts            = join(", ", local.china_fqdns)
    ingress_upstream = var.ingress_upstream
  })
  relay_user_data = <<-EOT
    #cloud-config
    package_update: true
    package_upgrade: false
    manage_etc_hosts: true
    timezone: Asia/Shanghai
    packages: [wireguard, wireguard-tools, caddy]
    write_files:
      - path: /etc/wireguard/wg0.conf
        permissions: "0600"
        content: |
          [Interface]
          Address = ${var.wireguard_relay_address}
          ListenPort = ${var.wireguard_port}
          PrivateKey = ${var.wireguard_relay_private_key}
          # Masquerade forwarded traffic so the homelab peer only has to accept
          # packets sourced from the relay's tunnel address. Without this the
          # homelab peer's AllowedIPs would drop the original client source IP.
          PostUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
          PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE
          [Peer]
          PublicKey = ${var.wireguard_homelab_public_key}
          AllowedIPs = ${var.wireguard_homelab_allowed_ips}
          PersistentKeepalive = 25
      - path: /etc/sysctl.d/99-wireguard.conf
        content: |
          net.ipv4.ip_forward = 1
      - path: /etc/caddy/Caddyfile
        permissions: "0644"
        content: ${jsonencode(local.caddyfile)}
    runcmd:
      - sysctl --system
      - systemctl enable --now wg-quick@wg0
      - caddy validate --config /etc/caddy/Caddyfile
      - systemctl enable --now caddy
      - systemctl restart caddy
  EOT
 }
 resource "tencentcloud_instance" "relay" {
  instance_name     = "${replace(var.china_domain, ".", "_")}_edge"
  availability_zone = var.tencent_availability_zone
  image_id          = var.relay_image_id
  instance_type     = var.relay_instance_type
  vpc_id             = tencentcloud_vpc.edge.id
  subnet_id          = tencentcloud_subnet.edge.id
  security_groups    = [tencentcloud_security_group.edge.id]
  key_ids            = [tencentcloud_key_pair.edge.id]
  allocate_public_ip = true
  # Outbound only needs a few Mbps; inbound TLS is the same order.
  internet_max_bandwidth_out = var.relay_internet_max_bandwidth_out
  system_disk_type           = var.relay_system_disk_type
  system_disk_size           = var.relay_system_disk_size
  # The private key is embedded in this bootstrap script. sensitive() keeps it
  # out of CLI output; it does not remove it from state or instance metadata.
  user_data = sensitive(local.relay_user_data)
  tags = {
    role   = "china-edge"
    domain = var.china_domain
  }
  # Accidental `terraform destroy` would take the only China-facing edge down.
  # Remove this block deliberately when a replacement path exists.
  lifecycle {
    prevent_destroy = true
  }
 }
 # DNS for the .cn domain lives at DNSPod (Tencent). Each subdomain resolves to
 # the relay's public address, and the relay proxies to the homelab over the
 # tunnel. The "中国境内" line is the default line; no separate line records.
 resource "tencentcloud_dnspod_record" "china" {
  for_each = toset(var.china_hostnames)
  domain      = var.china_domain
  sub_domain  = each.value
  record_type = "A"
  record_line = var.china_dns_record_line
  value       = tencentcloud_instance.relay.public_ip
  ttl         = var.china_dns_ttl
  remark      = "China edge relay; origin is the homelab ingress over WireGuard"
 }
@@ -0,0 +1,45 @@
 # Copy to terraform.tfvars.local (git-ignored) and fill in.
 # Secrets never belong in a committed file: pass them as environment variables
 # instead — TF_VAR_wireguard_relay_private_key, CLOUDFLARE_API_TOKEN,
 # TENCENTCLOUD_SECRET_ID, TENCENTCLOUD_SECRET_KEY.
 # ── Cloudflare (existing zone) ───────────────────────────────────────────────
 cloudflare_zone_id = "REPLACE_WITH_ZONE_ID"
 tunnel_id          = "REPLACE_WITH_TUNNEL_UUID"
 # Adopt existing records with `terraform import` before the first apply, or the
 # create call collides with the hand-made record of the same name:
 #   terraform import 'cloudflare_dns_record.tunnel["zitadel"]' \
 #     <zone_id>/<record_id>
 #
 # A tunnel CNAME is only half of the path. The tunnel's Public Hostname list
 # (Zero Trust -> Networks -> Tunnels -> Public Hostname) must also map the name
 # to https://<name>.riotpiao.com, or Cloudflare answers 404 from the tunnel even
 # though DNS resolves.
 cloudflare_tunnel_hostnames = ["paperless", "img", "poimen", "zitadel"]
 # Leave false until the tunnel config import is understood. True makes
 # Terraform replace the entire ingress list with tunnel_ingress.
 manage_tunnel_config = false
 tunnel_ingress       = {}
 # ── China edge (Tencent Cloud) ───────────────────────────────────────────────
 china_domain   = "REPLACE.example.cn"
 china_hostnames = ["paperless", "img"]
 # Mainland regions require an ICP filing for the domain. Use ap-hongkong or
 # ap-singapore when no filing exists.
 tencent_region            = "ap-guangzhou"
 tencent_availability_zone = "ap-guangzhou-6"
 relay_instance_type       = "S5.SMALL2"
 relay_image_id            = "REPLACE_WITH_IMAGE_ID"
 relay_ssh_public_key      = "ssh-ed25519 AAAA... operator@laptop"
 relay_admin_cidr          = "REPLACE_WITH_OPERATOR_CIDR"
 # ── WireGuard between the relay and the cluster ──────────────────────────────
 # The relay private key comes from `wg genkey`. Keep it in the environment, or
 # in a SOPS file mounted for the run. It is not written by this module.
 wireguard_port                 = 51820
 wireguard_homelab_public_key   = "REPLACE_WITH_HOMELAB_WG_PUBKEY"
 wireguard_relay_address        = "10.88.0.1/24"
 wireguard_homelab_address      = "10.88.0.2/24"
@@ -0,0 +1,227 @@
 variable "cloudflare_api_token" {
  description = "Cloudflare API token with Zone:DNS:Edit and, for tunnel config, Account:Cloudflare Tunnel:Edit."
  type        = string
  sensitive   = true
  default     = null
 }
 variable "cloudflare_zone_id" {
  description = "Zone ID of the existing public zone (riotpiao.com)."
  type        = string
 }
 variable "cloudflare_account_id" {
  description = "Account ID. Required only when manage_tunnel_config is true."
  type        = string
  default     = null
 }
 variable "tunnel_id" {
  description = "UUID of the cloudflared tunnel that the DNS records point at."
  type        = string
 }
 # The dashboard remains the source of truth for the remotely-managed tunnel
 # today. Setting this true hands that config to Terraform, which then rewrites
 # the whole ingress list: every hostname not listed in tunnel_ingress is
 # removed from the tunnel. Import first and read the plan before applying.
 variable "manage_tunnel_config" {
  description = "Take ownership of the remotely-managed tunnel configuration."
  type        = bool
  default     = false
 }
 variable "tunnel_ingress" {
  description = "hostname => origin service, used only when manage_tunnel_config is true."
  type        = map(string)
  default     = {}
 }
 # ── DNS records managed in the Cloudflare zone ───────────────────────────────
 # These names already exist, created by hand in the dashboard. Adopt them with
 # `terraform import` before the first apply, or Terraform will try to create a
 # second record with the same name and Cloudflare will reject it.
 variable "cloudflare_tunnel_hostnames" {
  description = "Hostnames in the Cloudflare zone that must CNAME to the tunnel."
  type        = list(string)
  default     = []
  validation {
    condition     = alltrue([for h in var.cloudflare_tunnel_hostnames : can(regex("^[a-zA-Z0-9._-]+$", h))])
    error_message = "cloudflare_tunnel_hostnames entries must be DNS labels or relative names (letters, digits, dot, underscore, hyphen)."
  }
 }
 variable "cloudflare_dns_ttl" {
  description = "TTL for tunnel CNAME records. 1 means automatic, which is required when proxied is true."
  type        = number
  default     = 1
 }
 variable "cloudflare_dns_proxied" {
  description = "Whether tunnel CNAME records are proxied through Cloudflare. Must stay true for a tunnel origin."
  type        = bool
  default     = true
 }
 # ── China edge ───────────────────────────────────────────────────────────────
 variable "china_domain" {
  description = "The .cn domain served from Tencent Cloud, without a trailing dot."
  type        = string
 }
 variable "china_hostnames" {
  description = "Subdomains of china_domain that point at the Tencent relay."
  type        = list(string)
  default     = ["paperless", "img"]
 }
 variable "tencent_secret_id" {
  description = "Tencent Cloud SecretId. Prefer TENCENTCLOUD_SECRET_ID in the environment."
  type        = string
  sensitive   = true
  default     = null
 }
 variable "tencent_secret_key" {
  description = "Tencent Cloud SecretKey. Prefer TENCENTCLOUD_SECRET_KEY in the environment."
  type        = string
  sensitive   = true
  default     = null
 }
 variable "tencent_region" {
  description = "Tencent Cloud region for the relay. Mainland regions need an ICP filing for the domain."
  type        = string
  default     = "ap-guangzhou"
 }
 variable "tencent_availability_zone" {
  description = "Availability zone inside tencent_region."
  type        = string
  default     = "ap-guangzhou-6"
 }
 variable "relay_instance_type" {
  description = "CVM spec for the relay. A relay only terminates TLS and forwards bytes."
  type        = string
  default     = "S5.SMALL2"
 }
 variable "relay_image_id" {
  description = "CVM image ID. Find one with: tencentcloud images --image-name 'Ubuntu 24.04' (or the console)."
  type        = string
 }
 variable "relay_ssh_public_key" {
  description = "Public key injected into the relay for break-glass SSH. Never the private key."
  type        = string
 }
 variable "relay_admin_cidr" {
  description = "CIDR allowed to reach SSH and WireGuard on the relay. Keep it to the operator, not 0.0.0.0/0."
  type        = string
  validation {
    condition     = var.relay_admin_cidr != "0.0.0.0/0" && var.relay_admin_cidr != "::/0"
    error_message = "relay_admin_cidr must not be the whole internet. Set the operator's egress CIDR."
  }
 }
 variable "wireguard_port" {
  description = "UDP port of the WireGuard listener on the relay."
  type        = number
  default     = 51820
 }
 variable "wireguard_allowed_cidr" {
  description = "CIDR allowed to reach the WireGuard UDP port. The homelab dials outbound, so this is often 0.0.0.0/0; WireGuard authenticates by key. Restrict it to the homelab egress CIDR when that address is static."
  type        = string
  default     = "0.0.0.0/0"
 }
 variable "wireguard_relay_private_key" {
  description = "WireGuard private key of the relay side. Generate with `wg genkey`; pass via TF_VAR_wireguard_relay_private_key."
  type        = string
  sensitive   = true
 }
 variable "wireguard_homelab_public_key" {
  description = "WireGuard public key of the homelab peer that dials the relay."
  type        = string
 }
 variable "wireguard_relay_address" {
  description = "Tunnel address of the relay side."
  type        = string
  default     = "10.88.0.1/24"
 }
 variable "wireguard_homelab_address" {
  description = "Tunnel address of the homelab side."
  type        = string
  default     = "10.88.0.2/24"
 }
 # ── Relay networking / sizing ───────────────────────────────────────────────
 variable "edge_vpc_cidr" {
  description = "CIDR of the relay VPC."
  type        = string
  default     = "10.66.0.0/16"
 }
 variable "edge_subnet_cidr" {
  description = "CIDR of the relay subnet."
  type        = string
  default     = "10.66.1.0/24"
 }
 variable "relay_internet_max_bandwidth_out" {
  description = "Peak outbound bandwidth in Mbps. The relay only terminates TLS and forwards bytes."
  type        = number
  default     = 20
 }
 variable "relay_system_disk_type" {
  description = "System disk type for the relay."
  type        = string
  default     = "CLOUD_PREMIUM"
 }
 variable "relay_system_disk_size" {
  description = "System disk size in GB for the relay."
  type        = number
  default     = 50
 }
 # ── DNS records ──────────────────────────────────────────────────────────────
 variable "china_dns_ttl" {
  description = "TTL in seconds for the .cn A records."
  type        = number
  default     = 600
 }
 variable "china_dns_record_line" {
  description = "DNSPod line for the .cn A records. Use 默认 for the default line."
  type        = string
  default     = "默认"
 }
 # ── Caddy reverse proxy ──────────────────────────────────────────────────────
 variable "wireguard_homelab_allowed_ips" {
  description = "AllowedIPs for the homelab peer. Must include the homelab LAN so Caddy can reach ingress-nginx over the tunnel."
  type        = string
  default     = "10.88.0.2/32,192.168.1.0/24"
 }
 variable "ingress_upstream" {
  description = "Upstream URL Caddy proxies to over WireGuard (the homelab ingress-nginx)."
  type        = string
  default     = "http://192.168.1.160:80"
 }
 variable "acme_email" {
  description = "Contact email for Caddy ACME (Let's Encrypt)."
  type        = string
  default     = "[email protected]"
 }
@@ -23,6 +23,7 @@
    # via nginx (not the backend service) is critical: a direct rewrite bypasses
    # nginx TLS and exposes each app's own self-signed cert, trusted by nothing.
    rewrite name authentik.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name zitadel.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name grafana.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name minio.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name minio-api.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
@@ -1,97 +0,0 @@
 # RECOVERED from live cluster state via talosctl get machineconfig.
 # Regenerated after the original tfvars.local was lost/corrupted.
 cluster_id  = "Rtc4g2av9EP0mOdxA4M0-QQitzHLWICz-rLBNfrOjgw="
 cluster_secret = "8E0CAeylAPKmmUEQmIGmHQAhQf+8c7NUf43CdrQZ+vg="
 bootstrap_token = "b2q7lh.9w7hwgrulrd65gr3"
 machine_token = "hq9wlf.96l9z46efd79jtr2"
 machine_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQekNCOHFBREFnRUNBaEVBMkUwRWZqbG41L1J3eTVjMVJmNkNSakFGQmdNclpYQXdFREVPTUF3R0ExVUUKQ2hNRmRHRnNiM013SGhjTk1qWXdOekE1TURVd056VTRXaGNOTXpZd056QTJNRFV3TnpVNFdqQVFNUTR3REFZRApWUVFLRXdWMFlXeHZjekFxTUFVR0F5dGxjQU1oQUNwR3NQZkVHdEU4anVmNG9JTkNHNnlCQmN6aURFaEpLbDV3CnJib0hSR0tUbzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkdmaExZUUdaOGYzd3lZZAo0SFI3KzlONnZKQXJNQVVHQXl0bGNBTkJBQ1ZQVlAwcGYxMkdUakYvSHJMZmcwZHBLemdBMlE1OGR5Y0paY0ZvClQvNDdPQk8ra29VZm11dXNjVVNNQnBXS0VuWHNYMFNocmNab3hQd1FHM3diblFFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
 machine_ca_key = "LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0KTUM0Q0FRQXdCUVlESzJWd0JDSUVJSlR4MXRqZEVOTTg1cGNRRFR0WWRtMFd0QXNBd0tzL1VESlZLN0ZqYU5uUgotLS0tLUVORCBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K"
 kubernetes_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpVENDQVRDZ0F3SUJBZ0lSQUtwT1ZaK29CRzljNHFycEpwaUc4TkV3Q2dZSUtvWkl6ajBFQXdJd0ZURVQKTUJFR0ExVUVDaE1LYTNWaVpYSnVaWFJsY3pBZUZ3MHlOakEzTURrd05UQTNOVGhhRncwek5qQTNNRFl3TlRBMwpOVGhhTUJVeEV6QVJCZ05WQkFvVENtdDFZbVZ5Ym1WMFpYTXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVQ1VjJvNkliMUpRZkcza3lCTTBCeTRkd2FLbnlGY2tVdjNVem9yOG1Ba3RaN2JrT2ZKZDlmL2VmcVkKYXBzUFpLV1RHQnYxM3JZbFdvOGtuU3ZnNm83Um8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXdIUVlEVlIwbApCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPCkJCWUVGQ1dPZVJUVHdnN2tnNVNWMG9raFQwY0VDNGwzTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSURURCtNYXUKb2JXdkp6UkNMVEdJaHJHc1daalFnalVmRWl2MnhCTXB6RnNVQWlCQzNNWkYwMjVqVHk4Uno2YjI5czVJQmpkNgpZQTVWd0kvNVFieXlwSGFXQlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
 kubernetes_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUrSEdPcUJJYXpJMzdqNmJJUlA1emVxeXEwZzhBU2xZeGplZUlJcEpIcXBvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFK1ZkcU9pRzlTVUh4dDVNZ1ROQWN1SGNHaXA4aFhKRkw5MU02Sy9KZ0pMV2UyNURueVhmWAovM242bUdxYkQyU2xreGdiOWQ2MkpWcVBKSjByNE9xTzBRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
 etcd_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJmVENDQVNTZ0F3SUJBZ0lSQVAyaHkwdUhvTm5vSzQyZE9LTk1nU2t3Q2dZSUtvWkl6ajBFQXdJd0R6RU4KTUFzR0ExVUVDaE1FWlhSalpEQWVGdzB5TmpBM01Ea3dOVEEzTlRoYUZ3MHpOakEzTURZd05UQTNOVGhhTUE4eApEVEFMQmdOVkJBb1RCR1YwWTJRd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFRVGlKYUJpSEJPCmJha3JuL0dVdkUxVFd5czRVUkVzVGtVNEM4OG1EdWZYQURjV0NnN2RTRzc0QjkzOGFwSWgybGc4UDhKRDFFRUoKN0RkU1lQVG5zYWh1bzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSApBd0VHQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkliYTBLaEhmM3hhClBQNk1hb3dESUNWVXBLdDVNQW9HQ0NxR1NNNDlCQU1DQTBjQU1FUUNJRzZMYUJGeGgvcys2WXpGdVlwaUxUWlYKS2prOGh5UVNvMmVTZTJTUy81MG5BaUI0a0RNWnR4b1UzTitHc3BEOHVEbXFrNlhxbzZoeE0vbG0yU21OMzlPNAovdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
 etcd_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUUwZ3JiMk0yblA4c1hxWjVhd3NzNThwbUdvb1FlSWg3a3RoWVlxNzhZZThvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFRTRpV2dZaHdUbTJwSzUveGxMeE5VMXNyT0ZFUkxFNUZPQXZQSmc3bjF3QTNGZ29PM1VodQorQWZkL0dxU0lkcFlQRC9DUTlSQkNldzNVbUQwNTdHb2JnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
 aggregator_ca_crt = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJZVENDQVFhZ0F3SUJBZ0lSQUpxeFI3alBzcmhzRUp2cmtEWndYR3N3Q2dZSUtvWkl6ajBFQXdJd0FEQWUKRncweU5qQTNNRGt3TlRBM05UaGFGdzB6TmpBM01EWXdOVEEzTlRoYU1BQXdXVEFUQmdjcWhrak9QUUlCQmdncQpoa2pPUFFNQkJ3TkNBQVFBWWlieHY1ZDVFRGdTbWhlRHlhYjJLZGh4ZFZnZ3lQc2pNcjBET0JMVmxtQmpMcXFqClpNSkk2d1ZmV2hkUjBRVGxVdEFLZDJvREJZLy9TeDBzQi9qY28yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXcKSFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4dwpIUVlEVlIwT0JCWUVGSTkrWm1EYVBybDhrTnhXUUxOT3ErTmVzeEh0TUFvR0NDcUdTTTQ5QkFNQ0Ewa0FNRVlDCklRRC9DRUZ2SUVHMW9qcmRZRVUzUldmTklLV1FwVXlZQWJ1ZGE0T0ZWQS9yOUFJaEFPS1VOZFF6bUk2WVZOdzgKeklDejlLblRxazQrT3dvV3RjNVl5SmlKR29zUgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
 aggregator_ca_key = "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUFoOEwycXFLbUxoYkpmczJ2M3ZRWXBFZHF6Ri9hTGZhSmoraEZRazdPY2NvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFQUdJbThiK1hlUkE0RXBvWGc4bW05aW5ZY1hWWUlNajdJeks5QXpnUzFaWmdZeTZxbzJUQwpTT3NGWDFvWFVkRUU1VkxRQ25kcUF3V1AvMHNkTEFmNDNBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo="
 service_account_key = "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKSndJQkFBS0NBZ0VBd3NNZU40eE1YQkZ0VVE2dVZ2NEZFU1dNT2ZLc1RKdmxHa0ptVG1URjJIazg1SlJvCkhuQU1nMEkvMEJiMGFUQTJ3MjlkSEUrdUN6dFd5eVVqTzV5eWdJSDQ0Q2RtVnNHVzVua1Z0TmFGNzNpcFllbjYKaG11bmRoWDY1VndqOHpVUmFYZUxtUEc0Y2d4dk5SOXdGYUZXS0ZZdzQzdGtlWjg5S0F5QTNibjFXN3JHTloyOQovcXRYWEcrQnFSSlVuUzhXQmt4dkErUktyRE1OVHF4bjVGOEl1VkpTWkw5bDVEd3lSUnNqMjVMZVdRZUFYenl6CllWcmFKZmVubSt6L3hLZjhLdHFZMSs5U1pwbkZsS2N6TGlWVzg4WkJ5TnR0KytYZjB4N3FSL2lQOHhRQXV0ckIKV1pTbytoa01FRjV5YU0wQWdxcG5mbDBsUmhsM0I0UUx3NHkyTUpoTEpyQ1ltQjl1QllLWU5oRE5NOXk0Z1lrZgpacmdEdm9NdGpsSUJUUHRyVjMyemxYN05Od3d5UjdXOXp5ZlBSVjEvbjhpMk9KR0szeittdGNCR2Y0TS8veDFsClJRNEhzWWhDUVJRaE8vQWp2U2QvQlhFRnB3dEhsb1VYRU5nQ2lPSW00MnFrZTNWb3lLbjJ2Q3g2QlhodlJVZDAKUHgvK0JpM0d4RmRoa2Uxa0doelJLQTdySGhEMVBkVGsyM04wUXV3WHNNVDZRWXVrRDdPZG5KdFZtZVN4TkxqcwptdVpybmpwRzZsaHRDSmdvMGdVeXBYQ2RlWGVnZ0dIc0JCMks4NjIrdThVa0dwMk9IQUhGeldsdDF5L2VXTW9PCmhneWx1SEVLcjkzUU5uNDZsTGtZTDViWjlVTUZ0NXBMRFVrSFQxNkV5dTh0V21ET2ZoQ3Z3M1lmRWY4Q0F3RUEKQVFLQ0FnQUdSbUlSV1JoV3VRc0VHd3g3NmdoQXdxeHZhNFdvbkRjMzd0Njc5Tnc0K3NMKy9GY1ViL2kvTytHeApjeVBoeGJkbCtZOE82L1JJRVZ2ZEJLL0xhbU9INTJnYzFMZ2o0RzNidEJnQ2NRejBwN2NSWEFnQno3TWdCMXBECmpJSHVBbzR5anpMMHRRa0R4Nm5IbE9FNEdUQWM4WlgycGxHWTU0d0JYOUhCRXc0NEs5N1orR0NZTlc0Rm9PUVYKRGUyaStOTGxWZzRYbW9IYlpYT3V6cmcwTCttb2l1SHp0QVQwNHdtZGwxL0M0Y3IvSkZJNi8wb3FQMUthK1kweApaV1BpTXFWWnZoeEJqTWpqWEYzMHlhUkkvdFA3MjYzZjZrM3pXVGNxWnFzV3NZZjF4WFcyajNpK1NaOWVHM043CmpZZHpIL085d2Y2K29BS2s3UW9jT0dGbXBnQnlwdm9JbytlSFdjZlpZNFNXNloyeTRacUl0M2xTSWFHMEtmQjEKUnVrSVBtMlYzNDNlc3Azdy9TV2liRVU2elhvd01sWjlZc0dPalY1MTZDQmJZK0lQcmY2RXY0UFhDWjlnUERxcgpnUFFIZ0lFS294aUdYK2dENG9pYWdUMWVVSk1iYWZkaEkzSHJMWk9YbUpBZU40RHpOZXR1SGRsTk13YnpZZHN5CnpyMllSMkhqNTZydk9hcWZZUFJKV1NmS3ozRlhQdDNDMHJRVmNRekVqV1ZZbm9MZG9yR3FkaU1uMyt1NzF2RW4KSnluZksvN3VRSDY5VTU3NGNKK3FjOFNtNTlPeHBnN1RoODljMUZTeSszTk03bTE2czlJcjQvcG94Q2pWaHNWeQpKRmN2cHE4UTMwek9CQk0waUxyblNKcVRXaXJDRU8xNHlRY0VLbFBVd2VGUWdJK05pUUtDQVFFQTMwMWNrTnZuCmpaRHduRXFQaWp4SlBDSUNEeHcxYWtnVTZUNUpMVkt5SUFnK3BYNHlPZDZtendvV2hpWE5oVUQ5eWxPL21lTEsKSDRPUXVzeWdJUDdkOVNKZi9ZVTV5a1RZaVZLMUdzM1loZXovRmhRQXdBQUg4UmdBdW8rZU9UTUcyT0IxM2loLworQlFYL1lrOG9VR1M0YlpsZWNGWG5pSTZ5S0g1MUJUQ3g0ZTZZcStvdGMrMFc2RzBRVHVBT3JWcjdXWklGY1htClVDdElReXRJN2s2UVd6SXU0K3dnVnU3OUdrUEJTMHNBaXhzU0ppYXRTNzk1TjlhYjYwRzNDeVFWZDZWajJaVmEKWGpCR1oyQXREalpEWS9MMDlZQTlRVUxDbWVqT2hYVFlGNkJuNnRkYmVjeVlZTHdNS2MzWEhqNEt2U2ZaQ01HTwpvUXROdnFoUFFlSWpxUUtDQVFFQTMwZnFCSThDK2QwdXJlM1JhNFFRdUlzUC9xVTMyejIzMUdzbTZOOXlQd0hUClM4ZzlrS0ZDYzhMVGlmWFdnajhIRWxTVjVLSXJseERwZEVWQ1pEM05qdy9GVDdHcHV5SVplN1E3VlhiNld3MnYKY1I0M1FkdUQvOEc4ZkdlZGkrZ1BnKzlzeURCQXFIVzFGSHl1RmVaWGNsVTF1cndOV3V3TlFKUnZHczFoK1NuQQpXVmJxd1ZUL01GMmYxTjlBaFN0alhkM0Nscm1rb0o4ek05YW0zVWg5VUprQ2xSZi9mZVRONXZndE1odS9NNGtGCmVQYUlBMTVqT3h1cEFMN3ptVVVxZ0h1NE9yMm5mYTVNNHMzWTR6TVRYYm9VZGNVT09oRTRzU0J2bGlaak5KL3QKV2pSTmJmUWhxbWRBTTNZZjFGUC8vRW9CYTBPejBxMHNXci92cEhDUlp3S0NBUUFxMlkySnZxa1FZVi9LbmdRdApZcVFyQmR1ZlNxcDFXcCtvb21zb1oxWUhENDMxOCtGdmVXcEpFSWFCOTM4WXN3QUFjMUd4RmZQeldDdk5yTGFOCm5scTVUMzljQnRTd0c4WHhsQTFzdDFOMVg2VVRkNE10Vk5ReFQ0blVRdnI1dnZEeGJTRXhJRlJ1Sm16MEdnR28KY0F6ZmcwQzF2SVF6dEIzVG9rRnVrUTFQZkp3bms4MnNGYzltUmdGeEF4bjRLaGdyMWhTL0dOcTVSNVQyVHJnUQpBc053dkpDQzdDeklnZFBQMW5DaElpTllqamxOV042b1NuWFlZVFpLVHJIeFVWdE5PaytPMFRvbUdOMXB1T3JzCmJ6MC9VTC93M0VyazJ3cTh2Zy9qVENpclgveVE5QUo1dk9rQXB4VXVjSEYzUERDVFc3SXFHL3Bpck9pZVRXM28KRnAwQkFvSUJBRjBxa3JsSU8wT3JTUmtHRE1aQ0d3QUY5cXlZb0EvNVZzVnAySmgrOUJyYVZpSmU4V0Z5Q0ZwcApSdjlmOXh2dDFMT1BXK1JFenMrQUhRbUpCTVR6RE56UEJkUFZIQytiY09xdkw3cmZwR050K0hESTNPRzhDUDRsCkJ0TWFJU0VKdWIraG5kQ0NZZGhwRlIveFRtcVE3SmdtZWY3ckROK05jNUlvM1p0ZmE2d2VBY2JGZjdzZ0RrTk8KTGEwVFlzYXViZzN5eElsRCtTK1VmamI1TUROUlZnalZiOEJxZlE4NDg3bVdnTFZSNHB4TVpsNHM4R0FIZUh4bgpkRU45YWdQZ1duVzJLZzlJcDZUSG9BbGJQMDYrTnl4NndxTEprTUFtQTNQVlJ2cHVGaU1WUUdMTlJDbkhIbTBPCkhEbmM1amNndmNXMTA1WEFjRDVPU0IydHpQN2VnYTBDZ2dFQUZhTHJxMDJ6M2tXaW1iVzdoNi9Pby9YRHdjZ2YKSmdXMDYyYWdWUlpEMjM3Mm4valZSL25kMnFybmwxTFdWaXpzWW91b3hvY2N6NUwvQ2h3MktCOC9Rc1Zxai9KOApOUWh1ZDJ2ZUxjQlUvVzNseVhENnJpajJZMythNzRvM1A4b1psOGdDQmJEck1jajNsMHRZMXRWbm9nOWo0eHQ3Ckp3UXA3djNXb3ArSFVuRWVjbGpscTdlN1VuTGNwZlRDOE5PUmdxbjBGSUtCbXdFNlpJQnQxUzB6NmxSKytJdVMKQk1oTTZTSkZmeUFOVVdjTCt2cHJoRzBjWUZmcUYzajlPZ1dXRnBGTHFWQmdSTlZBVDRoM0U1Ymh4L0lsMTVHMAo2USs2aWlkRElqMEhNNFAzcjhja2lRUlVPdFI1R2NVS1RYZGh1TjkwMjZGRTdEckRvSURVZ0F5dGVnPT0KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K"
 secretbox_encryption_secret = "RLkR4RmeaLmH/abyK3eYf6q22umJ/byLhheVCnh/yrA="
 controlplane_configs = {
  "talos-cp-1" = {
    hostname     = "talos-cp-1"
    lan_ip       = "192.168.1.166"
    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/nvme0n1"
    longhorn_disks = []
    zone             = "az-a"
    allow_scheduling = true
    cloudflare_talos_sans     = []
    cloudflare_apiserver_sans = []
  },
  "talos-cp-2" = {
    hostname     = "talos-cp-2"
    lan_ip       = "192.168.1.214"
    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/disk/by-id/wwn-0x644a842029bd3f002720989b07d7143d"
    longhorn_disks = [
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b60f4375d97062", mountpoint = "/var/lib/longhorn-disk1" },
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b5ffeb8bb8b47f", mountpoint = "/var/lib/longhorn-disk2" },
        { device = "/dev/disk/by-id/wwn-0x644a842029bd3f0031b6000c8dabb266", mountpoint = "/var/lib/longhorn-disk3" },
        # /var/lib/longhorn-disk4 (wwn-0x6b083fe0c5782700321370dc23fd765b) was
        # retired 2026-10-12: the 2TB PERC virtual disk failed with continuous
        # SCSI I/O errors, hung its XFS filesystem, and wedged Longhorn replica
        # rebuilds on talos-cp-2 with unkillable D-state processes. Do not
        # re-add it.
      ]
    zone             = "az-b"
    allow_scheduling = true
    cloudflare_talos_sans     = []
    cloudflare_apiserver_sans = []
  },
  "talos-cp-3" = {
    hostname     = "talos-cp-3"
    lan_ip       = "192.168.1.162"
    public_ip    = "https://k8s-api.riotpiao.com"
    lan_subnet   = "192.168.1.0/24"
    lan_gateway  = "192.168.1.254"
    install_disk = "/dev/nvme0n1"
    longhorn_disks = [
        { device = "/dev/disk/by-id/usb-Seagate_One_Touch_w_PW_00000000NABV3H34-0:0", mountpoint = "/var/lib/longhorn-paperless-media" },
      ]
    zone             = "az-c"
    allow_scheduling = true
    cloudflare_talos_sans     = []
    cloudflare_apiserver_sans = []
  },
 }
 worker_configs = {
  "worker-1" = {
    hostname          = "worker-1"
    lan_ip            = "192.168.1.223"
    lan_subnet        = "192.168.1.0/24"
    lan_gateway       = "192.168.1.254"
    install_disk      = "/dev/nvme0n1"
    network_interface = "enp28s0f0np0"
    zone              = "az-a"
    gpu_count         = 4
    node_labels       = {}
    node_taints       = []
    factory_image     = "factory.talos.dev/metal-installer/0a2153a6dc099a371bf2f63d6c3c22d275c876bf6302dd154c5813072924cb3f:v1.13.3"
    swap_size         = "64GiB"
    ephemeral_max_size = "700GiB"
    extra_disks       = []
  }
 }
 cluster_config = {
  controlplane_ip = "192.168.1.166"
  pod_subnets     = ["10.244.0.0/16"]
  service_subnets = ["10.96.0.0/12"]
  dns_servers     = ["8.8.8.8", "1.1.1.1"]
  dns_domain      = "cluster.local"
 }
Author	SHA1	Message	Date
rock	227a67a8c1	feat(cloudflared): manage tunnel ingress locally via ConfigMap The tunnel was remotely managed from the Zero Trust dashboard: the Deployment ran 'run --token' and ignored the ConfigMap that already existed in the tree. Switch to a locally-managed tunnel so the ingress routing table lives in git. - deployment: mount cloudflared-config (config.yml) and the cloudflared-credentials Secret; drop TUNNEL_TOKEN env - configmap.yaml: per-hostname ingress rules to ingress-nginx via the CoreDNS rewrite, plus a 404 catch-all - secret-generator: register the SOPS-encrypted credentials file - homelab-ca: add the cloudflared namespace copy so the connector trusts the ingress certificate (force-ssl-redirect makes plaintext impossible) Credentials were extracted from the live cloudflared-token Secret and re-encrypted with SOPS; no plaintext secret is committed.	2026-10-25 19:57:37 -07:00
pi	d4e9dbf350	refactor(iam): retire terraform/zitadel, provision through the API The Terraform root module duplicated a writer that the provisioning script could be, and its state carried generated client secrets. Declare the objects in the script instead: project, 19 roles, 12 OIDC apps, applied through the management API with the iam-admin PAT. State is a 0600 JSON file under ~/.config/zitadel. Also add zitadel to the Cloudflare tunnel hostname list, with the import command and the note that the tunnel's Public Hostname entry is a separate dashboard step: DNS alone yields a 404 from the tunnel.	2026-10-25 19:35:01 -07:00
pi	be44a576d4	fix(iam): own the Zitadel login keypair through ksops The chart generated the login RSA keypair on every render, so checksum/secret-login-service-key changed each time and both Zitadel Deployments stayed permanently OutOfSync. Adopt the live keypair into k8s/argocd/secrets/zitadel-login-service-key.enc.yaml (SOPS, age recipient from .sops.yaml), list it in the ksops generator, and point login.loginServiceKeySecretName at it. The chart then stops generating the keypair and the Deployment checksums become deterministic.	2026-10-25 19:16:18 -07:00
pi	39081222a3	fix(iam): stop the setup hook running after first boot A PreSync setup run is not harmless once the instance exists: it replays every repeatable migration on every sync, and interrupting one leaves a system.migration.started event with no matching done. ZITADEL then waits forever on 'migration already started'. That deadlocked this instance twice (projections.projects4, then repeatable_delete_stale_org_fields). Move the setup job's hook from pre-install,pre-upgrade to post-delete so ArgoCD sync never runs it. Re-enable deliberately for a schema upgrade.	2026-10-25 19:10:32 -07:00
pi	7e5b793670	fix(iam,edge): freeze first-boot and wire the WireGuard CIDR variable Zitadel first boot completed on 2026-10-26, so set FirstInstance.Skip: true and initJob.enabled: false. Without Skip, every sync re-runs machinekeyWriter and silently rotates the iam-admin key that terraform/zitadel authenticates with. Also wire wireguard_allowed_cidr into the WireGuard security-group rule; the variable was declared but the rule still used relay_admin_cidr, which blocks the homelab peer unless its egress address happens to match the operator CIDR.	2026-10-25 18:48:15 -07:00
pi	354814a531	security: stop tracking Talos secret backup; rotate edge WireGuard key - remove terraform/terraform.tfvars.local.bak from the index (it carries the live cluster PKI: CA keys, service-account key, secretbox secret) - ignore terraform/*.bak and terraform/edge/.secrets/ - generate a fresh WireGuard relay keypair into the git-ignored local tfvars; the committed example now documents rotation instead of holding key material	2026-10-25 18:15:48 -07:00
pi	54fac02783	fix(cloudflared): pin tunnel protocol to http2 QUIC loses long-lived SSE streams when Chinese ISP gateways throttle UDP. http2 over TCP is the deliberate fallback already in use.	2026-10-25 18:04:53 -07:00
pi	eb83aa6290	refactor(edge,iam): apply best practices to relay and Zitadel config - edge: extract VPC/subnet/bandwidth/disk/DNS/Caddy settings into variables - edge: add Caddyfile template with SSE-safe flush_interval -1 and security headers - edge: add WireGuard NAT masquerade so homelab peer accepts forwarded client IPs - edge: mark user_data sensitive and add prevent_destroy to the relay instance - edge: validate relay_admin_cidr and Cloudflare hostnames - edge: parameterise Cloudflare TTL/proxied and enforce proxied tunnel CNAMEs - iam: fix PDB key to chart-supported pdb and remove unsupported extraVolumes - iam: keep login hardened; drop unsupported login PDB	2026-10-25 18:03:55 -07:00
pi	6ebecec1c4	fix(zitadel): raise setup job deadline to 20m The chart default killed zitadel-setup at ~5.5 minutes while it was still replaying projections.projects4, which failed the whole PreSync and left the Application OutOfSync/Missing. 1200s leaves room for a cold database without masking a genuine hang.	2026-10-25 09:47:33 -07:00
rock	091ecdd7bb	feat(edge): Terraform for the China-facing relay and Cloudflare DNS Adds terraform/edge, a root module with two providers: - Cloudflare 5.28.1: CNAMEs to the existing tunnel, plus an opt-in resource that can take the tunnel ingress list over when manage_tunnel_config is true. It is off by default because that resource REPLACES the whole list, and the tunnel is remotely managed from the dashboard today. - Tencent Cloud 1.83.38: VPC, subnet, security group with three rules, key pair, one CVM relay with cloud-init (WireGuard listener, forwarding sysctls, Caddy), and DNSPod A records for the .cn hostnames. The design keeps the homelab inbound-free: the relay terminates TLS in China and forwards over a WireGuard tunnel the cluster dials outbound. Cloudflare keeps serving every existing hostname untouched. Verified: terraform init succeeds for both providers, fmt clean, validate reports 0 errors, and plan resolves to 10 creates with placeholder variables. Provider-side name validation caught a real bug during that run: key pair and security group names allow only letters, digits and underscore, so the derived names use underscores. Not applied. Credentials come from CLOUDFLARE_API_TOKEN, TENCENTCLOUD_SECRET_ID and TENCENTCLOUD_SECRET_KEY; the WireGuard private key is TF_VAR only. State is local and git-ignored like terraform/zitadel. Blocked on facts this module cannot invent: the real .cn domain, whether it has an ICP filing, the CVM image ID, the operator CIDR, and the homelab WireGuard public key.	2026-10-24 22:59:54 -07:00
rock	7251032081	Merge pull request 'fix(monitoring): cap Prometheus retention below the PVC size' (#68 ) from fix/prometheus-retention-size into main	2026-10-25 04:43:21 +00:00
rock	b7fc6ed60b	fix(monitoring): cap Prometheus retention below the PVC size retentionSize was 95GB against a 25Gi volume, so the limit never fired. The volume filled, and Prometheus crashlooped with: write /prometheus/queries.active: no space left on device failed to initialize active query tracker 18GB keeps roughly 7GiB of headroom for the WAL and block overhead.	2026-10-24 21:43:01 -07:00
rock	6312674058	chore(sms): retire the stuck macOS namespace The macOS workload is scaled to 0 and its PVC macos-disk has been Pending for 72 days against storage class longhorn-imessage-local. No node carries the workload=imessage label it pins to. The namespace only holds that dead PVC plus two unused Services. Remove the Application, the manifests and the namespace so ArgoCD prunes them.	2026-10-24 20:05:20 -07:00
rock	2fe6d50ffd	Merge pull request 'fix(argocd): allow the Zitadel chart repo in the homelab AppProject' (#67 ) from fix/allow-zitadel-chart-repo into main	2026-10-25 01:26:38 +00:00
rock	e2d9717ce1	fix(argocd): allow the Zitadel chart repo in the homelab AppProject The zitadel Application failed its first sync with InvalidSpecError: application repo https://charts.zitadel.com is not permitted in project 'homelab'. AppProject sourceRepos is an allowlist, so the chart repo must be listed before any chart from it can be reconciled.	2026-10-24 18:26:21 -07:00
rock	111b19dac2	Merge pull request 'fix(sops): repair the ksops generator list and rotate the Zitadel masterkey' (#66 ) from fix/sops-generator-indent into main	2026-10-25 01:24:43 +00:00
rock	e335694cbb	fix(sops): repair the generator list and rotate the Zitadel masterkey Two defects in the ksops generator: 1. The `zitadel-masterkey.enc.yaml` entry was added with two-space indentation while every other entry uses four. YAML rejected the list, and kustomize reported it as `loading generator plugins ... MalformedYAMLError: yaml: line 39`. That failed the whole sops-secrets application, so no Secret in that app could be created. Verified by running the same kustomize build inside argocd-repo-server: exit 1 before the fix, exit 0 with 33 manifests after it. 2. The masterkey value was generated during the first verification pass and appeared in that session's output. Rotate it now, before the instance exists: a fresh 32-byte value, SOPS-encrypted, with the offline copy updated. No ZITADEL data exists yet, so rotation costs nothing. After the first sync it would mean re-encrypting system.encryption_keys. The offline copy lives outside the repo at ~/zitadel-masterkey.offline (0600).	2026-10-24 18:24:28 -07:00
rock	d1a7d3be6e	fix(oidc): remove the API-server restart Jobs from GitOps The oidc-bootstrap ArgoCD Application runs with selfHeal and prune. Three Jobs in restart-apiserver.yaml kill the kube-apiserver process on each control plane, and bootstrap.yaml carried a fourth Job that deletes the mirror pods. ArgoCD recreated a deleted Job within two minutes, so the kill repeated and every API server reached CrashLoopBackOff. Remove all four. The OIDC authenticator already loaded, so the one-shot restart is not needed again.	2026-10-24 15:06:32 -07:00
rock	2f664c32d9	fix(cluster): grant queue-operator watches and retire the macOS workload queue-operator could not list temporalworkers.kmsvc.io or deployments.apps at cluster scope, so its informers failed on every retry and it reconciled nothing. Add those rules to the ClusterRole. macos-bluebubbles pins itself to a node labelled workload=imessage, and no such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local and stayed unbound for 71 days. Scale the Deployment to 0 and record why.	2026-10-24 04:03:02 -07:00
rock	6c98ca7436	Merge pull request 'fix(sops): drop the orphan forgejo-registry-pat file' (#65 ) from fix/sops-drop-orphan-forgejo-pat into main	2026-10-24 10:47:14 +00:00
rock	6ee22401fd	fix(sops): drop the orphan forgejo-registry-pat file ksops failed the whole sops-secrets application on this file: it declares Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists. The failure blocks every other file in the generator list, including zitadel-masterkey. The same Secret name is already generated in namespace api by forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace forgejo was never created. Remove the file and its generator entry. ArgoCD never created the Secret, so pruning has nothing to remove.	2026-10-24 03:46:29 -07:00
rock	e9817283e6	Merge pull request 'feat(iam): wire Zitadel as the Authentik replacement' (#63 ) from harness/zitadel-iam-wiring into main	2026-10-24 10:41:17 +00:00
rock	6e9586aa15	Merge pull request 'chore(argocd): delete the azhi-riotpiao Application' (#64 ) from chore/remove-azhi-riotpiao-app into main	2026-10-24 10:41:09 +00:00
rock	979a413a5b	feat(iam): read the Zitadel bootstrap token from .env ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the same shape the Authentik variables use. The line is present and empty: the PAT does not exist until the chart's setup job creates the iam-admin machine user on the first sync, so a value cannot be minted ahead of time. The provisioning script loads .env itself instead of relying on the caller to source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN, then the cached file under ~/.config/zitadel, then the cluster Secret. The .env value wins so a human can paste a rotated token without touching the cluster; the cluster Secret remains the only source that always reflects a rotation. .env stays git-ignored. It already holds the live Authentik secrets, and this change does not move them. Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.	2026-10-24 03:38:21 -07:00
rock	2f62351731	feat(iam): one script to provision Zitadel end to end scripts/iam/zitadel-provision.py replaces the manual sequence that provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users, verify. `all` runs keys -> apply -> secrets -> verify; users stays manual because the password hash must come from a human, one user at a time. - keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel at 0600, outside the repo. - apply: terraform apply terraform/zitadel with the machine key. - secrets: write generated client secrets into the SOPS files, write client ids into the plain manifests that carry them, write machine keys per agent. Requires an exactly-one-match regex before it edits a manifest, prints the diff without --apply-config, and is a no-op when the value already matches. - users: import a human user with the Django pbkdf2_sha256 hash converted to the passwap MCF form ZITADEL accepts. - verify: discovery endpoint, client count, and it refuses to clear cutover while the groups-claim action is disabled. Design points that came out of review: - Multi-document SOPS files (minio) are rewritten whole, because `sops set` cannot address one document. sops reads the creation rule from the filename, so the temp file is passed with --filename-override; without it sops finds no matching rule and refuses. Ciphertext is staged in the destination directory and installed with os.replace, so the mode is preserved and EXDEV is impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file. - homarr is listed as manual: the live workload mounts Secret homarr-oidc, which no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no consumer. Writing the new secret into the orphan file would report success while the dashboard kept the old one. - Secrets never enter argv: curl reads its config from stdin, so the PAT and the password hash do not appear in ps. .gitignore: allow this one script through the scripts/iam/*.py ignore rule (it embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M tokens and was one `git add -A` away from being committed. Checks: py_compile clean, --help lists every phase, convert_hash verified on a synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents survive, value set, second run unchanged, mode 644 kept, no temp left), all three client-id regexes match exactly once, dry-run leaves the manifests untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr Secret, the missing client-id writes and argv exposure; all fixed and re-reviewed to SIGN-OFF. Not run against a live ZITADEL: no instance exists yet.	2026-10-24 03:38:21 -07:00
rock	f81ff0e2fc	feat(iam): declare Zitadel identity objects in Terraform Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open. It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users plus keys, one API application for the gateway, project grants, per-agent claim metadata, and a disabled groups-claim action. Facts established while writing it: - Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest has no client_id field, and provider 3.8.7 rejects a settable application id (terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a client_id-preserving re-point; every consumer now also changes client ID. - Role keys equal the existing group names, so the groups claim can be byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200 characters with no character-set restriction. - Machine users get JWT profile keys, not passwords. Per-agent claims move to user metadata and reach tokens only through the groups-claim action. - The action stays gated: enable_groups_claim_action defaults to false because the action runtime API is unverified. Two validations stop a null script from reaching file(). Checks: terraform fmt -check -recursive clean, terraform validate 0 errors, plan reaches only the expected missing-credentials error, structural gate 0 error / 0 warning, and one reviewer round: BLOCK on a plan-time fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed to SIGN-OFF. State is local and git-ignored; it carries generated client secrets and machine keys. Move it to MinIO before a second operator applies. Not applied: no Zitadel instance is running yet.	2026-10-24 03:38:21 -07:00
rock	cb1219bf96	feat(iam): wire Zitadel into namespace iam under ArgoCD Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam. Authentik keeps serving; no consumer is moved and no Authentik object changes. - k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances - k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat - k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3, automated sync deliberately absent for the first sync (chart PreSync hook re-run risk) - k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS - k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI, crash-loop and PreSync-job-failed rules - terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1, authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and kube-state-metrics has 1925 restarts. Two manual steps still gate the first sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned). The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it to sealed storage and delete it before the first sync. Losing it makes every encrypted ZITADEL value unreadable.	2026-10-24 03:38:21 -07:00
rock	8e9ac37583	chore(argocd): delete the azhi-riotpiao Application The Application pointed at github.com/ToyoGlobal/azhi-infra, branch feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled: status stayed Unknown because the source repository could not be read, and the destination namespace azi-riotpiao was never created. ArgoCD reported it as an OutOfSync managed resource of homelab-root with nothing behind it. Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from that repository is referenced any more, so the allowlist entry is dead weight. Not touched here, because it is not in git: Secret azhi-infra-github in namespace argocd holds the read-only GitHub credential for that repository. Delete it separately once no other Application needs the repo. The live Application object was deleted by hand. Until this commit reaches main, the next homelab-root sync recreates it, because main still declares it.	2026-10-24 03:35:52 -07:00
rock	3c851b16c9	Merge branch 'fix/certmanager-dns01' Restores wildcard renewal: pin the ACME DNS-01 self-check to the public resolvers and drop the per-host TLS blocks that competed with the wildcard.	2026-10-24 02:39:38 -07:00