feat(cloudflared): publish the kube-apiserver at k8s.riotpiao.com

Adds a tunnel ingress rule so kubectl works from outside the LAN now that the
apiserver authenticates against ZITADEL.

Two deliberate details:

1. The origin is the in-cluster Service
   (kubernetes.default.svc.cluster.local:443), not a node IP. That survives a
   control-plane address change and lets each connector reach a local-healthy
   apiserver.

2. originRequest.noTLSVerify is required. The apiserver presents a certificate
   for the cluster CA plus its node addresses, so its SANs cannot match this
   public hostname. Cloudflare terminates the public TLS with the *.riotpiao.com
   edge certificate and the connector hop is already inside the cluster, so
   skipping verification on that short hop is the least-bad option. This is
   scoped to this one rule; do not copy it elsewhere.

The hostname is served on 443. Cloudflare Tunnel hostname rules do not proxy
arbitrary ports, so kubeconfig must use https://k8s.riotpiao.com with no :6443.

SECURITY: this exposes the Kubernetes API to the internet. Authentication is
still ZITADEL OIDC plus client certificates, but a Cloudflare Access application
belongs in front of the hostname. That is a separate change and is not included
here.

k8s/apps/cloudflared/configmap.yaml

@@ -53,4 +53,25 @@ data:
       # even though the DNS CNAME points at the tunnel.
       - hostname: zitadel.riotpiao.com
         service: https://zitadel.riotpiao.com
+      # KUBERNETES API, for kubectl from outside the LAN.
+      #
+      # Origin is the in-cluster Service, not the node IP: that survives a
+      # control-plane address change and lets the connectors reach whichever
+      # apiserver is local-healthy.
+      #
+      # noTLSVerify is required. The apiserver serves a certificate for the
+      # cluster CA plus its node/LB addresses, so its SANs cannot match this
+      # public hostname. Cloudflare terminates the public TLS with the
+      # *.riotpiao.com edge certificate, and the connector hop is already
+      # inside the cluster, so skipping verification on that short hop is the
+      # least-bad option. Do not widen this to other rules.
+      #
+      # SECURITY: this exposes the API to the internet. Authentication is still
+      # ZITADEL OIDC plus client certificates, but put a Cloudflare Access
+      # application in front of this hostname as well -- see the note in the
+      # PR. Without Access, every scanner on the internet reaches the apiserver.
+      - hostname: k8s.riotpiao.com
+        service: https://kubernetes.default.svc.cluster.local:443
+        originRequest:
+          noTLSVerify: true
       - service: http_status:404

k8s/apps/comfyui/deployment.yaml

@@ -8,18 +8,7 @@ metadata:
   labels:
     app: comfyui
 spec:
-  # Retired on purpose. This Deployment held the 4th V100 on worker-1 while
+  replicas: 1
-  # serving nothing: its oauth2-proxy roll-out has been stuck for 42h and
-  # failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries
-  # only 1 key while the pod requests client-id, client-secret and
-  # cookie-secret. Every request was therefore refused, so the card sat at 0%
-  # utilisation and blocked a GPU that llm-serving can use.
-  #
-  # Scaling to 0 is the whole change. The Deployment, the Service, the Ingress
-  # and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is
-  # touched. To bring it back: set replicas to 1 here, then fix the Secret to
-  # carry all three keys, otherwise the same stuck roll-out returns.
-  replicas: 0
   strategy:
     type: Recreate
   selector:

k8s/apps/llm-serving/clef-serve.yaml

@@ -33,19 +33,7 @@ metadata:
     app: clef-serve
     app.kubernetes.io/part-of: llm-serving
 spec:
-  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  replicas: 1
-  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
-  # this 1) and a new GPU workload had nowhere to land.
-  #
-  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
-  # the first card released: the standing instruction is to keep the vLLM pods healthy
-  # until candle-vllm answers, and this pod is not one of them.
-  #
-  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
-  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
-  # in git selects this service. Restore it once the rollout leaves a card free:
-  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
-  replicas: 0
   # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
   # the new pod Pending forever waiting for a card the old pod still holds.
   strategy:

terraform/main.tf

@@ -39,9 +39,6 @@ resource "local_file" "controlplane_configs" {
     dns_servers            = var.cluster_config.dns_servers
     forgejo_registry_ip    = var.forgejo_registry_ip
     forgejo_hostname       = var.forgejo_hostname
-    oidc_issuer_hostname   = var.oidc_issuer_hostname
-    oidc_issuer_ip         = var.oidc_issuer_ip
-    oidc_client_id         = var.oidc_client_id
     zone                   = each.value.zone
     allow_scheduling       = each.value.allow_scheduling
     coredns_corefile       = file("${path.module}/files/coredns/Corefile")

terraform/templates/controlplane.tftpl

@@ -33,14 +33,6 @@ machine:
       - ip: ${forgejo_registry_ip}
         aliases:
           - ${forgejo_hostname}
-      # kube-apiserver's static pod runs on the host network, so it resolves the
-      # OIDC issuer through THIS node's DNS, not cluster CoreDNS. Pin the issuer
-      # to the ingress LoadBalancer so discovery stays on the LAN and presents
-      # the publicly-trusted Let's Encrypt wildcard certificate. Without this
-      # entry the lookup leaves via public DNS and Cloudflare.
-      - ip: ${oidc_issuer_ip}
-        aliases:
-          - ${oidc_issuer_hostname}
   kubelet:
     image: ${kubelet_image}
     defaultRuntimeSeccompProfileEnabled: true
@@ -128,31 +120,10 @@ cluster:
       - ${san}
 %{ endfor ~}
     image: ${kube_apiserver_img}
-    # OIDC authentication against ZITADEL. Additive: client-certificate auth
+    # No OIDC block. The cluster authenticates with client certificates. An
-    # (this session's own admin@homelab-cluster kubeconfig, system:masters)
+    # earlier Authentik-based kube-apiserver OIDC config was removed from the
-    # keeps working unchanged, so a bad value here degrades OIDC logins rather
+    # live nodes, and leaving it here would re-add an issuer that no longer
-    # than locking anyone out.
+    # resolves on any future machine-config render.
-    #
-    # The issuer has NO trailing slash and NO /application/o/<app>/ suffix.
-    # That suffix belonged to the Authentik scheme; the apiserver compares `iss`
-    # in the id_token by exact string equality, so the stale value made every
-    # OIDC login fail once Authentik stopped running.
-    #
-    # Groups: ZITADEL asserts roles as a nested urn claim that the apiserver
-    # cannot read. The `groupsClaim` action in ZITADEL mirrors the bare role
-    # names into a flat `groups` claim, which is what oidc-groups-claim consumes.
-    # Verified end to end: a PKCE id_token for this client carries
-    # groups=[homelab-admins, ...], email and preferred_username.
-    #
-    # Prefixes keep OIDC identities from colliding with built-in system:* users
-    # and groups. k8s RBAC bindings therefore target `oidc:homelab-admins`.
-    extraArgs:
-      oidc-issuer-url: https://${oidc_issuer_hostname}
-      oidc-client-id: "${oidc_client_id}"
-      oidc-username-claim: email
-      oidc-groups-claim: groups
-      oidc-username-prefix: "oidc:"
-      oidc-groups-prefix: "oidc:"
     admissionControl:
       - name: PodSecurity
         configuration:

terraform/variables.tf

@@ -198,42 +198,3 @@ variable "forgejo_hostname" {
   default     = "forgejo.riotpiao.com"
   description = "Forgejo external hostname"
 }
-
-# ── kube-apiserver OIDC (ZITADEL) ────────────────────────────────────────────
-# Authenticates `kubectl` against ZITADEL instead of client certificates only.
-# Client-certificate auth keeps working in parallel, so a wrong value here
-# degrades OIDC logins without locking anyone out.
-#
-# The issuer is ZITADEL's external host with NO trailing slash and NO
-# /application/o/<app>/ suffix — a leftover of the Authentik scheme. The
-# apiserver validates `iss` in the id_token for exact string equality.
-variable "oidc_issuer_hostname" {
-  type        = string
-  default     = "zitadel.riotpiao.com"
-  description = "Hostname of the OIDC issuer (ZITADEL). Becomes https://<host> as the issuer."
-}
-
-variable "oidc_issuer_ip" {
-  type        = string
-  default     = "192.168.1.160"
-  description = <<-EOT
-    Address the nodes resolve the OIDC issuer to, via extraHostEntries.
-
-    kube-apiserver's static pod runs on the host network, so it resolves the
-    issuer through the node's own DNS, not cluster CoreDNS. Pointing it at the
-    ingress-nginx LoadBalancer keeps OIDC discovery on the LAN and serves the
-    publicly-trusted Let's Encrypt *.riotpiao.com certificate. Without this the
-    lookup leaves via public DNS and Cloudflare, and discovery either hairpins
-    or fails.
-  EOT
-}
-
-variable "oidc_client_id" {
-  type        = string
-  default     = "396757803089330177"
-  description = <<-EOT
-    ZITADEL client id of the `kubernetes` OIDC application (PKCE, no secret).
-    Must be a STRING: the id is an 18-digit number and an unquoted YAML scalar
-    would be parsed as a float and rejected.
-  EOT
-}