chore(llm-serving): park clef-serve to free a V100 for candle-vllm

candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.

This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.

It was not serving anything. At park time its last 30 log lines were 30 of 30
kubelet /health probes, with no inference request since it started on
2026-10-30T06:23:40Z. Nothing in git selects this Service. It held a card for a
day to answer liveness probes, which is the same shape as the comfyui
retirement.

Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.

Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick.

k8s/apps/cloudflared/configmap.yaml

@@ -53,25 +53,4 @@ data:
       # even though the DNS CNAME points at the tunnel.
       - hostname: zitadel.riotpiao.com
         service: https://zitadel.riotpiao.com
-      # KUBERNETES API, for kubectl from outside the LAN.
-      #
-      # Origin is the in-cluster Service, not the node IP: that survives a
-      # control-plane address change and lets the connectors reach whichever
-      # apiserver is local-healthy.
-      #
-      # noTLSVerify is required. The apiserver serves a certificate for the
-      # cluster CA plus its node/LB addresses, so its SANs cannot match this
-      # public hostname. Cloudflare terminates the public TLS with the
-      # *.riotpiao.com edge certificate, and the connector hop is already
-      # inside the cluster, so skipping verification on that short hop is the
-      # least-bad option. Do not widen this to other rules.
-      #
-      # SECURITY: this exposes the API to the internet. Authentication is still
-      # ZITADEL OIDC plus client certificates, but put a Cloudflare Access
-      # application in front of this hostname as well -- see the note in the
-      # PR. Without Access, every scanner on the internet reaches the apiserver.
-      - hostname: k8s.riotpiao.com
-        service: https://kubernetes.default.svc.cluster.local:443
-        originRequest:
-          noTLSVerify: true
       - service: http_status:404

k8s/apps/comfyui/deployment.yaml

@@ -8,7 +8,18 @@ metadata:
   labels:
     app: comfyui
 spec:
-  replicas: 1
+  # Retired on purpose. This Deployment held the 4th V100 on worker-1 while
+  # serving nothing: its oauth2-proxy roll-out has been stuck for 42h and
+  # failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries
+  # only 1 key while the pod requests client-id, client-secret and
+  # cookie-secret. Every request was therefore refused, so the card sat at 0%
+  # utilisation and blocked a GPU that llm-serving can use.
+  #
+  # Scaling to 0 is the whole change. The Deployment, the Service, the Ingress
+  # and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is
+  # touched. To bring it back: set replicas to 1 here, then fix the Secret to
+  # carry all three keys, otherwise the same stuck roll-out returns.
+  replicas: 0
   strategy:
     type: Recreate
   selector:

k8s/apps/llm-serving/clef-serve.yaml

@@ -33,7 +33,19 @@ metadata:
     app: clef-serve
     app.kubernetes.io/part-of: llm-serving
 spec:
-  replicas: 1
+  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
+  # this 1) and a new GPU workload had nowhere to land.
+  #
+  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
+  # the first card released: the standing instruction is to keep the vLLM pods healthy
+  # until candle-vllm answers, and this pod is not one of them.
+  #
+  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
+  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
+  # in git selects this service. Restore it once the rollout leaves a card free:
+  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
+  replicas: 0
   # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
   # the new pod Pending forever waiting for a card the old pod still holds.
   strategy:

terraform/main.tf

@@ -39,6 +39,9 @@ resource "local_file" "controlplane_configs" {
     dns_servers            = var.cluster_config.dns_servers
     forgejo_registry_ip    = var.forgejo_registry_ip
     forgejo_hostname       = var.forgejo_hostname
+    oidc_issuer_hostname   = var.oidc_issuer_hostname
+    oidc_issuer_ip         = var.oidc_issuer_ip
+    oidc_client_id         = var.oidc_client_id
     zone                   = each.value.zone
     allow_scheduling       = each.value.allow_scheduling
     coredns_corefile       = file("${path.module}/files/coredns/Corefile")

terraform/templates/controlplane.tftpl

@@ -33,6 +33,14 @@ machine:
       - ip: ${forgejo_registry_ip}
         aliases:
           - ${forgejo_hostname}
+      # kube-apiserver's static pod runs on the host network, so it resolves the
+      # OIDC issuer through THIS node's DNS, not cluster CoreDNS. Pin the issuer
+      # to the ingress LoadBalancer so discovery stays on the LAN and presents
+      # the publicly-trusted Let's Encrypt wildcard certificate. Without this
+      # entry the lookup leaves via public DNS and Cloudflare.
+      - ip: ${oidc_issuer_ip}
+        aliases:
+          - ${oidc_issuer_hostname}
   kubelet:
     image: ${kubelet_image}
     defaultRuntimeSeccompProfileEnabled: true
@@ -120,10 +128,31 @@ cluster:
       - ${san}
 %{ endfor ~}
     image: ${kube_apiserver_img}
-    # No OIDC block. The cluster authenticates with client certificates. An
+    # OIDC authentication against ZITADEL. Additive: client-certificate auth
-    # earlier Authentik-based kube-apiserver OIDC config was removed from the
+    # (this session's own admin@homelab-cluster kubeconfig, system:masters)
-    # live nodes, and leaving it here would re-add an issuer that no longer
+    # keeps working unchanged, so a bad value here degrades OIDC logins rather
-    # resolves on any future machine-config render.
+    # than locking anyone out.
+    #
+    # The issuer has NO trailing slash and NO /application/o/<app>/ suffix.
+    # That suffix belonged to the Authentik scheme; the apiserver compares `iss`
+    # in the id_token by exact string equality, so the stale value made every
+    # OIDC login fail once Authentik stopped running.
+    #
+    # Groups: ZITADEL asserts roles as a nested urn claim that the apiserver
+    # cannot read. The `groupsClaim` action in ZITADEL mirrors the bare role
+    # names into a flat `groups` claim, which is what oidc-groups-claim consumes.
+    # Verified end to end: a PKCE id_token for this client carries
+    # groups=[homelab-admins, ...], email and preferred_username.
+    #
+    # Prefixes keep OIDC identities from colliding with built-in system:* users
+    # and groups. k8s RBAC bindings therefore target `oidc:homelab-admins`.
+    extraArgs:
+      oidc-issuer-url: https://${oidc_issuer_hostname}
+      oidc-client-id: "${oidc_client_id}"
+      oidc-username-claim: email
+      oidc-groups-claim: groups
+      oidc-username-prefix: "oidc:"
+      oidc-groups-prefix: "oidc:"
     admissionControl:
       - name: PodSecurity
         configuration:

terraform/variables.tf

@@ -198,3 +198,42 @@ variable "forgejo_hostname" {
   default     = "forgejo.riotpiao.com"
   description = "Forgejo external hostname"
 }
+
+# ── kube-apiserver OIDC (ZITADEL) ────────────────────────────────────────────
+# Authenticates `kubectl` against ZITADEL instead of client certificates only.
+# Client-certificate auth keeps working in parallel, so a wrong value here
+# degrades OIDC logins without locking anyone out.
+#
+# The issuer is ZITADEL's external host with NO trailing slash and NO
+# /application/o/<app>/ suffix — a leftover of the Authentik scheme. The
+# apiserver validates `iss` in the id_token for exact string equality.
+variable "oidc_issuer_hostname" {
+  type        = string
+  default     = "zitadel.riotpiao.com"
+  description = "Hostname of the OIDC issuer (ZITADEL). Becomes https://<host> as the issuer."
+}
+
+variable "oidc_issuer_ip" {
+  type        = string
+  default     = "192.168.1.160"
+  description = <<-EOT
+    Address the nodes resolve the OIDC issuer to, via extraHostEntries.
+
+    kube-apiserver's static pod runs on the host network, so it resolves the
+    issuer through the node's own DNS, not cluster CoreDNS. Pointing it at the
+    ingress-nginx LoadBalancer keeps OIDC discovery on the LAN and serves the
+    publicly-trusted Let's Encrypt *.riotpiao.com certificate. Without this the
+    lookup leaves via public DNS and Cloudflare, and discovery either hairpins
+    or fails.
+  EOT
+}
+
+variable "oidc_client_id" {
+  type        = string
+  default     = "396757803089330177"
+  description = <<-EOT
+    ZITADEL client id of the `kubernetes` OIDC application (PKCE, no secret).
+    Must be a STRING: the id is an 18-digit number and an unquoted YAML scalar
+    would be parsed as a float and rejected.
+  EOT
+}