fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi

The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:

  Package registry API internal error: 500 write
  /data/tmp/package-upload/<id>: no space left on device

The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.

Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.

This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.

NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:

  disk ... does not have sufficient physical space for expansion:
  physical free space would drop below minimal:
  left=14596177920 < minimal=49931236556

60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.

k8s/apps/cloudflared/configmap.yaml

@@ -53,25 +53,4 @@ data:
       # even though the DNS CNAME points at the tunnel.
       - hostname: zitadel.riotpiao.com
         service: https://zitadel.riotpiao.com
-      # KUBERNETES API, for kubectl from outside the LAN.
-      #
-      # Origin is the in-cluster Service, not the node IP: that survives a
-      # control-plane address change and lets the connectors reach whichever
-      # apiserver is local-healthy.
-      #
-      # noTLSVerify is required. The apiserver serves a certificate for the
-      # cluster CA plus its node/LB addresses, so its SANs cannot match this
-      # public hostname. Cloudflare terminates the public TLS with the
-      # *.riotpiao.com edge certificate, and the connector hop is already
-      # inside the cluster, so skipping verification on that short hop is the
-      # least-bad option. Do not widen this to other rules.
-      #
-      # SECURITY: this exposes the API to the internet. Authentication is still
-      # ZITADEL OIDC plus client certificates, but put a Cloudflare Access
-      # application in front of this hostname as well -- see the note in the
-      # PR. Without Access, every scanner on the internet reaches the apiserver.
-      - hostname: k8s.riotpiao.com
-        service: https://kubernetes.default.svc.cluster.local:443
-        originRequest:
-          noTLSVerify: true
       - service: http_status:404

k8s/apps/comfyui/deployment.yaml

@@ -8,7 +8,18 @@ metadata:
   labels:
     app: comfyui
 spec:
-  replicas: 1
+  # Retired on purpose. This Deployment held the 4th V100 on worker-1 while
+  # serving nothing: its oauth2-proxy roll-out has been stuck for 42h and
+  # failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries
+  # only 1 key while the pod requests client-id, client-secret and
+  # cookie-secret. Every request was therefore refused, so the card sat at 0%
+  # utilisation and blocked a GPU that llm-serving can use.
+  #
+  # Scaling to 0 is the whole change. The Deployment, the Service, the Ingress
+  # and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is
+  # touched. To bring it back: set replicas to 1 here, then fix the Secret to
+  # carry all three keys, otherwise the same stuck roll-out returns.
+  replicas: 0
   strategy:
     type: Recreate
   selector:

k8s/bootstrap/phase3-forgejo/forgejo-values.yaml

@@ -97,10 +97,25 @@ gitea:
       # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
 
 # Persistence (shared storage for repos)
+#
+# 60Gi, not the original 20Gi. The container registry stores its blobs here,
+# and a single candle-vllm sm_70 image is 14GB uncompressed. At 20Gi the volume
+# filled, and every blob upload failed with:
+#   Package registry API internal error: 500 write /data/tmp/package-upload/...:
+#   no space left on device
+# The push client surfaces that as a bare `received unexpected HTTP status: 500`
+# after retrying every layer, which looks like a network fault and is not one.
+#
+# Note for whoever expands this again: Longhorn's admission webhook checks the
+# PHYSICAL free space of the disk holding each replica, not the thin volume
+# size. A request for 100Gi was rejected while the disk had 23.5Gi free:
+#   disk ... does not have sufficient physical space for expansion:
+#   physical free space would drop below minimal: left=14596177920 < minimal=49931236556
+# 60Gi fit. Raise it in steps and watch the webhook message.
 persistence:
   enabled: true
   storageClass: longhorn
-  size: 20Gi
+  size: 60Gi
   accessModes:
     - ReadWriteOnce
 
@@ -134,12 +149,31 @@ ingress:
   # wildcard. The wildcard covers this host.
 
 # Resources
+#
+# The 1000m ceiling throttled Gitea badly. Measured on a running pod:
+#   nr_throttled   189528 of 1169014 CFS periods  (16.2%)
+#   throttled_usec 4424898602 us                   (~74 minutes lost)
+#   actual usage   951m against the 1000m limit
+# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and
+# repository compare endpoints took 3091-3811ms, so runners timed out and
+# retried, which added more load to an already saturated process.
+#
+# The database was never the bottleneck. A real action_runner UPDATE measured
+# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself.
+#
+# requests.cpu rises to 750m because a request below real usage earns the pod
+# little scheduling weight during contention. limits.cpu rises to 3000m
+# because diff computation is bursty and needs room to run without throttling.
+#
+# Caution: this pod runs on talos-cp-3, a control-plane node. 3000m is 25% of
+# that node's 11950m allocatable CPU. If Gitea sustains near this limit, move
+# the workload off the control plane rather than raising the limit again.
 resources:
   requests:
-    cpu: 200m
-    memory: 512Mi
+    cpu: 750m
+    memory: 768Mi
   limits:
-    cpu: 1000m
+    cpu: 3000m
     memory: 2Gi
 
 # Tolerations for control-plane

terraform/main.tf

@@ -39,6 +39,9 @@ resource "local_file" "controlplane_configs" {
     dns_servers            = var.cluster_config.dns_servers
     forgejo_registry_ip    = var.forgejo_registry_ip
     forgejo_hostname       = var.forgejo_hostname
+    oidc_issuer_hostname   = var.oidc_issuer_hostname
+    oidc_issuer_ip         = var.oidc_issuer_ip
+    oidc_client_id         = var.oidc_client_id
     zone                   = each.value.zone
     allow_scheduling       = each.value.allow_scheduling
     coredns_corefile       = file("${path.module}/files/coredns/Corefile")

terraform/templates/controlplane.tftpl

@@ -33,6 +33,14 @@ machine:
       - ip: ${forgejo_registry_ip}
         aliases:
           - ${forgejo_hostname}
+      # kube-apiserver's static pod runs on the host network, so it resolves the
+      # OIDC issuer through THIS node's DNS, not cluster CoreDNS. Pin the issuer
+      # to the ingress LoadBalancer so discovery stays on the LAN and presents
+      # the publicly-trusted Let's Encrypt wildcard certificate. Without this
+      # entry the lookup leaves via public DNS and Cloudflare.
+      - ip: ${oidc_issuer_ip}
+        aliases:
+          - ${oidc_issuer_hostname}
   kubelet:
     image: ${kubelet_image}
     defaultRuntimeSeccompProfileEnabled: true
@@ -120,10 +128,31 @@ cluster:
       - ${san}
 %{ endfor ~}
     image: ${kube_apiserver_img}
-    # No OIDC block. The cluster authenticates with client certificates. An
-    # earlier Authentik-based kube-apiserver OIDC config was removed from the
-    # live nodes, and leaving it here would re-add an issuer that no longer
-    # resolves on any future machine-config render.
+    # OIDC authentication against ZITADEL. Additive: client-certificate auth
+    # (this session's own admin@homelab-cluster kubeconfig, system:masters)
+    # keeps working unchanged, so a bad value here degrades OIDC logins rather
+    # than locking anyone out.
+    #
+    # The issuer has NO trailing slash and NO /application/o/<app>/ suffix.
+    # That suffix belonged to the Authentik scheme; the apiserver compares `iss`
+    # in the id_token by exact string equality, so the stale value made every
+    # OIDC login fail once Authentik stopped running.
+    #
+    # Groups: ZITADEL asserts roles as a nested urn claim that the apiserver
+    # cannot read. The `groupsClaim` action in ZITADEL mirrors the bare role
+    # names into a flat `groups` claim, which is what oidc-groups-claim consumes.
+    # Verified end to end: a PKCE id_token for this client carries
+    # groups=[homelab-admins, ...], email and preferred_username.
+    #
+    # Prefixes keep OIDC identities from colliding with built-in system:* users
+    # and groups. k8s RBAC bindings therefore target `oidc:homelab-admins`.
+    extraArgs:
+      oidc-issuer-url: https://${oidc_issuer_hostname}
+      oidc-client-id: "${oidc_client_id}"
+      oidc-username-claim: email
+      oidc-groups-claim: groups
+      oidc-username-prefix: "oidc:"
+      oidc-groups-prefix: "oidc:"
     admissionControl:
       - name: PodSecurity
         configuration:

terraform/variables.tf

@@ -198,3 +198,42 @@ variable "forgejo_hostname" {
   default     = "forgejo.riotpiao.com"
   description = "Forgejo external hostname"
 }
+
+# ── kube-apiserver OIDC (ZITADEL) ────────────────────────────────────────────
+# Authenticates `kubectl` against ZITADEL instead of client certificates only.
+# Client-certificate auth keeps working in parallel, so a wrong value here
+# degrades OIDC logins without locking anyone out.
+#
+# The issuer is ZITADEL's external host with NO trailing slash and NO
+# /application/o/<app>/ suffix — a leftover of the Authentik scheme. The
+# apiserver validates `iss` in the id_token for exact string equality.
+variable "oidc_issuer_hostname" {
+  type        = string
+  default     = "zitadel.riotpiao.com"
+  description = "Hostname of the OIDC issuer (ZITADEL). Becomes https://<host> as the issuer."
+}
+
+variable "oidc_issuer_ip" {
+  type        = string
+  default     = "192.168.1.160"
+  description = <<-EOT
+    Address the nodes resolve the OIDC issuer to, via extraHostEntries.
+
+    kube-apiserver's static pod runs on the host network, so it resolves the
+    issuer through the node's own DNS, not cluster CoreDNS. Pointing it at the
+    ingress-nginx LoadBalancer keeps OIDC discovery on the LAN and serves the
+    publicly-trusted Let's Encrypt *.riotpiao.com certificate. Without this the
+    lookup leaves via public DNS and Cloudflare, and discovery either hairpins
+    or fails.
+  EOT
+}
+
+variable "oidc_client_id" {
+  type        = string
+  default     = "396757803089330177"
+  description = <<-EOT
+    ZITADEL client id of the `kubernetes` OIDC application (PKCE, no secret).
+    Must be a STRING: the id is an 18-digit number and an unquoted YAML scalar
+    would be parsed as a float and rejected.
+  EOT
+}