chore(llm-serving): park clef-serve to free a V100 for candle-vllm

candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.

This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.

Measured activity over the last 5,000 log lines, since it started on
2026-10-30T06:23:40Z:

  4,994  GET /health        (kubelet probes)
      2  GET /v1/models
      1  GET /v1/completions
      1  GET /v1/chat/completions
      1  GET /docs
      1  GET /

No successful inference. The four inference-path requests are GET, and these
routes answer POST, so they cannot have produced a completion. It held a card
for a day to answer liveness probes, which is the same shape as the comfyui
retirement. Nothing in git selects this Service.

Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.

Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick. Apply through
kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the
generated ConfigMap by its hashed name, and a raw apply drops the hash.

k8s/apps/llm-serving/clef-serve.yaml

@@ -33,7 +33,19 @@ metadata:
     app: clef-serve
     app.kubernetes.io/part-of: llm-serving
 spec:
-  replicas: 1
+  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
+  # this 1) and a new GPU workload had nowhere to land.
+  #
+  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
+  # the first card released: the standing instruction is to keep the vLLM pods healthy
+  # until candle-vllm answers, and this pod is not one of them.
+  #
+  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
+  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
+  # in git selects this service. Restore it once the rollout leaves a card free:
+  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
+  replicas: 0
   # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
   # the new pod Pending forever waiting for a card the old pod still holds.
   strategy:

k8s/bootstrap/phase3-forgejo/forgejo-values.yaml

@@ -97,25 +97,10 @@ gitea:
       # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
 
 # Persistence (shared storage for repos)
-#
-# 60Gi, not the original 20Gi. The container registry stores its blobs here,
-# and a single candle-vllm sm_70 image is 14GB uncompressed. At 20Gi the volume
-# filled, and every blob upload failed with:
-#   Package registry API internal error: 500 write /data/tmp/package-upload/...:
-#   no space left on device
-# The push client surfaces that as a bare `received unexpected HTTP status: 500`
-# after retrying every layer, which looks like a network fault and is not one.
-#
-# Note for whoever expands this again: Longhorn's admission webhook checks the
-# PHYSICAL free space of the disk holding each replica, not the thin volume
-# size. A request for 100Gi was rejected while the disk had 23.5Gi free:
-#   disk ... does not have sufficient physical space for expansion:
-#   physical free space would drop below minimal: left=14596177920 < minimal=49931236556
-# 60Gi fit. Raise it in steps and watch the webhook message.
 persistence:
   enabled: true
   storageClass: longhorn
-  size: 60Gi
+  size: 20Gi
   accessModes:
     - ReadWriteOnce
 
@@ -149,29 +134,10 @@ ingress:
   # wildcard. The wildcard covers this host.
 
 # Resources
-#
-# The 1000m ceiling throttled Gitea badly. Measured on a running pod:
-#   nr_throttled   189528 of 1169014 CFS periods  (16.2%)
-#   throttled_usec 4424898602 us                   (~74 minutes lost)
-#   actual usage   951m against the 1000m limit
-# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and
-# repository compare endpoints took 3091-3811ms, so runners timed out and
-# retried, which added more load to an already saturated process.
-#
-# The database was never the bottleneck. A real action_runner UPDATE measured
-# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself.
-#
-# requests.cpu stays 200m and limits.cpu stays 1000m. The 3000m limit was a
-# temporary measure while the Actions stall was diagnosed; the real fix landed
-# elsewhere, so the limit is back to the original.
-#
-# Caution: this pod runs on talos-cp-3, a control-plane node. If Gitea
-# throttles again at 1000m, move the workload off the control plane rather
-# than raising the limit.
 resources:
   requests:
     cpu: 200m
-    memory: 768Mi
+    memory: 512Mi
   limits:
     cpu: 1000m
     memory: 2Gi

k8s/infra/longhorn/disk-scan-job.yaml

@@ -1,90 +0,0 @@
-# One-shot destructive health scan of a candidate Longhorn disk.
-#
-# WHY THIS IS A MANUAL JOB AND NOT AN ARGOCD HOOK
-#   It writes over the entire raw device. An ArgoCD PostSync hook would run it
-#   on every sync and destroy whatever the volume holds. Apply it by hand,
-#   once, on a disk that has no partitions and no mounts.
-#
-# WHY NOT TALOS
-#   Talos declares disks by mounting them (machine.disks in the machine
-#   config). It has no primitive for "run this command on the host", so a scan
-#   cannot live in the Talos config. This Job is the IaC-correct home.
-#
-# ORDER OF OPERATIONS
-#   1. Scan the raw device with this Job        <- the disk must be UNMOUNTED
-#   2. If it passes, add the disk to terraform.tfvars.local `longhorn_disks`
-#      with device /dev/disk/by-id/<wwid> and run `make apply-cp`
-#   3. Register it with Longhorn via the pattern in longhorn-add-disks-job.yaml
-#
-# What it does
-#   fio writes a CRC32C-verified pattern across the whole device, then reads
-#   every block back and compares. Any unreadable or mismatched sector fails
-#   the Job. This is a full write and read pass, not a sample.
-#
-# Expected duration
-#   2 TB on a 5400 rpm 2.5" drive at roughly 90 MB/s is about 6 h to write and
-#   6 h to verify. Budget 12 hours. Check progress with:
-#     kubectl -n llm-io logs job/disk-scan-cp1-sdb -f
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: disk-scan-cp1-sdb
-  namespace: llm-io
-  labels:
-    app: disk-scan
-spec:
-  backoffLimit: 0
-  # Do not let a stuck scan retry silently.
-  template:
-    metadata:
-      labels:
-        app: disk-scan
-    spec:
-      restartPolicy: Never
-      nodeSelector:
-        kubernetes.io/hostname: talos-cp-1
-      containers:
-        - name: scan
-          image: ubuntu:22.04
-          securityContext:
-            privileged: true
-          command:
-            - /bin/bash
-            - -c
-            - |
-              set -euo pipefail
-              apt-get update -qq
-              apt-get install -y -qq smartmontools fio
-
-              echo "=== device identity ==="
-              smartctl -i /dev/sdb | tail -12
-
-              echo "=== SMART before ==="
-              smartctl -H /dev/sdb | tail -2
-
-              echo "=== full write + verify (destructive) ==="
-              fio --name=devverify \
-                  --filename=/dev/sdb \
-                  --rw=write \
-                  --bs=1M \
-                  --iodepth=4 \
-                  --ioengine=libaio \
-                  --direct=1 \
-                  --verify=crc32c \
-                  --do_verify=1 \
-                  --verify_fatal=1 \
-                  --eta=always \
-                  --output-format=normal
-
-              echo "=== SMART after ==="
-              smartctl -H /dev/sdb | tail -2
-              smartctl -A /dev/sdb | awk '/Reallocated_Sector|Current_Pending|Offline_Uncorrect|UDMA_CRC/{print "  "$0}'
-
-              echo "=== SCAN PASSED ==="
-          volumeMounts:
-            - name: dev
-              mountPath: /dev
-      volumes:
-        - name: dev
-          hostPath:
-            path: /dev