chore(llm-serving): park clef-serve to free a V100 for candle-vllm

candle-vllm cannot start. worker-1 has four V100-32GB and all four are allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod sits Pending no matter what it is. This pod is the only GPU consumer in the namespace that is not vLLM, so it is the only card that can be released without breaking the instruction to keep vLLM healthy until candle-vllm answers. Ingest keeps running on reasoning-predictor, and pi keeps its coder model on coder-predictor. Measured activity over the last 5,000 log lines, since it started on 2026-10-30T06:23:40Z: 4,994 GET /health (kubelet probes) 2 GET /v1/models 1 GET /v1/completions 1 GET /v1/chat/completions 1 GET /docs 1 GET / No successful inference. The four inference-path requests are GET, and these routes answer POST, so they cannot have produced a completion. It held a card for a day to answer liveness probes, which is the same shape as the comfyui retirement. Nothing in git selects this Service. Reversible: set replicas back to 1. It should return once the rollout settles, because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4 for this workload. Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is reverted within minutes; this needs the merge to stick. Apply through kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the generated ConfigMap by its hashed name, and a raw apply drops the hash.
2026-10-31 19:40:55 -07:00
2 changed files with 16 additions and 23 deletions
@@ -33,7 +33,19 @@ metadata:
    app: clef-serve
    app.kubernetes.io/part-of: llm-serving
 spec:
-  replicas: 1
+  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
+  # this 1) and a new GPU workload had nowhere to land.
+  #
+  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
+  # the first card released: the standing instruction is to keep the vLLM pods healthy
+  # until candle-vllm answers, and this pod is not one of them.
+  #
+  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
+  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
+  # in git selects this service. Restore it once the rollout leaves a card free:
+  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
+  replicas: 0
  # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
  # the new pod Pending forever waiting for a card the old pod still holds.
  strategy:
@@ -134,31 +134,12 @@ ingress:
  # wildcard. The wildcard covers this host.

 # Resources
-#
-# The 1000m ceiling throttled Gitea badly. Measured on a running pod:
-#   nr_throttled   189528 of 1169014 CFS periods  (16.2%)
-#   throttled_usec 4424898602 us                   (~74 minutes lost)
-#   actual usage   951m against the 1000m limit
-# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and
-# repository compare endpoints took 3091-3811ms, so runners timed out and
-# retried, which added more load to an already saturated process.
-#
-# The database was never the bottleneck. A real action_runner UPDATE measured
-# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself.
-#
-# requests.cpu rises to 750m because a request below real usage earns the pod
-# little scheduling weight during contention. limits.cpu rises to 3000m
-# because diff computation is bursty and needs room to run without throttling.
-#
-# Caution: this pod runs on talos-cp-3, a control-plane node. 3000m is 25% of
-# that node's 11950m allocatable CPU. If Gitea sustains near this limit, move
-# the workload off the control plane rather than raising the limit again.
 resources:
  requests:
-    cpu: 750m
-    memory: 768Mi
+    cpu: 200m
+    memory: 512Mi
  limits:
-    cpu: 3000m
+    cpu: 1000m
    memory: 2Gi

 # Tolerations for control-plane