chore(llm-serving): park clef-serve to free a V100 for candle-vllm

candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.

This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.

It was not serving anything. At park time its last 30 log lines were 30 of 30
kubelet /health probes, with no inference request since it started on
2026-10-30T06:23:40Z. Nothing in git selects this Service. It held a card for a
day to answer liveness probes, which is the same shape as the comfyui
retirement.

Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.

Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick.
This commit is contained in:
2026-10-31 19:40:27 -07:00
parent 3afb7b1a65
commit 44fbf071e1
+13 -1
View File
@@ -33,7 +33,19 @@ metadata:
app: clef-serve app: clef-serve
app.kubernetes.io/part-of: llm-serving app.kubernetes.io/part-of: llm-serving
spec: spec:
replicas: 1 # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
# run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
# this 1) and a new GPU workload had nowhere to land.
#
# This is the only GPU consumer in the namespace that is not vLLM, which is why it is
# the first card released: the standing instruction is to keep the vLLM pods healthy
# until candle-vllm answers, and this pod is not one of them.
#
# It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
# /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
# in git selects this service. Restore it once the rollout leaves a card free:
# 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
replicas: 0
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
# the new pod Pending forever waiting for a card the old pod still holds. # the new pod Pending forever waiting for a card the old pod still holds.
strategy: strategy: