chore(llm-serving): park clef-serve to free a V100 for candle-vllm
candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.
This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.
Measured activity over the last 5,000 log lines, since it started on
2026-10-30T06:23:40Z:
4,994 GET /health (kubelet probes)
2 GET /v1/models
1 GET /v1/completions
1 GET /v1/chat/completions
1 GET /docs
1 GET /
No successful inference. The four inference-path requests are GET, and these
routes answer POST, so they cannot have produced a completion. It held a card
for a day to answer liveness probes, which is the same shape as the comfyui
retirement. Nothing in git selects this Service.
Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.
Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick. Apply through
kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the
generated ConfigMap by its hashed name, and a raw apply drops the hash.
This commit is contained in:
@@ -33,7 +33,19 @@ metadata:
|
||||
app: clef-serve
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
replicas: 1
|
||||
# Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
|
||||
# run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
|
||||
# this 1) and a new GPU workload had nowhere to land.
|
||||
#
|
||||
# This is the only GPU consumer in the namespace that is not vLLM, which is why it is
|
||||
# the first card released: the standing instruction is to keep the vLLM pods healthy
|
||||
# until candle-vllm answers, and this pod is not one of them.
|
||||
#
|
||||
# It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
|
||||
# /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
|
||||
# in git selects this service. Restore it once the rollout leaves a card free:
|
||||
# 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
|
||||
replicas: 0
|
||||
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
|
||||
# the new pod Pending forever waiting for a card the old pod still holds.
|
||||
strategy:
|
||||
|
||||
Reference in New Issue
Block a user