homelab

riotpiao-poimen/homelab

Fork 0

Author	SHA1	Message	Date
rock	d821718375	chore(llm-serving): park clef-serve to free a V100 for candle-vllm candle-vllm cannot start. worker-1 has four V100-32GB and all four are allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod sits Pending no matter what it is. This pod is the only GPU consumer in the namespace that is not vLLM, so it is the only card that can be released without breaking the instruction to keep vLLM healthy until candle-vllm answers. Ingest keeps running on reasoning-predictor, and pi keeps its coder model on coder-predictor. Measured activity over the last 5,000 log lines, since it started on 2026-10-30T06:23:40Z: 4,994 GET /health (kubelet probes) 2 GET /v1/models 1 GET /v1/completions 1 GET /v1/chat/completions 1 GET /docs 1 GET / No successful inference. The four inference-path requests are GET, and these routes answer POST, so they cannot have produced a completion. It held a card for a day to answer liveness probes, which is the same shape as the comfyui retirement. Nothing in git selects this Service. Reversible: set replicas back to 1. It should return once the rollout settles, because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4 for this workload. Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is reverted within minutes; this needs the merge to stick. Apply through kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the generated ConfigMap by its hashed name, and a raw apply drops the hash.	2026-10-31 19:40:55 -07:00

Author

SHA1

Message

Date

rock

d821718375

chore(llm-serving): park clef-serve to free a V100 for candle-vllm

candle-vllm cannot start. worker-1 has four V100-32GB and all four are
allocated: coder 2, reasoning 1, clef-serve 1. Free is 0, so a new GPU pod
sits Pending no matter what it is.

This pod is the only GPU consumer in the namespace that is not vLLM, so it is
the only card that can be released without breaking the instruction to keep
vLLM healthy until candle-vllm answers. Ingest keeps running on
reasoning-predictor, and pi keeps its coder model on coder-predictor.

Measured activity over the last 5,000 log lines, since it started on
2026-10-30T06:23:40Z:

  4,994  GET /health        (kubelet probes)
      2  GET /v1/models
      1  GET /v1/completions
      1  GET /v1/chat/completions
      1  GET /docs
      1  GET /

No successful inference. The four inference-path requests are GET, and these
routes answer POST, so they cannot have produced a completion. It held a card
for a day to answer liveness probes, which is the same shape as the comfyui
retirement. Nothing in git selects this Service.

Reversible: set replicas back to 1. It should return once the rollout settles,
because 30B-A3B takes 1 card and Coder-Next TP=2 takes 2, which leaves card 4
for this workload.

Not applied. ArgoCD runs selfHeal for llm-serving, so a live scale to 0 is
reverted within minutes; this needs the merge to stick. Apply through
kustomize, not `kubectl apply -f clef-serve.yaml`: the Deployment mounts the
generated ConfigMap by its hashed name, and a raw apply drops the hash.

2026-10-31 19:40:55 -07:00

Compare commits

Diff Content Not Available