Without it the container runs under runc, libcuda.so is never injected and
candle-vllm panics in cudarc at startup. The pod reached Running but crashed
on every restart.
Verified live: pod Ready, coder-predictor endpoints populated, /v1/models
reports coder-14b-fp16 and a completion returns OK.
Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at
fp16 on TP=2.
The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a
dequantisation pass on the SM. Profiling showed SM at 98% with the memory
controller at 19%, and the batch-1 step cost matched a full-model read.
Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s.
The deployment keeps the candle-vllm name and the app: candle-vllm label,
so coder-service.yaml and every client keep working unchanged.
clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds
four V100s, so the coder takes one and clef takes one, leaving two free
for a TP=2 candle run.
The gateway already routes the clef-flash model to clef-serve:80 on
/v1/systemone, so this single replica count is what turns that route on.
Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so
clients keep one stable host name across the vLLM -> candle-vllm handoff and
only this selector changes from now on.
The retired KTML predictor requested two V100s and never reached Ready.
rock