feat(llm-serving): run coder as fp16 14B across two V100s

Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at
fp16 on TP=2.

The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a
dequantisation pass on the SM. Profiling showed SM at 98% with the memory
controller at 19%, and the batch-1 step cost matched a full-model read.
Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s.

The deployment keeps the candle-vllm name and the app: candle-vllm label,
so coder-service.yaml and every client keep working unchanged.
This commit is contained in:
2026-11-05 09:11:46 +08:00
parent db97159905
commit f6fca4705b
2 changed files with 111 additions and 0 deletions
+9
View File
@@ -13,6 +13,15 @@ resources:
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
# coder-14b-fp16.yaml owns the `candle-vllm` Deployment name and the
# `app: candle-vllm` label that coder-service.yaml selects. It must be the only
# workload with that label, or the Service splits traffic across two models.
#
# fp16 on two V100s, not GPTQ on one: measured 39.7 ms ITL and 287 tok/s at 16
# streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
# tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
# the memory controller idles at 19%.
- coder-14b-fp16.yaml
- clef-serve.yaml
# Stable client-facing name for the chat model. poimen-memory points at this Service,
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.