feat(llm-serving): run coder as fp16 14B across two V100s
Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at fp16 on TP=2. The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a dequantisation pass on the SM. Profiling showed SM at 98% with the memory controller at 19%, and the batch-1 step cost matched a full-model read. Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s. The deployment keeps the candle-vllm name and the app: candle-vllm label, so coder-service.yaml and every client keep working unchanged.
This commit is contained in:
@@ -13,6 +13,15 @@ resources:
|
||||
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
|
||||
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
|
||||
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
|
||||
# coder-14b-fp16.yaml owns the `candle-vllm` Deployment name and the
|
||||
# `app: candle-vllm` label that coder-service.yaml selects. It must be the only
|
||||
# workload with that label, or the Service splits traffic across two models.
|
||||
#
|
||||
# fp16 on two V100s, not GPTQ on one: measured 39.7 ms ITL and 287 tok/s at 16
|
||||
# streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
|
||||
# tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
|
||||
# the memory controller idles at 19%.
|
||||
- coder-14b-fp16.yaml
|
||||
- clef-serve.yaml
|
||||
# Stable client-facing name for the chat model. poimen-memory points at this Service,
|
||||
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
|
||||
|
||||
Reference in New Issue
Block a user