feat(llm-serving): run coder as fp16 14B across two V100s #108

2026-11-05T01:12:02Z

rock commented

2026-11-05 01:12:02 +00:00

Replaces the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at fp16 on TP=2.

Why

The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit checkpoint ran a dequantisation pass on the SM. Profiling: SM 98%, memory controller 19%, and the batch-1 step cost matched a full-model read.

Measured, same harness, 16 streams

GPTQ 30B-A3B TP=1: ITL 71.5 ms, 215 tok/s aggregate
fp16 14B TP=2: ITL 39.7 ms, 287 tok/s aggregate

Single-stream latency still favours the MoE (17.6 vs 27.9 ms) because it activates ~3B params per token.

TP=2 notes

No code change needed: heads 40->20, KV heads 8->4, intermediate 13824->6912, vocab 152064->76032.
The earlier TP=2 crash was GPTQ-specific (g_idx sharding in linear.rs), not TP itself.
No NVLink on these PCIE cards, so all-reduce crosses PCIe and shows in TTFT.
KV is 11.6 GB per rank, about 59K context.

Wiring

Keeps the candle-vllm name and app: candle-vllm label so coder-service.yaml and clients are unchanged.

Before merge

Gateway model id changes from a content hash to coder-14b-fp16.
28 GB of weights must be present at /mnt/models/coder-14b-fp16.
Live test Deployment coder14b-tp2 must be deleted, or two workloads claim the same label.

Replaces the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at fp16 on TP=2. ## Why The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit checkpoint ran a dequantisation pass on the SM. Profiling: SM 98%, memory controller 19%, and the batch-1 step cost matched a full-model read. ## Measured, same harness, 16 streams - GPTQ 30B-A3B TP=1: ITL 71.5 ms, 215 tok/s aggregate - fp16 14B TP=2: ITL 39.7 ms, 287 tok/s aggregate Single-stream latency still favours the MoE (17.6 vs 27.9 ms) because it activates ~3B params per token. ## TP=2 notes - No code change needed: heads 40->20, KV heads 8->4, intermediate 13824->6912, vocab 152064->76032. - The earlier TP=2 crash was GPTQ-specific (g_idx sharding in linear.rs), not TP itself. - No NVLink on these PCIE cards, so all-reduce crosses PCIe and shows in TTFT. - KV is 11.6 GB per rank, about 59K context. ## Wiring Keeps the candle-vllm name and app: candle-vllm label so coder-service.yaml and clients are unchanged. ## Before merge - Gateway model id changes from a content hash to coder-14b-fp16. - 28 GB of weights must be present at /mnt/models/coder-14b-fp16. - Live test Deployment coder14b-tp2 must be deleted, or two workloads claim the same label.

rock added 3 commits 2026-11-05 01:12:04 +00:00

k8s(llm-serving): client-facing Service for candle-vllm coder 67760720fc

Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so
clients keep one stable host name across the vLLM -> candle-vllm handoff and
only this selector changes from now on.

The retired KTML predictor requested two V100s and never reached Ready.

feat(llm-serving): run clef-serve so /v1/systemone answers db97159905

clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds
four V100s, so the coder takes one and clef takes one, leaving two free
for a TP=2 candle run.

The gateway already routes the clef-flash model to clef-serve:80 on
/v1/systemone, so this single replica count is what turns that route on.

feat(llm-serving): run coder as fp16 14B across two V100s f6fca4705b

Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at
fp16 on TP=2.

The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a
dequantisation pass on the SM. Profiling showed SM at 98% with the memory
controller at 19%, and the batch-1 step cost matched a full-model read.
Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s.

The deployment keeps the candle-vllm name and the app: candle-vllm label,
so coder-service.yaml and every client keep working unchanged.

rock added 1 commit 2026-11-05 01:22:54 +00:00

fix(llm-serving): set runtimeClassName nvidia on the candle deployment 7bdc8b2de2

Without it the container runs under runc, libcuda.so is never injected and
candle-vllm panics in cudarc at startup. The pod reached Running but crashed
on every restart.

Verified live: pod Ready, coder-predictor endpoints populated, /v1/models
reports coder-14b-fp16 and a completion returns OK.

rock merged commit 32f1d50e5e into main

2026-11-05 01:40:25 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#108