rock
7bdc8b2de2
Without it the container runs under runc, libcuda.so is never injected and candle-vllm panics in cudarc at startup. The pod reached Running but crashed on every restart. Verified live: pod Ready, coder-predictor endpoints populated, /v1/models reports coder-14b-fp16 and a completion returns OK.
107 lines
4.3 KiB
YAML
107 lines
4.3 KiB
YAML
# Qwen2.5-Coder-14B-Instruct, fp16, tensor-parallel across two V100s.
|
|
#
|
|
# Why this model and this precision. The previous workload was Qwen3-Coder-30B-A3B
|
|
# in GPTQ W4A16 on one V100. Profiling that deployment showed SM utilisation at
|
|
# 98% while the memory controller sat at 19%: the V100 has FP16 tensor cores and
|
|
# no INT4 ones, so every decode step ran a dequantisation pass on the SM. That is
|
|
# a hardware-generation mismatch, not a tunable. Sequence throughput matches the
|
|
# arithmetic: at batch 1 the step cost equalled a full-model read (17.6 ms against
|
|
# 15 GB / 900 GB/s = 16.7 ms) while the routed experts alone are ~1.5 GB.
|
|
#
|
|
# Measured on this hardware, same harness, same 16-stream load:
|
|
#
|
|
# GPTQ W4A16 30B-A3B, TP=1 : ITL 71.5 ms, 215 tok/s aggregate
|
|
# fp16 14B, TP=2 : ITL 39.7 ms, 287 tok/s aggregate
|
|
#
|
|
# The dense model degrades only 1.4x from 1 to 16 streams (27.9 -> 39.7 ms) while
|
|
# the MoE degrades 4.1x (17.6 -> 71.5 ms), because a dense model reads its weights
|
|
# once per step and amortises across the batch. The MoE pays expert weight traffic
|
|
# per sequence. Single-stream latency is the one case where the MoE still wins
|
|
# (17.6 ms against 27.9 ms), since it activates ~3B parameters per token.
|
|
#
|
|
# TP=2 notes that cost time to establish, so they are recorded here:
|
|
# - Sharding needs no code change. Every dimension divides by 2: attention heads
|
|
# 40 -> 20, KV heads 8 -> 4, intermediate_size 13824 -> 6912, vocab
|
|
# 152064 -> 76032. Verify this first for any replacement model.
|
|
# - TP=2 on a GPTQ checkpoint is what crashed (CUDA_ERROR_ILLEGAL_ADDRESS in
|
|
# forward()) on the previous model. The cause was quantised weight sharding
|
|
# and the g_idx guard in src/openai/models/linear.rs, not TP itself. A dense
|
|
# fp16 checkpoint has no g_idx, no packed weights and no scales, so it takes
|
|
# the plain column/row-parallel path and the crash does not reproduce.
|
|
# - These are V100-PCIE cards with no NVLink, so every layer's all-reduce
|
|
# crosses PCIe. It shows up in TTFT (275 ms at 1 stream, 1090 ms at 16)
|
|
# more than in ITL.
|
|
# - KV capacity is 11635 MB per rank, about 59K context tokens. The fp16
|
|
# weights consume the room the KV pool would otherwise use. This is well
|
|
# short of the 256K the previous model targeted, and it is the price of
|
|
# leaving the dequantisation path.
|
|
#
|
|
# Weights must be in the PVC as /mnt/models/coder-14b-fp16 (28 GB, 6 shards).
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: candle-vllm
|
|
namespace: llm-serving
|
|
labels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
app: candle-vllm
|
|
spec:
|
|
replicas: 1
|
|
strategy:
|
|
# One replica holds both GPUs, so a rolling update would need a second pair.
|
|
# Recreate keeps the rollout to a single weights load.
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app: candle-vllm
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
app: candle-vllm
|
|
spec:
|
|
# Required. Without this the container runs under runc, libcuda.so is never
|
|
# injected, and candle-vllm panics in cudarc at startup with
|
|
# "Unable to dynamically load the cuda shared library".
|
|
runtimeClassName: nvidia
|
|
containers:
|
|
- name: candle
|
|
image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70-gptq-slim
|
|
command: ["/usr/local/bin/candle-vllm"]
|
|
# fp16, not GPTQ, and two ranks, not one. See the header for the
|
|
# measurements and the TP=2 findings.
|
|
args:
|
|
- --w
|
|
- /mnt/models/coder-14b-fp16
|
|
- --d
|
|
- 0,1
|
|
- --dtype
|
|
- f16
|
|
- --p
|
|
- "8080"
|
|
- --max-num-seqs
|
|
- "32"
|
|
# 11.6 GB of KV per rank, about 59K context tokens. The fp16 weights
|
|
# leave no room for more.
|
|
- --kv-fraction
|
|
- "0.85"
|
|
ports:
|
|
- containerPort: 8080
|
|
resources:
|
|
limits:
|
|
cpu: "8"
|
|
memory: 36Gi
|
|
nvidia.com/gpu: "2"
|
|
requests:
|
|
cpu: "4"
|
|
memory: 12Gi
|
|
nvidia.com/gpu: "2"
|
|
volumeMounts:
|
|
- name: models
|
|
mountPath: /mnt/models
|
|
readOnly: true
|
|
volumes:
|
|
- name: models
|
|
persistentVolumeClaim:
|
|
claimName: llm-models
|