Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100
could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the
weights per card to 7.85 GiB each and frees room for the larger pool.
Measured after boot, both cards worker-1:
Available KV cache memory per GPU : 19.96 GiB
GPU KV cache size : 436,112 tokens
Max concurrency at 262,144 : 1.66x
max_model_len (both served names) : 262144
POST /v1/chat/completions : 200 in 2.83s
Weights, per worker : 7.8516 GiB
torch.compile : 78 s
Graph capture : 6 s
One fatal detail, measured twice. With vLLM's default custom all-reduce the
engine reaches "Graph capturing finished in 6 secs" and then hangs. The only
output afterwards was "No available shared memory broadcast block found in
60 seconds", repeated for 14 minutes, and the front-end never received the
ready message. The first attempt died on its startup probe at 20 min.
`--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own
CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the
fallback and costs nothing here, because TP=2 across two PCIe V100s gains
nothing from the custom kernel. Both workers now register cleanly and the
API binds at 17:18.
Two supporting changes, both required:
- memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the
earlier 24Gi request could never schedule.
- /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default
shows up as an init hang.
`reasoning` remains a served name at the same 262,144 so the client needs no
model-name change.
rock
42617fbd7c