Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100
could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the
weights per card to 7.85 GiB each and frees room for the larger pool.
Measured after boot, both cards worker-1:
Available KV cache memory per GPU : 19.96 GiB
GPU KV cache size : 436,112 tokens
Max concurrency at 262,144 : 1.66x
max_model_len (both served names) : 262144
POST /v1/chat/completions : 200 in 2.83s
Weights, per worker : 7.8516 GiB
torch.compile : 78 s
Graph capture : 6 s
One fatal detail, measured twice. With vLLM's default custom all-reduce the
engine reaches "Graph capturing finished in 6 secs" and then hangs. The only
output afterwards was "No available shared memory broadcast block found in
60 seconds", repeated for 14 minutes, and the front-end never received the
ready message. The first attempt died on its startup probe at 20 min.
`--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own
CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the
fallback and costs nothing here, because TP=2 across two PCIe V100s gains
nothing from the custom kernel. Both workers now register cleanly and the
API binds at 17:18.
Two supporting changes, both required:
- memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the
earlier 24Gi request could never schedule.
- /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default
shows up as an init hang.
`reasoning` remains a served name at the same 262,144 so the client needs no
model-name change.
The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.
Measured, on worker-1, driver 580.159.04, compute capability 7.0:
GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
POST /v1/chat/completions model=coder -> 200 in 3.18s,
"fn main() { println!(\"hi\"); }"
POST /v1/chat/completions model=reasoning -> 200 in 0.21s
Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.
Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.
Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:
TORCH 2.9.1+cu129
ARCH_LIST [] <- empty
RuntimeError: cudaGetDeviceCount() ... Error 803
fla: ModuleNotFoundError <- no GatedDeltaNet kernels
vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.
Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.
reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.
rock