feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100

The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.

Measured, on worker-1, driver 580.159.04, compute capability 7.0:

GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
POST /v1/chat/completions model=coder -> 200 in 3.18s,
"fn main() { println!(\"hi\"); }"
POST /v1/chat/completions model=reasoning -> 200 in 0.21s

Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.

Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.

Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:

TORCH 2.9.1+cu129
ARCH_LIST [] <- empty
RuntimeError: cudaGetDeviceCount() ... Error 803
fla: ModuleNotFoundError <- no GatedDeltaNet kernels

vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.

Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.

reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.

This commit is contained in:

rock

2026-10-31 15:26:08 -07:00

parent 47c67f9bd9

commit 03e3e8393a

2 changed files with 122 additions and 0 deletions

									
										k8s/apps/llm-serving/kustomization.yaml
									
		+4
		
												View File
												
				@@ -10,8 +10,12 @@ kind: Kustomization

				# way after any edit here: a GPU predictor restart is a weights reload measured

				# in tens of seconds, not a rolling update.

				resources:

				  - coder.yaml

				  - embeddings.yaml

				  - ornith.yaml

				  # reasoning stays listed only until poimen-memory's LLM_CHAT_BASE moves to

				  # coder-predictor. coder already serves the name `reasoning`, so the client

				  # needs no model-name change, only the host change. Drop this line after that.

				  - reasoning.yaml

				  - reranker.yaml

				  - qwen-cpu.yaml