feat(llm-serving): serve Qwen3-30B-A3B int4 as reasoning, on one card

Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context.
The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same
64k window fits on one card.

Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV
cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token
prompt, and clean reasoning_content separation through the gateway.

Three settings are load-bearing on sm_70 rather than tuning:
  - VLLM_USE_FLASHINFER_SAMPLER=0  flashinfer's check_cuda_arch raises on CC 7.0
  - VLLM_ATTENTION_BACKEND=TRITON_ATTN  FlashInfer and FA2 both need sm75+
  - deploymentStrategy Recreate  a rolling update deadlocks once every card is allocated

The manifest is committed as well as applied: reasoning.yaml had never been in git.
This commit is contained in:
2026-10-29 03:56:17 -07:00
parent ad2720a724
commit 3eb6a91311
2 changed files with 106 additions and 6 deletions
+4 -6
View File
@@ -26,12 +26,10 @@ resources:
#
# Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is
# kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning.
# ornith and reasoning are live but unmanaged: their manifests were never committed
# to this directory, so listing them made `kustomize build` fail and left the whole
# Application in ComparisonError. Remove the entries rather than the models. The
# Application runs with prune=false, so an object that is no longer in the desired
# state is left alone, not deleted. Re-export both from live state when they are
# next edited.
- reasoning.yaml
# ornith is retired: its InferenceService still reports Ready=True while running zero
# replicas, because desired equals available at 0. Its manifest was never committed,
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
- embeddings.yaml
- reranker.yaml
- qwen-cpu.yaml