feat(llm-serving): serve Qwen3-30B-A3B int4 as reasoning, on one card
Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context. The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same 64k window fits on one card. Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token prompt, and clean reasoning_content separation through the gateway. Three settings are load-bearing on sm_70 rather than tuning: - VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0 - VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+ - deploymentStrategy Recreate a rolling update deadlocks once every card is allocated The manifest is committed as well as applied: reasoning.yaml had never been in git.
This commit is contained in:
@@ -26,12 +26,10 @@ resources:
|
||||
#
|
||||
# Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is
|
||||
# kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning.
|
||||
# ornith and reasoning are live but unmanaged: their manifests were never committed
|
||||
# to this directory, so listing them made `kustomize build` fail and left the whole
|
||||
# Application in ComparisonError. Remove the entries rather than the models. The
|
||||
# Application runs with prune=false, so an object that is no longer in the desired
|
||||
# state is left alone, not deleted. Re-export both from live state when they are
|
||||
# next edited.
|
||||
- reasoning.yaml
|
||||
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
||||
# replicas, because desired equals available at 0. Its manifest was never committed,
|
||||
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
|
||||
- embeddings.yaml
|
||||
- reranker.yaml
|
||||
- qwen-cpu.yaml
|
||||
|
||||
Reference in New Issue
Block a user