reasoning: halve max-num-seqs to 2 for Kimi swap's first boot

New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves
less confirmed KV-cache headroom on the 32GB card than the old one had --
reducing concurrent-sequence worst case until real memory use is verified.
This commit is contained in:
Story Crater Bot
2026-08-19 18:14:22 -07:00
parent f21679f601
commit d42ee34bd5
+7 -1
View File
@@ -40,7 +40,13 @@ spec:
- --tensor-parallel-size=1 - --tensor-parallel-size=1
- --max-model-len=16384 - --max-model-len=16384
- --gpu-memory-utilization=0.90 - --gpu-memory-utilization=0.90
- --max-num-seqs=4 # Halved from 4 -- weights alone (35B total, MoE, all experts loaded
# regardless of ~3B active) at on-the-fly bnb 4bit are ~19-21GB on a
# 32GB card at 0.90 util (~29.5GB budget), leaving only ~8-10GB for KV
# cache + activations. Old DeepSeek-32B was in the same weight
# ballpark but MoE bnb overhead here is untested. Starting at 2 to
# confirm real headroom via nvidia-smi/vLLM logs before raising back.
- --max-num-seqs=2
- --enable-chunked-prefill - --enable-chunked-prefill
- --enable-prefix-caching - --enable-prefix-caching
# qwen3 parser handles this family's <think> reasoning blocks (best # qwen3 parser handles this family's <think> reasoning blocks (best