reasoning: halve max-num-seqs to 2 for Kimi swap's first boot
New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves less confirmed KV-cache headroom on the 32GB card than the old one had -- reducing concurrent-sequence worst case until real memory use is verified.
This commit is contained in:
@@ -40,7 +40,13 @@ spec:
|
||||
- --tensor-parallel-size=1
|
||||
- --max-model-len=16384
|
||||
- --gpu-memory-utilization=0.90
|
||||
- --max-num-seqs=4
|
||||
# Halved from 4 -- weights alone (35B total, MoE, all experts loaded
|
||||
# regardless of ~3B active) at on-the-fly bnb 4bit are ~19-21GB on a
|
||||
# 32GB card at 0.90 util (~29.5GB budget), leaving only ~8-10GB for KV
|
||||
# cache + activations. Old DeepSeek-32B was in the same weight
|
||||
# ballpark but MoE bnb overhead here is untested. Starting at 2 to
|
||||
# confirm real headroom via nvidia-smi/vLLM logs before raising back.
|
||||
- --max-num-seqs=2
|
||||
- --enable-chunked-prefill
|
||||
- --enable-prefix-caching
|
||||
# qwen3 parser handles this family's <think> reasoning blocks (best
|
||||
|
||||
Reference in New Issue
Block a user