reasoning: swap to dense Qwen3-32B-bnb-4bit for reliable tool calling

DeepSeek-R1-distill's tool_choice=auto narration bug needed a real fix,
not a workaround -- Qwen3's native tool-call format (hermes-compatible
chat template) solves it at the source instead of parsing around it.
Dense Qwen3-32B avoids the MoE arch/quantization pitfalls hit by the two
prior swap attempts (Kimi-distilled Qwen3.6 MoE, AWQ Qwen3-30B-A3B) --
same bnb-4bit path already proven working on this sm70 (V100) node.
This commit is contained in:
Story Crater Bot
2026-08-19 18:40:15 -07:00
parent 9ae0b90d4c
commit ea0c00f76e
+17 -14
View File
@@ -12,7 +12,18 @@ spec:
predictor:
containers:
- args:
- --model=unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit
# DeepSeek-R1-Distill-32B retired: tool_choice="auto" (what pi sends)
# hit a documented vLLM/R1-family conflict -- the model narrated fake
# tool_calls in its <think> block instead of emitting real ones,
# regardless of parser. Tried swapping to a Kimi-distilled Qwen3.6
# MoE checkpoint and an AWQ-quantized Qwen3-30B-A3B first -- both
# failed on real, separate blockers (unrecognized model_type; then
# marlin INT4 kernels needing compute capability 80+, but worker-1's
# GPU is sm70/V100). Landed on dense Qwen3-32B instead: native Qwen3
# tool-call format (no narration bug), bnb-4bit works fine on sm70
# (proven by the old DeepSeek config already), and no MoE
# arch/quantization risk this time.
- --model=unsloth/Qwen3-32B-bnb-4bit
- --served-model-name=reasoning
- --quantization=bitsandbytes
- --dtype=float16
@@ -23,19 +34,11 @@ spec:
- --max-num-seqs=4
- --enable-chunked-prefill
- --enable-prefix-caching
- --reasoning-parser=deepseek_r1
# Without these, any client sending tool_choice="auto" (pi does, for
# Read/Bash/etc.) gets a 400: "auto" tool choice requires
# --enable-auto-tool-choice and --tool-call-parser to be set.
# deepseek_v3 (matching --reasoning-parser above) 400s here --
# "DeepSeek-V3 Tool parser could not locate tool call start/end
# tokens in the tokenizer" -- this checkpoint is a Qwen2.5-32B base
# distilled on R1 traces, so its tokenizer never got DeepSeek-V3's
# own special tool-call tokens registered even though it kept R1's
# <think> reasoning format. hermes parses tool calls from plain text
# patterns instead of special tokens, so it works against the
# underlying Qwen tokenizer regardless. Verified live: deepseek_v3
# 400s, hermes returns a real tool_calls response.
# qwen3 is vLLM's dedicated reasoning parser for this family's <think>
# blocks.
- --reasoning-parser=qwen3
# hermes is the documented tool-call parser for general (non-Coder)
# Qwen3 models -- native chat template support, not narrated text.
- --enable-auto-tool-choice
- --tool-call-parser=hermes
# vLLM 0.11.0's native OffloadingConnector -- spills KV cache blocks