fix(llm-serving): use hermes tool-call parser, not deepseek_v3
deepseek_v3 400s on this checkpoint: "could not locate tool call start/end tokens in the tokenizer". unsloth/DeepSeek-R1-Distill-Qwen-32B is a Qwen2.5 base distilled on R1 reasoning traces -- it kept R1's <think> format but never got DeepSeek-V3's own special tool-call tokens registered in its tokenizer. hermes parses from text patterns instead of special tokens, so it works against the underlying Qwen tokenizer.
This commit is contained in:
@@ -26,14 +26,18 @@ spec:
|
||||
- --reasoning-parser=deepseek_r1
|
||||
# Without these, any client sending tool_choice="auto" (pi does, for
|
||||
# Read/Bash/etc.) gets a 400: "auto" tool choice requires
|
||||
# --enable-auto-tool-choice and --tool-call-parser to be set. Verified
|
||||
# against this exact deployed vLLM v0.11.0's registered parsers
|
||||
# (ToolParserManager.tool_parsers) -- deepseek_v3 matches, same family
|
||||
# as the deepseek_r1 reasoning-parser already set above (this model
|
||||
# emits DeepSeek's own special-token format, not Qwen's Hermes-style,
|
||||
# despite being a Qwen-base distillation).
|
||||
# --enable-auto-tool-choice and --tool-call-parser to be set.
|
||||
# deepseek_v3 (matching --reasoning-parser above) 400s here --
|
||||
# "DeepSeek-V3 Tool parser could not locate tool call start/end
|
||||
# tokens in the tokenizer" -- this checkpoint is a Qwen2.5-32B base
|
||||
# distilled on R1 traces, so its tokenizer never got DeepSeek-V3's
|
||||
# own special tool-call tokens registered even though it kept R1's
|
||||
# <think> reasoning format. hermes parses tool calls from plain text
|
||||
# patterns instead of special tokens, so it works against the
|
||||
# underlying Qwen tokenizer regardless. Verified live: deepseek_v3
|
||||
# 400s, hermes returns a real tool_calls response.
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser=deepseek_v3
|
||||
- --tool-call-parser=hermes
|
||||
- --host=0.0.0.0
|
||||
- --port=8080
|
||||
env:
|
||||
|
||||
Reference in New Issue
Block a user