cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor --
config.json declares compressed-tensors. Passing awq_marlin explicitly
conflicted with the checkpoint's own declared format and 400d at
config-validation time.
Kimi-K2.6-distilled Qwen3.6-35B-A3B crashed on boot -- model type
qwen3_5_moe unrecognized by transformers/vLLM 0.11.0, a genuinely
unsupported architecture, not a config issue. Using cpatonn's pre-quantized
AWQ-4bit build of the official Qwen3-30B-A3B-Thinking-2507 instead: native
vLLM support confirmed, no Kimi distillation but Qwen3's own tool-call
format is natively supported (the actual root problem being solved).
Restored max-num-seqs=4 since AWQ-4bit weight footprint leaves more KV
headroom than the bnb attempts did.
New model's weight footprint (35B total MoE at on-the-fly bnb-4bit) leaves
less confirmed KV-cache headroom on the 32GB card than the old one had --
reducing concurrent-sequence worst case until real memory use is verified.
R1-family tool_choice=auto is a documented vLLM architecture conflict --
the model narrates fake tool_calls in <think> instead of emitting real
ones, regardless of parser (deepseek_v3 400s, hermes parses but the model
still doesn't call out). Qwen3's native tool-call format sidesteps this.
No pre-quantized AWQ/GPTQ/bnb checkpoint exists for this specific distill
(only GGUF, llama.cpp/Ollama-only) -- using on-the-fly bitsandbytes
quantization against the full bf16 checkpoint instead.
vLLM 0.11.0's native OffloadingConnector -- spills KV blocks to CPU RAM on
preemption instead of discarding them, avoiding recompute. Built into vLLM
core, no extra dependency. Bumped memory request/limit (+4Gi/replica) to
give the CPU block pool real room; worker-1 had ~18Gi of request headroom
across both replicas.
deepseek_v3 400s on this checkpoint: "could not locate tool call start/end tokens in the tokenizer". unsloth/DeepSeek-R1-Distill-Qwen-32B is a Qwen2.5 base distilled on R1 reasoning traces -- it kept R1's <think> format but never got DeepSeek-V3's own special tool-call tokens registered in its tokenizer. hermes parses from text patterns instead of special tokens, so it works against the underlying Qwen tokenizer.
pi sends tool_choice="auto" for every session (Read/Bash/etc.) -- vLLM 400s on that without --enable-auto-tool-choice and a --tool-call-parser. Verified this deployed vLLM v0.11.0's registered parsers directly; deepseek_v3 matches, same family as the deepseek_r1 reasoning-parser already set (this Qwen-base distillation still emits DeepSeek's own tool-call format).
reasoning keeps its 2 GPUs untouched. verifier's freed GPU goes to a second ornith replica instead of a standalone qwen-only pod -- both replicas load ornith:35b + qwen2.5:3b-instruct, k8s Service load-balances across them, so 2 concurrent implementer-style calls get independent instances.
Frees verifier's GPU from an underused vLLM PRM deployment. qwen2.5:3b-instruct moves off ornith-predictor's shared pod onto its own dedicated GPU (grm.yaml), so verification/judge traffic stops contending with ornith:35b's agent traffic. /v1/qwen/chat/completions now points at grm-predictor; path unchanged.
Story Crater Bot