From ff9f0e99e32e13dce8efd967fb4800ce51bb5b37 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed, 19 Aug 2026 18:20:34 -0700 Subject: [PATCH] reasoning: fall back to official Qwen3-30B-A3B-Thinking-2507 AWQ-4bit Kimi-K2.6-distilled Qwen3.6-35B-A3B crashed on boot -- model type qwen3_5_moe unrecognized by transformers/vLLM 0.11.0, a genuinely unsupported architecture, not a config issue. Using cpatonn's pre-quantized AWQ-4bit build of the official Qwen3-30B-A3B-Thinking-2507 instead: native vLLM support confirmed, no Kimi distillation but Qwen3's own tool-call format is natively supported (the actual root problem being solved). Restored max-num-seqs=4 since AWQ-4bit weight footprint leaves more KV headroom than the bnb attempts did. --- k8s/apps/llm-serving/reasoning.yaml | 69 ++++++++++++----------------- 1 file changed, 29 insertions(+), 40 deletions(-) diff --git a/k8s/apps/llm-serving/reasoning.yaml b/k8s/apps/llm-serving/reasoning.yaml index 859929e..8aab953 100644 --- a/k8s/apps/llm-serving/reasoning.yaml +++ b/k8s/apps/llm-serving/reasoning.yaml @@ -18,61 +18,50 @@ spec: # block instead of emitting real tool_calls, regardless of # parser combo tried (deepseek_v3 400s outright, hermes parsed but the # model itself never called out to the real tool-call path). Root - # cause is upstream in the R1 distillation, not this config -- moving - # to a Qwen3-family model with a Kimi-K2.6 reasoning distillation - # instead, since Qwen3's own tool-call format is natively supported. + # cause is upstream in the R1 distillation, not this config -- Qwen3's + # own tool-call format is natively supported instead. # - # No pre-quantized AWQ/GPTQ/bnb checkpoint exists for this specific - # distilled model (only a GGUF, which is llama.cpp/Ollama-only and not - # usable here) -- pointing --quantization=bitsandbytes at the full - # bf16 checkpoint directly, which makes vLLM quantize on load instead - # of requiring a pre-quantized repo. This on-the-fly bnb path is - # well-trodden for dense models but less battle-tested for MoE - # (this model is 35B total / ~3B active) -- watch first boot closely; - # if it OOMs or errors on the MoE expert weights, that's the likely - # cause. - - --model=lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled + # First attempt was a Kimi-K2.6-distilled Qwen3.6-35B-A3B checkpoint + # (lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled) -- crashed on + # boot: `model type qwen3_5_moe` unrecognized by transformers/vLLM + # 0.11.0, not a config problem, a genuinely unsupported/obscure + # architecture. Falling back to the official Qwen3-30B-A3B-Thinking-2507 + # (no Kimi distillation, but native vLLM support confirmed) using + # cpatonn's pre-quantized AWQ-4bit build -- avoids repeating the + # on-the-fly bnb gamble a second time. + - --model=cpatonn/Qwen3-30B-A3B-Thinking-2507-AWQ-4bit - --served-model-name=reasoning - - --quantization=bitsandbytes - - --trust-remote-code - - --dtype=bfloat16 + - --quantization=awq_marlin + - --dtype=float16 - --kv-cache-dtype=auto - --tensor-parallel-size=1 - --max-model-len=16384 - --gpu-memory-utilization=0.90 - # Halved from 4 -- weights alone (35B total, MoE, all experts loaded - # regardless of ~3B active) at on-the-fly bnb 4bit are ~19-21GB on a - # 32GB card at 0.90 util (~29.5GB budget), leaving only ~8-10GB for KV - # cache + activations. Old DeepSeek-32B was in the same weight - # ballpark but MoE bnb overhead here is untested. Starting at 2 to - # confirm real headroom via nvidia-smi/vLLM logs before raising back. - - --max-num-seqs=2 + # 30B total MoE at pre-quantized AWQ-4bit is ~15-16GB weights on a + # 32GB card at 0.90 util (~29.5GB budget) -- meaningfully more KV-cache + # headroom than the old DeepSeek-32B config had, so restoring the + # original max-num-seqs=4 rather than starting conservative again. + - --max-num-seqs=4 - --enable-chunked-prefill - --enable-prefix-caching - # qwen3 parser handles this family's reasoning blocks (best - # match for this architecture; unverified against this exact - # checkpoint -- if it 400s or fails to strip tags, that's the - # first thing to check). + # qwen3 is vLLM's dedicated reasoning parser for this family's + # blocks -- official pairing per Qwen3-Thinking's own deployment docs. - --reasoning-parser=qwen3 - # hermes previously verified (on the old model) to work against a - # Qwen tokenizer's plain-text tool-call patterns without needing - # special tokens; Qwen3's native tool-call format is also - # hermes-style, so kept as-is. + # hermes is the documented tool-call parser for general Qwen3 models + # (qwen3_coder/qwen3_xml are Coder-variant-only). - --enable-auto-tool-choice - --tool-call-parser=hermes # vLLM 0.11.0's native OffloadingConnector -- spills KV cache blocks # to CPU DRAM instead of discarding them on preemption (max-num-seqs=4 # + max-model-len=16384 means concurrent long sequences compete for # the same GPU KV space). No extra dependency, built into vLLM core. - # num_cpu_blocks=2000 hung the pod at startup ("Allocating 64 CPU - # tensors..." then nothing -- 64 is this model's layer count, one - # pinned host tensor per layer, each sized for every CPU block; 2000 - # was oversized enough to stall pinning that much host memory, likely - # blowing well past the pod's memory limit). Dropped to a small, - # known-safe starting point -- confirm it actually comes up healthy, - # then watch real host memory usage and raise it deliberately rather - # than guessing a round number again. block_size=128 tokens matches - # vLLM's own example. + # num_cpu_blocks=2000 hung the pod at startup on the old model (one + # pinned host tensor per layer, sized per CPU block -- oversized enough + # to stall pinning that much host memory). 32 is a small, known-safe + # starting point independent of this model's own layer count -- + # confirm it comes up healthy, then watch real host memory usage and + # raise it deliberately rather than guessing a round number again. + # block_size=128 tokens matches vLLM's own example. # Note: 0.11.0 ships the original (fragmented, small-transfer-block) # version of this connector -- 0.12.0 consolidates KV data into one # contiguous block per request and is reported an order of magnitude