Files
homelab/k8s/apps/llm-serving
Story Crater Bot f21679f601 reasoning: swap DeepSeek-R1-Distill-32B for Kimi-K2.6-distilled Qwen3.6-35B-A3B
R1-family tool_choice=auto is a documented vLLM architecture conflict --
the model narrates fake tool_calls in <think> instead of emitting real
ones, regardless of parser (deepseek_v3 400s, hermes parses but the model
still doesn't call out). Qwen3's native tool-call format sidesteps this.

No pre-quantized AWQ/GPTQ/bnb checkpoint exists for this specific distill
(only GGUF, llama.cpp/Ollama-only) -- using on-the-fly bitsandbytes
quantization against the full bf16 checkpoint instead.
2026-08-19 18:11:45 -07:00
..