reasoning: fix quantization flag mismatch (compressed-tensors, not awq_marlin)

cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor --
config.json declares compressed-tensors. Passing awq_marlin explicitly
conflicted with the checkpoint's own declared format and 400d at
config-validation time.
This commit is contained in:
Story Crater Bot
2026-08-19 18:23:15 -07:00
parent ff9f0e99e3
commit 5470d55b79
+6 -1
View File
@@ -31,7 +31,12 @@ spec:
# on-the-fly bnb gamble a second time.
- --model=cpatonn/Qwen3-30B-A3B-Thinking-2507-AWQ-4bit
- --served-model-name=reasoning
- --quantization=awq_marlin
# Named "AWQ" but actually quantized via llm-compressor -- config.json
# declares compressed-tensors, not classic AWQ. vLLM auto-detects this
# from the checkpoint; passing awq_marlin explicitly mismatches and
# 400s at config-validation time. Letting vLLM read it from the
# checkpoint instead of asserting the wrong format.
- --quantization=compressed-tensors
- --dtype=float16
- --kv-cache-dtype=auto
- --tensor-parallel-size=1