reasoning: fix quantization flag mismatch (compressed-tensors, not awq_marlin)
cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor -- config.json declares compressed-tensors. Passing awq_marlin explicitly conflicted with the checkpoint's own declared format and 400d at config-validation time.
This commit is contained in:
@@ -31,7 +31,12 @@ spec:
|
||||
# on-the-fly bnb gamble a second time.
|
||||
- --model=cpatonn/Qwen3-30B-A3B-Thinking-2507-AWQ-4bit
|
||||
- --served-model-name=reasoning
|
||||
- --quantization=awq_marlin
|
||||
# Named "AWQ" but actually quantized via llm-compressor -- config.json
|
||||
# declares compressed-tensors, not classic AWQ. vLLM auto-detects this
|
||||
# from the checkpoint; passing awq_marlin explicitly mismatches and
|
||||
# 400s at config-validation time. Letting vLLM read it from the
|
||||
# checkpoint instead of asserting the wrong format.
|
||||
- --quantization=compressed-tensors
|
||||
- --dtype=float16
|
||||
- --kv-cache-dtype=auto
|
||||
- --tensor-parallel-size=1
|
||||
|
||||
Reference in New Issue
Block a user