rock/homelab
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

reasoning: fix quantization flag mismatch (compressed-tensors, not awq_marlin)

Browse Source 
cpatonn's "AWQ-4bit" repo is actually quantized via llm-compressor --
config.json declares compressed-tensors. Passing awq_marlin explicitly
conflicted with the checkpoint's own declared format and 400d at
config-validation time.
This commit is contained in:
Story Crater Bot
2026-08-19 18:23:15 -07:00
parent ff9f0e99e3
commit 5470d55b79
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/apps/llm-serving/reasoning.yaml
+6 -1
View File
@@ -31,7 +31,12 @@ spec:
# on-the-fly bnb gamble a second time. # on-the-fly bnb gamble a second time.
- --model=cpatonn/Qwen3-30B-A3B-Thinking-2507-AWQ-4bit - --model=cpatonn/Qwen3-30B-A3B-Thinking-2507-AWQ-4bit
- --served-model-name=reasoning - --served-model-name=reasoning
- --quantization=awq_marlin # Named "AWQ" but actually quantized via llm-compressor -- config.json
# declares compressed-tensors, not classic AWQ. vLLM auto-detects this
# from the checkpoint; passing awq_marlin explicitly mismatches and
# 400s at config-validation time. Letting vLLM read it from the
# checkpoint instead of asserting the wrong format.
- --quantization=compressed-tensors
- --dtype=float16 - --dtype=float16
- --kv-cache-dtype=auto - --kv-cache-dtype=auto
- --tensor-parallel-size=1 - --tensor-parallel-size=1