feat(llm-serving): also serve the 30B-A3B as reasoning-long
The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at
64K on one card, the model qwen-finetune wiring and the models.json overlay
target. It was deployed and healthy but only answerable as `reasoning`, so
that name did not exist.
It is a second served name on the existing pod, not a new
InferenceService. All four V100s are allocated (coder 2, this 1,
clef-serve 1), so a separate reasoning-long predictor would sit Pending
forever. It would also load the same 15.56 GiB of weights a second time to
serve a name, which buys nothing.
The two names are separate list entries. The comma form does NOT work here,
and fails silently in a way that looks like success: this image's parser
turns `--served-model-name=reasoning,reasoning-long` into
['reasoning,reasoning-long'], one model whose name contains a comma, and
api_server.py iterates args.served_model_name when it builds the model list.
Verified against vLLM 0.11.0 itself rather than assumed:
FlexibleArgumentParser().parse_args(["--model","m",
"--served-model-name=reasoning,reasoning-long"])
-> ['reasoning,reasoning-long']
FlexibleArgumentParser().parse_args(["--model","m",
"--served-model-name","reasoning","reasoning-long"])
-> ['reasoning', 'reasoning-long']
`reasoning` stays served from this pod on purpose. poimen-memory pins
LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest
outright. The duplicate with coder-predictor is known and is resolved by
moving the client host, not by deleting a name here.
Not verified end to end: a live apply was reverted by ArgoCD selfHeal before
the new pod finished loading, so the served-name list was never observed on
a running pod with both names. The parser result above is what this commit
rests on. Confirm with `GET /v1/models` after the merge syncs.
This commit is contained in:
@@ -30,7 +30,28 @@ spec:
|
||||
# Measured on this cluster, one V100-32GB, 64k context: 15.55 GiB of weights,
|
||||
# 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request.
|
||||
- --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4
|
||||
- --served-model-name=reasoning
|
||||
# Two served names, one pod, on purpose. `reasoning-long` is the name the goal map
|
||||
# uses for this model, and the name qwen-finetune wiring targets. It is NOT a
|
||||
# separate InferenceService: this model already occupies the only free card, and a
|
||||
# second pod on the same weights cannot schedule (4 of 4 GPUs allocated: coder 2,
|
||||
# this 1, clef-serve 1). Serving both names from one process costs nothing and
|
||||
# keeps the two callers working without a second weights copy.
|
||||
#
|
||||
# `reasoning` is deliberately still served here. poimen-memory pins
|
||||
# LLM_CHAT_BASE=reasoning-predictor, so dropping the name here breaks ingest
|
||||
# immediately. coder-predictor also serves `reasoning`; that ambiguity is known
|
||||
# and documented in the PR. Resolve it by moving the client host, not by
|
||||
# removing the name here.
|
||||
#
|
||||
# The two names are separate list entries, NOT one comma-joined value. This
|
||||
# image's parser returns ['reasoning,reasoning-long'] for the comma form, i.e. a
|
||||
# single model whose name contains a comma, and api_server.py then iterates
|
||||
# args.served_model_name to build the model list. Verified against vLLM 0.11.0:
|
||||
# FlexibleArgumentParser().parse_args([... "--served-model-name", "a", "b"])
|
||||
# -> ['a', 'b']
|
||||
- --served-model-name
|
||||
- reasoning
|
||||
- reasoning-long
|
||||
- --quantization=gptq
|
||||
- --dtype=float16
|
||||
- --kv-cache-dtype=auto
|
||||
|
||||
Reference in New Issue
Block a user