From 871bb8bfd46cc282269348c5f8b3e901fd650b0c Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 19:38:26 -0700 Subject: [PATCH] feat(llm-serving): also serve the 30B-A3B as reasoning-long The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at 64K on one card, the model qwen-finetune wiring and the models.json overlay target. It was deployed and healthy but only answerable as `reasoning`, so that name did not exist. It is a second served name on the existing pod, not a new InferenceService. All four V100s are allocated (coder 2, this 1, clef-serve 1), so a separate reasoning-long predictor would sit Pending forever. It would also load the same 15.56 GiB of weights a second time to serve a name, which buys nothing. The two names are separate list entries. The comma form does NOT work here, and fails silently in a way that looks like success: this image's parser turns `--served-model-name=reasoning,reasoning-long` into ['reasoning,reasoning-long'], one model whose name contains a comma, and api_server.py iterates args.served_model_name when it builds the model list. Verified against vLLM 0.11.0 itself rather than assumed: FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name=reasoning,reasoning-long"]) -> ['reasoning,reasoning-long'] FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name","reasoning","reasoning-long"]) -> ['reasoning', 'reasoning-long'] `reasoning` stays served from this pod on purpose. poimen-memory pins LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest outright. The duplicate with coder-predictor is known and is resolved by moving the client host, not by deleting a name here. Not verified end to end: a live apply was reverted by ArgoCD selfHeal before the new pod finished loading, so the served-name list was never observed on a running pod with both names. The parser result above is what this commit rests on. Confirm with `GET /v1/models` after the merge syncs. --- k8s/apps/llm-serving/reasoning.yaml | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/k8s/apps/llm-serving/reasoning.yaml b/k8s/apps/llm-serving/reasoning.yaml index 77f5203..de87548 100644 --- a/k8s/apps/llm-serving/reasoning.yaml +++ b/k8s/apps/llm-serving/reasoning.yaml @@ -30,7 +30,28 @@ spec: # Measured on this cluster, one V100-32GB, 64k context: 15.55 GiB of weights, # 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request. - --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4 - - --served-model-name=reasoning + # Two served names, one pod, on purpose. `reasoning-long` is the name the goal map + # uses for this model, and the name qwen-finetune wiring targets. It is NOT a + # separate InferenceService: this model already occupies the only free card, and a + # second pod on the same weights cannot schedule (4 of 4 GPUs allocated: coder 2, + # this 1, clef-serve 1). Serving both names from one process costs nothing and + # keeps the two callers working without a second weights copy. + # + # `reasoning` is deliberately still served here. poimen-memory pins + # LLM_CHAT_BASE=reasoning-predictor, so dropping the name here breaks ingest + # immediately. coder-predictor also serves `reasoning`; that ambiguity is known + # and documented in the PR. Resolve it by moving the client host, not by + # removing the name here. + # + # The two names are separate list entries, NOT one comma-joined value. This + # image's parser returns ['reasoning,reasoning-long'] for the comma form, i.e. a + # single model whose name contains a comma, and api_server.py then iterates + # args.served_model_name to build the model list. Verified against vLLM 0.11.0: + # FlexibleArgumentParser().parse_args([... "--served-model-name", "a", "b"]) + # -> ['a', 'b'] + - --served-model-name + - reasoning + - reasoning-long - --quantization=gptq - --dtype=float16 - --kv-cache-dtype=auto