Compare commits

...
Author SHA1 Message Date
rock 2c5f510951 feat(llm-serving): give the chat model a stable client-facing name
poimen-memory will address `reasoning-long-predictor`, and the plan replaces
vLLM with candle-vllm underneath. Without an indirection, that handoff changes
a hostname the client is pinned to, and the ingest cutover has a window where
the name is wrong.

This Service is that indirection. It selects the vLLM reasoning pod today. The
handoff becomes a selector edit with no client change, so poimen-memory's
LLM_CHAT_BASE stays fixed across both engines.

Two ports, both verified through the Service from inside the poimen-memory pod:

  80   -> 8080   HTTP 200 in 0.0057s   matches every other predictor Service and
                                       poimen-memory's LLM_CHAT_BASE, which has
                                       no port in the URL
  8080 -> 8080   HTTP 200 in 0.0182s   the container port, the only port the two
                                       NetworkPolicies permit, and the port the
                                       handoff readiness check uses

Existing predictor Services publish port 80 only, so a curl to :8080 on one of
them fails to connect. Publishing both here means neither caller has to know
which convention the other follows.

Verified the selector resolves to the intended pod and nothing else: one
endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb.

Residual state, deliberately: /v1/models through this Service returns
['reasoning'], not ['reasoning-long'], because the second served name lands in
the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod
spec. Both names appear after this branch merges and syncs.

Not deleted and recreated at handoff. A delete drops the ClusterIP, and a
client that cached the name across the gap fails until it re-resolves.
2026-10-31 19:43:03 -07:00
rock 871bb8bfd4 feat(llm-serving): also serve the 30B-A3B as reasoning-long
The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at
64K on one card, the model qwen-finetune wiring and the models.json overlay
target. It was deployed and healthy but only answerable as `reasoning`, so
that name did not exist.

It is a second served name on the existing pod, not a new
InferenceService. All four V100s are allocated (coder 2, this 1,
clef-serve 1), so a separate reasoning-long predictor would sit Pending
forever. It would also load the same 15.56 GiB of weights a second time to
serve a name, which buys nothing.

The two names are separate list entries. The comma form does NOT work here,
and fails silently in a way that looks like success: this image's parser
turns `--served-model-name=reasoning,reasoning-long` into
['reasoning,reasoning-long'], one model whose name contains a comma, and
api_server.py iterates args.served_model_name when it builds the model list.
Verified against vLLM 0.11.0 itself rather than assumed:

  FlexibleArgumentParser().parse_args(["--model","m",
      "--served-model-name=reasoning,reasoning-long"])
    -> ['reasoning,reasoning-long']
  FlexibleArgumentParser().parse_args(["--model","m",
      "--served-model-name","reasoning","reasoning-long"])
    -> ['reasoning', 'reasoning-long']

`reasoning` stays served from this pod on purpose. poimen-memory pins
LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest
outright. The duplicate with coder-predictor is known and is resolved by
moving the client host, not by deleting a name here.

Not verified end to end: a live apply was reverted by ArgoCD selfHeal before
the new pod finished loading, so the served-name list was never observed on
a running pod with both names. The parser result above is what this commit
rests on. Confirm with `GET /v1/models` after the merge syncs.
2026-10-31 19:38:26 -07:00
3 changed files with 68 additions and 1 deletions
+3
View File
@@ -19,6 +19,9 @@ resources:
# model-name change. reasoning.yaml below can be dropped once that host change lands.
- coder.yaml
- reasoning.yaml
# Stable client-facing name for the chat model. poimen-memory points at this Service,
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
- reasoning-long-service.yaml
# ornith is retired: its InferenceService still reports Ready=True while running zero
# replicas, because desired equals available at 0. Its manifest was never committed,
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
@@ -0,0 +1,43 @@
# Stable client-facing name for the chat model, deliberately decoupled from the
# process that currently serves it.
#
# poimen-memory addresses this Service, not a pod and not an InferenceService. That
# means the vLLM -> candle-vllm handoff changes only this file's selector. The client
# config never changes again, so the ingest cutover has no window in which the host
# name is wrong.
#
# Two ports on purpose:
# - 80 matches every other predictor Service in this namespace, and matches
# poimen-memory's LLM_CHAT_BASE, which carries no port number.
# - 8080 is the container port. It is also the only port llm-serving-ingress and
# poimen-memory-llm-egress permit, and the port the handoff readiness check
# uses. Publishing it directly avoids a port translation that is easy to get
# wrong in a curl.
#
# The selector names the vLLM reasoning pod today. To hand off, replace the three
# selector labels with candle-vllm's pod labels and nothing else. Do not delete and
# recreate this Service: a delete drops the ClusterIP, and a client that cached the
# name during the gap will fail until it re-resolves.
apiVersion: v1
kind: Service
metadata:
name: reasoning-long-predictor
namespace: llm-serving
labels:
app.kubernetes.io/part-of: llm-serving
app: reasoning-long-predictor
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: llm-reasoning
component: predictor
serving.kserve.io/inferenceservice: reasoning
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
- name: http-8080
protocol: TCP
port: 8080
targetPort: 8080
+22 -1
View File
@@ -30,7 +30,28 @@ spec:
# Measured on this cluster, one V100-32GB, 64k context: 15.55 GiB of weights,
# 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request.
- --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4
- --served-model-name=reasoning
# Two served names, one pod, on purpose. `reasoning-long` is the name the goal map
# uses for this model, and the name qwen-finetune wiring targets. It is NOT a
# separate InferenceService: this model already occupies the only free card, and a
# second pod on the same weights cannot schedule (4 of 4 GPUs allocated: coder 2,
# this 1, clef-serve 1). Serving both names from one process costs nothing and
# keeps the two callers working without a second weights copy.
#
# `reasoning` is deliberately still served here. poimen-memory pins
# LLM_CHAT_BASE=reasoning-predictor, so dropping the name here breaks ingest
# immediately. coder-predictor also serves `reasoning`; that ambiguity is known
# and documented in the PR. Resolve it by moving the client host, not by
# removing the name here.
#
# The two names are separate list entries, NOT one comma-joined value. This
# image's parser returns ['reasoning,reasoning-long'] for the comma form, i.e. a
# single model whose name contains a comma, and api_server.py then iterates
# args.served_model_name to build the model list. Verified against vLLM 0.11.0:
# FlexibleArgumentParser().parse_args([... "--served-model-name", "a", "b"])
# -> ['a', 'b']
- --served-model-name
- reasoning
- reasoning-long
- --quantization=gptq
- --dtype=float16
- --kv-cache-dtype=auto