From a2344515487061480ebf90eefe5bec7a324d826c Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 14:58:39 -0700 Subject: [PATCH 1/5] fix(llm-serving): allow poimen namespace ingress so memory ingest works poimen-memory could not reach any inference pod. The egress side was already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to the pods labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no matching ingress rule, so Cilium allowed the packet out and dropped it at the destination. The observable was a bare timeout. The request never reached vLLM, so the predictor log stayed empty, and /health still answered because a probe from inside the pod does not cross the policy. That made the failure look like a hung inference engine. Evidence, measured on 2026-10-31 against the running predictors: 1. A probe pod in llm-serving WITH label part-of=llm-serving: /v1/chat/completions -> 200 in 3.1s, 8 completion tokens. 2. The same probe pod WITHOUT that label: 8s timeout, all services. 3. From inside poimen-memory-bfc8k: embeddings /health -> 12s timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic still worked, which limits the fault to this policy. 4. memory logged "readiness: embedder probe timed out elapsed_ms=10000" every 60s. All five services already serve inference correctly. They are reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real 768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu (4.1s, qwen2.5:3b-instruct). This adds the missing ingress rule. It mirrors the existing paperless rule, which bypasses the gateway for the same reason: poimen-memory has no service-account JWT yet. Refs: #TBD --- k8s/apps/llm-serving/networkpolicy.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/k8s/apps/llm-serving/networkpolicy.yaml b/k8s/apps/llm-serving/networkpolicy.yaml index d403e60..17a1ecd 100644 --- a/k8s/apps/llm-serving/networkpolicy.yaml +++ b/k8s/apps/llm-serving/networkpolicy.yaml @@ -66,6 +66,21 @@ spec: port: 80 - protocol: TCP port: 8000 + # Allow from poimen namespace (poimen-memory ingest + query). + # The client-side egress rule is the other half and must stay in step: + # NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods + # labelled part-of=llm-serving, which is the same selector used here. This + # policy had no matching ingress rule at all, so the packet was allowed out + # and dropped here. The client then saw a bare timeout and the predictor log + # stayed empty, which is why this failure looked like a hung inference engine + # for 34 days. + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: poimen + ports: + - protocol: TCP + port: 8080 # Allow intra-namespace (pod-to-pod within llm-serving) - from: - podSelector: -- 2.54.0 From dc5ec40c7d1cf1403a3e49fb237f7db4104155b1 Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 15:26:08 -0700 Subject: [PATCH 2/5] feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100 The name now matches the model. `coder` serves Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to 4-bit GPTQ. It is measured working on this node, not assumed. Measured, on worker-1, driver 580.159.04, compute capability 7.0: GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A POST /v1/chat/completions model=coder -> 200 in 3.18s, "fn main() { println!(\"hi\"); }" POST /v1/chat/completions model=reasoning -> 200 in 0.21s Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached 1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest matches live state. Two names are served on purpose: `coder` and `reasoning`. The rename therefore needs no change to poimen-memory's LLM_MODEL, only its host. That keeps the client working through the cutover. Why not Qwen3-Coder-Next, the model in the original plan. It cannot run here, and that is measured too. A v0.15.0 image on this free V100 gave: TORCH 2.9.1+cu129 ARCH_LIST [] <- empty RuntimeError: cudaGetDeviceCount() ... Error 803 fla: ModuleNotFoundError <- no GatedDeltaNet kernels vLLM 0.15.0 requires compute capability 7.5 or higher and fails here before it loads a model. Qwen3-Next's hybrid attention needs the `fla` module, which the image does not ship. So this stays on the pinned v0.11.0 image, which reports sm_70..sm_120 and works. Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that fits one card. The checkpoint is a community quant; the manifest says so and names the reason. reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves from reasoning-predictor to coder-predictor. Two GPUs are free, so coder rolled out beside reasoning with no downtime. --- k8s/apps/llm-serving/coder.yaml | 118 ++++++++++++++++++++++++ k8s/apps/llm-serving/kustomization.yaml | 4 + 2 files changed, 122 insertions(+) create mode 100644 k8s/apps/llm-serving/coder.yaml diff --git a/k8s/apps/llm-serving/coder.yaml b/k8s/apps/llm-serving/coder.yaml new file mode 100644 index 0000000..8b1b05f --- /dev/null +++ b/k8s/apps/llm-serving/coder.yaml @@ -0,0 +1,118 @@ +apiVersion: serving.kserve.io/v1beta1 +kind: InferenceService +metadata: + name: coder + namespace: llm-serving + labels: + app.kubernetes.io/name: llm-coder + app.kubernetes.io/part-of: llm-serving + annotations: + serving.kserve.io/deploymentMode: RawDeployment +spec: + predictor: + minReplicas: 1 + maxReplicas: 1 + nodeSelector: + kubernetes.io/hostname: worker-1 + # Two GPUs are free, so this rolls out beside `reasoning` instead of over it. + # Recreate still matters for the later swap: a rolling update needs a card the + # old pod holds and would sit Pending forever. + deploymentStrategy: + type: Recreate + containers: + - name: kserve-container + # Pinned to v0.11.0 on purpose. This is the last image measured to work on this + # node: torch 2.8.0+cu128, arch list sm_70..sm_120, driver 580.159.04. + # v0.15.0 was measured and FAILED here: arch list empty, and + # `cudaGetDeviceCount()` raised Error 803 (unsupported driver combination) + # before any model load. Do not raise this tag without re-running that probe. + image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c + args: + # Qwen3-Coder-30B-A3B-Instruct: 30.5B total, 3.3B active. Architecture is + # Qwen3MoeForCausalLM, the same class as the Qwen3-30B-A3B already serving + # here, so it takes the proven code path on sm70. + # + # Qwen ships this model only as BF16 (61.1 GB) and FP8 (no FP8 on Volta). + # 4-bit GPTQ is the only form that fits one 32 GB card. This checkpoint is a + # community quant: ModelCloud maintains GPTQModel, and the repo is complete + # (config + tokenizer + 4 shards, 16.9 GB). Verified by loading, not by trust. + # + # Qwen3-Coder-Next (80B/3B, the hybrid-attention model) is NOT an option: + # its GatedDeltaNet kernels need compute capability 7.5+ and the `fla` module, + # and sm70 has neither. Measured, not assumed. + - --model=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A + # `reasoning` stays a served name so an existing client that sends + # model=reasoning keeps working after the DNS rename. vLLM takes the names + # as a space-separated list, so they go in as separate argv items. + - --served-model-name + - coder + - reasoning + - --quantization=gptq + - --dtype=float16 + - --kv-cache-dtype=auto + # Native window is 262144. 64k keeps the KV cache comfortable on one card; + # raise only after measuring. + - --max-model-len=65536 + - --tensor-parallel-size=1 + - --pipeline-parallel-size=1 + - --gpu-memory-utilization=0.90 + - --max-num-seqs=16 + - --max-num-batched-tokens=8192 + - --long-prefill-token-threshold=2048 + - --enable-chunked-prefill + - --enable-prefix-caching + - --reasoning-parser=qwen3 + - --enable-auto-tool-choice + - --tool-call-parser=hermes + - --host=0.0.0.0 + - --port=8080 + env: + # Required on this hardware, not tuning: + # - flashinfer's check_cuda_arch raises on compute capability 7.0, + # so the sampler must stay off + # - FlashInfer and FlashAttention-2 need sm75+, so Triton serves attention + - name: VLLM_USE_FLASHINFER_SAMPLER + value: "0" + - name: VLLM_ATTENTION_BACKEND + value: TRITON_ATTN + - name: HF_HOME + value: /mnt/models + ports: + - containerPort: 8080 + protocol: TCP + readinessProbe: + httpGet: + path: /health + port: 8080 + periodSeconds: 10 + startupProbe: + httpGet: + path: /health + port: 8080 + # The coder checkpoint is a 16.9 GB cold download on first start, then + # ~20 s of weight load and ~80 s of torch.compile. 80 x 15 s covers it. + failureThreshold: 80 + periodSeconds: 15 + resources: + requests: + cpu: "4" + memory: 12Gi + nvidia.com/gpu: "1" + limits: + cpu: "8" + memory: 36Gi + nvidia.com/gpu: "1" + volumeMounts: + - mountPath: /mnt/models + name: models + - mountPath: /dev/shm + name: shm + runtimeClassName: nvidia + volumes: + - name: models + persistentVolumeClaim: + claimName: llm-models + - name: shm + emptyDir: + medium: Memory + sizeLimit: 2Gi diff --git a/k8s/apps/llm-serving/kustomization.yaml b/k8s/apps/llm-serving/kustomization.yaml index 8f0fb81..5680790 100644 --- a/k8s/apps/llm-serving/kustomization.yaml +++ b/k8s/apps/llm-serving/kustomization.yaml @@ -14,6 +14,10 @@ resources: # v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model # instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100. - clef-serve.yaml + # coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ, TP=2 at 262,144 tokens. It serves + # both `coder` and `reasoning`, so poimen-memory needs a host change only, never a + # model-name change. reasoning.yaml below can be dropped once that host change lands. + - coder.yaml - reasoning.yaml # ornith is retired: its InferenceService still reports Ready=True while running zero # replicas, because desired equals available at 0. Its manifest was never committed, -- 2.54.0 From 42617fbd7cf0150516d03b8b72fd61de5ce52d56 Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 17:21:54 -0700 Subject: [PATCH 3/5] feat(coder): serve 256K context on TP=2, disable custom all-reduce Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100 could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the weights per card to 7.85 GiB each and frees room for the larger pool. Measured after boot, both cards worker-1: Available KV cache memory per GPU : 19.96 GiB GPU KV cache size : 436,112 tokens Max concurrency at 262,144 : 1.66x max_model_len (both served names) : 262144 POST /v1/chat/completions : 200 in 2.83s Weights, per worker : 7.8516 GiB torch.compile : 78 s Graph capture : 6 s One fatal detail, measured twice. With vLLM's default custom all-reduce the engine reaches "Graph capturing finished in 6 secs" and then hangs. The only output afterwards was "No available shared memory broadcast block found in 60 seconds", repeated for 14 minutes, and the front-end never received the ready message. The first attempt died on its startup probe at 20 min. `--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the fallback and costs nothing here, because TP=2 across two PCIe V100s gains nothing from the custom kernel. Both workers now register cleanly and the API binds at 17:18. Two supporting changes, both required: - memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the earlier 24Gi request could never schedule. - /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default shows up as an init hang. `reasoning` remains a served name at the same 262,144 so the client needs no model-name change. --- k8s/apps/llm-serving/coder.yaml | 46 ++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/k8s/apps/llm-serving/coder.yaml b/k8s/apps/llm-serving/coder.yaml index 8b1b05f..b4386e3 100644 --- a/k8s/apps/llm-serving/coder.yaml +++ b/k8s/apps/llm-serving/coder.yaml @@ -50,17 +50,28 @@ spec: - --quantization=gptq - --dtype=float16 - --kv-cache-dtype=auto - # Native window is 262144. 64k keeps the KV cache comfortable on one card; - # raise only after measuring. - - --max-model-len=65536 - - --tensor-parallel-size=1 + # Native window is 262144 and rope_scaling is null, so 256K needs no YaRN. + # One card cannot hold it: measured KV pool on a single V100 was 132,672 + # tokens. TP=2 halves the weights per card (7.8 GiB each), which frees room + # for the larger pool. Measured after boot: see the KV line in the log. + - --max-model-len=262144 + - --tensor-parallel-size=2 - --pipeline-parallel-size=1 - --gpu-memory-utilization=0.90 - - --max-num-seqs=16 + # A single 262,144-token request consumes the whole pool, so keep few slots. + - --max-num-seqs=4 - --max-num-batched-tokens=8192 - --long-prefill-token-threshold=2048 - --enable-chunked-prefill - --enable-prefix-caching + # Measured: with the default (custom all-reduce ON) the engine finished CUDA graph + # capture in 6s at 17:01:56 and then went silent for 14+ minutes, emitting only + # "No available shared memory broadcast block found in 60 seconds". The front-end + # never received the ready message. vLLM's custom all-reduce registers its own + # CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the fallback + # and is correct here, because TP=2 across two PCIe V100s gains nothing from the + # custom kernel anyway. + - --disable-custom-all-reduce - --reasoning-parser=qwen3 - --enable-auto-tool-choice - --tool-call-parser=hermes @@ -89,19 +100,22 @@ spec: httpGet: path: /health port: 8080 - # The coder checkpoint is a 16.9 GB cold download on first start, then - # ~20 s of weight load and ~80 s of torch.compile. 80 x 15 s covers it. - failureThreshold: 80 + # TP=2 at 256K needs ~10 min before the API binds: 22s weights, 80s torch.compile, + # 85s KV profiling, then graph capture. 120 x 15s = 30 min of headroom. + failureThreshold: 120 periodSeconds: 15 resources: + # worker-1 exposes only 63,495 MiB of allocatable memory, so a 24Gi request + # never schedules. Two vLLM workers keep the weights on the GPU and need + # little host RAM, so the request stays small and the limit allows burst. requests: - cpu: "4" - memory: 12Gi - nvidia.com/gpu: "1" - limits: cpu: "8" - memory: 36Gi - nvidia.com/gpu: "1" + memory: 12Gi + nvidia.com/gpu: "2" + limits: + cpu: "16" + memory: 40Gi + nvidia.com/gpu: "2" volumeMounts: - mountPath: /mnt/models name: models @@ -115,4 +129,6 @@ spec: - name: shm emptyDir: medium: Memory - sizeLimit: 2Gi + # NCCL all-reduce over TP=2 moves through /dev/shm. The 64Mi default is + # too small and shows up as an init hang or a bus error. + sizeLimit: 8Gi -- 2.54.0 From 871bb8bfd46cc282269348c5f8b3e901fd650b0c Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 19:38:26 -0700 Subject: [PATCH 4/5] feat(llm-serving): also serve the 30B-A3B as reasoning-long The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at 64K on one card, the model qwen-finetune wiring and the models.json overlay target. It was deployed and healthy but only answerable as `reasoning`, so that name did not exist. It is a second served name on the existing pod, not a new InferenceService. All four V100s are allocated (coder 2, this 1, clef-serve 1), so a separate reasoning-long predictor would sit Pending forever. It would also load the same 15.56 GiB of weights a second time to serve a name, which buys nothing. The two names are separate list entries. The comma form does NOT work here, and fails silently in a way that looks like success: this image's parser turns `--served-model-name=reasoning,reasoning-long` into ['reasoning,reasoning-long'], one model whose name contains a comma, and api_server.py iterates args.served_model_name when it builds the model list. Verified against vLLM 0.11.0 itself rather than assumed: FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name=reasoning,reasoning-long"]) -> ['reasoning,reasoning-long'] FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name","reasoning","reasoning-long"]) -> ['reasoning', 'reasoning-long'] `reasoning` stays served from this pod on purpose. poimen-memory pins LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest outright. The duplicate with coder-predictor is known and is resolved by moving the client host, not by deleting a name here. Not verified end to end: a live apply was reverted by ArgoCD selfHeal before the new pod finished loading, so the served-name list was never observed on a running pod with both names. The parser result above is what this commit rests on. Confirm with `GET /v1/models` after the merge syncs. --- k8s/apps/llm-serving/reasoning.yaml | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/k8s/apps/llm-serving/reasoning.yaml b/k8s/apps/llm-serving/reasoning.yaml index 77f5203..de87548 100644 --- a/k8s/apps/llm-serving/reasoning.yaml +++ b/k8s/apps/llm-serving/reasoning.yaml @@ -30,7 +30,28 @@ spec: # Measured on this cluster, one V100-32GB, 64k context: 15.55 GiB of weights, # 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request. - --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4 - - --served-model-name=reasoning + # Two served names, one pod, on purpose. `reasoning-long` is the name the goal map + # uses for this model, and the name qwen-finetune wiring targets. It is NOT a + # separate InferenceService: this model already occupies the only free card, and a + # second pod on the same weights cannot schedule (4 of 4 GPUs allocated: coder 2, + # this 1, clef-serve 1). Serving both names from one process costs nothing and + # keeps the two callers working without a second weights copy. + # + # `reasoning` is deliberately still served here. poimen-memory pins + # LLM_CHAT_BASE=reasoning-predictor, so dropping the name here breaks ingest + # immediately. coder-predictor also serves `reasoning`; that ambiguity is known + # and documented in the PR. Resolve it by moving the client host, not by + # removing the name here. + # + # The two names are separate list entries, NOT one comma-joined value. This + # image's parser returns ['reasoning,reasoning-long'] for the comma form, i.e. a + # single model whose name contains a comma, and api_server.py then iterates + # args.served_model_name to build the model list. Verified against vLLM 0.11.0: + # FlexibleArgumentParser().parse_args([... "--served-model-name", "a", "b"]) + # -> ['a', 'b'] + - --served-model-name + - reasoning + - reasoning-long - --quantization=gptq - --dtype=float16 - --kv-cache-dtype=auto -- 2.54.0 From 2c5f510951797a80f7df4a51f2bf14ca11ed4021 Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 19:43:03 -0700 Subject: [PATCH 5/5] feat(llm-serving): give the chat model a stable client-facing name poimen-memory will address `reasoning-long-predictor`, and the plan replaces vLLM with candle-vllm underneath. Without an indirection, that handoff changes a hostname the client is pinned to, and the ingest cutover has a window where the name is wrong. This Service is that indirection. It selects the vLLM reasoning pod today. The handoff becomes a selector edit with no client change, so poimen-memory's LLM_CHAT_BASE stays fixed across both engines. Two ports, both verified through the Service from inside the poimen-memory pod: 80 -> 8080 HTTP 200 in 0.0057s matches every other predictor Service and poimen-memory's LLM_CHAT_BASE, which has no port in the URL 8080 -> 8080 HTTP 200 in 0.0182s the container port, the only port the two NetworkPolicies permit, and the port the handoff readiness check uses Existing predictor Services publish port 80 only, so a curl to :8080 on one of them fails to connect. Publishing both here means neither caller has to know which convention the other follows. Verified the selector resolves to the intended pod and nothing else: one endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb. Residual state, deliberately: /v1/models through this Service returns ['reasoning'], not ['reasoning-long'], because the second served name lands in the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod spec. Both names appear after this branch merges and syncs. Not deleted and recreated at handoff. A delete drops the ClusterIP, and a client that cached the name across the gap fails until it re-resolves. --- k8s/apps/llm-serving/kustomization.yaml | 3 ++ .../llm-serving/reasoning-long-service.yaml | 43 +++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 k8s/apps/llm-serving/reasoning-long-service.yaml diff --git a/k8s/apps/llm-serving/kustomization.yaml b/k8s/apps/llm-serving/kustomization.yaml index 5680790..801ae29 100644 --- a/k8s/apps/llm-serving/kustomization.yaml +++ b/k8s/apps/llm-serving/kustomization.yaml @@ -19,6 +19,9 @@ resources: # model-name change. reasoning.yaml below can be dropped once that host change lands. - coder.yaml - reasoning.yaml + # Stable client-facing name for the chat model. poimen-memory points at this Service, + # so the vLLM -> candle-vllm handoff is a selector edit here and no client change. + - reasoning-long-service.yaml # ornith is retired: its InferenceService still reports Ready=True while running zero # replicas, because desired equals available at 0. Its manifest was never committed, # so it is left unmanaged and is not listed here. Delete the CR when convenient. diff --git a/k8s/apps/llm-serving/reasoning-long-service.yaml b/k8s/apps/llm-serving/reasoning-long-service.yaml new file mode 100644 index 0000000..2b05330 --- /dev/null +++ b/k8s/apps/llm-serving/reasoning-long-service.yaml @@ -0,0 +1,43 @@ +# Stable client-facing name for the chat model, deliberately decoupled from the +# process that currently serves it. +# +# poimen-memory addresses this Service, not a pod and not an InferenceService. That +# means the vLLM -> candle-vllm handoff changes only this file's selector. The client +# config never changes again, so the ingest cutover has no window in which the host +# name is wrong. +# +# Two ports on purpose: +# - 80 matches every other predictor Service in this namespace, and matches +# poimen-memory's LLM_CHAT_BASE, which carries no port number. +# - 8080 is the container port. It is also the only port llm-serving-ingress and +# poimen-memory-llm-egress permit, and the port the handoff readiness check +# uses. Publishing it directly avoids a port translation that is easy to get +# wrong in a curl. +# +# The selector names the vLLM reasoning pod today. To hand off, replace the three +# selector labels with candle-vllm's pod labels and nothing else. Do not delete and +# recreate this Service: a delete drops the ClusterIP, and a client that cached the +# name during the gap will fail until it re-resolves. +apiVersion: v1 +kind: Service +metadata: + name: reasoning-long-predictor + namespace: llm-serving + labels: + app.kubernetes.io/part-of: llm-serving + app: reasoning-long-predictor +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: llm-reasoning + component: predictor + serving.kserve.io/inferenceservice: reasoning + ports: + - name: http + protocol: TCP + port: 80 + targetPort: 8080 + - name: http-8080 + protocol: TCP + port: 8080 + targetPort: 8080 -- 2.54.0