feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff #90
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: riotpiao-poimen/homelab#90
Reference in New Issue
Block a user
Five commits. Originally: make
llm-servingreachable bypoimen-memoryand put a 256K coder online. Now also: the handoff surface for the vLLM -> candle-vllm migration.1. The NetworkPolicy gap (34 days of silent ingest failure)
llm-serving-ingressallowed ingress only fromapi,monitoring,paperlessand intra-namespace pods. It had no rule for namespacepoimen.The client-side egress rule already existed and was correct. So Cilium allowed the packet out and dropped it at the destination. The client saw a bare timeout while
/healthkept answering 200, because a probe from inside the pod does not cross the policy. Nothing appeared in the predictor log.Measured, same node, same minute:
part-of=llm-servingThis commit adds
namespaceSelector: poimenon port 8080, mirroring the existingpaperlessrule, which bypasses the gateway for the same reason:poimen-memoryhas no service-account JWT yet.2. coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ
The name now matches the model. Measured:
/v1/modelsreportscoder, weights load in 15.61 GiB,torch.compile78 s, andPOST /v1/chat/completionsreturns real Rust code.Two served names,
coderandreasoning, so the rename costs the client a host change only, not a model-name change.Why not Qwen3-Coder-Next. Measured on a free V100 with the real v0.15.0 image:
It fails before loading any weight, and Qwen3-Next's hybrid attention needs
fla, which the image does not ship. vLLM requires compute capability 7.5; this node is 7.0.3. 256K context via TP=2
Measured after boot:
One card could not hold it (single-card pool was 132,672 tokens).
The fatal detail: with vLLM's default custom all-reduce the engine logs
Graph capturing finished in 6 secsand then hangs for 14+ minutes, emitting onlyNo available shared memory broadcast block found in 60 seconds. The first attempt died on its startup probe.--disable-custom-all-reducefixes it: the custom kernel registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL costs nothing, because TP=2 across two PCIe V100s gains nothing from it.Also required: memory request 12Gi (worker-1 exposes only 63,495 MiB allocatable), and
/dev/shm8Gi for NCCL.Deployment shape
coderrolls out besidereasoning(2 GPUs were free), so there is no downtime. GPU total after sync: clef 1 + coder 2 + reasoning 1 = 4 of 4.Verification
kubectl diff -f coder.yamlreturns empty against live state.Added for the candle-vllm handoff
vLLM is retired. candle-vllm replaces it, starting with Qwen3-30B-A3B on one card. Two commits here make that swap invisible to the client.
reasoning-longserved name. The 30B-A3B answers toreasoning-longas well asreasoning. Two separate list entries, not a comma-joined value: this image's parser returns['reasoning,reasoning-long']for the comma form, one model whose name contains a comma. Verified against vLLM 0.11.0's ownFlexibleArgumentParser.Service
reasoning-long-predictor. poimen-memory addresses this name. The vLLM-to-candle switch becomes a selector edit with no client change. It publishes port 80 and port 8080, both verified 200 through the Service from inside the poimen-memory pod, because the existing predictor Services publish 80 only while the NetworkPolicies and the readiness check use 8080.Merge order with #91 (park clef-serve): independent. #91 touches only
clef-serve.yaml, this touchescoder.yaml,kustomization.yaml,networkpolicy.yaml,reasoning.yaml,reasoning-long-service.yaml. Either can land first.Readiness after this merges and syncs, run in-cluster:
expect
reasoning-longin the model list. Before the merge it returns['reasoning']only.poimen-memory could not reach any inference pod. The egress side was already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to the pods labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no matching ingress rule, so Cilium allowed the packet out and dropped it at the destination. The observable was a bare timeout. The request never reached vLLM, so the predictor log stayed empty, and /health still answered because a probe from inside the pod does not cross the policy. That made the failure look like a hung inference engine. Evidence, measured on 2026-10-31 against the running predictors: 1. A probe pod in llm-serving WITH label part-of=llm-serving: /v1/chat/completions -> 200 in 3.1s, 8 completion tokens. 2. The same probe pod WITHOUT that label: 8s timeout, all services. 3. From inside poimen-memory-bfc8k: embeddings /health -> 12s timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic still worked, which limits the fault to this policy. 4. memory logged "readiness: embedder probe timed out elapsed_ms=10000" every 60s. All five services already serve inference correctly. They are reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real 768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu (4.1s, qwen2.5:3b-instruct). This adds the missing ingress rule. It mirrors the existing paperless rule, which bypasses the gateway for the same reason: poimen-memory has no service-account JWT yet. Refs: #TBDThe name now matches the model. `coder` serves Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to 4-bit GPTQ. It is measured working on this node, not assumed. Measured, on worker-1, driver 580.159.04, compute capability 7.0: GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A POST /v1/chat/completions model=coder -> 200 in 3.18s, "fn main() { println!(\"hi\"); }" POST /v1/chat/completions model=reasoning -> 200 in 0.21s Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached 1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest matches live state. Two names are served on purpose: `coder` and `reasoning`. The rename therefore needs no change to poimen-memory's LLM_MODEL, only its host. That keeps the client working through the cutover. Why not Qwen3-Coder-Next, the model in the original plan. It cannot run here, and that is measured too. A v0.15.0 image on this free V100 gave: TORCH 2.9.1+cu129 ARCH_LIST [] <- empty RuntimeError: cudaGetDeviceCount() ... Error 803 fla: ModuleNotFoundError <- no GatedDeltaNet kernels vLLM 0.15.0 requires compute capability 7.5 or higher and fails here before it loads a model. Qwen3-Next's hybrid attention needs the `fla` module, which the image does not ship. So this stays on the pinned v0.11.0 image, which reports sm_70..sm_120 and works. Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that fits one card. The checkpoint is a community quant; the manifest says so and names the reason. reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves from reasoning-predictor to coder-predictor. Two GPUs are free, so coder rolled out beside reasoning with no downtime.Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100 could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the weights per card to 7.85 GiB each and frees room for the larger pool. Measured after boot, both cards worker-1: Available KV cache memory per GPU : 19.96 GiB GPU KV cache size : 436,112 tokens Max concurrency at 262,144 : 1.66x max_model_len (both served names) : 262144 POST /v1/chat/completions : 200 in 2.83s Weights, per worker : 7.8516 GiB torch.compile : 78 s Graph capture : 6 s One fatal detail, measured twice. With vLLM's default custom all-reduce the engine reaches "Graph capturing finished in 6 secs" and then hangs. The only output afterwards was "No available shared memory broadcast block found in 60 seconds", repeated for 14 minutes, and the front-end never received the ready message. The first attempt died on its startup probe at 20 min. `--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the fallback and costs nothing here, because TP=2 across two PCIe V100s gains nothing from the custom kernel. Both workers now register cleanly and the API binds at 17:18. Two supporting changes, both required: - memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the earlier 24Gi request could never schedule. - /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default shows up as an init hang. `reasoning` remains a served name at the same 262,144 so the client needs no model-name change.d97fd6aba7to42617fbd7cThe goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at 64K on one card, the model qwen-finetune wiring and the models.json overlay target. It was deployed and healthy but only answerable as `reasoning`, so that name did not exist. It is a second served name on the existing pod, not a new InferenceService. All four V100s are allocated (coder 2, this 1, clef-serve 1), so a separate reasoning-long predictor would sit Pending forever. It would also load the same 15.56 GiB of weights a second time to serve a name, which buys nothing. The two names are separate list entries. The comma form does NOT work here, and fails silently in a way that looks like success: this image's parser turns `--served-model-name=reasoning,reasoning-long` into ['reasoning,reasoning-long'], one model whose name contains a comma, and api_server.py iterates args.served_model_name when it builds the model list. Verified against vLLM 0.11.0 itself rather than assumed: FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name=reasoning,reasoning-long"]) -> ['reasoning,reasoning-long'] FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name","reasoning","reasoning-long"]) -> ['reasoning', 'reasoning-long'] `reasoning` stays served from this pod on purpose. poimen-memory pins LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest outright. The duplicate with coder-predictor is known and is resolved by moving the client host, not by deleting a name here. Not verified end to end: a live apply was reverted by ArgoCD selfHeal before the new pod finished loading, so the served-name list was never observed on a running pod with both names. The parser result above is what this commit rests on. Confirm with `GET /v1/models` after the merge syncs.poimen-memory will address `reasoning-long-predictor`, and the plan replaces vLLM with candle-vllm underneath. Without an indirection, that handoff changes a hostname the client is pinned to, and the ingest cutover has a window where the name is wrong. This Service is that indirection. It selects the vLLM reasoning pod today. The handoff becomes a selector edit with no client change, so poimen-memory's LLM_CHAT_BASE stays fixed across both engines. Two ports, both verified through the Service from inside the poimen-memory pod: 80 -> 8080 HTTP 200 in 0.0057s matches every other predictor Service and poimen-memory's LLM_CHAT_BASE, which has no port in the URL 8080 -> 8080 HTTP 200 in 0.0182s the container port, the only port the two NetworkPolicies permit, and the port the handoff readiness check uses Existing predictor Services publish port 80 only, so a curl to :8080 on one of them fails to connect. Publishing both here means neither caller has to know which convention the other follows. Verified the selector resolves to the intended pod and nothing else: one endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb. Residual state, deliberately: /v1/models through this Service returns ['reasoning'], not ['reasoning-long'], because the second served name lands in the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod spec. Both names appear after this branch merges and syncs. Not deleted and recreated at handoff. A delete drops the ClusterIP, and a client that cached the name across the gap fails until it re-resolves.feat(llm-serving): poimen ingress rule + coder at 256K on TP=2to feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff