riotpiao-poimen/homelab

feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff #90

Open

rock wants to merge 5 commits from fix/llm-serving-poimen-netpol into main

Author	SHA1	Message	Date
rock	2c5f510951	feat(llm-serving): give the chat model a stable client-facing name poimen-memory will address `reasoning-long-predictor`, and the plan replaces vLLM with candle-vllm underneath. Without an indirection, that handoff changes a hostname the client is pinned to, and the ingest cutover has a window where the name is wrong. This Service is that indirection. It selects the vLLM reasoning pod today. The handoff becomes a selector edit with no client change, so poimen-memory's LLM_CHAT_BASE stays fixed across both engines. Two ports, both verified through the Service from inside the poimen-memory pod: 80 -> 8080 HTTP 200 in 0.0057s matches every other predictor Service and poimen-memory's LLM_CHAT_BASE, which has no port in the URL 8080 -> 8080 HTTP 200 in 0.0182s the container port, the only port the two NetworkPolicies permit, and the port the handoff readiness check uses Existing predictor Services publish port 80 only, so a curl to :8080 on one of them fails to connect. Publishing both here means neither caller has to know which convention the other follows. Verified the selector resolves to the intended pod and nothing else: one endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb. Residual state, deliberately: /v1/models through this Service returns ['reasoning'], not ['reasoning-long'], because the second served name lands in the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod spec. Both names appear after this branch merges and syncs. Not deleted and recreated at handoff. A delete drops the ClusterIP, and a client that cached the name across the gap fails until it re-resolves.	2026-10-31 19:43:03 -07:00
rock	871bb8bfd4	feat(llm-serving): also serve the 30B-A3B as reasoning-long The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at 64K on one card, the model qwen-finetune wiring and the models.json overlay target. It was deployed and healthy but only answerable as `reasoning`, so that name did not exist. It is a second served name on the existing pod, not a new InferenceService. All four V100s are allocated (coder 2, this 1, clef-serve 1), so a separate reasoning-long predictor would sit Pending forever. It would also load the same 15.56 GiB of weights a second time to serve a name, which buys nothing. The two names are separate list entries. The comma form does NOT work here, and fails silently in a way that looks like success: this image's parser turns `--served-model-name=reasoning,reasoning-long` into ['reasoning,reasoning-long'], one model whose name contains a comma, and api_server.py iterates args.served_model_name when it builds the model list. Verified against vLLM 0.11.0 itself rather than assumed: FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name=reasoning,reasoning-long"]) -> ['reasoning,reasoning-long'] FlexibleArgumentParser().parse_args(["--model","m", "--served-model-name","reasoning","reasoning-long"]) -> ['reasoning', 'reasoning-long'] `reasoning` stays served from this pod on purpose. poimen-memory pins LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest outright. The duplicate with coder-predictor is known and is resolved by moving the client host, not by deleting a name here. Not verified end to end: a live apply was reverted by ArgoCD selfHeal before the new pod finished loading, so the served-name list was never observed on a running pod with both names. The parser result above is what this commit rests on. Confirm with `GET /v1/models` after the merge syncs.	2026-10-31 19:38:26 -07:00
rock	42617fbd7c	feat(coder): serve 256K context on TP=2, disable custom all-reduce Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100 could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the weights per card to 7.85 GiB each and frees room for the larger pool. Measured after boot, both cards worker-1: Available KV cache memory per GPU : 19.96 GiB GPU KV cache size : 436,112 tokens Max concurrency at 262,144 : 1.66x max_model_len (both served names) : 262144 POST /v1/chat/completions : 200 in 2.83s Weights, per worker : 7.8516 GiB torch.compile : 78 s Graph capture : 6 s One fatal detail, measured twice. With vLLM's default custom all-reduce the engine reaches "Graph capturing finished in 6 secs" and then hangs. The only output afterwards was "No available shared memory broadcast block found in 60 seconds", repeated for 14 minutes, and the front-end never received the ready message. The first attempt died on its startup probe at 20 min. `--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the fallback and costs nothing here, because TP=2 across two PCIe V100s gains nothing from the custom kernel. Both workers now register cleanly and the API binds at 17:18. Two supporting changes, both required: - memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the earlier 24Gi request could never schedule. - /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default shows up as an init hang. `reasoning` remains a served name at the same 262,144 so the client needs no model-name change.	2026-10-31 19:11:29 -07:00
rock	dc5ec40c7d	feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100 The name now matches the model. `coder` serves Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to 4-bit GPTQ. It is measured working on this node, not assumed. Measured, on worker-1, driver 580.159.04, compute capability 7.0: GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A POST /v1/chat/completions model=coder -> 200 in 3.18s, "fn main() { println!(\"hi\"); }" POST /v1/chat/completions model=reasoning -> 200 in 0.21s Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached 1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest matches live state. Two names are served on purpose: `coder` and `reasoning`. The rename therefore needs no change to poimen-memory's LLM_MODEL, only its host. That keeps the client working through the cutover. Why not Qwen3-Coder-Next, the model in the original plan. It cannot run here, and that is measured too. A v0.15.0 image on this free V100 gave: TORCH 2.9.1+cu129 ARCH_LIST [] <- empty RuntimeError: cudaGetDeviceCount() ... Error 803 fla: ModuleNotFoundError <- no GatedDeltaNet kernels vLLM 0.15.0 requires compute capability 7.5 or higher and fails here before it loads a model. Qwen3-Next's hybrid attention needs the `fla` module, which the image does not ship. So this stays on the pinned v0.11.0 image, which reports sm_70..sm_120 and works. Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that fits one card. The checkpoint is a community quant; the manifest says so and names the reason. reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves from reasoning-predictor to coder-predictor. Two GPUs are free, so coder rolled out beside reasoning with no downtime.	2026-10-31 19:11:29 -07:00
rock	a234451548	fix(llm-serving): allow poimen namespace ingress so memory ingest works poimen-memory could not reach any inference pod. The egress side was already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to the pods labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no matching ingress rule, so Cilium allowed the packet out and dropped it at the destination. The observable was a bare timeout. The request never reached vLLM, so the predictor log stayed empty, and /health still answered because a probe from inside the pod does not cross the policy. That made the failure look like a hung inference engine. Evidence, measured on 2026-10-31 against the running predictors: 1. A probe pod in llm-serving WITH label part-of=llm-serving: /v1/chat/completions -> 200 in 3.1s, 8 completion tokens. 2. The same probe pod WITHOUT that label: 8s timeout, all services. 3. From inside poimen-memory-bfc8k: embeddings /health -> 12s timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic still worked, which limits the fault to this policy. 4. memory logged "readiness: embedder probe timed out elapsed_ms=10000" every 60s. All five services already serve inference correctly. They are reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real 768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu (4.1s, qwen2.5:3b-instruct). This adds the missing ingress rule. It mirrors the existing paperless rule, which bypasses the gateway for the same reason: poimen-memory has no service-account JWT yet. Refs: #TBD	2026-10-31 19:11:29 -07:00