riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 24 Pull Requests 3 Actions Packages Projects 1 Releases Wiki Activity

feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff #90

Open
rock wants to merge 5 commits from fix/llm-serving-poimen-netpol into main
pull from: fix/llm-serving-poimen-netpol
merge into: :main
Conversation 0 Commits 5 Files Changed 5
+221 -1
rock commented 2026-11-01 02:04:11 +00:00
Owner
Copy Link
Copy Source

Five commits. Originally: make llm-serving reachable by poimen-memory and put a 256K coder online. Now also: the handoff surface for the vLLM -> candle-vllm migration.

1. The NetworkPolicy gap (34 days of silent ingest failure)

llm-serving-ingress allowed ingress only from api, monitoring, paperless and intra-namespace pods. It had no rule for namespace poimen.

The client-side egress rule already existed and was correct. So Cilium allowed the packet out and dropped it at the destination. The client saw a bare timeout while /health kept answering 200, because a probe from inside the pod does not cross the policy. Nothing appeared in the predictor log.

Measured, same node, same minute:

Client Label Result
probe pod in llm-serving part-of=llm-serving 200 in 5 ms
same pod without that label 8 s timeout, every service

This commit adds namespaceSelector: poimen on port 8080, mirroring the existing paperless rule, which bypasses the gateway for the same reason: poimen-memory has no service-account JWT yet.

2. coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ

The name now matches the model. Measured: /v1/models reports coder, weights load in 15.61 GiB, torch.compile 78 s, and POST /v1/chat/completions returns real Rust code.

Two served names, coder and reasoning, so the rename costs the client a host change only, not a model-name change.

Why not Qwen3-Coder-Next. Measured on a free V100 with the real v0.15.0 image:

TORCH 2.9.1+cu129
ARCH_LIST []                       <- empty
RuntimeError: cudaGetDeviceCount() ... Error 803
fla: ModuleNotFoundError           <- no GatedDeltaNet kernels

It fails before loading any weight, and Qwen3-Next's hybrid attention needs fla, which the image does not ship. vLLM requires compute capability 7.5; this node is 7.0.

3. 256K context via TP=2

Measured after boot:

Available KV cache per GPU   : 19.96 GiB
GPU KV cache size            : 436,112 tokens
Max concurrency at 262,144   : 1.66x
max_model_len (both names)   : 262144
completion                   : 200 in 2.83s

One card could not hold it (single-card pool was 132,672 tokens).

The fatal detail: with vLLM's default custom all-reduce the engine logs Graph capturing finished in 6 secs and then hangs for 14+ minutes, emitting only No available shared memory broadcast block found in 60 seconds. The first attempt died on its startup probe. --disable-custom-all-reduce fixes it: the custom kernel registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL costs nothing, because TP=2 across two PCIe V100s gains nothing from it.

Also required: memory request 12Gi (worker-1 exposes only 63,495 MiB allocatable), and /dev/shm 8Gi for NCCL.

Deployment shape

coder rolls out beside reasoning (2 GPUs were free), so there is no downtime. GPU total after sync: clef 1 + coder 2 + reasoning 1 = 4 of 4.

Verification

kubectl diff -f coder.yaml returns empty against live state.

Added for the candle-vllm handoff

vLLM is retired. candle-vllm replaces it, starting with Qwen3-30B-A3B on one card. Two commits here make that swap invisible to the client.

reasoning-long served name. The 30B-A3B answers to reasoning-long as well as reasoning. Two separate list entries, not a comma-joined value: this image's parser returns ['reasoning,reasoning-long'] for the comma form, one model whose name contains a comma. Verified against vLLM 0.11.0's own FlexibleArgumentParser.

Service reasoning-long-predictor. poimen-memory addresses this name. The vLLM-to-candle switch becomes a selector edit with no client change. It publishes port 80 and port 8080, both verified 200 through the Service from inside the poimen-memory pod, because the existing predictor Services publish 80 only while the NetworkPolicies and the readiness check use 8080.

Merge order with #91 (park clef-serve): independent. #91 touches only clef-serve.yaml, this touches coder.yaml, kustomization.yaml, networkpolicy.yaml, reasoning.yaml, reasoning-long-service.yaml. Either can land first.

Readiness after this merges and syncs, run in-cluster:

curl -sS -m 30 http://reasoning-long-predictor.llm-serving.svc.cluster.local:8080/v1/models

expect reasoning-long in the model list. Before the merge it returns ['reasoning'] only.

Five commits. Originally: make `llm-serving` reachable by `poimen-memory` and put a 256K coder online. Now also: the handoff surface for the vLLM -> candle-vllm migration. ## 1. The NetworkPolicy gap (34 days of silent ingest failure) `llm-serving-ingress` allowed ingress only from `api`, `monitoring`, `paperless` and intra-namespace pods. It had **no rule for namespace `poimen`**. The client-side egress rule already existed and was correct. So Cilium allowed the packet out and dropped it at the destination. The client saw a bare timeout while `/health` kept answering 200, because a probe from inside the pod does not cross the policy. Nothing appeared in the predictor log. Measured, same node, same minute: | Client | Label | Result | |---|---|---| | probe pod in llm-serving | `part-of=llm-serving` | 200 in **5 ms** | | same pod | without that label | **8 s timeout**, every service | This commit adds `namespaceSelector: poimen` on port 8080, mirroring the existing `paperless` rule, which bypasses the gateway for the same reason: `poimen-memory` has no service-account JWT yet. ## 2. coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ The name now matches the model. Measured: `/v1/models` reports `coder`, weights load in 15.61 GiB, `torch.compile` 78 s, and `POST /v1/chat/completions` returns real Rust code. Two served names, `coder` and `reasoning`, so the rename costs the client a host change only, not a model-name change. **Why not Qwen3-Coder-Next.** Measured on a free V100 with the real v0.15.0 image: ``` TORCH 2.9.1+cu129 ARCH_LIST [] <- empty RuntimeError: cudaGetDeviceCount() ... Error 803 fla: ModuleNotFoundError <- no GatedDeltaNet kernels ``` It fails before loading any weight, and Qwen3-Next's hybrid attention needs `fla`, which the image does not ship. vLLM requires compute capability 7.5; this node is 7.0. ## 3. 256K context via TP=2 Measured after boot: ``` Available KV cache per GPU : 19.96 GiB GPU KV cache size : 436,112 tokens Max concurrency at 262,144 : 1.66x max_model_len (both names) : 262144 completion : 200 in 2.83s ``` One card could not hold it (single-card pool was 132,672 tokens). **The fatal detail**: with vLLM's default custom all-reduce the engine logs `Graph capturing finished in 6 secs` and then hangs for 14+ minutes, emitting only `No available shared memory broadcast block found in 60 seconds`. The first attempt died on its startup probe. `--disable-custom-all-reduce` fixes it: the custom kernel registers its own CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL costs nothing, because TP=2 across two PCIe V100s gains nothing from it. Also required: memory request 12Gi (worker-1 exposes only 63,495 MiB allocatable), and `/dev/shm` 8Gi for NCCL. ## Deployment shape `coder` rolls out **beside** `reasoning` (2 GPUs were free), so there is no downtime. GPU total after sync: clef 1 + coder 2 + reasoning 1 = 4 of 4. ## Verification `kubectl diff -f coder.yaml` returns empty against live state. ## Added for the candle-vllm handoff vLLM is retired. candle-vllm replaces it, starting with Qwen3-30B-A3B on one card. Two commits here make that swap invisible to the client. **`reasoning-long` served name.** The 30B-A3B answers to `reasoning-long` as well as `reasoning`. Two separate list entries, not a comma-joined value: this image's parser returns `['reasoning,reasoning-long']` for the comma form, one model whose name contains a comma. Verified against vLLM 0.11.0's own `FlexibleArgumentParser`. **Service `reasoning-long-predictor`.** poimen-memory addresses this name. The vLLM-to-candle switch becomes a selector edit with no client change. It publishes port 80 and port 8080, both verified 200 through the Service from inside the poimen-memory pod, because the existing predictor Services publish 80 only while the NetworkPolicies and the readiness check use 8080. Merge order with #91 (park clef-serve): independent. #91 touches only `clef-serve.yaml`, this touches `coder.yaml`, `kustomization.yaml`, `networkpolicy.yaml`, `reasoning.yaml`, `reasoning-long-service.yaml`. Either can land first. Readiness after this merges and syncs, run in-cluster: ``` curl -sS -m 30 http://reasoning-long-predictor.llm-serving.svc.cluster.local:8080/v1/models ``` expect `reasoning-long` in the model list. Before the merge it returns `['reasoning']` only.
rock added 3 commits 2026-11-01 02:11:45 +00:00
fix(llm-serving): allow poimen namespace ingress so memory ingest works a234451548 
poimen-memory could not reach any inference pod. The egress side was
already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits
port 8080 to the pods labelled llm-reasoning, llm-embeddings and
llm-reranker. This policy had no matching ingress rule, so Cilium
allowed the packet out and dropped it at the destination.

The observable was a bare timeout. The request never reached vLLM, so
the predictor log stayed empty, and /health still answered because a
probe from inside the pod does not cross the policy. That made the
failure look like a hung inference engine.

Evidence, measured on 2026-10-31 against the running predictors:

  1. A probe pod in llm-serving WITH label part-of=llm-serving:
     /v1/chat/completions -> 200 in 3.1s, 8 completion tokens.
  2. The same probe pod WITHOUT that label: 8s timeout, all services.
  3. From inside poimen-memory-bfc8k: embeddings /health -> 12s
     timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic
     still worked, which limits the fault to this policy.
  4. memory logged "readiness: embedder probe timed out
     elapsed_ms=10000" every 60s.

All five services already serve inference correctly. They are
reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real
768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu
(4.1s, qwen2.5:3b-instruct).

This adds the missing ingress rule. It mirrors the existing paperless
rule, which bypasses the gateway for the same reason: poimen-memory
has no service-account JWT yet.

Refs: #TBD
feat(llm-serving): add coder, serving Qwen3-Coder-30B-A3B on one V100 dc5ec40c7d 
The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.

Measured, on worker-1, driver 580.159.04, compute capability 7.0:

  GET  /v1/models        -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
  POST /v1/chat/completions model=coder -> 200 in 3.18s,
       "fn main() { println!(\"hi\"); }"
  POST /v1/chat/completions model=reasoning -> 200 in 0.21s

Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.

Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.

Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:

  TORCH 2.9.1+cu129
  ARCH_LIST []                       <- empty
  RuntimeError: cudaGetDeviceCount() ... Error 803
  fla: ModuleNotFoundError           <- no GatedDeltaNet kernels

vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.

Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.

reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.
feat(coder): serve 256K context on TP=2, disable custom all-reduce 42617fbd7c 
Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100
could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the
weights per card to 7.85 GiB each and frees room for the larger pool.

Measured after boot, both cards worker-1:

  Available KV cache memory per GPU : 19.96 GiB
  GPU KV cache size                 : 436,112 tokens
  Max concurrency at 262,144        : 1.66x
  max_model_len (both served names) : 262144
  POST /v1/chat/completions         : 200 in 2.83s
  Weights, per worker               : 7.8516 GiB
  torch.compile                     : 78 s
  Graph capture                     : 6 s

One fatal detail, measured twice. With vLLM's default custom all-reduce the
engine reaches "Graph capturing finished in 6 secs" and then hangs. The only
output afterwards was "No available shared memory broadcast block found in
60 seconds", repeated for 14 minutes, and the front-end never received the
ready message. The first attempt died on its startup probe at 20 min.

`--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own
CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the
fallback and costs nothing here, because TP=2 across two PCIe V100s gains
nothing from the custom kernel. Both workers now register cleanly and the
API binds at 17:18.

Two supporting changes, both required:
  - memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the
    earlier 24Gi request could never schedule.
  - /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default
    shows up as an init hang.

`reasoning` remains a served name at the same 262,144 so the client needs no
model-name change.
rock force-pushed fix/llm-serving-poimen-netpol from d97fd6aba7 to 42617fbd7c 2026-11-01 02:11:45 +00:00 Compare
rock added 1 commit 2026-11-01 02:38:29 +00:00
feat(llm-serving): also serve the 30B-A3B as reasoning-long 871bb8bfd4 
The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at
64K on one card, the model qwen-finetune wiring and the models.json overlay
target. It was deployed and healthy but only answerable as `reasoning`, so
that name did not exist.

It is a second served name on the existing pod, not a new
InferenceService. All four V100s are allocated (coder 2, this 1,
clef-serve 1), so a separate reasoning-long predictor would sit Pending
forever. It would also load the same 15.56 GiB of weights a second time to
serve a name, which buys nothing.

The two names are separate list entries. The comma form does NOT work here,
and fails silently in a way that looks like success: this image's parser
turns `--served-model-name=reasoning,reasoning-long` into
['reasoning,reasoning-long'], one model whose name contains a comma, and
api_server.py iterates args.served_model_name when it builds the model list.
Verified against vLLM 0.11.0 itself rather than assumed:

  FlexibleArgumentParser().parse_args(["--model","m",
      "--served-model-name=reasoning,reasoning-long"])
    -> ['reasoning,reasoning-long']
  FlexibleArgumentParser().parse_args(["--model","m",
      "--served-model-name","reasoning","reasoning-long"])
    -> ['reasoning', 'reasoning-long']

`reasoning` stays served from this pod on purpose. poimen-memory pins
LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest
outright. The duplicate with coder-predictor is known and is resolved by
moving the client host, not by deleting a name here.

Not verified end to end: a live apply was reverted by ArgoCD selfHeal before
the new pod finished loading, so the served-name list was never observed on
a running pod with both names. The parser result above is what this commit
rests on. Confirm with `GET /v1/models` after the merge syncs.
rock added 1 commit 2026-11-01 02:43:07 +00:00
feat(llm-serving): give the chat model a stable client-facing name 2c5f510951 
poimen-memory will address `reasoning-long-predictor`, and the plan replaces
vLLM with candle-vllm underneath. Without an indirection, that handoff changes
a hostname the client is pinned to, and the ingest cutover has a window where
the name is wrong.

This Service is that indirection. It selects the vLLM reasoning pod today. The
handoff becomes a selector edit with no client change, so poimen-memory's
LLM_CHAT_BASE stays fixed across both engines.

Two ports, both verified through the Service from inside the poimen-memory pod:

  80   -> 8080   HTTP 200 in 0.0057s   matches every other predictor Service and
                                       poimen-memory's LLM_CHAT_BASE, which has
                                       no port in the URL
  8080 -> 8080   HTTP 200 in 0.0182s   the container port, the only port the two
                                       NetworkPolicies permit, and the port the
                                       handoff readiness check uses

Existing predictor Services publish port 80 only, so a curl to :8080 on one of
them fails to connect. Publishing both here means neither caller has to know
which convention the other follows.

Verified the selector resolves to the intended pod and nothing else: one
endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb.

Residual state, deliberately: /v1/models through this Service returns
['reasoning'], not ['reasoning-long'], because the second served name lands in
the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod
spec. Both names appear after this branch merges and syncs.

Not deleted and recreated at handoff. A delete drops the ClusterIP, and a
client that cached the name across the gap fails until it re-resolves.
rock changed title from feat(llm-serving): poimen ingress rule + coder at 256K on TP=2 to feat(llm-serving): poimen ingress, 256K coder, and a stable name for the candle-vllm handoff 2026-11-01 02:43:18 +00:00
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/llm-serving-poimen-netpol:fix/llm-serving-poimen-netpol
git checkout fix/llm-serving-poimen-netpol
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
area/auth
Authentication & authorization
 area/authentik
Authentik OIDC provider config
 area/downstream
Downstream service changes
 area/event-bus
Event routing and notification pipeline
 area/gateway
API gateway code (homelab-frontend)
 area/notifications
Gotify, SMTP, push notifications
 area/observer
Poimen goal-driven observation and pattern detection
 priority/high
Must have
 priority/medium
Should have
 status/blocked
Blocked by dependency
 status/done
Completed
 status/in-progress
Currently being worked on
 status/review
Ready for review
 status/todo
Not started
 type/docs
Documentation
 type/feature
New feature
 type/infra
Infrastructure change
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
poimen rock (Rock)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#90