fix(llm-serving): run clef-candle from the fixed binary on the models PVC #112

2026-11-05T02:19:41Z

rock commented

2026-11-05 02:19:41 +00:00

Makes clef servable now, without waiting on an image rebuild.

Problem

The published :sm70 predates both the /v1/systemone route and the GDN conv fix. candle-vllm f3c00b2 (merge of #5) contains both and is on main, but CI has not republished the tag: the pod still runs digest 8533d742 and POST /v1/systemone returns 404.

Change

command: /mnt/models/bin/candle-vllm (the build from f3c00b2, on the models PVC).

Verified live

HTTP 200 on /v1/systemone
severity medium 0.5029 (reference 0.5422)
rollback 0.6581 (reference 0.7489)
confidence 2.6079 (reference 2.3778)
All three decisions match the reference.

Retire

When CI publishes a :sm70 from f3c00b2, switch the command back to /usr/local/bin/candle-vllm and delete /mnt/models/bin/candle-vllm.

Makes clef servable now, without waiting on an image rebuild. ## Problem The published :sm70 predates both the /v1/systemone route and the GDN conv fix. candle-vllm f3c00b2 (merge of #5) contains both and is on main, but CI has not republished the tag: the pod still runs digest 8533d742 and POST /v1/systemone returns 404. ## Change command: /mnt/models/bin/candle-vllm (the build from f3c00b2, on the models PVC). ## Verified live - HTTP 200 on /v1/systemone - severity medium 0.5029 (reference 0.5422) - rollback 0.6581 (reference 0.7489) - confidence 2.6079 (reference 2.3778) All three decisions match the reference. ## Retire When CI publishes a :sm70 from f3c00b2, switch the command back to /usr/local/bin/candle-vllm and delete /mnt/models/bin/candle-vllm.

rock added 10 commits 2026-11-05 02:19:43 +00:00

k8s(llm-serving): client-facing Service for candle-vllm coder 67760720fc

Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so
clients keep one stable host name across the vLLM -> candle-vllm handoff and
only this selector changes from now on.

The retired KTML predictor requested two V100s and never reached Ready.

feat(llm-serving): run clef-serve so /v1/systemone answers db97159905

clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds
four V100s, so the coder takes one and clef takes one, leaving two free
for a TP=2 candle run.

The gateway already routes the clef-flash model to clef-serve:80 on
/v1/systemone, so this single replica count is what turns that route on.

feat(llm-serving): run coder as fp16 14B across two V100s f6fca4705b

Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at
fp16 on TP=2.

The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a
dequantisation pass on the SM. Profiling showed SM at 98% with the memory
controller at 19%, and the batch-1 step cost matched a full-model read.
Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s.

The deployment keeps the candle-vllm name and the app: candle-vllm label,
so coder-service.yaml and every client keep working unchanged.

fix(llm-serving): set runtimeClassName nvidia on the candle deployment 7bdc8b2de2

Without it the container runs under runc, libcuda.so is never injected and
candle-vllm panics in cudarc at startup. The pod reached Running but crashed
on every restart.

Verified live: pod Ready, coder-predictor endpoints populated, /v1/models
reports coder-14b-fp16 and a completion returns OK.

feat(llm-serving): serve clef-flash on candle-vllm 3ab127e581

Adds clef-candle.yaml: clef-flash (hybrid Qwen3_5, 8 attention + 24 GDN
layers) on one V100 via candle-vllm, and repoints the clef-serve Service
selector at it. The Service name and ClusterIP do not move, so no client
changes.

Runs on one card because the GDN recurrent state is per-sequence and cannot
be sharded the way attention KV can. --kv-fraction 0.30 gives about 66.8K KV
tokens, which is ample for a single-shot classifier-style endpoint.

The image is candle-vllm:sm70, built by .gitea/workflows/candle-vllm-sm70.yaml
on every merge to main. It must be a build containing the GDN conv fix: older
images zero the key channel band in causal_conv1d_fwd, starve the delta rule
and make every linear-attention layer return zero. That failure is silent -
the response is well formed, just wrong.

Verified against the reference implementation: token ids 420/420 identical,
beta 8647.021 vs 8646.982, delta-rule and gated-norm absmax equal to six
decimals, all three head answers agree.

merge: resolve kustomization conflict, keep clef-candle and coder-14b-fp16 64a3f131b5

hold clef-candle at replicas 0 until the image carries the GDN fix 9617bbe814

candle-vllm PR #5 (the conv fix) could not be merged, so CI has not published
a :sm70 image containing it. Deploying clef against an older :sm70 fails
silently: the delta rule returns zero on all 24 linear-attention layers and
/v1/systemone still answers 200 with well-formed but wrong classifications.

Raise to 1 once a :sm70 build from the fix merge exists.

fix(llm-serving): pull clef-candle's mutable tag on every start ea8299c417

:sm70 is rebuilt by CI on every merge to candle-vllm main. The default
IfNotPresent means a node with a cached copy of that tag keeps running the old
binary, so a fix can be merged and deployed and still never reach the pod. The
symptom is a 404 on /v1/systemone while the image tag looks correct.

Always forces the pull.

merge: resolve add/add on clef-candle.yaml, keep imagePullPolicy Always ef764e97c7

fix(llm-serving): run clef-candle from the fixed binary on the models PVC aa41e8c509

The published :sm70 predates /v1/systemone and the GDN conv fix. candle-vllm
f3c00b2 carries both and is on main, but CI has not republished the tag, so the
in-image binary returns 404 on the route and would zero every linear-attention
layer if it did.

/mnt/models/bin/candle-vllm is that build. Verified serving: HTTP 200,
severity medium 0.5029, rollback 0.6581, confidence 2.6079, matching the
reference's decisions.

Temporary: retire once CI publishes a :sm70 from f3c00b2.

rock merged commit b790f5fb9b into main

2026-11-05 02:20:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#112