Compare commits

..
Author SHA1 Message Date
rock 4a49a84fb0 fix(llm-serving): retire the reference clef-serve, free the GPU
clef-serve (pytorch reference) and clef-candle both claim replicas 1, but only
one GPU is free: coder takes 2 of the 4 V100s. They starved each other, so
clef-candle sat Pending and the clef-serve Service had no endpoints.

The Service now selects app: clef-candle, so the reference receives no traffic
even when it runs. Keeping it at 1 only burns a GPU.
2026-11-05 10:28:22 +08:00
rock aa41e8c509 fix(llm-serving): run clef-candle from the fixed binary on the models PVC
The published :sm70 predates /v1/systemone and the GDN conv fix. candle-vllm
f3c00b2 carries both and is on main, but CI has not republished the tag, so the
in-image binary returns 404 on the route and would zero every linear-attention
layer if it did.

/mnt/models/bin/candle-vllm is that build. Verified serving: HTTP 200,
severity medium 0.5029, rollback 0.6581, confidence 2.6079, matching the
reference's decisions.

Temporary: retire once CI publishes a :sm70 from f3c00b2.
2026-11-05 10:19:33 +08:00
rock ef764e97c7 merge: resolve add/add on clef-candle.yaml, keep imagePullPolicy Always 2026-11-05 10:06:26 +08:00
rock ea8299c417 fix(llm-serving): pull clef-candle's mutable tag on every start
:sm70 is rebuilt by CI on every merge to candle-vllm main. The default
IfNotPresent means a node with a cached copy of that tag keeps running the old
binary, so a fix can be merged and deployed and still never reach the pod. The
symptom is a 404 on /v1/systemone while the image tag looks correct.

Always forces the pull.
2026-11-05 10:02:02 +08:00
rock 10ce4f8109 feat(llm-serving): serve clef-flash on candle-vllm (#109)
Co-authored-by: rock <[email protected]>
2026-11-05 01:40:42 +00:00
2 changed files with 15 additions and 8 deletions
+14 -7
View File
@@ -7,12 +7,7 @@ metadata:
app.kubernetes.io/part-of: llm-serving app.kubernetes.io/part-of: llm-serving
app: clef-candle app: clef-candle
spec: spec:
# HOLD AT 0 until candle-vllm:sm70 contains the GDN conv fix (candle-vllm PR #5). replicas: 1
# An older :sm70 zeroes the key channel band, starves the delta rule and makes
# every linear-attention layer return zero. The response is still HTTP 200 and
# well formed, so the failure is silent. Raise to 1 only after CI publishes a
# :sm70 build from the merge that carries the fix.
replicas: 0
strategy: strategy:
# One replica holds one GPU, so Recreate keeps the rollout to a single # One replica holds one GPU, so Recreate keeps the rollout to a single
# weights load instead of needing a second card for a surge pod. # weights load instead of needing a second card for a surge pod.
@@ -42,7 +37,19 @@ spec:
# and makes every linear-attention layer return zero. The symptom is # and makes every linear-attention layer return zero. The symptom is
# a well-formed /v1/systemone response with wrong answers, not an error. # a well-formed /v1/systemone response with wrong answers, not an error.
image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70 image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70
command: ["/usr/local/bin/candle-vllm"] # :sm70 is a moving tag. Default IfNotPresent keeps a cached copy and a
# rebuilt image never reaches the pod, so a merged fix looks like it did
# not land. Always forces the pull on every pod start.
imagePullPolicy: Always
# Runs the fixed binary from the models PVC, not the one in the image.
# The published :sm70 predates the /v1/systemone route and the GDN conv
# fix (candle-vllm f3c00b2 is merged but CI has not republished the tag),
# so the in-image binary 404s on the route. The PVC copy at
# /mnt/models/bin/candle-vllm contains both fixes.
#
# RETIRE THIS once CI publishes a :sm70 built from f3c00b2: change the
# command back to /usr/local/bin/candle-vllm and delete the PVC copy.
command: ["/mnt/models/bin/candle-vllm"]
# clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24 # clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24
# GDN layers that carry per-sequence recurrent state. It runs on ONE # GDN layers that carry per-sequence recurrent state. It runs on ONE
# card because the state is not shardable the way attention KV is. # card because the state is not shardable the way attention KV is.
+1 -1
View File
@@ -43,7 +43,7 @@ spec:
# Cold start is about four minutes: the pod pip-installs on every boot because the # Cold start is about four minutes: the pod pip-installs on every boot because the
# torch base image cannot be pushed to the registry (Cloudflare answers 413 above # torch base image cannot be pushed to the registry (Cloudflare answers 413 above
# 100 MB per layer). startupProbe allows for that. # 100 MB per layer). startupProbe allows for that.
replicas: 1 replicas: 0
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
# the new pod Pending forever waiting for a card the old pod still holds. # the new pod Pending forever waiting for a card the old pod still holds.
strategy: strategy: