merge: resolve add/add on clef-candle.yaml, keep imagePullPolicy Always

fix(llm-serving): pull clef-candle's mutable tag on every start
:sm70 is rebuilt by CI on every merge to candle-vllm main. The default IfNotPresent means a node with a cached copy of that tag keeps running the old binary, so a fix can be merged and deployed and still never reach the pod. The symptom is a 404 on /v1/systemone while the image tag looks correct. Always forces the pull.
2026-11-05 10:06:26 +08:00 · 2026-11-05 10:02:02 +08:00 · 2026-11-05 09:41:28 +08:00 · 2026-11-05 09:40:36 +08:00 · 2026-11-05 09:35:47 +08:00 · 2026-11-05 09:22:48 +08:00
1 changed files with 1 additions and 9 deletions
@@ -41,15 +41,7 @@ spec:
          # rebuilt image never reaches the pod, so a merged fix looks like it did
          # not land. Always forces the pull on every pod start.
          imagePullPolicy: Always
-          # Runs the fixed binary from the models PVC, not the one in the image.
+          command: ["/usr/local/bin/candle-vllm"]
          # The published :sm70 predates the /v1/systemone route and the GDN conv
          # fix (candle-vllm f3c00b2 is merged but CI has not republished the tag),
          # so the in-image binary 404s on the route. The PVC copy at
          # /mnt/models/bin/candle-vllm contains both fixes.
          #
          # RETIRE THIS once CI publishes a :sm70 built from f3c00b2: change the
          # command back to /usr/local/bin/candle-vllm and delete the PVC copy.
          command: ["/mnt/models/bin/candle-vllm"]
          # clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24
          # GDN layers that carry per-sequence recurrent state. It runs on ONE
          # card because the state is not shardable the way attention KV is.
Author	SHA1	Message	Date
rock	ef764e97c7	merge: resolve add/add on clef-candle.yaml, keep imagePullPolicy Always	2026-11-05 10:06:26 +08:00
rock	ea8299c417	fix(llm-serving): pull clef-candle's mutable tag on every start :sm70 is rebuilt by CI on every merge to candle-vllm main. The default IfNotPresent means a node with a cached copy of that tag keeps running the old binary, so a fix can be merged and deployed and still never reach the pod. The symptom is a 404 on /v1/systemone while the image tag looks correct. Always forces the pull.	2026-11-05 10:02:02 +08:00
rock	9617bbe814	hold clef-candle at replicas 0 until the image carries the GDN fix candle-vllm PR #5 (the conv fix) could not be merged, so CI has not published a :sm70 image containing it. Deploying clef against an older :sm70 fails silently: the delta rule returns zero on all 24 linear-attention layers and /v1/systemone still answers 200 with well-formed but wrong classifications. Raise to 1 once a :sm70 build from the fix merge exists.	2026-11-05 09:41:28 +08:00
rock	64a3f131b5	merge: resolve kustomization conflict, keep clef-candle and coder-14b-fp16	2026-11-05 09:40:36 +08:00
rock	3ab127e581	feat(llm-serving): serve clef-flash on candle-vllm Adds clef-candle.yaml: clef-flash (hybrid Qwen3_5, 8 attention + 24 GDN layers) on one V100 via candle-vllm, and repoints the clef-serve Service selector at it. The Service name and ClusterIP do not move, so no client changes. Runs on one card because the GDN recurrent state is per-sequence and cannot be sharded the way attention KV can. --kv-fraction 0.30 gives about 66.8K KV tokens, which is ample for a single-shot classifier-style endpoint. The image is candle-vllm:sm70, built by .gitea/workflows/candle-vllm-sm70.yaml on every merge to main. It must be a build containing the GDN conv fix: older images zero the key channel band in causal_conv1d_fwd, starve the delta rule and make every linear-attention layer return zero. That failure is silent - the response is well formed, just wrong. Verified against the reference implementation: token ids 420/420 identical, beta 8647.021 vs 8646.982, delta-rule and gated-norm absmax equal to six decimals, all three head answers agree.	2026-11-05 09:35:47 +08:00
rock	7bdc8b2de2	fix(llm-serving): set runtimeClassName nvidia on the candle deployment Without it the container runs under runc, libcuda.so is never injected and candle-vllm panics in cudarc at startup. The pod reached Running but crashed on every restart. Verified live: pod Ready, coder-predictor endpoints populated, /v1/models reports coder-14b-fp16 and a completion returns OK.	2026-11-05 09:22:48 +08:00
rock	f6fca4705b	feat(llm-serving): run coder as fp16 14B across two V100s Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at fp16 on TP=2. The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a dequantisation pass on the SM. Profiling showed SM at 98% with the memory controller at 19%, and the batch-1 step cost matched a full-model read. Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s. The deployment keeps the candle-vllm name and the app: candle-vllm label, so coder-service.yaml and every client keep working unchanged.	2026-11-05 09:11:46 +08:00
rock	db97159905	feat(llm-serving): run clef-serve so /v1/systemone answers clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds four V100s, so the coder takes one and clef takes one, leaving two free for a TP=2 candle run. The gateway already routes the clef-flash model to clef-serve:80 on /v1/systemone, so this single replica count is what turns that route on.	2026-11-04 18:59:56 +08:00
rock	67760720fc	k8s(llm-serving): client-facing Service for candle-vllm coder Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so clients keep one stable host name across the vLLM -> candle-vllm handoff and only this selector changes from now on. The retired KTML predictor requested two V100s and never reached Ready.	2026-11-04 18:59:56 +08:00