fix(llm-serving): set runtimeClassName nvidia on the candle deployment

Without it the container runs under runc, libcuda.so is never injected and candle-vllm panics in cudarc at startup. The pod reached Running but crashed on every restart. Verified live: pod Ready, coder-predictor endpoints populated, /v1/models reports coder-14b-fp16 and a completion returns OK.
feat(llm-serving): run coder as fp16 14B across two V100s
2026-11-05 09:22:48 +08:00 · 2026-11-05 09:11:46 +08:00 · 2026-11-04 18:59:56 +08:00 · 2026-11-04 18:59:56 +08:00
3 changed files with 1 additions and 88 deletions
@@ -1,80 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: clef-candle
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: clef-candle
-spec:
-  replicas: 1
-  strategy:
-    # One replica holds one GPU, so Recreate keeps the rollout to a single
-    # weights load instead of needing a second card for a surge pod.
-    type: Recreate
-  selector:
-    matchLabels:
-      app: clef-candle
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/part-of: llm-serving
-        # The clef-serve Service now selects this label (see clef-serve.yaml),
-        # so clients keep addressing clef-serve:80 unchanged. The reference
-        # pytorch Deployment must stay at replicas 0 while this runs, or the
-        # Service splits traffic across two different implementations.
-        app: clef-candle
-    spec:
-      # Required. Without the nvidia runtime class libcuda.so is never injected
-      # and candle-vllm panics in cudarc at startup.
-      runtimeClassName: nvidia
-      containers:
-        - name: candle
-          # :sm70 is built by .gitea/workflows/candle-vllm-sm70.yaml on every
-          # merge to main. It MUST be a build that contains the GDN conv fix
-          # (candle-vllm PR: feat/clef-systemone). Older :sm70 images zero the
-          # key channel band in causal_conv1d_fwd, which starves the delta rule
-          # and makes every linear-attention layer return zero. The symptom is
-          # a well-formed /v1/systemone response with wrong answers, not an error.
-          image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70
-          command: ["/usr/local/bin/candle-vllm"]
-          # clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24
-          # GDN layers that carry per-sequence recurrent state. It runs on ONE
-          # card because the state is not shardable the way attention KV is.
-          args:
-            - --w
-            - /mnt/models/clef-flash
-            - --d
-            - "0"
-            - --dtype
-            - f16
-            - --p
-            - "8080"
-            - --max-num-seqs
-            - "4"
-            # About 66.8K KV tokens. Clef is a single-shot classifier-style
-            # endpoint, so it does not need the coder's pool.
-            - --kv-fraction
-            - "0.30"
-            # Prefix caching stays on: clef requests repeat the schema block.
-            - --prefill-chunk-size
-            - "1024"
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: "8"
-              memory: 32Gi
-              nvidia.com/gpu: "1"
-            requests:
-              cpu: "4"
-              memory: 16Gi
-              nvidia.com/gpu: "1"
-          volumeMounts:
-            - name: models
-              mountPath: /mnt/models
-              readOnly: true
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llm-models
@@ -127,11 +127,8 @@ metadata:
  labels:
    app.kubernetes.io/part-of: llm-serving
 spec:
-  # Points at the candle-vllm implementation (clef-candle.yaml). Switching back
-  # to the reference pytorch server is this line plus that Deployment's replicas;
-  # no client changes, because the Service name and ClusterIP do not move.
  selector:
-    app: clef-candle
+    app: clef-serve
  ports:
  - name: http
    port: 80
@@ -21,10 +21,6 @@ resources:
  # streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
  # tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
  # the memory controller idles at 19%.
-  # clef-candle.yaml serves clef-flash on candle-vllm (hybrid Qwen3_5, one GPU)
-  # and owns the clef-candle label that clef-serve.yaml's Service now selects.
-  # It must not sync against an image built before the GDN conv fix.
-  - clef-candle.yaml
  - coder-14b-fp16.yaml
  - clef-serve.yaml
  # Stable client-facing name for the chat model. poimen-memory points at this Service,
Author	SHA1	Message	Date
rock	7bdc8b2de2	fix(llm-serving): set runtimeClassName nvidia on the candle deployment Without it the container runs under runc, libcuda.so is never injected and candle-vllm panics in cudarc at startup. The pod reached Running but crashed on every restart. Verified live: pod Ready, coder-predictor endpoints populated, /v1/models reports coder-14b-fp16 and a completion returns OK.	2026-11-05 09:22:48 +08:00
rock	f6fca4705b	feat(llm-serving): run coder as fp16 14B across two V100s Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at fp16 on TP=2. The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a dequantisation pass on the SM. Profiling showed SM at 98% with the memory controller at 19%, and the batch-1 step cost matched a full-model read. Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s. The deployment keeps the candle-vllm name and the app: candle-vllm label, so coder-service.yaml and every client keep working unchanged.	2026-11-05 09:11:46 +08:00
rock	db97159905	feat(llm-serving): run clef-serve so /v1/systemone answers clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds four V100s, so the coder takes one and clef takes one, leaving two free for a TP=2 candle run. The gateway already routes the clef-flash model to clef-serve:80 on /v1/systemone, so this single replica count is what turns that route on.	2026-11-04 18:59:56 +08:00
rock	67760720fc	k8s(llm-serving): client-facing Service for candle-vllm coder Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so clients keep one stable host name across the vLLM -> candle-vllm handoff and only this selector changes from now on. The retired KTML predictor requested two V100s and never reached Ready.	2026-11-04 18:59:56 +08:00