k8s(llm-serving): stop managing the coder InferenceService

coder.yaml requests 2 V100s and never reaches Ready, so it permanently
occupies half the GPU pool. candle-vllm serves the coder model on the
other two cards; the unmanaged predictor starves clef-serve and blocks
its own rollout on every sync.

ArgoCD has prune:false, so this removes management only. The live object
is deleted manually after the sync, otherwise selfHeal recreates it.

k8s/apps/llm-serving/clef-candle.yaml

@@ -1,92 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: clef-candle
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: clef-candle
-spec:
-  replicas: 1
-  strategy:
-    # One replica holds one GPU, so Recreate keeps the rollout to a single
-    # weights load instead of needing a second card for a surge pod.
-    type: Recreate
-  selector:
-    matchLabels:
-      app: clef-candle
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/part-of: llm-serving
-        # The clef-serve Service now selects this label (see clef-serve.yaml),
-        # so clients keep addressing clef-serve:80 unchanged. The reference
-        # pytorch Deployment must stay at replicas 0 while this runs, or the
-        # Service splits traffic across two different implementations.
-        app: clef-candle
-    spec:
-      # Required. Without the nvidia runtime class libcuda.so is never injected
-      # and candle-vllm panics in cudarc at startup.
-      runtimeClassName: nvidia
-      containers:
-        - name: candle
-          # :sm70 is built by .gitea/workflows/candle-vllm-sm70.yaml on every
-          # merge to main. It MUST be a build that contains the GDN conv fix
-          # (candle-vllm PR: feat/clef-systemone). Older :sm70 images zero the
-          # key channel band in causal_conv1d_fwd, which starves the delta rule
-          # and makes every linear-attention layer return zero. The symptom is
-          # a well-formed /v1/systemone response with wrong answers, not an error.
-          image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70
-          # :sm70 is a moving tag. Default IfNotPresent keeps a cached copy and a
-          # rebuilt image never reaches the pod, so a merged fix looks like it did
-          # not land. Always forces the pull on every pod start.
-          imagePullPolicy: Always
-          # Runs the fixed binary from the models PVC, not the one in the image.
-          # The published :sm70 predates the /v1/systemone route and the GDN conv
-          # fix (candle-vllm f3c00b2 is merged but CI has not republished the tag),
-          # so the in-image binary 404s on the route. The PVC copy at
-          # /mnt/models/bin/candle-vllm contains both fixes.
-          #
-          # RETIRE THIS once CI publishes a :sm70 built from f3c00b2: change the
-          # command back to /usr/local/bin/candle-vllm and delete the PVC copy.
-          command: ["/mnt/models/bin/candle-vllm"]
-          # clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24
-          # GDN layers that carry per-sequence recurrent state. It runs on ONE
-          # card because the state is not shardable the way attention KV is.
-          args:
-            - --w
-            - /mnt/models/clef-flash
-            - --d
-            - "0"
-            - --dtype
-            - f16
-            - --p
-            - "8080"
-            - --max-num-seqs
-            - "4"
-            # About 66.8K KV tokens. Clef is a single-shot classifier-style
-            # endpoint, so it does not need the coder's pool.
-            - --kv-fraction
-            - "0.30"
-            # Prefix caching stays on: clef requests repeat the schema block.
-            - --prefill-chunk-size
-            - "1024"
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: "8"
-              memory: 32Gi
-              nvidia.com/gpu: "1"
-            requests:
-              cpu: "4"
-              memory: 16Gi
-              nvidia.com/gpu: "1"
-          volumeMounts:
-            - name: models
-              mountPath: /mnt/models
-              readOnly: true
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llm-models

k8s/apps/llm-serving/clef-serve.yaml

@@ -33,17 +33,19 @@ metadata:
     app: clef-serve
     app.kubernetes.io/part-of: llm-serving
 spec:
-  # One card. worker-1 holds four V100s: candle-vllm takes one for the coder, and
-  # this pod takes one, so two stay free for a TP=2 candle run.
+  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
+  # this 1) and a new GPU workload had nowhere to land.
   #
-  # Enabled 2026-11-04 to serve /v1/systemone, which the gateway already routes to
-  # this Service. It was parked at 0 while candle-vllm was brought up on the same
-  # cards, and it served no inference request between 2026-10-30T06:23:40Z and then.
+  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
+  # the first card released: the standing instruction is to keep the vLLM pods healthy
+  # until candle-vllm answers, and this pod is not one of them.
   #
-  # Cold start is about four minutes: the pod pip-installs on every boot because the
-  # torch base image cannot be pushed to the registry (Cloudflare answers 413 above
-  # 100 MB per layer). startupProbe allows for that.
-  replicas: 1
+  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
+  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
+  # in git selects this service. Restore it once the rollout leaves a card free:
+  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
+  replicas: 0
   # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
   # the new pod Pending forever waiting for a card the old pod still holds.
   strategy:
@@ -127,11 +129,8 @@ metadata:
   labels:
     app.kubernetes.io/part-of: llm-serving
 spec:
-  # Points at the candle-vllm implementation (clef-candle.yaml). Switching back
-  # to the reference pytorch server is this line plus that Deployment's replicas;
-  # no client changes, because the Service name and ClusterIP do not move.
   selector:
-    app: clef-candle
+    app: clef-serve
   ports:
   - name: http
     port: 80

k8s/apps/llm-serving/coder-14b-fp16.yaml

@@ -1,106 +0,0 @@
-# Qwen2.5-Coder-14B-Instruct, fp16, tensor-parallel across two V100s.
-#
-# Why this model and this precision. The previous workload was Qwen3-Coder-30B-A3B
-# in GPTQ W4A16 on one V100. Profiling that deployment showed SM utilisation at
-# 98% while the memory controller sat at 19%: the V100 has FP16 tensor cores and
-# no INT4 ones, so every decode step ran a dequantisation pass on the SM. That is
-# a hardware-generation mismatch, not a tunable. Sequence throughput matches the
-# arithmetic: at batch 1 the step cost equalled a full-model read (17.6 ms against
-# 15 GB / 900 GB/s = 16.7 ms) while the routed experts alone are ~1.5 GB.
-#
-# Measured on this hardware, same harness, same 16-stream load:
-#
-#   GPTQ W4A16 30B-A3B, TP=1  : ITL 71.5 ms, 215 tok/s aggregate
-#   fp16 14B, TP=2            : ITL 39.7 ms, 287 tok/s aggregate
-#
-# The dense model degrades only 1.4x from 1 to 16 streams (27.9 -> 39.7 ms) while
-# the MoE degrades 4.1x (17.6 -> 71.5 ms), because a dense model reads its weights
-# once per step and amortises across the batch. The MoE pays expert weight traffic
-# per sequence. Single-stream latency is the one case where the MoE still wins
-# (17.6 ms against 27.9 ms), since it activates ~3B parameters per token.
-#
-# TP=2 notes that cost time to establish, so they are recorded here:
-#   - Sharding needs no code change. Every dimension divides by 2: attention heads
-#     40 -> 20, KV heads 8 -> 4, intermediate_size 13824 -> 6912, vocab
-#     152064 -> 76032. Verify this first for any replacement model.
-#   - TP=2 on a GPTQ checkpoint is what crashed (CUDA_ERROR_ILLEGAL_ADDRESS in
-#     forward()) on the previous model. The cause was quantised weight sharding
-#     and the g_idx guard in src/openai/models/linear.rs, not TP itself. A dense
-#     fp16 checkpoint has no g_idx, no packed weights and no scales, so it takes
-#     the plain column/row-parallel path and the crash does not reproduce.
-#   - These are V100-PCIE cards with no NVLink, so every layer's all-reduce
-#     crosses PCIe. It shows up in TTFT (275 ms at 1 stream, 1090 ms at 16)
-#     more than in ITL.
-#   - KV capacity is 11635 MB per rank, about 59K context tokens. The fp16
-#     weights consume the room the KV pool would otherwise use. This is well
-#     short of the 256K the previous model targeted, and it is the price of
-#     leaving the dequantisation path.
-#
-# Weights must be in the PVC as /mnt/models/coder-14b-fp16 (28 GB, 6 shards).
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: candle-vllm
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: candle-vllm
-spec:
-  replicas: 1
-  strategy:
-    # One replica holds both GPUs, so a rolling update would need a second pair.
-    # Recreate keeps the rollout to a single weights load.
-    type: Recreate
-  selector:
-    matchLabels:
-      app: candle-vllm
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/part-of: llm-serving
-        app: candle-vllm
-    spec:
-      # Required. Without this the container runs under runc, libcuda.so is never
-      # injected, and candle-vllm panics in cudarc at startup with
-      # "Unable to dynamically load the cuda shared library".
-      runtimeClassName: nvidia
-      containers:
-        - name: candle
-          image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70-gptq-slim
-          command: ["/usr/local/bin/candle-vllm"]
-          # fp16, not GPTQ, and two ranks, not one. See the header for the
-          # measurements and the TP=2 findings.
-          args:
-            - --w
-            - /mnt/models/coder-14b-fp16
-            - --d
-            - 0,1
-            - --dtype
-            - f16
-            - --p
-            - "8080"
-            - --max-num-seqs
-            - "32"
-            # 11.6 GB of KV per rank, about 59K context tokens. The fp16 weights
-            # leave no room for more.
-            - --kv-fraction
-            - "0.85"
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: "8"
-              memory: 36Gi
-              nvidia.com/gpu: "2"
-            requests:
-              cpu: "4"
-              memory: 12Gi
-              nvidia.com/gpu: "2"
-          volumeMounts:
-            - name: models
-              mountPath: /mnt/models
-              readOnly: true
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llm-models

k8s/apps/llm-serving/coder-service.yaml

@@ -1,40 +0,0 @@
-# Stable client-facing name for the coder chat model, deliberately decoupled
-# from the process that serves it.
-#
-# poimen-memory addresses a Service, not a pod and not an InferenceService. The
-# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
-# else, so the client config never changes again and there is no window in which
-# the host name is wrong.
-#
-# The previous KTML predictor requested two V100s and never reached Ready, so it
-# held half the GPU pool while serving nothing. candle-vllm replaces it.
-#
-# Two ports on purpose:
-#   - 80    matches every other predictor Service in this namespace, and matches
-#           a client base URL that carries no port number.
-#   - 8080  is the container port. Publishing it directly avoids a port
-#           translation that is easy to get wrong in a curl.
-#
-# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
-# client that cached the name during the gap fails until it re-resolves.
-apiVersion: v1
-kind: Service
-metadata:
-  name: coder-predictor
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: coder-predictor
-spec:
-  type: ClusterIP
-  selector:
-    app: candle-vllm
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080
-    - name: http-8080
-      protocol: TCP
-      port: 8080
-      targetPort: 8080

k8s/apps/llm-serving/kustomization.yaml

@@ -13,23 +13,9 @@ resources:
   # clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
   # v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
   # instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
-  # coder-14b-fp16.yaml owns the `candle-vllm` Deployment name and the
-  # `app: candle-vllm` label that coder-service.yaml selects. It must be the only
-  # workload with that label, or the Service splits traffic across two models.
-  #
-  # fp16 on two V100s, not GPTQ on one: measured 39.7 ms ITL and 287 tok/s at 16
-  # streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
-  # tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
-  # the memory controller idles at 19%.
-  # clef-candle.yaml serves clef-flash on candle-vllm (hybrid Qwen3_5, one GPU)
-  # and owns the clef-candle label that clef-serve.yaml's Service now selects.
-  # It must not sync against an image built before the GDN conv fix.
-  - clef-candle.yaml
-  - coder-14b-fp16.yaml
   - clef-serve.yaml
   # Stable client-facing name for the chat model. poimen-memory points at this Service,
   # so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
-  - coder-service.yaml
   - reasoning-long-service.yaml
   # ornith is retired: its InferenceService still reports Ready=True while running zero
   # replicas, because desired equals available at 0. Its manifest was never committed,