revert(cloudflared): do not publish the kube-apiserver at k8s.riotpiao.com

#88 added a tunnel rule mapping k8s.riotpiao.com to the in-cluster apiserver.
Its own comment warned that without a Cloudflare Access application in front,
every scanner on the internet reaches the apiserver, and no Access
application exists.

ArgoCD applied the rule before the exposure could be stopped, so the live
ConfigMap was patched and cloudflared restarted to drop it. All paths now
return 404. This commit removes the rule from the source so it cannot return.

The hostname keeps its DNS record and returns the tunnel's 404 fallback. That
is deliberate: publish it again only together with an Access policy.

k8s/apps/llm-serving/clef-candle.yaml

@@ -1,84 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: clef-candle
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: clef-candle
-spec:
-  replicas: 1
-  strategy:
-    # One replica holds one GPU, so Recreate keeps the rollout to a single
-    # weights load instead of needing a second card for a surge pod.
-    type: Recreate
-  selector:
-    matchLabels:
-      app: clef-candle
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/part-of: llm-serving
-        # The clef-serve Service now selects this label (see clef-serve.yaml),
-        # so clients keep addressing clef-serve:80 unchanged. The reference
-        # pytorch Deployment must stay at replicas 0 while this runs, or the
-        # Service splits traffic across two different implementations.
-        app: clef-candle
-    spec:
-      # Required. Without the nvidia runtime class libcuda.so is never injected
-      # and candle-vllm panics in cudarc at startup.
-      runtimeClassName: nvidia
-      containers:
-        - name: candle
-          # :sm70 is built by .gitea/workflows/candle-vllm-sm70.yaml on every
-          # merge to main. It MUST be a build that contains the GDN conv fix
-          # (candle-vllm PR: feat/clef-systemone). Older :sm70 images zero the
-          # key channel band in causal_conv1d_fwd, which starves the delta rule
-          # and makes every linear-attention layer return zero. The symptom is
-          # a well-formed /v1/systemone response with wrong answers, not an error.
-          image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70
-          # :sm70 is a moving tag. Default IfNotPresent keeps a cached copy and a
-          # rebuilt image never reaches the pod, so a merged fix looks like it did
-          # not land. Always forces the pull on every pod start.
-          imagePullPolicy: Always
-          command: ["/usr/local/bin/candle-vllm"]
-          # clef-flash is a hybrid Qwen3_5 model: 8 full-attention layers plus 24
-          # GDN layers that carry per-sequence recurrent state. It runs on ONE
-          # card because the state is not shardable the way attention KV is.
-          args:
-            - --w
-            - /mnt/models/clef-flash
-            - --d
-            - "0"
-            - --dtype
-            - f16
-            - --p
-            - "8080"
-            - --max-num-seqs
-            - "4"
-            # About 66.8K KV tokens. Clef is a single-shot classifier-style
-            # endpoint, so it does not need the coder's pool.
-            - --kv-fraction
-            - "0.30"
-            # Prefix caching stays on: clef requests repeat the schema block.
-            - --prefill-chunk-size
-            - "1024"
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: "8"
-              memory: 32Gi
-              nvidia.com/gpu: "1"
-            requests:
-              cpu: "4"
-              memory: 16Gi
-              nvidia.com/gpu: "1"
-          volumeMounts:
-            - name: models
-              mountPath: /mnt/models
-              readOnly: true
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llm-models

k8s/apps/llm-serving/clef-serve.yaml

@@ -33,17 +33,19 @@ metadata:
     app: clef-serve
     app.kubernetes.io/part-of: llm-serving
 spec:
-  # One card. worker-1 holds four V100s: candle-vllm takes one for the coder, and
-  # this pod takes one, so two stay free for a TP=2 candle run.
+  # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
+  # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
+  # this 1) and a new GPU workload had nowhere to land.
   #
-  # Enabled 2026-11-04 to serve /v1/systemone, which the gateway already routes to
-  # this Service. It was parked at 0 while candle-vllm was brought up on the same
-  # cards, and it served no inference request between 2026-10-30T06:23:40Z and then.
+  # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
+  # the first card released: the standing instruction is to keep the vLLM pods healthy
+  # until candle-vllm answers, and this pod is not one of them.
   #
-  # Cold start is about four minutes: the pod pip-installs on every boot because the
-  # torch base image cannot be pushed to the registry (Cloudflare answers 413 above
-  # 100 MB per layer). startupProbe allows for that.
-  replicas: 1
+  # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
+  # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
+  # in git selects this service. Restore it once the rollout leaves a card free:
+  # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
+  replicas: 0
   # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
   # the new pod Pending forever waiting for a card the old pod still holds.
   strategy:
@@ -127,11 +129,8 @@ metadata:
   labels:
     app.kubernetes.io/part-of: llm-serving
 spec:
-  # Points at the candle-vllm implementation (clef-candle.yaml). Switching back
-  # to the reference pytorch server is this line plus that Deployment's replicas;
-  # no client changes, because the Service name and ClusterIP do not move.
   selector:
-    app: clef-candle
+    app: clef-serve
   ports:
   - name: http
     port: 80

k8s/apps/llm-serving/coder-14b-fp16.yaml

@@ -1,106 +0,0 @@
-# Qwen2.5-Coder-14B-Instruct, fp16, tensor-parallel across two V100s.
-#
-# Why this model and this precision. The previous workload was Qwen3-Coder-30B-A3B
-# in GPTQ W4A16 on one V100. Profiling that deployment showed SM utilisation at
-# 98% while the memory controller sat at 19%: the V100 has FP16 tensor cores and
-# no INT4 ones, so every decode step ran a dequantisation pass on the SM. That is
-# a hardware-generation mismatch, not a tunable. Sequence throughput matches the
-# arithmetic: at batch 1 the step cost equalled a full-model read (17.6 ms against
-# 15 GB / 900 GB/s = 16.7 ms) while the routed experts alone are ~1.5 GB.
-#
-# Measured on this hardware, same harness, same 16-stream load:
-#
-#   GPTQ W4A16 30B-A3B, TP=1  : ITL 71.5 ms, 215 tok/s aggregate
-#   fp16 14B, TP=2            : ITL 39.7 ms, 287 tok/s aggregate
-#
-# The dense model degrades only 1.4x from 1 to 16 streams (27.9 -> 39.7 ms) while
-# the MoE degrades 4.1x (17.6 -> 71.5 ms), because a dense model reads its weights
-# once per step and amortises across the batch. The MoE pays expert weight traffic
-# per sequence. Single-stream latency is the one case where the MoE still wins
-# (17.6 ms against 27.9 ms), since it activates ~3B parameters per token.
-#
-# TP=2 notes that cost time to establish, so they are recorded here:
-#   - Sharding needs no code change. Every dimension divides by 2: attention heads
-#     40 -> 20, KV heads 8 -> 4, intermediate_size 13824 -> 6912, vocab
-#     152064 -> 76032. Verify this first for any replacement model.
-#   - TP=2 on a GPTQ checkpoint is what crashed (CUDA_ERROR_ILLEGAL_ADDRESS in
-#     forward()) on the previous model. The cause was quantised weight sharding
-#     and the g_idx guard in src/openai/models/linear.rs, not TP itself. A dense
-#     fp16 checkpoint has no g_idx, no packed weights and no scales, so it takes
-#     the plain column/row-parallel path and the crash does not reproduce.
-#   - These are V100-PCIE cards with no NVLink, so every layer's all-reduce
-#     crosses PCIe. It shows up in TTFT (275 ms at 1 stream, 1090 ms at 16)
-#     more than in ITL.
-#   - KV capacity is 11635 MB per rank, about 59K context tokens. The fp16
-#     weights consume the room the KV pool would otherwise use. This is well
-#     short of the 256K the previous model targeted, and it is the price of
-#     leaving the dequantisation path.
-#
-# Weights must be in the PVC as /mnt/models/coder-14b-fp16 (28 GB, 6 shards).
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: candle-vllm
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: candle-vllm
-spec:
-  replicas: 1
-  strategy:
-    # One replica holds both GPUs, so a rolling update would need a second pair.
-    # Recreate keeps the rollout to a single weights load.
-    type: Recreate
-  selector:
-    matchLabels:
-      app: candle-vllm
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/part-of: llm-serving
-        app: candle-vllm
-    spec:
-      # Required. Without this the container runs under runc, libcuda.so is never
-      # injected, and candle-vllm panics in cudarc at startup with
-      # "Unable to dynamically load the cuda shared library".
-      runtimeClassName: nvidia
-      containers:
-        - name: candle
-          image: forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70-gptq-slim
-          command: ["/usr/local/bin/candle-vllm"]
-          # fp16, not GPTQ, and two ranks, not one. See the header for the
-          # measurements and the TP=2 findings.
-          args:
-            - --w
-            - /mnt/models/coder-14b-fp16
-            - --d
-            - 0,1
-            - --dtype
-            - f16
-            - --p
-            - "8080"
-            - --max-num-seqs
-            - "32"
-            # 11.6 GB of KV per rank, about 59K context tokens. The fp16 weights
-            # leave no room for more.
-            - --kv-fraction
-            - "0.85"
-          ports:
-            - containerPort: 8080
-          resources:
-            limits:
-              cpu: "8"
-              memory: 36Gi
-              nvidia.com/gpu: "2"
-            requests:
-              cpu: "4"
-              memory: 12Gi
-              nvidia.com/gpu: "2"
-          volumeMounts:
-            - name: models
-              mountPath: /mnt/models
-              readOnly: true
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llm-models

k8s/apps/llm-serving/coder-service.yaml

@@ -1,40 +0,0 @@
-# Stable client-facing name for the coder chat model, deliberately decoupled
-# from the process that serves it.
-#
-# poimen-memory addresses a Service, not a pod and not an InferenceService. The
-# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
-# else, so the client config never changes again and there is no window in which
-# the host name is wrong.
-#
-# The previous KTML predictor requested two V100s and never reached Ready, so it
-# held half the GPU pool while serving nothing. candle-vllm replaces it.
-#
-# Two ports on purpose:
-#   - 80    matches every other predictor Service in this namespace, and matches
-#           a client base URL that carries no port number.
-#   - 8080  is the container port. Publishing it directly avoids a port
-#           translation that is easy to get wrong in a curl.
-#
-# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
-# client that cached the name during the gap fails until it re-resolves.
-apiVersion: v1
-kind: Service
-metadata:
-  name: coder-predictor
-  namespace: llm-serving
-  labels:
-    app.kubernetes.io/part-of: llm-serving
-    app: coder-predictor
-spec:
-  type: ClusterIP
-  selector:
-    app: candle-vllm
-  ports:
-    - name: http
-      protocol: TCP
-      port: 80
-      targetPort: 8080
-    - name: http-8080
-      protocol: TCP
-      port: 8080
-      targetPort: 8080

k8s/apps/llm-serving/kustomization.yaml

@@ -13,23 +13,14 @@ resources:
   # clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
   # v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
   # instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
-  # coder-14b-fp16.yaml owns the `candle-vllm` Deployment name and the
-  # `app: candle-vllm` label that coder-service.yaml selects. It must be the only
-  # workload with that label, or the Service splits traffic across two models.
-  #
-  # fp16 on two V100s, not GPTQ on one: measured 39.7 ms ITL and 287 tok/s at 16
-  # streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
-  # tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
-  # the memory controller idles at 19%.
-  # clef-candle.yaml serves clef-flash on candle-vllm (hybrid Qwen3_5, one GPU)
-  # and owns the clef-candle label that clef-serve.yaml's Service now selects.
-  # It must not sync against an image built before the GDN conv fix.
-  - clef-candle.yaml
-  - coder-14b-fp16.yaml
   - clef-serve.yaml
+  # coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ, TP=2 at 262,144 tokens. It serves
+  # both `coder` and `reasoning`, so poimen-memory needs a host change only, never a
+  # model-name change. reasoning.yaml below can be dropped once that host change lands.
+  - coder.yaml
+  - reasoning.yaml
   # Stable client-facing name for the chat model. poimen-memory points at this Service,
   # so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
-  - coder-service.yaml
   - reasoning-long-service.yaml
   # ornith is retired: its InferenceService still reports Ready=True while running zero
   # replicas, because desired equals available at 0. Its manifest was never committed,