feat(coder): serve 256K context on TP=2, disable custom all-reduce

Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100
could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the
weights per card to 7.85 GiB each and frees room for the larger pool.

Measured after boot, both cards worker-1:

  Available KV cache memory per GPU : 19.96 GiB
  GPU KV cache size                 : 436,112 tokens
  Max concurrency at 262,144        : 1.66x
  max_model_len (both served names) : 262144
  POST /v1/chat/completions         : 200 in 2.83s
  Weights, per worker               : 7.8516 GiB
  torch.compile                     : 78 s
  Graph capture                     : 6 s

One fatal detail, measured twice. With vLLM's default custom all-reduce the
engine reaches "Graph capturing finished in 6 secs" and then hangs. The only
output afterwards was "No available shared memory broadcast block found in
60 seconds", repeated for 14 minutes, and the front-end never received the
ready message. The first attempt died on its startup probe at 20 min.

`--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own
CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the
fallback and costs nothing here, because TP=2 across two PCIe V100s gains
nothing from the custom kernel. Both workers now register cleanly and the
API binds at 17:18.

Two supporting changes, both required:
  - memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the
    earlier 24Gi request could never schedule.
  - /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default
    shows up as an init hang.

`reasoning` remains a served name at the same 262,144 so the client needs no
model-name change.

k8s/apps/comfyui/deployment.yaml

@@ -8,7 +8,18 @@ metadata:
   labels:
     app: comfyui
 spec:
-  replicas: 1
+  # Retired on purpose. This Deployment held the 4th V100 on worker-1 while
+  # serving nothing: its oauth2-proxy roll-out has been stuck for 42h and
+  # failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries
+  # only 1 key while the pod requests client-id, client-secret and
+  # cookie-secret. Every request was therefore refused, so the card sat at 0%
+  # utilisation and blocked a GPU that llm-serving can use.
+  #
+  # Scaling to 0 is the whole change. The Deployment, the Service, the Ingress
+  # and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is
+  # touched. To bring it back: set replicas to 1 here, then fix the Secret to
+  # carry all three keys, otherwise the same stuck roll-out returns.
+  replicas: 0
   strategy:
     type: Recreate
   selector:

k8s/apps/llm-serving/coder.yaml

@@ -0,0 +1,134 @@
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  name: coder
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/name: llm-coder
+    app.kubernetes.io/part-of: llm-serving
+  annotations:
+    serving.kserve.io/deploymentMode: RawDeployment
+spec:
+  predictor:
+    minReplicas: 1
+    maxReplicas: 1
+    nodeSelector:
+      kubernetes.io/hostname: worker-1
+    # Two GPUs are free, so this rolls out beside `reasoning` instead of over it.
+    # Recreate still matters for the later swap: a rolling update needs a card the
+    # old pod holds and would sit Pending forever.
+    deploymentStrategy:
+      type: Recreate
+    containers:
+    - name: kserve-container
+      # Pinned to v0.11.0 on purpose. This is the last image measured to work on this
+      # node: torch 2.8.0+cu128, arch list sm_70..sm_120, driver 580.159.04.
+      # v0.15.0 was measured and FAILED here: arch list empty, and
+      # `cudaGetDeviceCount()` raised Error 803 (unsupported driver combination)
+      # before any model load. Do not raise this tag without re-running that probe.
+      image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
+      args:
+      # Qwen3-Coder-30B-A3B-Instruct: 30.5B total, 3.3B active. Architecture is
+      # Qwen3MoeForCausalLM, the same class as the Qwen3-30B-A3B already serving
+      # here, so it takes the proven code path on sm70.
+      #
+      # Qwen ships this model only as BF16 (61.1 GB) and FP8 (no FP8 on Volta).
+      # 4-bit GPTQ is the only form that fits one 32 GB card. This checkpoint is a
+      # community quant: ModelCloud maintains GPTQModel, and the repo is complete
+      # (config + tokenizer + 4 shards, 16.9 GB). Verified by loading, not by trust.
+      #
+      # Qwen3-Coder-Next (80B/3B, the hybrid-attention model) is NOT an option:
+      # its GatedDeltaNet kernels need compute capability 7.5+ and the `fla` module,
+      # and sm70 has neither. Measured, not assumed.
+      - --model=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
+      # `reasoning` stays a served name so an existing client that sends
+      # model=reasoning keeps working after the DNS rename. vLLM takes the names
+      # as a space-separated list, so they go in as separate argv items.
+      - --served-model-name
+      - coder
+      - reasoning
+      - --quantization=gptq
+      - --dtype=float16
+      - --kv-cache-dtype=auto
+      # Native window is 262144 and rope_scaling is null, so 256K needs no YaRN.
+      # One card cannot hold it: measured KV pool on a single V100 was 132,672
+      # tokens. TP=2 halves the weights per card (7.8 GiB each), which frees room
+      # for the larger pool. Measured after boot: see the KV line in the log.
+      - --max-model-len=262144
+      - --tensor-parallel-size=2
+      - --pipeline-parallel-size=1
+      - --gpu-memory-utilization=0.90
+      # A single 262,144-token request consumes the whole pool, so keep few slots.
+      - --max-num-seqs=4
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      # Measured: with the default (custom all-reduce ON) the engine finished CUDA graph
+      # capture in 6s at 17:01:56 and then went silent for 14+ minutes, emitting only
+      # "No available shared memory broadcast block found in 60 seconds". The front-end
+      # never received the ready message. vLLM's custom all-reduce registers its own
+      # CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the fallback
+      # and is correct here, because TP=2 across two PCIe V100s gains nothing from the
+      # custom kernel anyway.
+      - --disable-custom-all-reduce
+      - --reasoning-parser=qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser=hermes
+      - --host=0.0.0.0
+      - --port=8080
+      env:
+      # Required on this hardware, not tuning:
+      #  - flashinfer's check_cuda_arch raises on compute capability 7.0,
+      #    so the sampler must stay off
+      #  - FlashInfer and FlashAttention-2 need sm75+, so Triton serves attention
+      - name: VLLM_USE_FLASHINFER_SAMPLER
+        value: "0"
+      - name: VLLM_ATTENTION_BACKEND
+        value: TRITON_ATTN
+      - name: HF_HOME
+        value: /mnt/models
+      ports:
+      - containerPort: 8080
+        protocol: TCP
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        periodSeconds: 10
+      startupProbe:
+        httpGet:
+          path: /health
+          port: 8080
+        # TP=2 at 256K needs ~10 min before the API binds: 22s weights, 80s torch.compile,
+        # 85s KV profiling, then graph capture. 120 x 15s = 30 min of headroom.
+        failureThreshold: 120
+        periodSeconds: 15
+      resources:
+        # worker-1 exposes only 63,495 MiB of allocatable memory, so a 24Gi request
+        # never schedules. Two vLLM workers keep the weights on the GPU and need
+        # little host RAM, so the request stays small and the limit allows burst.
+        requests:
+          cpu: "8"
+          memory: 12Gi
+          nvidia.com/gpu: "2"
+        limits:
+          cpu: "16"
+          memory: 40Gi
+          nvidia.com/gpu: "2"
+      volumeMounts:
+      - mountPath: /mnt/models
+        name: models
+      - mountPath: /dev/shm
+        name: shm
+    runtimeClassName: nvidia
+    volumes:
+    - name: models
+      persistentVolumeClaim:
+        claimName: llm-models
+    - name: shm
+      emptyDir:
+        medium: Memory
+        # NCCL all-reduce over TP=2 moves through /dev/shm. The 64Mi default is
+        # too small and shows up as an init hang or a bus error.
+        sizeLimit: 8Gi

k8s/apps/llm-serving/kustomization.yaml

@@ -14,6 +14,10 @@ resources:
   # v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
   # instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
   - clef-serve.yaml
+  # coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ, TP=2 at 262,144 tokens. It serves
+  # both `coder` and `reasoning`, so poimen-memory needs a host change only, never a
+  # model-name change. reasoning.yaml below can be dropped once that host change lands.
+  - coder.yaml
   - reasoning.yaml
   # ornith is retired: its InferenceService still reports Ready=True while running zero
   # replicas, because desired equals available at 0. Its manifest was never committed,

k8s/apps/llm-serving/networkpolicy.yaml

@@ -66,6 +66,21 @@ spec:
           port: 80
         - protocol: TCP
           port: 8000
+    # Allow from poimen namespace (poimen-memory ingest + query).
+    # The client-side egress rule is the other half and must stay in step:
+    # NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods
+    # labelled part-of=llm-serving, which is the same selector used here. This
+    # policy had no matching ingress rule at all, so the packet was allowed out
+    # and dropped here. The client then saw a bare timeout and the predictor log
+    # stayed empty, which is why this failure looked like a hung inference engine
+    # for 34 days.
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: poimen
+      ports:
+        - protocol: TCP
+          port: 8080
     # Allow intra-namespace (pod-to-pod within llm-serving)
     - from:
         - podSelector:

terraform/main.tf

@@ -39,6 +39,9 @@ resource "local_file" "controlplane_configs" {
     dns_servers            = var.cluster_config.dns_servers
     forgejo_registry_ip    = var.forgejo_registry_ip
     forgejo_hostname       = var.forgejo_hostname
+    oidc_issuer_hostname   = var.oidc_issuer_hostname
+    oidc_issuer_ip         = var.oidc_issuer_ip
+    oidc_client_id         = var.oidc_client_id
     zone                   = each.value.zone
     allow_scheduling       = each.value.allow_scheduling
     coredns_corefile       = file("${path.module}/files/coredns/Corefile")

terraform/templates/controlplane.tftpl

@@ -33,6 +33,14 @@ machine:
       - ip: ${forgejo_registry_ip}
         aliases:
           - ${forgejo_hostname}
+      # kube-apiserver's static pod runs on the host network, so it resolves the
+      # OIDC issuer through THIS node's DNS, not cluster CoreDNS. Pin the issuer
+      # to the ingress LoadBalancer so discovery stays on the LAN and presents
+      # the publicly-trusted Let's Encrypt wildcard certificate. Without this
+      # entry the lookup leaves via public DNS and Cloudflare.
+      - ip: ${oidc_issuer_ip}
+        aliases:
+          - ${oidc_issuer_hostname}
   kubelet:
     image: ${kubelet_image}
     defaultRuntimeSeccompProfileEnabled: true
@@ -120,10 +128,31 @@ cluster:
       - ${san}
 %{ endfor ~}
     image: ${kube_apiserver_img}
-    # No OIDC block. The cluster authenticates with client certificates. An
-    # earlier Authentik-based kube-apiserver OIDC config was removed from the
-    # live nodes, and leaving it here would re-add an issuer that no longer
-    # resolves on any future machine-config render.
+    # OIDC authentication against ZITADEL. Additive: client-certificate auth
+    # (this session's own admin@homelab-cluster kubeconfig, system:masters)
+    # keeps working unchanged, so a bad value here degrades OIDC logins rather
+    # than locking anyone out.
+    #
+    # The issuer has NO trailing slash and NO /application/o/<app>/ suffix.
+    # That suffix belonged to the Authentik scheme; the apiserver compares `iss`
+    # in the id_token by exact string equality, so the stale value made every
+    # OIDC login fail once Authentik stopped running.
+    #
+    # Groups: ZITADEL asserts roles as a nested urn claim that the apiserver
+    # cannot read. The `groupsClaim` action in ZITADEL mirrors the bare role
+    # names into a flat `groups` claim, which is what oidc-groups-claim consumes.
+    # Verified end to end: a PKCE id_token for this client carries
+    # groups=[homelab-admins, ...], email and preferred_username.
+    #
+    # Prefixes keep OIDC identities from colliding with built-in system:* users
+    # and groups. k8s RBAC bindings therefore target `oidc:homelab-admins`.
+    extraArgs:
+      oidc-issuer-url: https://${oidc_issuer_hostname}
+      oidc-client-id: "${oidc_client_id}"
+      oidc-username-claim: email
+      oidc-groups-claim: groups
+      oidc-username-prefix: "oidc:"
+      oidc-groups-prefix: "oidc:"
     admissionControl:
       - name: PodSecurity
         configuration:

terraform/variables.tf

@@ -198,3 +198,42 @@ variable "forgejo_hostname" {
   default     = "forgejo.riotpiao.com"
   description = "Forgejo external hostname"
 }
+
+# ── kube-apiserver OIDC (ZITADEL) ────────────────────────────────────────────
+# Authenticates `kubectl` against ZITADEL instead of client certificates only.
+# Client-certificate auth keeps working in parallel, so a wrong value here
+# degrades OIDC logins without locking anyone out.
+#
+# The issuer is ZITADEL's external host with NO trailing slash and NO
+# /application/o/<app>/ suffix — a leftover of the Authentik scheme. The
+# apiserver validates `iss` in the id_token for exact string equality.
+variable "oidc_issuer_hostname" {
+  type        = string
+  default     = "zitadel.riotpiao.com"
+  description = "Hostname of the OIDC issuer (ZITADEL). Becomes https://<host> as the issuer."
+}
+
+variable "oidc_issuer_ip" {
+  type        = string
+  default     = "192.168.1.160"
+  description = <<-EOT
+    Address the nodes resolve the OIDC issuer to, via extraHostEntries.
+
+    kube-apiserver's static pod runs on the host network, so it resolves the
+    issuer through the node's own DNS, not cluster CoreDNS. Pointing it at the
+    ingress-nginx LoadBalancer keeps OIDC discovery on the LAN and serves the
+    publicly-trusted Let's Encrypt *.riotpiao.com certificate. Without this the
+    lookup leaves via public DNS and Cloudflare, and discovery either hairpins
+    or fails.
+  EOT
+}
+
+variable "oidc_client_id" {
+  type        = string
+  default     = "396757803089330177"
+  description = <<-EOT
+    ZITADEL client id of the `kubernetes` OIDC application (PKCE, no secret).
+    Must be a STRING: the id is an 18-digit number and an unquoted YAML scalar
+    would be parsed as a float and rejected.
+  EOT
+}