From ad2720a724c465e487a7c860fa81effee1ed0d89 Mon Sep 17 00:00:00 2001 From: rock Date: Wed, 28 Oct 2026 10:29:24 -0700 Subject: [PATCH] fix(llm-serving): stop shipping clef-flash, it cannot run on a V100 Clef needs SGLang v0.5.22+, which upstream publishes only on CUDA 13. CUDA 13 dropped Volta kernels and every GPU here is a Tesla V100 (sm_70): torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device SGLang also states sm75 as its floor: the README hardware table omits V100, sm75 support was removed in sgl-project/sglang#9207, and FlashInfer requires sm75+. No build has both Clef and Volta kernels, so the model is not runnable here at any dtype or flag. Remove it from the kustomization so nothing crash-loops, and record the reason in both files. The manifest stays in git for the day an Ampere-or-newer GPU arrives. --- k8s/apps/llm-serving/kustomization.yaml | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/k8s/apps/llm-serving/kustomization.yaml b/k8s/apps/llm-serving/kustomization.yaml index 4d09759..31d77a3 100644 --- a/k8s/apps/llm-serving/kustomization.yaml +++ b/k8s/apps/llm-serving/kustomization.yaml @@ -10,7 +10,22 @@ kind: Kustomization # way after any edit here: a GPU predictor restart is a weights reload measured # in tens of seconds, not a rolling update. resources: - - clef-flash.yaml + # clef-flash.yaml is NOT listed: the model cannot run on this cluster. + # + # Clef needs SGLang v0.5.22 or newer, and SGLang publishes that only on CUDA 13 + # (the last CUDA 12 tag is v0.5.19-cu129, which predates Clef support). CUDA 13 + # dropped Volta kernels, and every GPU here is a Tesla V100 (sm_70): + # + # torch.AcceleratorError: CUDA error: no kernel image is available for + # execution on the device + # + # SGLang also states sm75 as its floor: the README hardware list omits V100, + # sm75 support was removed in sgl-project/sglang#9207, and FlashInfer needs + # sm75+. So no SGLang build exists that has both Clef and Volta kernels, and + # no flag or dtype change bridges that gap. + # + # Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is + # kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning. # ornith and reasoning are live but unmanaged: their manifests were never committed # to this directory, so listing them made `kustomize build` fail and left the whole # Application in ComparisonError. Remove the entries rather than the models. The