feat(llm-serving): manage clef-serve in git, generated from app.py

The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.

Three things move into git:
  - clef-serve.yaml: the Deployment and the Service the gateway routes to
  - configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
    instead of leaving a stale copy mounted
  - clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
    V100, and its slot is now clef-serve.yaml

Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.

Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
This commit is contained in:
2026-10-29 23:23:03 -07:00
parent 3eb6a91311
commit 41dcda819e
7 changed files with 525 additions and 184 deletions
+14 -16
View File
@@ -10,22 +10,10 @@ kind: Kustomization
# way after any edit here: a GPU predictor restart is a weights reload measured
# in tens of seconds, not a rolling update.
resources:
# clef-flash.yaml is NOT listed: the model cannot run on this cluster.
#
# Clef needs SGLang v0.5.22 or newer, and SGLang publishes that only on CUDA 13
# (the last CUDA 12 tag is v0.5.19-cu129, which predates Clef support). CUDA 13
# dropped Volta kernels, and every GPU here is a Tesla V100 (sm_70):
#
# torch.AcceleratorError: CUDA error: no kernel image is available for
# execution on the device
#
# SGLang also states sm75 as its floor: the README hardware list omits V100,
# sm75 support was removed in sgl-project/sglang#9207, and FlashInfer needs
# sm75+. So no SGLang build exists that has both Clef and Volta kernels, and
# no flag or dtype change bridges that gap.
#
# Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is
# kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning.
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
- clef-serve.yaml
- reasoning.yaml
# ornith is retired: its InferenceService still reports Ready=True while running zero
# replicas, because desired equals available at 0. Its manifest was never committed,
@@ -36,3 +24,13 @@ resources:
- networkpolicy.yaml
# No namespace transformer: every file sets its own, and the transformer would
# rewrite metadata.namespace on anything cross-namespace added later.
# app.py lives in git next to this file. The generator hashes the ConfigMap name, so an
# edit to app.py rolls the pod instead of leaving a stale copy mounted.
configMapGenerator:
- name: clef-serve-app
namespace: llm-serving
files:
- clef-serve/app.py
options:
disableNameSuffixHash: false