feat(llm-serving): manage clef-serve in git, generated from app.py
The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.
Three things move into git:
- clef-serve.yaml: the Deployment and the Service the gateway routes to
- configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
instead of leaving a stale copy mounted
- clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
V100, and its slot is now clef-serve.yaml
Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.
Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
This commit is contained in:
@@ -1,168 +0,0 @@
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
annotations:
|
||||
serving.kserve.io/deploymentMode: RawDeployment
|
||||
# KServe injects the prometheus.io/{scrape,port,path} annotations into the
|
||||
# generated pod only when this flag is on. SGLang exposes engine metrics on
|
||||
# the same port as its API, so this is what makes them visible at all.
|
||||
serving.kserve.io/enable-prometheus-scraping: "true"
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-clef-flash
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
name: clef-flash
|
||||
namespace: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
containers:
|
||||
- args:
|
||||
# Clef-flash: Cloudflare's 9.4B decision model, Apache-2.0, self-hosted.
|
||||
#
|
||||
# This is NOT a chat model and it does not replace `reasoning`. It reads
|
||||
# a state plus a schema of typed questions and returns a probability per
|
||||
# allowed option in a single prefill pass. There is no decode step and no
|
||||
# free-form text output. Consumers call POST /v1/systemone, not
|
||||
# /v1/chat/completions.
|
||||
#
|
||||
# Why SGLang and not vLLM: Clef needs the separate joint_schema_head
|
||||
# weights (joint_head.safetensors) loaded on top of the Qwen3.5-9B
|
||||
# backbone, and it needs the /v1/systemone route. Only SGLang implements
|
||||
# these (upstream PR #42721, released in v0.5.22). vLLM has a qwen3_5
|
||||
# model class but would ignore the joint head and expose no systemone
|
||||
# endpoint, so it cannot serve this model correctly.
|
||||
#
|
||||
# BLOCKED ON HARDWARE. worker-1 holds Tesla V100-PCIE-32GB cards, compute
|
||||
# capability 7.0. The pinned image ships torch 2.14.1+cu130, whose kernels cover
|
||||
# CC 8.0 and above only:
|
||||
#
|
||||
# Found GPU0 Tesla V100-PCIE-32GB which is of cuda capability 7.0.
|
||||
# Your installed torch==2.14.1+cu130 does not include kernels for this GPU.
|
||||
#
|
||||
# So the pod starts and then dies in the model loader. minReplicas is 0 until
|
||||
# either a build with sm_70 kernels exists or the model moves to newer hardware;
|
||||
# serving it on CPU is not viable for a 9.4B backbone.
|
||||
#
|
||||
# Image: the `dev-clef` tag is a pinned nightly because Clef support
|
||||
# landed after v0.5.21. Move to the official v0.5.22+ tag and re-pin by
|
||||
# digest when it ships; do not leave a nightly tag in git long term.
|
||||
#
|
||||
# Context: 24576. This is deliberate and it is NOT raised to 128K.
|
||||
# - 24576 is exactly what Cloudflare serves for clef-flash on Workers
|
||||
# AI, so self-hosted and hosted answers match.
|
||||
# - The SGLang cookbook defaults to 16384 and its published accuracy
|
||||
# runs stop near 16K. Nothing public validates quality above that.
|
||||
# - Cloudflare cut their hosted window from 64K to 24K on purpose and
|
||||
# points long-context users at the 27B `clef` instead.
|
||||
# - Latency, not memory, is the real cap: Clef has no decode, so cost
|
||||
# scales with input tokens. On a V100 without flash attention, 128K
|
||||
# would push a single decision into the tens of seconds, which
|
||||
# defeats the point of a classifier.
|
||||
#
|
||||
# KV cache math, from config.json: 32 layers, but only 8 use full
|
||||
# attention (every 4th). 8 * 2 * 4 kv_heads * 256 head_dim * 2 B
|
||||
# = 32 KB/token. At 24576 tokens that is only 0.75 GiB. The other 24
|
||||
# layers use linear attention and hold a fixed 48 MiB state regardless
|
||||
# of length. Memory is therefore dominated by the BF16 weights (18.8 GB),
|
||||
# not the cache, which is why one V100-32GB is enough.
|
||||
# The image ships its CLI at /opt/sglang/bin/sglang and puts no `serve` on
|
||||
# PATH, so a bare `serve` failed with "exec: serve: not found" on every start
|
||||
# (19 restarts). Name the binary by its full path.
|
||||
- /opt/sglang/bin/sglang
|
||||
- serve
|
||||
- --model-path=Cloudflare/clef-flash
|
||||
- --host=0.0.0.0
|
||||
- --port=8080
|
||||
- --context-length=24576
|
||||
- --tp-size=1
|
||||
# Leave headroom for the vision encoder and the joint head. The model
|
||||
# reports 41 GB at a 64K window on H200 including engine overhead; at
|
||||
# 24K on a 32 GB card this keeps the static pool bounded.
|
||||
- --mem-fraction-static=0.80
|
||||
# Clef runs a single prefill and disables radix cache, chunked prefill
|
||||
# and CUDA graphs itself. Set them explicitly so a future default change
|
||||
# cannot silently re-enable them.
|
||||
- --disable-radix-cache
|
||||
# Triton is the only attention backend with a chance on sm70. FlashInfer
|
||||
# requires sm75+ and FA3 requires sm90. SGLang's install guide states the
|
||||
# pairing explicitly: on backends below sm75 you must set
|
||||
# --attention-backend triton --sampling-backend pytorch
|
||||
# Both flags are required. Triton alone leaves the sampler on a FlashInfer
|
||||
# kernel, which fails on Volta.
|
||||
#
|
||||
# This is still the main untested assumption here: Cloudflare validated
|
||||
# the dev-clef image on H200, B200 and B300 only. SGLang upstream also
|
||||
# warns that Triton >=3.3.0 has a Dot() bug for compute capability < 80.
|
||||
# If the pod crashes, check the Triton version in the image first, then
|
||||
# fall back to the sglang-V100 fork, which ships purpose-built SM70
|
||||
# kernels.
|
||||
- --attention-backend=triton
|
||||
- --sampling-backend=pytorch
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /mnt/models
|
||||
- name: HF_HUB_CACHE
|
||||
value: /mnt/models/hub
|
||||
image: lmsysorg/sglang:dev-clef
|
||||
name: kserve-container
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 32Gi
|
||||
nvidia.com/gpu: '1'
|
||||
requests:
|
||||
cpu: '8'
|
||||
# 8Gi, not 16Gi. worker-1 holds 64Gi and the other predictors already
|
||||
# request about 55Gi, so a 16Gi request left the pod Pending forever:
|
||||
# "1 Insufficient memory". The weights live on the GPU, so host memory is
|
||||
# for SGLang's own bookkeeping. The limit stays at 32Gi, so a burst that
|
||||
# needs more is allowed; only the scheduler reservation is smaller.
|
||||
memory: 8Gi
|
||||
nvidia.com/gpu: '1'
|
||||
startupProbe:
|
||||
# The image is large and the checkpoint is 18.8 GB on first pull.
|
||||
# 120 * 15s = 30 min ceiling for the initial download.
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 15
|
||||
volumeMounts:
|
||||
- mountPath: /mnt/models
|
||||
name: models
|
||||
- mountPath: /dev/shm
|
||||
name: shm
|
||||
# podMetadata is NOT valid on this KServe version; annotations and labels
|
||||
# sit directly under predictor. See the note in qwen3-moe.yaml.
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8080"
|
||||
prometheus.io/path: "/metrics"
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-clef-flash
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
deploymentStrategy:
|
||||
type: Recreate
|
||||
maxReplicas: 1
|
||||
# 0 until the hardware or the build changes: see the hardware note above.
|
||||
minReplicas: 0
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
runtimeClassName: nvidia
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
# SGLang needs a large shared-memory segment for the tokenizer and
|
||||
# message-queue processes. The upstream run command uses --shm-size 32g.
|
||||
sizeLimit: 8Gi
|
||||
name: shm
|
||||
@@ -0,0 +1,125 @@
|
||||
# clef-flash: a decision model, not a chat model.
|
||||
#
|
||||
# It reads a `state` plus a schema of typed `questions` and returns one probability
|
||||
# per allowed option in a single prefill pass. There is no decoding, no sampling and
|
||||
# no free-form text. The public route is /v1/systemone, and its request and response
|
||||
# shapes are Clef's own: no `messages`, no `choices`.
|
||||
#
|
||||
# Why not SGLang: Clef support landed in SGLang v0.5.22, which upstream publishes only
|
||||
# on CUDA 13. CUDA 13 dropped Volta kernels and every card here is a Tesla V100
|
||||
# (compute capability 7.0), so SGLang aborts with "no kernel image is available for
|
||||
# execution on the device". SGLang also states sm75 as its floor regardless.
|
||||
#
|
||||
# So this runs the implementation the model ships: joint_schema_model.py, which has
|
||||
# load_release_model, encode_record, collate_records and systemone. The weights are
|
||||
# 17.75 GiB of BF16, and fp16 costs the same bytes per parameter. On one 32 GiB card
|
||||
# that leaves room for a 32k state, which is why this is one GPU and not two: with
|
||||
# device_map over two cards the embedding table and the tensors indexing into it land
|
||||
# on different devices, and the forward pass dies with
|
||||
# "indices should be either on cpu or on the same device as the indexed tensor".
|
||||
#
|
||||
# Known gap: the image cannot be pushed to the registry. Cloudflare fronts
|
||||
# forgejo.riotpiao.com and answers 413 Payload Too Large for any layer above 100 MB,
|
||||
# and the torch base alone is 7.58 GB. So this mounts app.py from a ConfigMap and pip
|
||||
# installs on every start. A cold start therefore needs PyPI reachable, and takes
|
||||
# about four minutes. Build the image in-cluster, or unproxy the registry host, before
|
||||
# treating this as production.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: clef-serve
|
||||
namespace: llm-serving
|
||||
labels:
|
||||
app: clef-serve
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
replicas: 1
|
||||
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
|
||||
# the new pod Pending forever waiting for a card the old pod still holds.
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: clef-serve
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: clef-serve
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
runtimeClassName: nvidia
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
containers:
|
||||
- name: clef
|
||||
image: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
|
||||
command: ["/bin/bash","-lc"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
pip install --no-cache-dir transformers==5.10.2 accelerate fastapi 'uvicorn[standard]' pillow safetensors sentencepiece protobuf
|
||||
exec uvicorn app:create_app --factory --host 0.0.0.0 --port 8080
|
||||
workingDir: /app
|
||||
env:
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
- name: MODEL_PATH
|
||||
value: /mnt/models/clef-flash
|
||||
# V100 has no bf16, and the release weights are bf16. fp16 is the same width.
|
||||
- name: DTYPE
|
||||
value: float16
|
||||
# The state is truncated at this length without warning, so raise it only with
|
||||
# spare VRAM: 17.75 GiB of weights plus 128 KiB of KV per token.
|
||||
- name: MAX_LEN
|
||||
value: "32768"
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
# A cold start pip-installs and loads 17.75 GiB, measured at about four minutes.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
failureThreshold: 120
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: "4"
|
||||
memory: 12Gi
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
cpu: "8"
|
||||
memory: 40Gi
|
||||
nvidia.com/gpu: "1"
|
||||
volumeMounts:
|
||||
- name: app
|
||||
mountPath: /app
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: app
|
||||
configMap:
|
||||
name: clef-serve-app
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: clef-serve
|
||||
namespace: llm-serving
|
||||
labels:
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
selector:
|
||||
app: clef-serve
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 8080
|
||||
@@ -0,0 +1,22 @@
|
||||
# Base image: torch 2.8.0+cu128 still lists sm_70 in torch.cuda.get_arch_list().
|
||||
# The CUDA 13 builds of torch drop sm_70. worker-1 holds Tesla V100 cards with
|
||||
# compute capability 7.0, so a newer base kills the kernel launch at load time.
|
||||
# Keep this pin until the model moves to newer hardware.
|
||||
FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
|
||||
|
||||
# Unbuffered output lets the pod log show the startup lines in order.
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# torch comes from the base image. Never reinstall it here, or pip pulls a
|
||||
# wheel built for another CUDA version.
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app.py .
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["uvicorn", "app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8080"]
|
||||
@@ -0,0 +1,201 @@
|
||||
"""HTTP server for the Clef-flash decision model.
|
||||
|
||||
Clef is a decision model and not a chat model. It reads one state plus a schema
|
||||
of typed questions and returns one probability per allowed option. Consumers
|
||||
call POST /v1/systemone and never /v1/chat/completions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import traceback
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
import torch
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from safetensors.torch import load_file
|
||||
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
|
||||
|
||||
# uvicorn configures its own loggers only. Without this call the root logger
|
||||
# stays at WARNING and the startup lines below never reach the pod log.
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
LOG = logging.getLogger("clef-serve")
|
||||
|
||||
MODEL_PATH = os.environ.get("MODEL_PATH", "/mnt/models/clef-flash")
|
||||
DTYPE = os.environ.get("DTYPE", "float16")
|
||||
MAX_LEN = int(os.environ.get("MAX_LEN", "65536"))
|
||||
PORT = int(os.environ.get("PORT", "8080"))
|
||||
|
||||
# HF_HOME stays untouched. The k8s manifest sets it, and the loader must reuse
|
||||
# that cache path instead of downloading a second copy of an 18.8 GB checkpoint.
|
||||
|
||||
_TORCH_DTYPES = {
|
||||
"float16": torch.float16,
|
||||
"bfloat16": torch.bfloat16,
|
||||
"float32": torch.float32,
|
||||
}
|
||||
|
||||
|
||||
def _resolve_dtype() -> torch.dtype:
|
||||
"""Map the DTYPE env string to a torch dtype. Fail fast on a typo."""
|
||||
try:
|
||||
return _TORCH_DTYPES[DTYPE]
|
||||
except KeyError:
|
||||
raise ValueError(
|
||||
f"The server does not support DTYPE={DTYPE!r}. Use one of {sorted(_TORCH_DTYPES)}."
|
||||
) from None
|
||||
|
||||
|
||||
def _load_runtime() -> tuple[Any, Any, list[str], Any]:
|
||||
"""Load backbone, joint head and processor. Runs in a worker thread."""
|
||||
# The released code imports its sibling modules by bare name. The model
|
||||
# directory is therefore the only place on sys.path that resolves them.
|
||||
sys.path.insert(0, MODEL_PATH)
|
||||
|
||||
# This import must follow the sys.path.insert above. joint_schema_model.py
|
||||
# ships inside MODEL_PATH and not in site-packages. An import before that
|
||||
# insert raises ModuleNotFoundError. A local import binds the name in this
|
||||
# function scope only, so the caller must receive the callable as a value.
|
||||
from joint_schema_model import (
|
||||
ClefModel,
|
||||
JointSchemaHead,
|
||||
collate_records,
|
||||
encode_record,
|
||||
systemone,
|
||||
)
|
||||
|
||||
dtype = _resolve_dtype()
|
||||
LOG.info("Loading backbone. path=%s dtype=%s", MODEL_PATH, DTYPE)
|
||||
|
||||
# load_release_model passes device_map={"": str(device)} and so pins the
|
||||
# whole backbone on one card. This loader asks accelerate to shard instead.
|
||||
backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
|
||||
MODEL_PATH,
|
||||
dtype=dtype,
|
||||
device_map="auto",
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
|
||||
# The head must sit on the device that produces the final hidden state,
|
||||
# because ClefModel applies it to the output of the last layer.
|
||||
head_cfg = json.loads((Path(MODEL_PATH) / "joint_head_config.json").read_text())
|
||||
head = JointSchemaHead(**head_cfg)
|
||||
head.load_state_dict(load_file(str(Path(MODEL_PATH) / "joint_head.safetensors")))
|
||||
# The output embeddings carry the last layer's device. Without them, take
|
||||
# the last parameter, which lives there too.
|
||||
output_embeddings = backbone.get_output_embeddings()
|
||||
if output_embeddings is not None:
|
||||
head_device = output_embeddings.weight.device
|
||||
else:
|
||||
head_device = next(reversed(list(backbone.parameters()))).device
|
||||
head.to(device=head_device, dtype=dtype)
|
||||
|
||||
model = ClefModel(backbone, head).eval()
|
||||
processor = AutoProcessor.from_pretrained(MODEL_PATH)
|
||||
|
||||
# hf_device_map maps layer names to ints for CUDA shards. Report it so a
|
||||
# human can see whether one card or two carried the weights.
|
||||
raw_map = getattr(backbone, "hf_device_map", None) or {}
|
||||
if not raw_map:
|
||||
raw_map = {"": str(next(backbone.parameters()).device)}
|
||||
devices = list(
|
||||
dict.fromkeys(
|
||||
f"cuda:{name}" if isinstance(name, int) else str(name)
|
||||
for name in raw_map.values()
|
||||
)
|
||||
)
|
||||
LOG.info(
|
||||
"Model ready. dtype=%s devices=%s device_map=%s head_device=%s",
|
||||
DTYPE,
|
||||
devices,
|
||||
raw_map,
|
||||
head_device,
|
||||
)
|
||||
return model, processor, devices, systemone
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
"""Load the model on startup without blocking the event loop."""
|
||||
# A 9.4B load takes minutes. asyncio.to_thread keeps /health answerable
|
||||
# during that window, so the startup probe sees 503 and not a dead socket.
|
||||
model, processor, devices, systemone_fn = await asyncio.to_thread(_load_runtime)
|
||||
app.state.model = model
|
||||
app.state.processor = processor
|
||||
app.state.devices = devices
|
||||
# The route handler runs at module scope and cannot see the local import
|
||||
# inside _load_runtime. Publish the callable here, or the handler raises
|
||||
# NameError on the first request.
|
||||
app.state.systemone = systemone_fn
|
||||
app.state.lock = asyncio.Lock()
|
||||
LOG.info("Runtime published. systemone_callable=%s", systemone_fn is not None)
|
||||
yield
|
||||
app.state.model = None
|
||||
app.state.processor = None
|
||||
app.state.systemone = None
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
app = FastAPI(title="clef-flash", lifespan=_lifespan)
|
||||
|
||||
@app.get("/health")
|
||||
async def health() -> JSONResponse:
|
||||
# A ready flag alone is not enough. The pod must not take traffic until
|
||||
# the weights and the joint head are both resident.
|
||||
if getattr(app.state, "model", None) is None:
|
||||
return JSONResponse({"status": "loading"}, status_code=503)
|
||||
return JSONResponse(
|
||||
{
|
||||
"status": "ok",
|
||||
"dtype": DTYPE,
|
||||
"devices": app.state.devices,
|
||||
},
|
||||
status_code=200,
|
||||
)
|
||||
|
||||
@app.post("/v1/systemone")
|
||||
async def systemone_route(request: Request) -> Any:
|
||||
model = getattr(app.state, "model", None)
|
||||
if model is None:
|
||||
return JSONResponse({"error": "model is not loaded"}, status_code=503)
|
||||
|
||||
body = await request.json()
|
||||
processor = app.state.processor
|
||||
|
||||
def _forward() -> dict:
|
||||
# inference_mode is thread-local. Enter it inside this call, or the
|
||||
# worker thread records no autograd state saving and still builds
|
||||
# the graph.
|
||||
with torch.inference_mode():
|
||||
return app.state.systemone(model, processor, body, max_length=MAX_LEN)
|
||||
|
||||
# Clef runs one prefill with shared buffers and a shared joint head.
|
||||
# Two calls at once corrupt each other, so serialize them here.
|
||||
async with app.state.lock:
|
||||
try:
|
||||
result = await asyncio.to_thread(_forward)
|
||||
except ValueError as exc:
|
||||
# A malformed request body is the caller's fault.
|
||||
return JSONResponse({"error": str(exc)}, status_code=400)
|
||||
except Exception as exc:
|
||||
LOG.error("systemone failed\n%s", traceback.format_exc())
|
||||
return JSONResponse(
|
||||
{"error": type(exc).__name__, "detail": str(exc)},
|
||||
status_code=500,
|
||||
)
|
||||
return result
|
||||
|
||||
return app
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run("app:create_app", factory=True, host="0.0.0.0", port=PORT)
|
||||
@@ -0,0 +1,12 @@
|
||||
# Runtime dependencies for the clef-flash server.
|
||||
# torch ships with the base image and stays out of this list.
|
||||
transformers==5.10.2
|
||||
# device_map="auto" sharding is an accelerate feature, not a transformers one
|
||||
accelerate>=1.1.0
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
pillow
|
||||
safetensors
|
||||
huggingface_hub
|
||||
sentencepiece
|
||||
protobuf
|
||||
@@ -10,22 +10,10 @@ kind: Kustomization
|
||||
# way after any edit here: a GPU predictor restart is a weights reload measured
|
||||
# in tens of seconds, not a rolling update.
|
||||
resources:
|
||||
# clef-flash.yaml is NOT listed: the model cannot run on this cluster.
|
||||
#
|
||||
# Clef needs SGLang v0.5.22 or newer, and SGLang publishes that only on CUDA 13
|
||||
# (the last CUDA 12 tag is v0.5.19-cu129, which predates Clef support). CUDA 13
|
||||
# dropped Volta kernels, and every GPU here is a Tesla V100 (sm_70):
|
||||
#
|
||||
# torch.AcceleratorError: CUDA error: no kernel image is available for
|
||||
# execution on the device
|
||||
#
|
||||
# SGLang also states sm75 as its floor: the README hardware list omits V100,
|
||||
# sm75 support was removed in sgl-project/sglang#9207, and FlashInfer needs
|
||||
# sm75+. So no SGLang build exists that has both Clef and Volta kernels, and
|
||||
# no flag or dtype change bridges that gap.
|
||||
#
|
||||
# Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is
|
||||
# kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning.
|
||||
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
|
||||
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
|
||||
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
|
||||
- clef-serve.yaml
|
||||
- reasoning.yaml
|
||||
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
||||
# replicas, because desired equals available at 0. Its manifest was never committed,
|
||||
@@ -36,3 +24,13 @@ resources:
|
||||
- networkpolicy.yaml
|
||||
# No namespace transformer: every file sets its own, and the transformer would
|
||||
# rewrite metadata.namespace on anything cross-namespace added later.
|
||||
|
||||
# app.py lives in git next to this file. The generator hashes the ConfigMap name, so an
|
||||
# edit to app.py rolls the pod instead of leaving a stale copy mounted.
|
||||
configMapGenerator:
|
||||
- name: clef-serve-app
|
||||
namespace: llm-serving
|
||||
files:
|
||||
- clef-serve/app.py
|
||||
options:
|
||||
disableNameSuffixHash: false
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
annotations:
|
||||
serving.kserve.io/deploymentMode: RawDeployment
|
||||
# KServe injects the prometheus.io/{scrape,port,path} annotations into the
|
||||
# generated pod only when this flag is on. Without it the predictor carries
|
||||
# no scrape annotations at all, which is why no vLLM/SGLang metrics have
|
||||
# ever reached Prometheus from this namespace.
|
||||
serving.kserve.io/enable-prometheus-scraping: "true"
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-reasoning
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
name: reasoning
|
||||
namespace: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
containers:
|
||||
- args:
|
||||
# Replaces the dense Qwen3-32B. Same serving name (`reasoning`), so every
|
||||
# consumer keeps its model string and no client changes.
|
||||
#
|
||||
# Why a MoE: Qwen3-30B-A3B has 30.5B total but only 3.3B active per token.
|
||||
# On sm70 the decode path is bandwidth- and kernel-launch-bound, not
|
||||
# FLOP-bound, so activating 3.3B instead of 32B is the single biggest
|
||||
# throughput win available on this hardware without changing GPUs.
|
||||
#
|
||||
# Quantization: GPTQ-Int4, pinned with --quantization=gptq. This repeats
|
||||
# the finding recorded on the previous checkpoint: bnb-4bit works poorly
|
||||
# on sm70 because there are no int4 tensor cores, and vLLM auto-upgrades
|
||||
# AWQ/GPTQ to the Marlin kernel which requires sm80+. Pinning `gptq`
|
||||
# forces the plain kernel, which is verified Volta-compatible here.
|
||||
# Do not remove this flag.
|
||||
#
|
||||
# Context: 131072 via YaRN. The base model card documents 32,768 native
|
||||
# and 131,072 with YaRN (factor 4.0, original_max_position_embeddings
|
||||
# 32768). That is Qwen's own validated ceiling, so unlike a stretched
|
||||
# window this is a documented capability.
|
||||
#
|
||||
# KV cache math, from config.json: 48 layers, 4 KV heads, head_dim 128,
|
||||
# fp16 -> 2 * 4 * 128 * 2 = 2048 B/token/layer -> 96 KB/token.
|
||||
# At 131072 tokens that is 12 GiB total. With TP=2 the KV heads split
|
||||
# 2 per GPU, so each GPU carries ~6 GiB of KV. Weights are ~17 GB total,
|
||||
# so ~8.5 GB per GPU. At gpu-memory-utilization=0.95 (about 30.4 GB
|
||||
# usable per V100-32GB) that leaves roughly 21 GB per GPU for KV plus
|
||||
# activations: comfortable for one full 128K request and a few shorter
|
||||
# ones. Re-read vllm:cache_config_info after rollout for the real
|
||||
# num_gpu_blocks value rather than trusting this projection.
|
||||
#
|
||||
# TP=2, not PP=2: the model has 4 KV heads, so TP must divide 4. TP=2
|
||||
# splits cleanly; TP=3 would not. TP also shortens prefill at long
|
||||
# context, which is the actual bottleneck at 128K on V100.
|
||||
- --model=Qwen/Qwen3-30B-A3B-GPTQ-Int4
|
||||
- --served-model-name=reasoning
|
||||
- --quantization=gptq
|
||||
- --dtype=float16
|
||||
- --kv-cache-dtype=auto
|
||||
- --rope-scaling={"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}
|
||||
- --tensor-parallel-size=2
|
||||
- --max-model-len=131072
|
||||
- --gpu-memory-utilization=0.95
|
||||
- --max-num-seqs=4
|
||||
# Prefill scheduling. The same reasoning as the previous checkpoint
|
||||
# applies: vLLM V1 has no separate prefill phase, so a single long
|
||||
# prompt can consume the whole step budget and starve every other
|
||||
# request. The threshold must be strictly below the budget or it never
|
||||
# binds, hence 8192 budget with a 2048 threshold.
|
||||
- --max-num-batched-tokens=8192
|
||||
- --long-prefill-token-threshold=2048
|
||||
- --enable-chunked-prefill
|
||||
- --enable-prefix-caching
|
||||
# qwen3 is vLLM's reasoning parser for this family's <think> blocks.
|
||||
- --reasoning-parser=qwen3
|
||||
# hermes is the documented tool-call parser for general Qwen3 models.
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser=hermes
|
||||
- --host=0.0.0.0
|
||||
- --port=8080
|
||||
env:
|
||||
# FlashInfer is sm80+. TRITON_ATTN is the Volta-compatible backend and is
|
||||
# already the proven setting on this node. Clef-flash runs on SGLang
|
||||
# instead, but Qwen stays on vLLM here because this exact combination is
|
||||
# tested on this hardware; moving it to SGLang would add risk for no
|
||||
# measured gain.
|
||||
- name: VLLM_USE_FLASHINFER_SAMPLER
|
||||
value: '0'
|
||||
- name: VLLM_ATTENTION_BACKEND
|
||||
value: TRITON_ATTN
|
||||
- name: HF_HOME
|
||||
value: /mnt/models
|
||||
image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
|
||||
name: kserve-container
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
limits:
|
||||
cpu: '16'
|
||||
memory: 36Gi
|
||||
nvidia.com/gpu: '2'
|
||||
requests:
|
||||
cpu: '8'
|
||||
memory: 12Gi
|
||||
nvidia.com/gpu: '2'
|
||||
startupProbe:
|
||||
# 30.5B of int4 weights plus a 128K rope config: allow a generous
|
||||
# first-boot window. failureThreshold 120 * 15s = 30 min ceiling.
|
||||
failureThreshold: 120
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 15
|
||||
volumeMounts:
|
||||
- mountPath: /mnt/models
|
||||
name: models
|
||||
- mountPath: /dev/shm
|
||||
name: shm
|
||||
# podMetadata is NOT a valid field on this KServe version. The CRD exposes
|
||||
# annotations and labels directly under predictor. The older manifests in
|
||||
# this directory still carry podMetadata and ArgoCD silently drops it on
|
||||
# server-side apply; do not copy that pattern.
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8080"
|
||||
prometheus.io/path: "/metrics"
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-reasoning
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
deploymentStrategy:
|
||||
# Recreate, not RollingUpdate: GPU memory is a hard resource. A second
|
||||
# pod cannot schedule alongside the first on a full 4-GPU node, so a
|
||||
# rolling update would wedge with the new pod Pending forever.
|
||||
type: Recreate
|
||||
maxReplicas: 1
|
||||
minReplicas: 1
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
runtimeClassName: nvidia
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 2Gi
|
||||
name: shm
|
||||
Reference in New Issue
Block a user