rock
3eb6a91311
Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context. The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same 64k window fits on one card. Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token prompt, and clean reasoning_content separation through the gateway. Three settings are load-bearing on sm_70 rather than tuning: - VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0 - VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+ - deploymentStrategy Recreate a rolling update deadlocks once every card is allocated The manifest is committed as well as applied: reasoning.yaml had never been in git.
39 lines
2.0 KiB
YAML
39 lines
2.0 KiB
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# Explicit allowlist, matching k8s/apps/api. Anything added to this directory
|
|
# and not listed here is silently dropped — no error, no drift shown.
|
|
#
|
|
# These five were adopted from live state on 2026-08-15; they had been applied
|
|
# by hand and carried no ArgoCD ownership. Each was exported and verified with
|
|
# `kubectl diff -f <file>` returning empty before the Application below was
|
|
# created, so the first sync was a no-op rather than a redeploy. Re-verify that
|
|
# way after any edit here: a GPU predictor restart is a weights reload measured
|
|
# in tens of seconds, not a rolling update.
|
|
resources:
|
|
# clef-flash.yaml is NOT listed: the model cannot run on this cluster.
|
|
#
|
|
# Clef needs SGLang v0.5.22 or newer, and SGLang publishes that only on CUDA 13
|
|
# (the last CUDA 12 tag is v0.5.19-cu129, which predates Clef support). CUDA 13
|
|
# dropped Volta kernels, and every GPU here is a Tesla V100 (sm_70):
|
|
#
|
|
# torch.AcceleratorError: CUDA error: no kernel image is available for
|
|
# execution on the device
|
|
#
|
|
# SGLang also states sm75 as its floor: the README hardware list omits V100,
|
|
# sm75 support was removed in sgl-project/sglang#9207, and FlashInfer needs
|
|
# sm75+. So no SGLang build exists that has both Clef and Volta kernels, and
|
|
# no flag or dtype change bridges that gap.
|
|
#
|
|
# Re-add it when the model gets an Ampere-or-newer GPU. The manifest itself is
|
|
# kept in git (k8s/apps/llm-serving/clef-flash.yaml) with the same reasoning.
|
|
- reasoning.yaml
|
|
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
|
# replicas, because desired equals available at 0. Its manifest was never committed,
|
|
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
|
|
- embeddings.yaml
|
|
- reranker.yaml
|
|
- qwen-cpu.yaml
|
|
- networkpolicy.yaml
|
|
# No namespace transformer: every file sets its own, and the transformer would
|
|
# rewrite metadata.namespace on anything cross-namespace added later.
|