feat(gpu): serve 6 models on worker-1 via KServe — vLLM v0.11.0 (bitsandbytes) + Ollama + TEI, plus RuntimeClass/privileged-PSA prereqs and a local-NVMe StorageClass, working around Volta sm_70 limits
Layout on 4x Tesla V100 32GB (PCIe, no NVLink), all TP=1: GPU0+1 vLLM DeepSeek-R1-Distill-Qwen-32B bnb-nf4 (2 replicas) GPU2 Ollama ornith:35b + qwen2.5:3b-instruct (co-resident) GPU3 vLLM Qwen2.5-Math-PRM-7B (reward model) CPU TEI nomic-embed-text-v2-moe, bge-reranker-base Volta constraints, each verified against live output rather than config: - vLLM pinned v0.11.0: sm_70 dropped from CUDA_SUPPORTED_ARCHS at v0.11.1. - AWQ hard-rejected (needs sm_75). GPTQ passes vLLM's min_capability=60 gate but is NUMERICALLY WRONG on sm_70 — emits garbage logits. Proven by an fp16 control run producing correct text on an identical backend. bitsandbytes nf4 verified correct by output. - flashinfer's check_cuda_arch() crashes on any sm_7x (calls .isdigit() on an int) -> VLLM_USE_FLASHINFER_SAMPLER=0. - xformers has no sm_70 kernel for V1's paged-attention bias, and V0 was removed in v0.11.0 -> TRITON_ATTN. - Ornith is Qwen3.5-MoE hybrid-attention; vLLM added that arch after dropping Volta, so no build has both -> Ollama, which also multiplexes a second model on the same card for free. Cluster prereqs that were absent: - RuntimeClass nvidia: the Talos toolkit extension registers the containerd handler but not the k8s object; without it every pod is rejected at admission. - gpu-system pinned to privileged PSA: a device plugin cannot satisfy the cluster-default baseline, it must mount hostPath. - device-plugin affinity=null: the chart requires NFD labels that do not exist here, so it matched zero nodes and reported desiredNumberScheduled=0 silently. - Recreate strategy on GPU services: with GPUs allocated exactly 4/4, a RollingUpdate surge pod has no card and deadlocks the rollout. - longhorn-llm-local SC (1 replica, strict-local, disk tag llm): the default 3-replica class could not place the volume at all (every control-plane disk was at its over-provisioning ceiling), and this keeps ~60GB of weights on worker-1's own NVMe instead of reading them over the network. deploy-gpu-serving.sh sequences ArgoCD syncs (or helm/kubectl in --manual mode) and never applies a manifest absent from git; doctor/unstick/teardown stages exist so this is diagnosable without ad-hoc kubectl archaeology.
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# No namespace: RuntimeClass is cluster-scoped.
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- runtimeclass.yaml
|
||||
@@ -0,0 +1,23 @@
|
||||
# Namespace for GPU node-level plumbing (device plugin, and later DCGM).
|
||||
#
|
||||
# PodSecurity must be `privileged` here. The cluster default from the Talos
|
||||
# controlplane config is `enforce: baseline` with exemptions only for
|
||||
# kube-system, and a device plugin cannot satisfy baseline: it has to mount the
|
||||
# kubelet device-plugin socket and the CDI/driver directories as hostPath
|
||||
# volumes, which baseline forbids outright:
|
||||
#
|
||||
# Error creating: pods "nvidia-device-plugin-xxxxx" is forbidden:
|
||||
# violates PodSecurity "baseline:latest": hostPath volumes
|
||||
# (volumes "kubelet-device-plugins-dir", "mps-root", "mps-shm", "cdi-root")
|
||||
#
|
||||
# This is inherent to how device plugins work, not a workaround. Scope is
|
||||
# limited to this namespace; the engine namespace (llm-serving) stays on the
|
||||
# cluster default.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: gpu-system
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
pod-security.kubernetes.io/audit: privileged
|
||||
pod-security.kubernetes.io/warn: privileged
|
||||
@@ -0,0 +1,18 @@
|
||||
# Cluster-scoped prerequisite for every GPU workload on worker-1.
|
||||
#
|
||||
# The Talos nvidia-container-toolkit extension already registers the containerd
|
||||
# runtime handler (/etc/cri/conf.d/10-nvidia-container-runtime.part ->
|
||||
# plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia), but the
|
||||
# Kubernetes RuntimeClass object is separate and is NOT created by the
|
||||
# extension. Without it every pod carrying runtimeClassName: nvidia is rejected
|
||||
# at admission with:
|
||||
# pods "..." is forbidden: pod rejected: RuntimeClass "nvidia" not found
|
||||
#
|
||||
# Deliberately NOT setting nvidia as containerd's default_runtime_name (the
|
||||
# 20-customization.part patch in the Talos guide): that would route every pod on
|
||||
# the node through the NVIDIA runtime. Opting in per-pod is narrower.
|
||||
apiVersion: node.k8s.io/v1
|
||||
kind: RuntimeClass
|
||||
metadata:
|
||||
name: nvidia
|
||||
handler: nvidia
|
||||
@@ -0,0 +1,74 @@
|
||||
# Embeddings — Nomic Embed Text v2 (MoE), on CPU via HuggingFace TEI.
|
||||
#
|
||||
# CPU, not GPU, deliberately. All 4 V100s are claimed by the generation models,
|
||||
# and the device plugin hands out WHOLE GPUs — a 5th GPU-requesting pod is
|
||||
# unschedulable no matter how much VRAM is free. Sharing would need global
|
||||
# time-slicing, which on a single node cannot be scoped to one card and would let
|
||||
# the scheduler co-locate two ~20GB models and OOM both.
|
||||
#
|
||||
# worker-1 has 96 cores with ~250m requested, and this is a 475M-param encoder
|
||||
# (305M active). Retrieval runs once per agent-loop iteration, not per token, so
|
||||
# CPU latency here is immaterial. This is also what Plan 1 originally specified.
|
||||
#
|
||||
# TEI (not vLLM) because it is purpose-built for encoders and explicitly lists
|
||||
# nomic-embed-text-v2-moe as supported.
|
||||
#
|
||||
# NOTE: Nomic v2 requires task prefixes on the CLIENT side —
|
||||
# documents: "search_document: <text>"
|
||||
# queries: "search_query: <text>"
|
||||
# Embedding without the prefix silently degrades retrieval quality.
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
name: embeddings
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-embeddings
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
# Pinned to worker-1 only so it can share the RWO models PVC with the GPU
|
||||
# pods (RWO = single node, any number of pods on it).
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
containers:
|
||||
- name: kserve-container
|
||||
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.2@sha256:4d632b76bd14cb57044a1ffb0ad48ab0ba4939e705a9a615ccc740658575c26e
|
||||
args:
|
||||
- --model-id=nomic-ai/nomic-embed-text-v2-moe
|
||||
- --port=8080
|
||||
- --hostname=0.0.0.0
|
||||
# Truncate rather than 413 on over-long input.
|
||||
- --auto-truncate
|
||||
env:
|
||||
- name: HUGGINGFACE_HUB_CACHE
|
||||
value: /mnt/models
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: "8"
|
||||
memory: 4Gi
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
@@ -0,0 +1,107 @@
|
||||
# Action engine — Ornith-1.0-35B on Ollama.
|
||||
#
|
||||
# Why not vLLM like the other two: Ornith is
|
||||
# Qwen3_5MoeForConditionalGeneration (Qwen3.5 MoE, 256 experts / 8 active,
|
||||
# hybrid attention — 30 linear_attention + 10 full_attention layers). vLLM's
|
||||
# Qwen3.5 support landed 2026-07-29, AFTER vLLM dropped Volta (sm_70) at
|
||||
# v0.11.1. No vLLM build has both, so Ornith cannot run on vLLM on a V100.
|
||||
#
|
||||
# Ollama ships `ornith:35b` in its library and runs a llama-server runner
|
||||
# underneath, which keeps Volta support. q4 is ~21GB — fits one 32GB V100 with
|
||||
# room for KV.
|
||||
#
|
||||
# OLLAMA_KEEP_ALIVE=-1 is load-bearing: the harness calls this every loop
|
||||
# iteration, and Ollama's default is to evict an idle model after 5m, which
|
||||
# would add a ~21GB reload to a random future request.
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
name: ornith
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-ornith
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
minReplicas: 1
|
||||
# Recreate, not the default RollingUpdate: GPUs are allocated exactly 4/4,
|
||||
# so a surge pod has no card to claim and sits Pending while the old pod is
|
||||
# never torn down — a deadlock. Recreate tears down first, accepting a brief
|
||||
# gap during updates.
|
||||
deploymentStrategy:
|
||||
type: Recreate
|
||||
maxReplicas: 1
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
runtimeClassName: nvidia
|
||||
containers:
|
||||
- name: kserve-container
|
||||
image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
|
||||
# `ollama serve` does not pull models, and `ollama pull` needs a running
|
||||
# server — so background the server, wait for it, pull, then hand the
|
||||
# foreground back to serve.
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
ollama serve &
|
||||
SERVE_PID=$!
|
||||
until ollama list >/dev/null 2>&1; do sleep 2; done
|
||||
ollama pull ornith:35b
|
||||
ollama pull qwen2.5:3b-instruct
|
||||
wait $SERVE_PID
|
||||
env:
|
||||
# Match the port the other two engines use.
|
||||
- name: OLLAMA_HOST
|
||||
value: "0.0.0.0:8080"
|
||||
- name: OLLAMA_MODELS
|
||||
value: /mnt/models/ollama
|
||||
# Ollama defaults to a 4096 context, far too small for an agentic
|
||||
# coding model. Ornith's hybrid attention means only 10 of its 40
|
||||
# layers hold a conventional KV cache, so 32K is affordable in the
|
||||
# ~11GiB left after its 21GB of weights.
|
||||
- name: OLLAMA_CONTEXT_LENGTH
|
||||
value: "32768"
|
||||
# Never evict — this model is on the harness's hot path.
|
||||
- name: OLLAMA_KEEP_ALIVE
|
||||
value: "-1"
|
||||
# Serial agent loop; no benefit from parallel slots.
|
||||
- name: OLLAMA_NUM_PARALLEL
|
||||
value: "1"
|
||||
# 2, so ornith and the small utility model stay co-resident on GPU2
|
||||
# instead of evicting one another on every alternating request.
|
||||
- name: OLLAMA_MAX_LOADED_MODELS
|
||||
value: "2"
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: "8"
|
||||
memory: 8Gi
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 16Gi
|
||||
nvidia.com/gpu: "1"
|
||||
volumeMounts:
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
# Probes must confirm the MODEL is present, not just that the server
|
||||
# answers. Ollama's `GET /` returns 200 ("Ollama is running") the moment
|
||||
# `ollama serve` binds — which is before the ~21GB pull finishes. An
|
||||
# httpGet probe would therefore mark this pod Ready with no model
|
||||
# loaded, and KServe would route traffic to it.
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | grep -q qwen2.5"]
|
||||
periodSeconds: 15
|
||||
failureThreshold: 120
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | grep -q qwen2.5"]
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
@@ -0,0 +1,133 @@
|
||||
# Reasoning engine — DeepSeek-R1-Distill-Qwen-32B, GPTQ INT4, vLLM.
|
||||
#
|
||||
# TP=1 with 2 data-parallel replicas (GPU0 + GPU1) rather than one TP=2 engine:
|
||||
# worker-1 has NO NVLink, so tensor-parallel's per-token all-reduce would cross
|
||||
# PCIe on every decode step. Two independent replicas need zero inter-GPU
|
||||
# communication and KServe load-balances them behind one Service.
|
||||
#
|
||||
# vLLM is pinned to v0.11.0 — the LAST release that compiles sm_70 (Volta)
|
||||
# kernels. v0.11.1 dropped 7.0 from CUDA_SUPPORTED_ARCHS. Do not bump this
|
||||
# without re-checking CMakeLists.txt, or every pod dies with "no kernel image".
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
name: reasoning
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-reasoning
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
minReplicas: 2
|
||||
# Recreate, not the default RollingUpdate: GPUs are allocated exactly 4/4,
|
||||
# so a surge pod has no card to claim and sits Pending while the old pod is
|
||||
# never torn down — a deadlock. Recreate tears down first, accepting a brief
|
||||
# gap during updates.
|
||||
deploymentStrategy:
|
||||
type: Recreate
|
||||
maxReplicas: 2
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
runtimeClassName: nvidia
|
||||
containers:
|
||||
- name: kserve-container
|
||||
image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
|
||||
args:
|
||||
# GPTQ, NOT AWQ. vLLM hard-refuses AWQ below compute capability 75:
|
||||
# "The quantization method awq is not supported for the current GPU.
|
||||
# Minimum capability: 75. Current capability: 70."
|
||||
# V100 is sm_70. GPTQ's min capability is 60, so it runs. (gptq_marlin
|
||||
# needs 80 and fp8 needs 80 — both also out.) Same 19.3GB footprint.
|
||||
# desc_act=False in this build: no activation reordering, faster.
|
||||
- --model=unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit
|
||||
- --served-model-name=reasoning
|
||||
# bitsandbytes nf4. GPTQ passed vLLM's min_capability=60 check but was
|
||||
# numerically WRONG on sm_70 (garbage logits) — proven by the fp16
|
||||
# control run producing correct text with the identical backend. bnb
|
||||
# declares min_capability=70, but treat that as unverified until the
|
||||
# output itself is checked.
|
||||
# NOTE the repo sets bnb_4bit_compute_dtype=bfloat16, which Volta does
|
||||
# not have; --dtype=float16 must override it.
|
||||
- --quantization=bitsandbytes
|
||||
# Volta has no bf16 — must be explicit, the repo's weights are bf16.
|
||||
- --dtype=float16
|
||||
# No FP8 KV on Volta; stays fp16.
|
||||
- --kv-cache-dtype=auto
|
||||
- --tensor-parallel-size=1
|
||||
- --max-model-len=16384
|
||||
# VRAM budget on a 32GiB V100: 0.92 => ~29.4GiB, minus ~18GiB of GPTQ
|
||||
# weights leaves ~11GiB for KV + activations. One full 32K sequence
|
||||
# costs 32768 x 256KB = 8GiB of KV, so 8 concurrent full-length
|
||||
# sequences is not physically possible here — 4 is honest, and a
|
||||
# serial single-user harness never needs more.
|
||||
- --gpu-memory-utilization=0.90
|
||||
- --max-num-seqs=4
|
||||
# Smooths Volta's slow prefill (no FlashAttention2 on sm_70).
|
||||
- --enable-chunked-prefill
|
||||
- --enable-prefix-caching
|
||||
# Splits <think>…</think> into its own channel.
|
||||
- --reasoning-parser=deepseek_r1
|
||||
- --host=0.0.0.0
|
||||
- --port=8080
|
||||
env:
|
||||
# FlashAttention2 requires sm_80; Volta must fall back to xformers.
|
||||
# flashinfer's check_cuda_arch() has an upstream bug that crashes on
|
||||
# ANY sm_7x GPU: `elif major == 7 and minor.isdigit()` calls .isdigit()
|
||||
# on an int, so instead of reporting "unsupported" it raises
|
||||
# AttributeError: 'int' object has no attribute 'isdigit'
|
||||
# and engine init dies. Default is None (auto-detect), which walks
|
||||
# straight into that path. 0 disables the flashinfer sampler outright.
|
||||
# Only affects the generate runner — the verifier (pooling) never hits
|
||||
# the sampler, which is why it started fine and this did not.
|
||||
- name: VLLM_USE_FLASHINFER_SAMPLER
|
||||
value: "0"
|
||||
# TRITON_ATTN, not XFORMERS. On sm_70 every xformers kernel is
|
||||
# rejected for V1's paged-attention bias type:
|
||||
# fa2F / triton_splitKF -> require sm_80
|
||||
# cutlassF -> supports sm_70 but not
|
||||
# PagedBlockDiagonalCausalWithOffsetPaddedKeysMask
|
||||
# -> NotImplementedError kills EngineCore on the FIRST request, which
|
||||
# takes the whole pod down (vLLM treats engine death as fatal).
|
||||
# V0, whose hand-written paged kernels did support sm_70, was REMOVED
|
||||
# in v0.11.0, so VLLM_USE_V1=0 has nothing to fall back to.
|
||||
# Triton JIT-compiles for the local arch, so it is the last option.
|
||||
- name: VLLM_ATTENTION_BACKEND
|
||||
value: TRITON_ATTN
|
||||
- name: HF_HOME
|
||||
value: /mnt/models
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: "8"
|
||||
memory: 8Gi
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 16Gi
|
||||
nvidia.com/gpu: "1"
|
||||
volumeMounts:
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
- name: shm
|
||||
mountPath: /dev/shm
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
# Cold start pulls ~18Gi of weights over Longhorn, then loads to VRAM.
|
||||
periodSeconds: 15
|
||||
failureThreshold: 80
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
- name: shm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 2Gi
|
||||
@@ -0,0 +1,71 @@
|
||||
# Reranker — BAAI/bge-reranker-base, on CPU via HuggingFace TEI.
|
||||
#
|
||||
# Second stage of retrieval: the embedding model fetches a coarse top-k by
|
||||
# vector similarity, this cross-encoder re-scores those candidates against the
|
||||
# query directly. That is what fixes the "semantic dilution" problem in Plan 1 —
|
||||
# a single embedding vector cannot represent a large chunk faithfully, so
|
||||
# ranking by cosine alone surfaces near-misses.
|
||||
#
|
||||
# CPU for the same reason as the embedding service: all 4 GPUs are claimed and
|
||||
# the device plugin allocates whole cards. A 568M cross-encoder scoring ~20-50
|
||||
# candidates per query is well within CPU budget.
|
||||
#
|
||||
# Arch is XLMRobertaForSequenceClassification, which TEI serves as /rerank.
|
||||
# 278M params — smaller than v2-m3 (568M) and English/Chinese rather than
|
||||
# multilingual, which suits code+docs retrieval and is faster on CPU.
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
name: reranker
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-reranker
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
# Same worker-1 pin as the embedding service, to share the RWO models PVC.
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
containers:
|
||||
- name: kserve-container
|
||||
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.2@sha256:4d632b76bd14cb57044a1ffb0ad48ab0ba4939e705a9a615ccc740658575c26e
|
||||
args:
|
||||
# bge-reranker-base, NOT v2-m3. TEI's CPU image starts the ONNX
|
||||
# Runtime backend and v2-m3 ships no ONNX files, so it dies with
|
||||
# "Model ONNX files not found in the repository". This build does.
|
||||
- --model-id=BAAI/bge-reranker-base
|
||||
- --port=8080
|
||||
- --hostname=0.0.0.0
|
||||
- --auto-truncate
|
||||
env:
|
||||
- name: HUGGINGFACE_HUB_CACHE
|
||||
value: /mnt/models
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: "8"
|
||||
memory: 4Gi
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 8Gi
|
||||
volumeMounts:
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
@@ -0,0 +1,101 @@
|
||||
# Verifier — Qwen2.5-Math-PRM-7B, step-level process reward model, vLLM.
|
||||
#
|
||||
# Model choice was constrained by vLLM v0.11.0's registry: its arch
|
||||
# (Qwen2ForProcessRewardModel) is natively registered, whereas the smaller
|
||||
# community PRMs are Qwen2ForTokenClassification / Qwen2ForPrmModel, neither of
|
||||
# which v0.11.0 can load (ForTokenClassification is absent from
|
||||
# _SUFFIX_TO_DEFAULTS, so it won't even auto-convert).
|
||||
#
|
||||
# --runner pooling, NOT --task reward: --task is [DEPRECATED] in v0.11.0.
|
||||
# Scoring goes to /pooling, not /v1/completions — this is a reward model, it
|
||||
# returns scores, not tokens.
|
||||
#
|
||||
# Gets a whole dedicated GPU despite only needing ~15Gi: it sits on the
|
||||
# harness's critical path (every reasoning->action->verify iteration waits on
|
||||
# it), so isolation from the generation engines' decode loops matters more than
|
||||
# the idle VRAM.
|
||||
apiVersion: serving.kserve.io/v1beta1
|
||||
kind: InferenceService
|
||||
metadata:
|
||||
name: verifier
|
||||
labels:
|
||||
app.kubernetes.io/name: llm-verifier
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
predictor:
|
||||
minReplicas: 1
|
||||
# Recreate, not the default RollingUpdate: GPUs are allocated exactly 4/4,
|
||||
# so a surge pod has no card to claim and sits Pending while the old pod is
|
||||
# never torn down — a deadlock. Recreate tears down first, accepting a brief
|
||||
# gap during updates.
|
||||
deploymentStrategy:
|
||||
type: Recreate
|
||||
maxReplicas: 1
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: worker-1
|
||||
runtimeClassName: nvidia
|
||||
containers:
|
||||
- name: kserve-container
|
||||
image: vllm/vllm-openai:v0.11.0@sha256:014a95f21c9edf6abe0aea6b07353f96baa4ec291c427bb1176dc7c93a85845c
|
||||
args:
|
||||
- --model=Qwen/Qwen2.5-Math-PRM-7B
|
||||
- --served-model-name=verifier
|
||||
# Pooling runner => reward scoring. Weights are bf16; Volta needs fp16.
|
||||
- --runner=pooling
|
||||
- --dtype=float16
|
||||
- --tensor-parallel-size=1
|
||||
- --max-model-len=4096
|
||||
- --max-num-seqs=8
|
||||
- --host=0.0.0.0
|
||||
- --port=8080
|
||||
env:
|
||||
# flashinfer's check_cuda_arch() has an upstream bug that crashes on
|
||||
# ANY sm_7x GPU: `elif major == 7 and minor.isdigit()` calls .isdigit()
|
||||
# on an int, so instead of reporting "unsupported" it raises
|
||||
# AttributeError: 'int' object has no attribute 'isdigit'
|
||||
# and engine init dies. Default is None (auto-detect), which walks
|
||||
# straight into that path. 0 disables the flashinfer sampler outright.
|
||||
# Only affects the generate runner — the verifier (pooling) never hits
|
||||
# the sampler, which is why it started fine and this did not.
|
||||
- name: VLLM_USE_FLASHINFER_SAMPLER
|
||||
value: "0"
|
||||
- name: VLLM_ATTENTION_BACKEND
|
||||
value: XFORMERS
|
||||
- name: HF_HOME
|
||||
value: /mnt/models
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: "4"
|
||||
memory: 8Gi
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
cpu: "16"
|
||||
memory: 16Gi
|
||||
nvidia.com/gpu: "1"
|
||||
volumeMounts:
|
||||
- name: models
|
||||
mountPath: /mnt/models
|
||||
- name: shm
|
||||
mountPath: /dev/shm
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 15
|
||||
failureThreshold: 60
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8080
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: models
|
||||
persistentVolumeClaim:
|
||||
claimName: llm-models
|
||||
- name: shm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 1Gi
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: llm-serving
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- storageclass.yaml
|
||||
- pvc-models.yaml
|
||||
- inferenceservice-reasoning.yaml
|
||||
- inferenceservice-ornith.yaml
|
||||
- inferenceservice-verifier.yaml
|
||||
- inferenceservice-embeddings.yaml
|
||||
- inferenceservice-reranker.yaml
|
||||
- networkpolicy.yaml
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: llm-serving
|
||||
@@ -0,0 +1,43 @@
|
||||
# Default-deny ingress for the serving pods.
|
||||
#
|
||||
# This is a real compensating control, not hygiene: vLLM is pinned to v0.11.0
|
||||
# (forced — last release with Volta kernels), which sits below the patch line on
|
||||
# several advisories that will never be backported to that branch, incl.
|
||||
# CVE-2026-54234 (remote DoS) and GHSA-7m6h-x95x-82q5 (cross-user data leak).
|
||||
# Those are all remote/unauthenticated attack surface, so keeping the engines
|
||||
# reachable only from opted-in in-cluster clients is what keeps exposure low.
|
||||
#
|
||||
# Consumers opt in with label `llm-client: "true"`. Do NOT expose these via
|
||||
# Ingress.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: llm-serving-default-deny
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
# Any pod, any namespace, that explicitly opts in as an LLM client.
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
llm-client: "true"
|
||||
# Sibling engines (harness may chain calls between them).
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- from:
|
||||
# Prometheus scraping /metrics.
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
@@ -0,0 +1,32 @@
|
||||
# Shared HuggingFace cache for all three engines.
|
||||
#
|
||||
# ReadWriteOnce is correct here despite six pods mounting it: RWO means "one
|
||||
# NODE", and every pod in this app is pinned to worker-1 via nodeSelector, so
|
||||
# they share the volume legally. If a pod is ever allowed onto another node,
|
||||
# this must become RWX first.
|
||||
#
|
||||
# StorageClass is longhorn-llm-local (1 replica, strict-local, disk tag `llm`)
|
||||
# — NOT the default 3-replica class, which could not place this volume at all:
|
||||
# every control-plane disk was already at its over-provisioning ceiling.
|
||||
#
|
||||
# Sizing (measured, not estimated):
|
||||
# reasoning GPTQ INT4 19.3 GB
|
||||
# ornith:35b q4 (ollama) 21.0 GB
|
||||
# verifier Qwen2.5-Math-PRM-7B fp16 15.3 GB
|
||||
# nomic-embed-text-v2-moe (CPU) 1.9 GB
|
||||
# bge-reranker-base (CPU) 1.1 GB
|
||||
# ------------------------------------------
|
||||
# total ~58.6 GB (+ HF temp during pulls)
|
||||
# The two reasoning replicas share ONE on-disk copy; they differ only in which
|
||||
# GPU they load it onto.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: llm-models
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn-llm-local
|
||||
resources:
|
||||
requests:
|
||||
storage: 120Gi
|
||||
@@ -0,0 +1,38 @@
|
||||
# Dedicated StorageClass for model weights on worker-1's local NVMe.
|
||||
#
|
||||
# Why not the default `longhorn` class (3 replicas, network-attached):
|
||||
#
|
||||
# 1. numberOfReplicas: 1 — model weights are re-downloadable from HuggingFace.
|
||||
# Replicating them 3x buys nothing; losing a replica costs a re-pull, not
|
||||
# data. The repo's "never delete a PVC without replicas/backups" rule exists
|
||||
# for irreplaceable data, which this is not.
|
||||
#
|
||||
# 2. dataLocality: strict-local — keeps the single replica on the SAME node as
|
||||
# the pod. All engines are pinned to worker-1, so weights are read from its
|
||||
# local 751GB NVMe instead of over the network from a control-plane node.
|
||||
# Removes ~60GB of network reads on every cold start.
|
||||
#
|
||||
# 3. diskSelector: llm — restricts this class to disks tagged `llm`, i.e. only
|
||||
# worker-1's disk. Equally important, worker-1's disk carries that tag so
|
||||
# UNTAGGED volumes (any ordinary cluster PVC) will not land on it. Before
|
||||
# tagging, worker-1 had been silently hosting a replica of cicd/runner-dind,
|
||||
# consuming GPU-node storage for general cluster workloads.
|
||||
#
|
||||
# The default 3-replica class also physically could not place this volume: all
|
||||
# three control-plane disks were already at their over-provisioning ceiling
|
||||
# (storage-over-provisioning-percentage=100, 30% reserved), so a 120Gi x3
|
||||
# request failed with ReplicaSchedulingFailure on every node.
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: longhorn-llm-local
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
numberOfReplicas: "1"
|
||||
dataLocality: "strict-local"
|
||||
diskSelector: "llm"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: "ext4"
|
||||
@@ -0,0 +1,156 @@
|
||||
# Wave 9-11 — GPU serving stack on worker-1 (4x Tesla V100 32GB).
|
||||
#
|
||||
# Ordering matters: device plugin must expose nvidia.com/gpu and the KServe CRDs
|
||||
# must exist before any InferenceService is applied, hence three waves.
|
||||
#
|
||||
# NOTE the chart versions below are v-PREFIXED (v0.15.2, not 0.15.2) — that is
|
||||
# how the KServe OCI tags are published; the unprefixed form 404s.
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: gpu-runtimeclass
|
||||
namespace: argocd
|
||||
annotations:
|
||||
# Wave 8: must precede the device plugin, whose DaemonSet sets
|
||||
# runtimeClassName: nvidia and is rejected at admission if the
|
||||
# RuntimeClass does not exist yet.
|
||||
argocd.argoproj.io/sync-wave: "8"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: [email protected]:Riotpiaole/riotpiao.homelab.com.git
|
||||
targetRevision: main
|
||||
path: k8s/apps/gpu-runtimeclass
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
# Cluster-scoped resource; namespace is only the app's default context.
|
||||
namespace: gpu-system
|
||||
syncPolicy:
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: nvidia-device-plugin
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "9"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://nvidia.github.io/k8s-device-plugin
|
||||
chart: nvidia-device-plugin
|
||||
targetRevision: "0.19.3"
|
||||
helm:
|
||||
values: |
|
||||
# Driver + container toolkit are supplied by Talos system extensions
|
||||
# baked into the installer image (nonfree-kmod-nvidia-lts /
|
||||
# nvidia-container-toolkit-lts). This chart ONLY advertises the GPUs to
|
||||
# the kubelet — it does not and must not install drivers.
|
||||
runtimeClassName: nvidia
|
||||
nodeSelector:
|
||||
nvidia.com/gpu: "true"
|
||||
# Drop the chart's default nodeAffinity. It requires one of three
|
||||
# Node-Feature-Discovery labels (feature.node.kubernetes.io/pci-10de.present,
|
||||
# .../cpu-model.vendor_id=NVIDIA, or nvidia.com/gpu.present). NFD is not
|
||||
# installed and Talos sets nvidia.com/gpu (no ".present" suffix), so the
|
||||
# affinity matches zero nodes and the DaemonSet silently reports
|
||||
# desiredNumberScheduled=0 with no events. nodeSelector is the constraint.
|
||||
affinity: null
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: gpu-system
|
||||
syncPolicy:
|
||||
# Manual sync for first bring-up: watch device-plugin -> KServe -> models
|
||||
# come up in order, and avoid auto-deploying while worker-1 is cordoned.
|
||||
# Switch to `automated: {prune: true, selfHeal: true}` once proven.
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: kserve-crd
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "9"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: oci://ghcr.io/kserve/charts
|
||||
chart: kserve-crd
|
||||
targetRevision: v0.15.2
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: kserve
|
||||
syncPolicy:
|
||||
# Manual sync for first bring-up: watch device-plugin -> KServe -> models
|
||||
# come up in order, and avoid auto-deploying while worker-1 is cordoned.
|
||||
# Switch to `automated: {prune: true, selfHeal: true}` once proven.
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
# InferenceService CRD exceeds the annotation size limit for
|
||||
# client-side apply.
|
||||
- ServerSideApply=true
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: kserve
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "10"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: oci://ghcr.io/kserve/charts
|
||||
chart: kserve
|
||||
targetRevision: v0.15.2
|
||||
helm:
|
||||
values: |
|
||||
kserve:
|
||||
controller:
|
||||
# RawDeployment => plain Deployments/Services, no Knative, no Istio.
|
||||
# v0.18 renames this mode "Standard"; do not bump without checking.
|
||||
deploymentMode: RawDeployment
|
||||
gateway:
|
||||
ingressGateway:
|
||||
# Route through the existing ingress-nginx, not Gateway API.
|
||||
# NOTE the nesting: it is gateway.ingressGateway.enableGatewayApi,
|
||||
# not gateway.enableGatewayApi — Helm silently ignores the wrong
|
||||
# key rather than erroring.
|
||||
enableGatewayApi: false
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: kserve
|
||||
syncPolicy:
|
||||
# Manual sync for first bring-up: watch device-plugin -> KServe -> models
|
||||
# come up in order, and avoid auto-deploying while worker-1 is cordoned.
|
||||
# Switch to `automated: {prune: true, selfHeal: true}` once proven.
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
- ServerSideApply=true
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: llm-serving
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "11"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: [email protected]:Riotpiaole/riotpiao.homelab.com.git
|
||||
targetRevision: main
|
||||
path: k8s/apps/llm-serving
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: llm-serving
|
||||
syncPolicy:
|
||||
# Manual sync for first bring-up: watch device-plugin -> KServe -> models
|
||||
# come up in order, and avoid auto-deploying while worker-1 is cordoned.
|
||||
# Switch to `automated: {prune: true, selfHeal: true}` once proven.
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
Executable
+381
@@ -0,0 +1,381 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Deploy the GPU serving stack on worker-1:
|
||||
# NVIDIA device plugin -> KServe (RawDeployment) -> 3 engines
|
||||
# GPU0,1 vLLM v0.11.0 deepseek-r1-distill-qwen-32b-awq (2 replicas)
|
||||
# GPU2 Ollama ornith:35b
|
||||
# GPU3 vLLM v0.11.0 Qwen2.5-Math-PRM-7B (reward model)
|
||||
#
|
||||
# Two modes:
|
||||
# argocd (default) -- syncs the ArgoCD Applications from
|
||||
# k8s/argocd/apps/70-gpu-serving.yaml. Git is the source
|
||||
# of truth; this only sequences the syncs.
|
||||
# manual (--manual) -- bootstraps directly with helm + kubectl, for first
|
||||
# bring-up before the Applications are committed. Applies
|
||||
# the SAME manifests from k8s/apps/llm-serving/, so ArgoCD
|
||||
# adopts them cleanly later (matches the repo's existing
|
||||
# k8s/bootstrap/phaseN-* pattern).
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/deploy-gpu-serving.sh --manual # full manual bootstrap
|
||||
# ./scripts/deploy-gpu-serving.sh --manual gpu-plugin # one stage
|
||||
# ./scripts/deploy-gpu-serving.sh --manual --dry-run # show, don't run
|
||||
# ./scripts/deploy-gpu-serving.sh # via ArgoCD
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
NODE=worker-1
|
||||
NS=llm-serving
|
||||
KSERVE_NS=kserve
|
||||
GPU_NS=gpu-system
|
||||
KSERVE_VER=v0.15.2 # v-PREFIXED; the unprefixed tag 404s
|
||||
NVDP_VER=0.19.3
|
||||
REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||
MODE=argocd
|
||||
DRY=false
|
||||
|
||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
run() { if $DRY; then info "[dry-run] $*"; else "$@"; fi; }
|
||||
need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }
|
||||
|
||||
sync_app() {
|
||||
local app=$1
|
||||
log "argocd sync: $app"
|
||||
run argocd app sync "$app" --timeout 600
|
||||
run argocd app wait "$app" --health --timeout 600
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- stages
|
||||
|
||||
preflight() {
|
||||
log "preflight"
|
||||
need kubectl
|
||||
[[ $MODE == argocd ]] && need argocd
|
||||
[[ $MODE == manual ]] && need helm
|
||||
|
||||
kubectl get node "$NODE" >/dev/null 2>&1 || die "node $NODE not found"
|
||||
local ready
|
||||
ready=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
|
||||
[[ $ready == True ]] || die "node $NODE is not Ready"
|
||||
info "node $NODE: Ready"
|
||||
|
||||
# Driver ships in the Talos installer image as a system extension, not from
|
||||
# the cluster. Missing label => node was built from the wrong schematic and
|
||||
# no amount of device-plugin will help.
|
||||
kubectl get node "$NODE" -o jsonpath='{.metadata.labels}' | grep -q 'nonfree-kmod-nvidia' \
|
||||
|| die "NVIDIA driver extension label absent — worker-1 not on the GPU schematic"
|
||||
info "NVIDIA driver extension: present"
|
||||
|
||||
kubectl get node "$NODE" -o jsonpath='{.metadata.labels.nvidia\.com/gpu}' 2>/dev/null | grep -q true \
|
||||
|| die "node label nvidia.com/gpu=true missing — device plugin nodeSelector will not match"
|
||||
info "node label nvidia.com/gpu=true: present"
|
||||
|
||||
[[ $(kubectl get node "$NODE" -o jsonpath='{.spec.unschedulable}') == true ]] \
|
||||
&& info "node is cordoned (expected; uncordoned before engines deploy)"
|
||||
info "mode: $MODE"
|
||||
}
|
||||
|
||||
runtimeclass() {
|
||||
# Must exist before ANY pod with runtimeClassName: nvidia is admitted —
|
||||
# including the device plugin itself. The Talos extension registers the
|
||||
# containerd handler but not this object.
|
||||
if [[ $MODE == manual ]]; then
|
||||
log "apply RuntimeClass nvidia"
|
||||
run kubectl apply -k "$REPO_ROOT/k8s/apps/gpu-runtimeclass"
|
||||
else
|
||||
sync_app gpu-runtimeclass
|
||||
fi
|
||||
$DRY || kubectl get runtimeclass nvidia >/dev/null 2>&1 \
|
||||
&& info "RuntimeClass nvidia: present"
|
||||
}
|
||||
|
||||
gpu_plugin() {
|
||||
runtimeclass
|
||||
if [[ $MODE == manual ]]; then
|
||||
log "helm install: nvidia-device-plugin $NVDP_VER"
|
||||
# Driver + container toolkit come from Talos system extensions. This chart
|
||||
# only advertises the GPUs to the kubelet — it must not install drivers.
|
||||
run helm upgrade --install nvidia-device-plugin nvidia-device-plugin \
|
||||
--repo https://nvidia.github.io/k8s-device-plugin \
|
||||
--version "$NVDP_VER" \
|
||||
-n "$GPU_NS" --create-namespace \
|
||||
--set runtimeClassName=nvidia \
|
||||
`# --set-string, NOT --set: nodeSelector is map[string]string in the` \
|
||||
`# PodSpec schema, and plain --set coerces "true" to a YAML boolean,` \
|
||||
`# which the API server rejects as a type violation.` \
|
||||
--set-string nodeSelector."nvidia\.com/gpu"=true \
|
||||
`# Drop the chart's default nodeAffinity. It requires one of three` \
|
||||
`# Node-Feature-Discovery labels (feature.node.kubernetes.io/pci-10de.present,` \
|
||||
`# .../cpu-model.vendor_id=NVIDIA, or nvidia.com/gpu.present). NFD is not` \
|
||||
`# installed here and Talos sets nvidia.com/gpu (no ".present" suffix), so` \
|
||||
`# the affinity matches zero nodes and the DaemonSet silently reports` \
|
||||
`# desiredNumberScheduled=0. The nodeSelector above is our constraint.` \
|
||||
`# Deliberately NO --wait: it blocks on DaemonSet readiness, and any` \
|
||||
`# admission failure then times helm out and wedges the release in` \
|
||||
`# pending-upgrade, blocking all later upgrades. The allocatable-GPU` \
|
||||
`# poll below is the real readiness signal.` \
|
||||
--set affinity=null
|
||||
else
|
||||
sync_app nvidia-device-plugin
|
||||
fi
|
||||
|
||||
log "waiting for nvidia.com/gpu to register on $NODE"
|
||||
$DRY && { info "[dry-run] would verify allocatable == 4"; return 0; }
|
||||
local n=""
|
||||
for _ in $(seq 1 60); do
|
||||
n=$(kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' 2>/dev/null || true)
|
||||
[[ -n $n && $n != 0 ]] && break
|
||||
sleep 5
|
||||
done
|
||||
[[ $n == 4 ]] || die "expected 4 allocatable GPUs, got '${n:-none}'"
|
||||
info "allocatable nvidia.com/gpu: $n"
|
||||
}
|
||||
|
||||
kserve() {
|
||||
if [[ $MODE == manual ]]; then
|
||||
log "helm install: kserve-crd $KSERVE_VER"
|
||||
# No --wait on helm (a timeout leaves the release wedged in pending-upgrade,
|
||||
# blocking every later upgrade). kubectl wait below is the readiness signal
|
||||
# and does not touch helm release state.
|
||||
run helm upgrade --install kserve-crd "oci://ghcr.io/kserve/charts/kserve-crd" \
|
||||
--version "$KSERVE_VER" -n "$KSERVE_NS" --create-namespace
|
||||
|
||||
run kubectl wait --for=condition=established --timeout=120s \
|
||||
crd/inferenceservices.serving.kserve.io
|
||||
|
||||
log "helm install: kserve $KSERVE_VER (RawDeployment)"
|
||||
# NOTE the nesting on enableGatewayApi: gateway.ingressGateway.enableGatewayApi.
|
||||
# Helm silently ignores a wrong key rather than erroring.
|
||||
run helm upgrade --install kserve "oci://ghcr.io/kserve/charts/kserve" \
|
||||
--version "$KSERVE_VER" -n "$KSERVE_NS" \
|
||||
--set kserve.controller.deploymentMode=RawDeployment \
|
||||
--set kserve.controller.gateway.ingressGateway.enableGatewayApi=false
|
||||
# Controller readiness via kubectl, not helm --wait, for the same reason.
|
||||
run kubectl rollout status deploy -n "$KSERVE_NS" --timeout=600s
|
||||
else
|
||||
sync_app kserve-crd
|
||||
run kubectl wait --for=condition=established --timeout=120s \
|
||||
crd/inferenceservices.serving.kserve.io
|
||||
sync_app kserve
|
||||
fi
|
||||
info "KServe ready (RawDeployment mode)"
|
||||
}
|
||||
|
||||
uncordon() {
|
||||
log "uncordon $NODE"
|
||||
# Must precede the engines. The device-plugin DaemonSet tolerates the
|
||||
# unschedulable taint automatically; the engine Deployments do not and would
|
||||
# sit Pending forever.
|
||||
run kubectl uncordon "$NODE"
|
||||
}
|
||||
|
||||
engines() {
|
||||
if [[ $MODE == manual ]]; then
|
||||
log "kubectl apply -k k8s/apps/llm-serving"
|
||||
run kubectl apply -k "$REPO_ROOT/k8s/apps/llm-serving"
|
||||
else
|
||||
sync_app llm-serving
|
||||
fi
|
||||
|
||||
log "waiting for InferenceServices (first start pulls ~60GB of weights)"
|
||||
run kubectl wait --for=condition=Ready --timeout=2400s \
|
||||
inferenceservice --all -n "$NS" \
|
||||
|| info "not all Ready yet — check: kubectl get pods -n $NS"
|
||||
}
|
||||
|
||||
smoke() {
|
||||
log "smoke test"
|
||||
$DRY && { info "[dry-run] would curl the three engines"; return 0; }
|
||||
|
||||
# Engines are ClusterIP behind a default-deny NetworkPolicy, so probe from
|
||||
# inside the cluster with the llm-client label that the policy allows.
|
||||
#
|
||||
# KServe names the Service "<isvc>-predictor" (constants.PredictorServiceName)
|
||||
# on port 80 -> containerPort 8080 — NOT "<isvc>" on 8080.
|
||||
probe() {
|
||||
local isvc=$1 path=$2 body=$3
|
||||
printf ' %-24s ' "$isvc"
|
||||
if kubectl run "smoke-$RANDOM" -n "$NS" --rm -i --restart=Never -q \
|
||||
--image=curlimages/curl:8.11.1 \
|
||||
--labels="llm-client=true" \
|
||||
--command -- curl -sf -m 300 -X POST \
|
||||
"http://${isvc}-predictor.${NS}.svc.cluster.local/${path}" \
|
||||
-H 'Content-Type: application/json' -d "$body" >/dev/null 2>&1; then
|
||||
printf '\033[32mOK\033[0m\n'
|
||||
else
|
||||
printf '\033[31mFAIL\033[0m\n'
|
||||
fi
|
||||
}
|
||||
|
||||
# Thinking models (reasoning, ornith) spend their first tokens inside a
|
||||
# <think> block, so a small max_tokens returns EMPTY content and looks like a
|
||||
# failure. Give them room.
|
||||
probe reasoning v1/completions \
|
||||
'{"model":"reasoning","prompt":"2+2=","max_tokens":200}'
|
||||
probe ornith v1/chat/completions \
|
||||
'{"model":"ornith:35b","messages":[{"role":"user","content":"hi"}],"max_tokens":400}'
|
||||
# Same Ollama endpoint, second co-resident model on GPU2.
|
||||
probe ornith v1/chat/completions \
|
||||
'{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'
|
||||
# Reward model: scores via /pooling with an `input` field, NOT /v1/completions.
|
||||
probe verifier pooling \
|
||||
'{"model":"verifier","input":"Step 1: 2+2=4."}'
|
||||
# TEI encoders (CPU): /embed and /rerank, not an OpenAI-shaped API.
|
||||
# Nomic v2 needs the search_document:/search_query: prefix from the client.
|
||||
probe embeddings embed \
|
||||
'{"inputs":"search_query: hello"}'
|
||||
probe reranker rerank \
|
||||
'{"query":"how to sort a list","texts":["use sorted()","unrelated text"]}'
|
||||
}
|
||||
|
||||
status() {
|
||||
log "status"
|
||||
kubectl get node "$NODE" -o wide
|
||||
echo
|
||||
kubectl get inferenceservice -n "$NS" 2>/dev/null || info "no InferenceServices yet"
|
||||
echo
|
||||
kubectl get pods -n "$NS" -o wide 2>/dev/null || true
|
||||
echo
|
||||
info "GPU allocatable: $(kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' 2>/dev/null || echo none)"
|
||||
}
|
||||
|
||||
# Every check that this bring-up has actually needed, in one command, so
|
||||
# troubleshooting never requires ad-hoc kubectl archaeology again. Read-only.
|
||||
doctor() {
|
||||
log "doctor — full diagnostic"
|
||||
set +e # advisory only: never abort on an absent resource
|
||||
local fail=0
|
||||
chk() { # chk <label> <expected> <actual>
|
||||
if [[ "$2" == "$3" ]]; then printf ' \033[32m✓\033[0m %-38s %s\n' "$1" "$3"
|
||||
else printf ' \033[31m✗\033[0m %-38s got=%s want=%s\n' "$1" "${3:-<empty>}" "$2"; fail=$((fail+1)); fi
|
||||
}
|
||||
|
||||
echo " [node]"
|
||||
chk "Ready" True "$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)"
|
||||
local cordon; cordon=$(kubectl get node "$NODE" -o jsonpath='{.spec.unschedulable}' 2>/dev/null || true)
|
||||
printf ' %-40s %s\n' "cordoned" "${cordon:-false} (must be false before engines)"
|
||||
chk "driver extension label" present \
|
||||
"$(kubectl get node "$NODE" -o jsonpath='{.metadata.labels}' 2>/dev/null | grep -q nonfree-kmod-nvidia && echo present)"
|
||||
chk "label nvidia.com/gpu" true "$(kubectl get node "$NODE" -o jsonpath='{.metadata.labels.nvidia\.com/gpu}' 2>/dev/null)"
|
||||
chk "allocatable nvidia.com/gpu" 4 "$(kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' 2>/dev/null)"
|
||||
|
||||
echo " [containerd / runtime]"
|
||||
# The Talos extension registers the handler; the RuntimeClass object is ours.
|
||||
chk "RuntimeClass nvidia" nvidia "$(kubectl get runtimeclass nvidia -o jsonpath='{.handler}' 2>/dev/null)"
|
||||
|
||||
echo " [device plugin]"
|
||||
local st; st=$(helm list -n "$GPU_NS" -a -o json 2>/dev/null \
|
||||
| python3 -c 'import json,sys;print(next((r["status"] for r in json.load(sys.stdin) if r["name"]=="nvidia-device-plugin"),""))' 2>/dev/null || true)
|
||||
chk "helm release status" deployed "$st"
|
||||
[[ $st == pending-* || $st == failed ]] && \
|
||||
info " -> stuck release: run '$0 unstick' (helm refuses upgrades in this state)"
|
||||
chk "DaemonSet desired" 1 "$(kubectl get ds -n "$GPU_NS" nvidia-device-plugin -o jsonpath='{.status.desiredNumberScheduled}' 2>/dev/null)"
|
||||
chk "DaemonSet ready" 1 "$(kubectl get ds -n "$GPU_NS" nvidia-device-plugin -o jsonpath='{.status.numberReady}' 2>/dev/null)"
|
||||
local aff; aff=$(kubectl get ds -n "$GPU_NS" nvidia-device-plugin -o jsonpath='{.spec.template.spec.affinity}' 2>/dev/null || true)
|
||||
[[ -z $aff ]] && printf ' \033[32m✓\033[0m %-38s removed\n' "chart nodeAffinity (needs NFD)" \
|
||||
|| { printf ' \033[31m✗\033[0m %-38s present — requires NFD labels, will match 0 nodes\n' "chart nodeAffinity"; fail=$((fail+1)); }
|
||||
local ev; ev=$(kubectl get events -n "$GPU_NS" --field-selector reason=FailedCreate \
|
||||
-o jsonpath='{.items[-1:].message}' 2>/dev/null || true)
|
||||
[[ -n $ev ]] && info " last FailedCreate: ${ev:0:110}"
|
||||
|
||||
echo " [kserve]"
|
||||
chk "InferenceService CRD" true \
|
||||
"$(kubectl get crd inferenceservices.serving.kserve.io >/dev/null 2>&1 && echo true)"
|
||||
local dm; dm=$(kubectl get cm inferenceservice-config -n "$KSERVE_NS" -o jsonpath='{.data.deploy}' 2>/dev/null \
|
||||
| grep -o '"defaultDeploymentMode": *"[^"]*"' | sed 's/.*"\([^"]*\)"$/\1/' || true)
|
||||
chk "deploymentMode" RawDeployment "$dm"
|
||||
|
||||
echo " [engines]"
|
||||
if kubectl get ns "$NS" >/dev/null 2>&1; then
|
||||
kubectl get inferenceservice -n "$NS" --no-headers 2>/dev/null | awk '{printf " %-24s ready=%s\n",$1,$3}'
|
||||
kubectl get pods -n "$NS" --no-headers 2>/dev/null \
|
||||
| awk '{printf " %-40s %s %s\n",$1,$3,$5}'
|
||||
else
|
||||
info "namespace $NS absent (engines not deployed yet)"
|
||||
fi
|
||||
|
||||
echo
|
||||
[[ $fail -eq 0 ]] && info "all checks passed" || info "$fail check(s) failed"
|
||||
set -e
|
||||
return 0
|
||||
}
|
||||
|
||||
# Clear a helm release wedged in pending-* / failed. Helm blocks every
|
||||
# subsequent upgrade with "another operation is in progress" until this is done.
|
||||
unstick() {
|
||||
log "unstick helm release"
|
||||
local st; st=$(helm list -n "$GPU_NS" -a -o json 2>/dev/null \
|
||||
| python3 -c 'import json,sys;print(next((r["status"] for r in json.load(sys.stdin) if r["name"]=="nvidia-device-plugin"),""))' 2>/dev/null)
|
||||
info "current status: ${st:-<no release>}"
|
||||
case $st in
|
||||
deployed) info "nothing to do" ;;
|
||||
"") info "no release; nothing to do" ;;
|
||||
*) # Roll back to the last deployed revision; if none, uninstall.
|
||||
if helm history nvidia-device-plugin -n "$GPU_NS" 2>/dev/null | grep -q deployed; then
|
||||
run helm rollback nvidia-device-plugin -n "$GPU_NS"
|
||||
else
|
||||
run helm uninstall nvidia-device-plugin -n "$GPU_NS"
|
||||
fi ;;
|
||||
esac
|
||||
# The DaemonSet controller backs off after repeated FailedCreate and will not
|
||||
# retry for many minutes even after the underlying cause is fixed. Force it.
|
||||
if kubectl get ds -n "$GPU_NS" nvidia-device-plugin >/dev/null 2>&1; then
|
||||
log "nudging DaemonSet (clears admission backoff)"
|
||||
run kubectl rollout restart ds/nvidia-device-plugin -n "$GPU_NS"
|
||||
fi
|
||||
}
|
||||
|
||||
# Remove everything this script creates, so a retry starts from a clean slate.
|
||||
# Does NOT touch the node's Talos config or the models PVC by default.
|
||||
teardown() {
|
||||
log "teardown"
|
||||
info "this removes: llm-serving ns, kserve, device plugin, RuntimeClass"
|
||||
if ! $DRY; then
|
||||
printf ' type "yes" to proceed: '; local a; read -r a
|
||||
[[ $a == yes ]] || die "aborted"
|
||||
fi
|
||||
run kubectl delete -k "$REPO_ROOT/k8s/apps/llm-serving" --ignore-not-found
|
||||
run helm uninstall kserve -n "$KSERVE_NS" --ignore-not-found 2>/dev/null || true
|
||||
run helm uninstall kserve-crd -n "$KSERVE_NS" --ignore-not-found 2>/dev/null || true
|
||||
run helm uninstall nvidia-device-plugin -n "$GPU_NS" --ignore-not-found 2>/dev/null || true
|
||||
run kubectl delete -k "$REPO_ROOT/k8s/apps/gpu-runtimeclass" --ignore-not-found
|
||||
info "PVC llm-models in ns $NS was NOT deleted (holds ~56Gi of downloaded weights)"
|
||||
info "delete explicitly if you want a cold re-download:"
|
||||
info " kubectl delete pvc llm-models -n $NS"
|
||||
}
|
||||
|
||||
all() { preflight; gpu_plugin; kserve; uncordon; engines; smoke; status; }
|
||||
|
||||
# ---------------------------------------------------------------- main
|
||||
|
||||
STAGE=all
|
||||
for a in "$@"; do
|
||||
case $a in
|
||||
--manual) MODE=manual ;;
|
||||
--argocd) MODE=argocd ;;
|
||||
--dry-run) DRY=true ;;
|
||||
preflight|runtimeclass|gpu-plugin|kserve|uncordon|engines|smoke|status|doctor|unstick|teardown|all) STAGE=$a ;;
|
||||
-h|--help) sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) die "unknown argument: $a" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
$DRY && info "DRY RUN — no changes will be made"
|
||||
case $STAGE in
|
||||
preflight) preflight ;;
|
||||
runtimeclass) runtimeclass ;;
|
||||
gpu-plugin) preflight; gpu_plugin ;;
|
||||
kserve) preflight; kserve ;;
|
||||
uncordon) uncordon ;;
|
||||
engines) preflight; engines ;;
|
||||
smoke) smoke ;;
|
||||
status) status ;;
|
||||
doctor) doctor ;;
|
||||
unstick) unstick ;;
|
||||
teardown) teardown ;;
|
||||
all) all ;;
|
||||
esac
|
||||
|
||||
log "done"
|
||||
Reference in New Issue
Block a user