Story Crater Bot
4fb1c6feeb
Layout on 4x Tesla V100 32GB (PCIe, no NVLink), all TP=1: GPU0+1 vLLM DeepSeek-R1-Distill-Qwen-32B bnb-nf4 (2 replicas) GPU2 Ollama ornith:35b + qwen2.5:3b-instruct (co-resident) GPU3 vLLM Qwen2.5-Math-PRM-7B (reward model) CPU TEI nomic-embed-text-v2-moe, bge-reranker-base Volta constraints, each verified against live output rather than config: - vLLM pinned v0.11.0: sm_70 dropped from CUDA_SUPPORTED_ARCHS at v0.11.1. - AWQ hard-rejected (needs sm_75). GPTQ passes vLLM's min_capability=60 gate but is NUMERICALLY WRONG on sm_70 — emits garbage logits. Proven by an fp16 control run producing correct text on an identical backend. bitsandbytes nf4 verified correct by output. - flashinfer's check_cuda_arch() crashes on any sm_7x (calls .isdigit() on an int) -> VLLM_USE_FLASHINFER_SAMPLER=0. - xformers has no sm_70 kernel for V1's paged-attention bias, and V0 was removed in v0.11.0 -> TRITON_ATTN. - Ornith is Qwen3.5-MoE hybrid-attention; vLLM added that arch after dropping Volta, so no build has both -> Ollama, which also multiplexes a second model on the same card for free. Cluster prereqs that were absent: - RuntimeClass nvidia: the Talos toolkit extension registers the containerd handler but not the k8s object; without it every pod is rejected at admission. - gpu-system pinned to privileged PSA: a device plugin cannot satisfy the cluster-default baseline, it must mount hostPath. - device-plugin affinity=null: the chart requires NFD labels that do not exist here, so it matched zero nodes and reported desiredNumberScheduled=0 silently. - Recreate strategy on GPU services: with GPUs allocated exactly 4/4, a RollingUpdate surge pod has no card and deadlocks the rollout. - longhorn-llm-local SC (1 replica, strict-local, disk tag llm): the default 3-replica class could not place the volume at all (every control-plane disk was at its over-provisioning ceiling), and this keeps ~60GB of weights on worker-1's own NVMe instead of reading them over the network. deploy-gpu-serving.sh sequences ArgoCD syncs (or helm/kubectl in --manual mode) and never applies a manifest absent from git; doctor/unstick/teardown stages exist so this is diagnosable without ad-hoc kubectl archaeology.
108 lines
4.2 KiB
YAML
108 lines
4.2 KiB
YAML
# Action engine — Ornith-1.0-35B on Ollama.
|
|
#
|
|
# Why not vLLM like the other two: Ornith is
|
|
# Qwen3_5MoeForConditionalGeneration (Qwen3.5 MoE, 256 experts / 8 active,
|
|
# hybrid attention — 30 linear_attention + 10 full_attention layers). vLLM's
|
|
# Qwen3.5 support landed 2026-07-29, AFTER vLLM dropped Volta (sm_70) at
|
|
# v0.11.1. No vLLM build has both, so Ornith cannot run on vLLM on a V100.
|
|
#
|
|
# Ollama ships `ornith:35b` in its library and runs a llama-server runner
|
|
# underneath, which keeps Volta support. q4 is ~21GB — fits one 32GB V100 with
|
|
# room for KV.
|
|
#
|
|
# OLLAMA_KEEP_ALIVE=-1 is load-bearing: the harness calls this every loop
|
|
# iteration, and Ollama's default is to evict an idle model after 5m, which
|
|
# would add a ~21GB reload to a random future request.
|
|
apiVersion: serving.kserve.io/v1beta1
|
|
kind: InferenceService
|
|
metadata:
|
|
name: ornith
|
|
labels:
|
|
app.kubernetes.io/name: llm-ornith
|
|
app.kubernetes.io/part-of: llm-serving
|
|
spec:
|
|
predictor:
|
|
minReplicas: 1
|
|
# Recreate, not the default RollingUpdate: GPUs are allocated exactly 4/4,
|
|
# so a surge pod has no card to claim and sits Pending while the old pod is
|
|
# never torn down — a deadlock. Recreate tears down first, accepting a brief
|
|
# gap during updates.
|
|
deploymentStrategy:
|
|
type: Recreate
|
|
maxReplicas: 1
|
|
nodeSelector:
|
|
kubernetes.io/hostname: worker-1
|
|
runtimeClassName: nvidia
|
|
containers:
|
|
- name: kserve-container
|
|
image: ollama/ollama:0.32.9@sha256:1685741456770df6e3cceb2a945a5f75e020f658d1701509668d6f4688f1dd3f
|
|
# `ollama serve` does not pull models, and `ollama pull` needs a running
|
|
# server — so background the server, wait for it, pull, then hand the
|
|
# foreground back to serve.
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
set -e
|
|
ollama serve &
|
|
SERVE_PID=$!
|
|
until ollama list >/dev/null 2>&1; do sleep 2; done
|
|
ollama pull ornith:35b
|
|
ollama pull qwen2.5:3b-instruct
|
|
wait $SERVE_PID
|
|
env:
|
|
# Match the port the other two engines use.
|
|
- name: OLLAMA_HOST
|
|
value: "0.0.0.0:8080"
|
|
- name: OLLAMA_MODELS
|
|
value: /mnt/models/ollama
|
|
# Ollama defaults to a 4096 context, far too small for an agentic
|
|
# coding model. Ornith's hybrid attention means only 10 of its 40
|
|
# layers hold a conventional KV cache, so 32K is affordable in the
|
|
# ~11GiB left after its 21GB of weights.
|
|
- name: OLLAMA_CONTEXT_LENGTH
|
|
value: "32768"
|
|
# Never evict — this model is on the harness's hot path.
|
|
- name: OLLAMA_KEEP_ALIVE
|
|
value: "-1"
|
|
# Serial agent loop; no benefit from parallel slots.
|
|
- name: OLLAMA_NUM_PARALLEL
|
|
value: "1"
|
|
# 2, so ornith and the small utility model stay co-resident on GPU2
|
|
# instead of evicting one another on every alternating request.
|
|
- name: OLLAMA_MAX_LOADED_MODELS
|
|
value: "2"
|
|
ports:
|
|
- containerPort: 8080
|
|
protocol: TCP
|
|
resources:
|
|
requests:
|
|
cpu: "8"
|
|
memory: 8Gi
|
|
nvidia.com/gpu: "1"
|
|
limits:
|
|
cpu: "16"
|
|
memory: 16Gi
|
|
nvidia.com/gpu: "1"
|
|
volumeMounts:
|
|
- name: models
|
|
mountPath: /mnt/models
|
|
# Probes must confirm the MODEL is present, not just that the server
|
|
# answers. Ollama's `GET /` returns 200 ("Ollama is running") the moment
|
|
# `ollama serve` binds — which is before the ~21GB pull finishes. An
|
|
# httpGet probe would therefore mark this pod Ready with no model
|
|
# loaded, and KServe would route traffic to it.
|
|
startupProbe:
|
|
exec:
|
|
command: ["/bin/sh", "-c", "ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | grep -q qwen2.5"]
|
|
periodSeconds: 15
|
|
failureThreshold: 120
|
|
readinessProbe:
|
|
exec:
|
|
command: ["/bin/sh", "-c", "ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | grep -q qwen2.5"]
|
|
periodSeconds: 10
|
|
volumes:
|
|
- name: models
|
|
persistentVolumeClaim:
|
|
claimName: llm-models
|