rock
dc5ec40c7d
The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.
Measured, on worker-1, driver 580.159.04, compute capability 7.0:
GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
POST /v1/chat/completions model=coder -> 200 in 3.18s,
"fn main() { println!(\"hi\"); }"
POST /v1/chat/completions model=reasoning -> 200 in 0.21s
Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.
Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.
Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:
TORCH 2.9.1+cu129
ARCH_LIST [] <- empty
RuntimeError: cudaGetDeviceCount() ... Error 803
fla: ModuleNotFoundError <- no GatedDeltaNet kernels
vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.
Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.
reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.
41 lines
2.0 KiB
YAML
41 lines
2.0 KiB
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# Explicit allowlist, matching k8s/apps/api. Anything added to this directory
|
|
# and not listed here is silently dropped — no error, no drift shown.
|
|
#
|
|
# These five were adopted from live state on 2026-08-15; they had been applied
|
|
# by hand and carried no ArgoCD ownership. Each was exported and verified with
|
|
# `kubectl diff -f <file>` returning empty before the Application below was
|
|
# created, so the first sync was a no-op rather than a redeploy. Re-verify that
|
|
# way after any edit here: a GPU predictor restart is a weights reload measured
|
|
# in tens of seconds, not a rolling update.
|
|
resources:
|
|
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
|
|
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
|
|
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
|
|
- clef-serve.yaml
|
|
# coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ, TP=2 at 262,144 tokens. It serves
|
|
# both `coder` and `reasoning`, so poimen-memory needs a host change only, never a
|
|
# model-name change. reasoning.yaml below can be dropped once that host change lands.
|
|
- coder.yaml
|
|
- reasoning.yaml
|
|
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
|
# replicas, because desired equals available at 0. Its manifest was never committed,
|
|
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
|
|
- embeddings.yaml
|
|
- reranker.yaml
|
|
- qwen-cpu.yaml
|
|
- networkpolicy.yaml
|
|
# No namespace transformer: every file sets its own, and the transformer would
|
|
# rewrite metadata.namespace on anything cross-namespace added later.
|
|
|
|
# app.py lives in git next to this file. The generator hashes the ConfigMap name, so an
|
|
# edit to app.py rolls the pod instead of leaving a stale copy mounted.
|
|
configMapGenerator:
|
|
- name: clef-serve-app
|
|
namespace: llm-serving
|
|
files:
|
|
- clef-serve/app.py
|
|
options:
|
|
disableNameSuffixHash: false
|