rock
3ab127e581
Adds clef-candle.yaml: clef-flash (hybrid Qwen3_5, 8 attention + 24 GDN layers) on one V100 via candle-vllm, and repoints the clef-serve Service selector at it. The Service name and ClusterIP do not move, so no client changes. Runs on one card because the GDN recurrent state is per-sequence and cannot be sharded the way attention KV can. --kv-fraction 0.30 gives about 66.8K KV tokens, which is ample for a single-shot classifier-style endpoint. The image is candle-vllm:sm70, built by .gitea/workflows/candle-vllm-sm70.yaml on every merge to main. It must be a build containing the GDN conv fix: older images zero the key channel band in causal_conv1d_fwd, starve the delta rule and make every linear-attention layer return zero. That failure is silent - the response is well formed, just wrong. Verified against the reference implementation: token ids 420/420 identical, beta 8647.021 vs 8646.982, delta-rule and gated-norm absmax equal to six decimals, all three head answers agree.
53 lines
2.7 KiB
YAML
53 lines
2.7 KiB
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# Explicit allowlist, matching k8s/apps/api. Anything added to this directory
|
|
# and not listed here is silently dropped — no error, no drift shown.
|
|
#
|
|
# These five were adopted from live state on 2026-08-15; they had been applied
|
|
# by hand and carried no ArgoCD ownership. Each was exported and verified with
|
|
# `kubectl diff -f <file>` returning empty before the Application below was
|
|
# created, so the first sync was a no-op rather than a redeploy. Re-verify that
|
|
# way after any edit here: a GPU predictor restart is a weights reload measured
|
|
# in tens of seconds, not a rolling update.
|
|
resources:
|
|
# clef-flash runs through clef-serve.yaml, not through an SGLang runtime. SGLang needs
|
|
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
|
|
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
|
|
# coder-14b-fp16.yaml owns the `candle-vllm` Deployment name and the
|
|
# `app: candle-vllm` label that coder-service.yaml selects. It must be the only
|
|
# workload with that label, or the Service splits traffic across two models.
|
|
#
|
|
# fp16 on two V100s, not GPTQ on one: measured 39.7 ms ITL and 287 tok/s at 16
|
|
# streams against 71.5 ms and 215 tok/s for the 4-bit MoE. The V100 has FP16
|
|
# tensor cores and no INT4 ones, so 4-bit pays dequantisation on the SM while
|
|
# the memory controller idles at 19%.
|
|
# clef-candle.yaml serves clef-flash on candle-vllm (hybrid Qwen3_5, one GPU)
|
|
# and owns the clef-candle label that clef-serve.yaml's Service now selects.
|
|
# It must not sync against an image built before the GDN conv fix.
|
|
- clef-candle.yaml
|
|
- coder-14b-fp16.yaml
|
|
- clef-serve.yaml
|
|
# Stable client-facing name for the chat model. poimen-memory points at this Service,
|
|
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
|
|
- coder-service.yaml
|
|
- reasoning-long-service.yaml
|
|
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
|
# replicas, because desired equals available at 0. Its manifest was never committed,
|
|
# so it is left unmanaged and is not listed here. Delete the CR when convenient.
|
|
- embeddings.yaml
|
|
- reranker.yaml
|
|
- qwen-cpu.yaml
|
|
- networkpolicy.yaml
|
|
# No namespace transformer: every file sets its own, and the transformer would
|
|
# rewrite metadata.namespace on anything cross-namespace added later.
|
|
|
|
# app.py lives in git next to this file. The generator hashes the ConfigMap name, so an
|
|
# edit to app.py rolls the pod instead of leaving a stale copy mounted.
|
|
configMapGenerator:
|
|
- name: clef-serve-app
|
|
namespace: llm-serving
|
|
files:
|
|
- clef-serve/app.py
|
|
options:
|
|
disableNameSuffixHash: false
|