clef-serve (pytorch reference) and clef-candle both claim replicas 1, but only
one GPU is free: coder takes 2 of the 4 V100s. They starved each other, so
clef-candle sat Pending and the clef-serve Service had no endpoints.
The Service now selects app: clef-candle, so the reference receives no traffic
even when it runs. Keeping it at 1 only burns a GPU.
The published :sm70 predates /v1/systemone and the GDN conv fix. candle-vllm
f3c00b2 carries both and is on main, but CI has not republished the tag, so the
in-image binary returns 404 on the route and would zero every linear-attention
layer if it did.
/mnt/models/bin/candle-vllm is that build. Verified serving: HTTP 200,
severity medium 0.5029, rollback 0.6581, confidence 2.6079, matching the
reference's decisions.
Temporary: retire once CI publishes a :sm70 from f3c00b2.
:sm70 is rebuilt by CI on every merge to candle-vllm main. The default
IfNotPresent means a node with a cached copy of that tag keeps running the old
binary, so a fix can be merged and deployed and still never reach the pod. The
symptom is a 404 on /v1/systemone while the image tag looks correct.
Always forces the pull.
candle-vllm PR #5 (the conv fix) could not be merged, so CI has not published
a :sm70 image containing it. Deploying clef against an older :sm70 fails
silently: the delta rule returns zero on all 24 linear-attention layers and
/v1/systemone still answers 200 with well-formed but wrong classifications.
Raise to 1 once a :sm70 build from the fix merge exists.
Adds clef-candle.yaml: clef-flash (hybrid Qwen3_5, 8 attention + 24 GDN
layers) on one V100 via candle-vllm, and repoints the clef-serve Service
selector at it. The Service name and ClusterIP do not move, so no client
changes.
Runs on one card because the GDN recurrent state is per-sequence and cannot
be sharded the way attention KV can. --kv-fraction 0.30 gives about 66.8K KV
tokens, which is ample for a single-shot classifier-style endpoint.
The image is candle-vllm:sm70, built by .gitea/workflows/candle-vllm-sm70.yaml
on every merge to main. It must be a build containing the GDN conv fix: older
images zero the key channel band in causal_conv1d_fwd, starve the delta rule
and make every linear-attention layer return zero. That failure is silent -
the response is well formed, just wrong.
Verified against the reference implementation: token ids 420/420 identical,
beta 8647.021 vs 8646.982, delta-rule and gated-norm absmax equal to six
decimals, all three head answers agree.
Without it the container runs under runc, libcuda.so is never injected and
candle-vllm panics in cudarc at startup. The pod reached Running but crashed
on every restart.
Verified live: pod Ready, coder-predictor endpoints populated, /v1/models
reports coder-14b-fp16 and a completion returns OK.
Replace the GPTQ W4A16 30B-A3B coder with Qwen2.5-Coder-14B-Instruct at
fp16 on TP=2.
The V100 has FP16 tensor cores and no INT4 ones, so the 4-bit path paid a
dequantisation pass on the SM. Profiling showed SM at 98% with the memory
controller at 19%, and the batch-1 step cost matched a full-model read.
Same harness, 16 streams: ITL 71.5 -> 39.7 ms, 215 -> 287 tok/s.
The deployment keeps the candle-vllm name and the app: candle-vllm label,
so coder-service.yaml and every client keep working unchanged.
clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds
four V100s, so the coder takes one and clef takes one, leaving two free
for a TP=2 candle run.
The gateway already routes the clef-flash model to clef-serve:80 on
/v1/systemone, so this single replica count is what turns that route on.
Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so
clients keep one stable host name across the vLLM -> candle-vllm handoff and
only this selector changes from now on.
The retired KTML predictor requested two V100s and never reached Ready.
Both ArgoCD Applications reported broken: 'kyverno' Unknown and
'kyverno-policies' OutOfSync/Missing. Neither the namespace nor a single
kyverno pod ever existed, so no policy was enforced and nothing is lost.
Kyverno is future work. Removing the two Applications and the bootstrap
directory keeps the root app from reporting two permanent failures.
The header of this file always said 2 instances while the spec said 3. The
third never ran: gotify-db-3's volume held a single replica that could not
start, because talos-cp-2 disk2 is at 2.9% free. That left the cluster
permanently in 'Creating a new replica' with one degraded Longhorn volume.
Two replicas on two different nodes is the redundancy that actually exists.
poimen-memory is a ClusterIP service on :8080, so a local pi session had no
route to the API. This adds an edge Ingress on the same nginx class and the
same Cloudflare-terminated shape as the frontend ingress.
Auth stays ZITADEL JWT (MEM_AUTH_MODE=jwt): a request without a valid bearer
token gets 401, so exposing the route does not expose the data.
Requires one Cloudflare proxied A record: api-poimen.riotpiao.com -> 192.168.1.160
poimen.riotpiao.com resolved only because someone edited the live coredns
ConfigMap by hand. Neither terraform/files/coredns/Corefile nor the Talos
machine config carried the line, so the next Talos reconcile would have
deleted it and broken in-cluster resolution of a heavily used service.
Live and source now agree on both hostnames the retirement touched.
comfyui served nothing and its oauth2-proxy could not start: the Secret
carried only client-secret while the Deployment asked for client-id,
cookie-secret and client-secret. Rather than repair a service nobody used,
remove it from every layer.
Gone:
- k8s/apps/comfyui/* and its ArgoCD Application
- the cloudflared tunnel rule for comfyui.riotpiao.com
- the CoreDNS rewrite in the terraform Corefile (the tracked source)
- the Cloudflare DNS record (terraform tfvars)
- the ZITADEL OIDC app and its two rotation tables
- the KeyRotation object
Not gone by this commit: the comfyui namespace and its 50Gi PVC. ArgoCD does
not own either, so a person deletes both by hand after the prune. The local
cluster-config/talos-*.yaml copies are gitignored; they carry the same
removal and need an apply-config to reach the live CoreDNS.
The rotation wrote a key the tenant never read, so minio restarted into
CreateContainerConfigError. The key name must match the consumer's
secretKeyRef, not a house style.
The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:
Package registry API internal error: 500 write
/data/tmp/package-upload/<id>: no space left on device
The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.
Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.
This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.
NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:
disk ... does not have sufficient physical space for expansion:
physical free space would drop below minimal:
left=14596177920 < minimal=49931236556
60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.
The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:
nr_throttled 189528 of 1169014 CFS periods (16.2%)
throttled_usec 4424898602 us (~74 minutes lost)
actual usage 951m against a 1000m limit
The user-visible symptom was slow Actions traffic in Gitea's own log:
slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms
So runners waited, timed out, and retried, which loaded the saturated process
further.
The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.
Why these numbers:
- requests.cpu 200m -> 750m. A request below real usage earns the pod little
scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
the problem and does not need room.
Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.
ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.
The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.
Three things move into git:
- clef-serve.yaml: the Deployment and the Service the gateway routes to
- configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
instead of leaving a stale copy mounted
- clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
V100, and its slot is now clef-serve.yaml
Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.
Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context.
The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same
64k window fits on one card.
Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV
cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token
prompt, and clean reasoning_content separation through the gateway.
Three settings are load-bearing on sm_70 rather than tuning:
- VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0
- VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+
- deploymentStrategy Recreate a rolling update deadlocks once every card is allocated
The manifest is committed as well as applied: reasoning.yaml had never been in git.
Clef needs SGLang v0.5.22+, which upstream publishes only on CUDA 13. CUDA 13 dropped
Volta kernels and every GPU here is a Tesla V100 (sm_70):
torch.AcceleratorError: CUDA error: no kernel image is available for execution on
the device
SGLang also states sm75 as its floor: the README hardware table omits V100, sm75 support
was removed in sgl-project/sglang#9207, and FlashInfer requires sm75+. No build has both
Clef and Volta kernels, so the model is not runnable here at any dtype or flag.
Remove it from the kustomization so nothing crash-loops, and record the reason in both
files. The manifest stays in git for the day an Ampere-or-newer GPU arrives.
forgejo-db-app is owned by the Cluster CR and was generated by CNPG: there is no
bootstrap secret, so nothing outside cloudnative-pg ever knew that password.
CNPG reads spec.bootstrap.initdb.secret only at creation. Recreate the Cluster CR and
CNPG mints a new password while the data directory keeps the old one — Forgejo starts,
stays Ready, and fails every query. That is a credential fault wearing a network fault's
clothes, and it is exactly the shape of tonight's outage.
This file is the record. Point spec.bootstrap.initdb.secret at it when the cluster is
rebuilt. The value matches the live Secret today; verified by decrypting and comparing.
The CLI path was wrong and is now correct, so the pod reaches the model loader — and
dies there:
Found GPU0 Tesla V100-PCIE-32GB which is of cuda capability 7.0.
Your installed torch==2.14.1+cu130 does not include kernels for this GPU.
Every GPU on worker-1 is a V100 (CC 7.0) and the pinned dev-clef image ships kernels for
CC 8.0 and above. Scale to zero so the pod stops crash-looping, and record the reason in
the manifest. Serving a 9.4B backbone on CPU is not a substitute.
The container args began with a bare 'serve'. This image installs its CLI at
/opt/sglang/bin/sglang and puts nothing called serve on PATH, so every start ended
with 'exec: serve: not found' — 19 restarts, and the InferenceService never became
ready. Name the binary, then the subcommand.
The gitea chart reads pod annotations from gitea.podAnnotations (values.yaml line 583).
A top-level key is silently ignored, so the previous commit changed nothing — the
Deployment came back without the annotation and Reloader still had no trigger.
GITEA__DATABASE_PASSWD comes from forgejo-db-app, which the CNPG Cluster CR owns and
generates. There is no bootstrap secret, so nothing but cloudnative-pg knows that
password. The Deployment reads it into the environment at start.
So a regenerated password leaves Forgejo authenticating with a value the database no
longer accepts: the pod stays Ready, every query fails, and the symptom looks like a
network problem rather than a credential problem. The cnpg.io/reload label on the
Secret is only a hint; this annotation is what makes Reloader act on it.
rock