clef-serve was parked at 0 while candle-vllm took a card. worker-1 holds
four V100s, so the coder takes one and clef takes one, leaving two free
for a TP=2 candle run.
The gateway already routes the clef-flash model to clef-serve:80 on
/v1/systemone, so this single replica count is what turns that route on.
Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so
clients keep one stable host name across the vLLM -> candle-vllm handoff and
only this selector changes from now on.
The retired KTML predictor requested two V100s and never reached Ready.
Both ArgoCD Applications reported broken: 'kyverno' Unknown and
'kyverno-policies' OutOfSync/Missing. Neither the namespace nor a single
kyverno pod ever existed, so no policy was enforced and nothing is lost.
Kyverno is future work. Removing the two Applications and the bootstrap
directory keeps the root app from reporting two permanent failures.
The header of this file always said 2 instances while the spec said 3. The
third never ran: gotify-db-3's volume held a single replica that could not
start, because talos-cp-2 disk2 is at 2.9% free. That left the cluster
permanently in 'Creating a new replica' with one degraded Longhorn volume.
Two replicas on two different nodes is the redundancy that actually exists.
poimen-memory is a ClusterIP service on :8080, so a local pi session had no
route to the API. This adds an edge Ingress on the same nginx class and the
same Cloudflare-terminated shape as the frontend ingress.
Auth stays ZITADEL JWT (MEM_AUTH_MODE=jwt): a request without a valid bearer
token gets 401, so exposing the route does not expose the data.
Requires one Cloudflare proxied A record: api-poimen.riotpiao.com -> 192.168.1.160
poimen.riotpiao.com resolved only because someone edited the live coredns
ConfigMap by hand. Neither terraform/files/coredns/Corefile nor the Talos
machine config carried the line, so the next Talos reconcile would have
deleted it and broken in-cluster resolution of a heavily used service.
Live and source now agree on both hostnames the retirement touched.
comfyui served nothing and its oauth2-proxy could not start: the Secret
carried only client-secret while the Deployment asked for client-id,
cookie-secret and client-secret. Rather than repair a service nobody used,
remove it from every layer.
Gone:
- k8s/apps/comfyui/* and its ArgoCD Application
- the cloudflared tunnel rule for comfyui.riotpiao.com
- the CoreDNS rewrite in the terraform Corefile (the tracked source)
- the Cloudflare DNS record (terraform tfvars)
- the ZITADEL OIDC app and its two rotation tables
- the KeyRotation object
Not gone by this commit: the comfyui namespace and its 50Gi PVC. ArgoCD does
not own either, so a person deletes both by hand after the prune. The local
cluster-config/talos-*.yaml copies are gitignored; they carry the same
removal and need an apply-config to reach the live CoreDNS.
The rotation wrote a key the tenant never read, so minio restarted into
CreateContainerConfigError. The key name must match the consumer's
secretKeyRef, not a house style.
The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:
Package registry API internal error: 500 write
/data/tmp/package-upload/<id>: no space left on device
The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.
Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.
This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.
NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:
disk ... does not have sufficient physical space for expansion:
physical free space would drop below minimal:
left=14596177920 < minimal=49931236556
60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.
The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:
nr_throttled 189528 of 1169014 CFS periods (16.2%)
throttled_usec 4424898602 us (~74 minutes lost)
actual usage 951m against a 1000m limit
The user-visible symptom was slow Actions traffic in Gitea's own log:
slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms
So runners waited, timed out, and retried, which loaded the saturated process
further.
The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.
Why these numbers:
- requests.cpu 200m -> 750m. A request below real usage earns the pod little
scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
the problem and does not need room.
Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.
ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.
The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.
Three things move into git:
- clef-serve.yaml: the Deployment and the Service the gateway routes to
- configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
instead of leaving a stale copy mounted
- clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
V100, and its slot is now clef-serve.yaml
Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.
Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context.
The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same
64k window fits on one card.
Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV
cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token
prompt, and clean reasoning_content separation through the gateway.
Three settings are load-bearing on sm_70 rather than tuning:
- VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0
- VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+
- deploymentStrategy Recreate a rolling update deadlocks once every card is allocated
The manifest is committed as well as applied: reasoning.yaml had never been in git.
Clef needs SGLang v0.5.22+, which upstream publishes only on CUDA 13. CUDA 13 dropped
Volta kernels and every GPU here is a Tesla V100 (sm_70):
torch.AcceleratorError: CUDA error: no kernel image is available for execution on
the device
SGLang also states sm75 as its floor: the README hardware table omits V100, sm75 support
was removed in sgl-project/sglang#9207, and FlashInfer requires sm75+. No build has both
Clef and Volta kernels, so the model is not runnable here at any dtype or flag.
Remove it from the kustomization so nothing crash-loops, and record the reason in both
files. The manifest stays in git for the day an Ampere-or-newer GPU arrives.
forgejo-db-app is owned by the Cluster CR and was generated by CNPG: there is no
bootstrap secret, so nothing outside cloudnative-pg ever knew that password.
CNPG reads spec.bootstrap.initdb.secret only at creation. Recreate the Cluster CR and
CNPG mints a new password while the data directory keeps the old one — Forgejo starts,
stays Ready, and fails every query. That is a credential fault wearing a network fault's
clothes, and it is exactly the shape of tonight's outage.
This file is the record. Point spec.bootstrap.initdb.secret at it when the cluster is
rebuilt. The value matches the live Secret today; verified by decrypting and comparing.
The CLI path was wrong and is now correct, so the pod reaches the model loader — and
dies there:
Found GPU0 Tesla V100-PCIE-32GB which is of cuda capability 7.0.
Your installed torch==2.14.1+cu130 does not include kernels for this GPU.
Every GPU on worker-1 is a V100 (CC 7.0) and the pinned dev-clef image ships kernels for
CC 8.0 and above. Scale to zero so the pod stops crash-looping, and record the reason in
the manifest. Serving a 9.4B backbone on CPU is not a substitute.
The container args began with a bare 'serve'. This image installs its CLI at
/opt/sglang/bin/sglang and puts nothing called serve on PATH, so every start ended
with 'exec: serve: not found' — 19 restarts, and the InferenceService never became
ready. Name the binary, then the subcommand.
The gitea chart reads pod annotations from gitea.podAnnotations (values.yaml line 583).
A top-level key is silently ignored, so the previous commit changed nothing — the
Deployment came back without the annotation and Reloader still had no trigger.
GITEA__DATABASE_PASSWD comes from forgejo-db-app, which the CNPG Cluster CR owns and
generates. There is no bootstrap secret, so nothing but cloudnative-pg knows that
password. The Deployment reads it into the environment at start.
So a regenerated password leaves Forgejo authenticating with a value the database no
longer accepts: the pod stays Ready, every query fails, and the symptom looks like a
network problem rather than a credential problem. The cnpg.io/reload label on the
Secret is only a hint; this annotation is what makes Reloader act on it.
The gateway compares the token audience with its allow-list of client ids. The scope
carried urn:zitadel:iam:org:project🆔<project>:aud, so every token said
aud = [396757796898471937] and the gateway answered
token validation failed: audience not allowed: [396757796898471937]
With plain openid the audience is the machine user, which is the entry that exists on
the list.
A Secret volume is owned by root and the sidecar runs as 65532, so defaultMode 0400
left it unable to open /var/run/llm-key/key: the container restarted with
'permission denied' on every start and never minted a token.
Set mode 0440 and give the pod fsGroup 65532, so the group that owns the volume can
read the key while nothing else on the host can.
worker-1 holds 64Gi and the other predictors request about 55Gi, so the 16Gi request
left clef-flash Pending with '0/4 nodes are available: 1 Insufficient memory'. The
weights live on the GPU; host memory covers SGLang's own bookkeeping, so 8Gi is the
honest reservation. The 32Gi limit stays, so a burst still gets the room.
The installed InferenceService CRD declares annotations and labels directly under
spec.predictor, not under spec.predictor.podMetadata. Two manifests used podMetadata,
so every diff and apply failed with 'field not declared in schema' and the whole
Application could not reconcile.
Move the prometheus scrape annotations and the labels up one level. Nothing is lost:
the same keys, in the place the CRD accepts.
The resource list named ornith.yaml and reasoning.yaml, and neither file has ever been
committed to this directory. kustomize therefore failed to build, the Application sat
in ComparisonError, and nothing in this directory could reconcile — including the
clef-flash predictor added a commit earlier.
Removing the entries does not remove the models: the Application runs with prune=false,
so an object it no longer lists is left untouched. Re-export both from live state the
next time either is edited, so the directory describes what actually runs.
The gateway already dispatches /v1/systemone and its config already names
clef-flash-predictor, but this directory is an explicit allowlist and clef-flash.yaml
was not on it, so the model was silently dropped: no error, no drift, just no
predictor.
clef-flash is a decision model, not a chat model. It reads a state plus a schema of
typed questions and returns one probability per allowed option in a single prefill
pass, with no decode step. It needs SGLang rather than vLLM because it loads separate
joint_schema_head weights on top of the Qwen3.5-9B backbone and it implements the
/v1/systemone route.
worker-1 has four GPUs: reasoning holds two, ornith holds one, so one is free and this
fits without retiring ornith first.
The init container fetched a token once at pod start with a client_credentials grant
against a web app. A web app has no such grant, so the fetch could not work; and a
token fetched once is dead within the hour while the pod runs for weeks.
llm-token-vendor runs beside the app. It signs a machine-key assertion for the
paperless-llm service account, exchanges it for a token, and rewrites
/app/data/llm_token.txt atomically at 80% of the token lifetime. The vendor image
keeps using CUSTOM_API_KEY_FILE, so it needs no change and no OIDC support.
The key is re-read on every mint, so a rotation lands without a restart: publish the
new key, the next mint uses it, retire the old one.
paperless-llm is a separate service account from poimen-llm, so revoking one caller
leaves the other working.
Adds svc-caller, a capability that names a machine-to-machine caller, separate from
the human roles so a service account can be identified by its grant alone.
Adds MACHINE_CALLERS: one ZITADEL machine user per caller, created once with its own
key and granted its roles. poimen-llm is the first, granted llm:inference and
svc-caller. Its key is separate from every other caller's, so revoking one service
leaves the rest working and ZITADEL can overlap keys during a rotation.
The role is granted even though ZITADEL does not assert roles into a machine-user
access token on this instance. The grant is the record of what the account may do, and
it takes effect the moment that behaviour changes. Until then the gateway authorises
the caller from its allow-list entry.
storage/minio-oidc is owned by the KeyRotation object iam/minio now, so its value
leaves git. minio-creds and minio-user-ollama stay: they are the root console
credential and an application key, and neither rotates through ZITADEL yet.
The file keeps a header that says so, because a prune of the sops-secrets application
would delete that Secret and the controller recreates it on the next rotation.
Two defects that a real rotation exposed:
merge writeSecret replaced the whole Secret map, so every key a consumer
keeps beside the credential disappeared. paperless lost
SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then
failed to start while the rotation reported success.
backoff a Failed attempt returned 0, so the controller retried in a tight loop.
It now waits two minutes and then retries.
All eight objects carry both identifiers ZITADEL needs: the app resource id for the
secret endpoint and the client id for introspection.
Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui,
argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload
exists for it.
Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
rock