queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.
macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
ksops failed the whole sops-secrets application on this file: it declares
Secret forgejo-registry-pat in namespace forgejo, and no such namespace exists.
The failure blocks every other file in the generator list, including
zitadel-masterkey.
The same Secret name is already generated in namespace api by
forgejo-registry-pat-api.enc.yaml, and that is the copy the gateway reads. The
forgejo namespace copy is stale: Forgejo runs in namespace cicd, and namespace
forgejo was never created.
Remove the file and its generator entry. ArgoCD never created the Secret, so
pruning has nothing to remove.
ZITADEL_BOOTSTRAP_TOKEN and ZITADEL_URL now live in .env and .env.example, the
same shape the Authentik variables use. The line is present and empty: the PAT
does not exist until the chart's setup job creates the iam-admin machine user on
the first sync, so a value cannot be minted ahead of time.
The provisioning script loads .env itself instead of relying on the caller to
source it, and resolves the token in a fixed order: ZITADEL_BOOTSTRAP_TOKEN,
then the cached file under ~/.config/zitadel, then the cluster Secret. The .env
value wins so a human can paste a rotated token without touching the cluster;
the cluster Secret remains the only source that always reflects a rotation.
.env stays git-ignored. It already holds the live Authentik secrets, and this
change does not move them.
Not exercised against a live instance: the kube-apiserver on talos-cp-1 is in
CrashLoopBackOff, so kubectl only works against cp-2 or cp-3 right now.
scripts/iam/zitadel-provision.py replaces the manual sequence that
provision-rbac.py did for Authentik. Phases: keys, apply, secrets, users,
verify. `all` runs keys -> apply -> secrets -> verify; users stays manual
because the password hash must come from a human, one user at a time.
- keys: extract the chart's iam-admin machine key and PAT to ~/.config/zitadel
at 0600, outside the repo.
- apply: terraform apply terraform/zitadel with the machine key.
- secrets: write generated client secrets into the SOPS files, write client ids
into the plain manifests that carry them, write machine keys per agent.
Requires an exactly-one-match regex before it edits a manifest, prints the
diff without --apply-config, and is a no-op when the value already matches.
- users: import a human user with the Django pbkdf2_sha256 hash converted to the
passwap MCF form ZITADEL accepts.
- verify: discovery endpoint, client count, and it refuses to clear cutover
while the groups-claim action is disabled.
Design points that came out of review:
- Multi-document SOPS files (minio) are rewritten whole, because `sops set`
cannot address one document. sops reads the creation rule from the filename,
so the temp file is passed with --filename-override; without it sops finds no
matching rule and refuses. Ciphertext is staged in the destination directory
and installed with os.replace, so the mode is preserved and EXDEV is
impossible. SIGTERM/SIGINT trigger cleanup of the decrypted temp file.
- homarr is listed as manual: the live workload mounts Secret homarr-oidc, which
no SOPS file in git defines, and the tracked homarr-auth-oidc.enc.yaml has no
consumer. Writing the new secret into the orphan file would report success
while the dashboard kept the old one.
- Secrets never enter argv: curl reads its config from stdin, so the PAT and the
password hash do not appear in ps.
.gitignore: allow this one script through the scripts/iam/*.py ignore rule (it
embeds no credential), and ignore authentik-m2m-tokens.env, which holds live M2M
tokens and was one `git add -A` away from being committed.
Checks: py_compile clean, --help lists every phase, convert_hash verified on a
synthetic hash, the 3-document SOPS rewrite tested on a real copy (3 documents
survive, value set, second run unchanged, mode 644 kept, no temp left), all
three client-id regexes match exactly once, dry-run leaves the manifests
untouched. Reviewer round: BLOCK on the sops creation rule, the orphan homarr
Secret, the missing client-id writes and argv exposure; all fixed and
re-reviewed to SIGN-OFF.
Not run against a live ZITADEL: no instance exists yet.
Adds terraform/zitadel: the provisioning half the ArgoCD wiring left open.
It declares 1 project, 19 project roles, 12 OIDC applications, 5 machine users
plus keys, one API application for the gateway, project grants, per-agent claim
metadata, and a disabled groups-claim action.
Facts established while writing it:
- Client IDs cannot be preserved. ZITADEL v4.19.2 CreateOIDCApplicationRequest
has no client_id field, and provider 3.8.7 rejects a settable application id
(terraform validate reports "Invalid or unknown key"). Plan §3.2 assumed a
client_id-preserving re-point; every consumer now also changes client ID.
- Role keys equal the existing group names, so the groups claim can be
byte-identical and no RBAC file changes. v4.19.2 allows role keys of 1-200
characters with no character-set restriction.
- Machine users get JWT profile keys, not passwords. Per-agent claims move to
user metadata and reach tokens only through the groups-claim action.
- The action stays gated: enable_groups_claim_action defaults to false because
the action runtime API is unverified. Two validations stop a null script from
reaching file().
Checks: terraform fmt -check -recursive clean, terraform validate 0 errors,
plan reaches only the expected missing-credentials error, structural gate
0 error / 0 warning, and one reviewer round: BLOCK on a plan-time
fileexists(null) and on the missing api-gw machine path, both fixed, re-reviewed
to SIGN-OFF.
State is local and git-ignored; it carries generated client secrets and machine
keys. Move it to MinIO before a second operator applies.
Not applied: no Zitadel instance is running yet.
Coexistence wiring for the Authentik replacement: ZITADEL gets its own CNPG
cluster, SOPS masterkey, Helm values and ArgoCD Application in namespace iam.
Authentik keeps serving; no consumer is moved and no Authentik object changes.
- k8s/infra/databases/zitadel-db.yaml: CNPG Postgres 16.2, 3 instances
- k8s/infra/iam/zitadel-values.yaml: chart 10.4.0 pinned, DSN from the CNPG
secret, masterkey from a Secret, metrics ServiceMonitor, second-sync caveat
- k8s/argocd/apps/30-security.yaml: Application named zitadel, wave 3,
automated sync deliberately absent for the first sync (chart PreSync hook
re-run risk)
- k8s/argocd/secrets/zitadel-masterkey.enc.yaml: 32-byte masterkey, SOPS
- k8s/infra/monitoring/alerts/svc-zitadel-rules.yaml: target-down, login-UI,
crash-loop and PreSync-job-failed rules
- terraform/files/coredns/Corefile: in-cluster rewrite for the zitadel host
Not applied. The cluster is mid-incident: vault-0 and authentik-server are 0/1,
authentik-db-2 has 14 restarts, prometheus-0 is 1/2 CrashLoopBackOff and
kube-state-metrics has 1925 restarts. Two manual steps still gate the first
sync: the Cloudflare Tunnel public hostname plus a proxied CNAME, and
terraform apply + make apply-cp for the CoreDNS rewrite (Talos-owned).
The offline masterkey copy is at ~/zitadel-masterkey.offline (mode 600). Move it
to sealed storage and delete it before the first sync. Losing it makes every
encrypted ZITADEL value unreadable.
The Application pointed at github.com/ToyoGlobal/azhi-infra, branch
feat/azi-riotpiao-cnpg, path k8s/overlays/azi-riotpiao. It never reconciled:
status stayed Unknown because the source repository could not be read, and the
destination namespace azi-riotpiao was never created. ArgoCD reported it as an
OutOfSync managed resource of homelab-root with nothing behind it.
Also drops the ToyoGlobal URL from the AppProject allowlist. No manifest from
that repository is referenced any more, so the allowlist entry is dead weight.
Not touched here, because it is not in git: Secret azhi-infra-github in
namespace argocd holds the read-only GitHub credential for that repository.
Delete it separately once no other Application needs the repo.
The live Application object was deleted by hand. Until this commit reaches
main, the next homelab-root sync recreates it, because main still declares it.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.
Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
disk is gone
The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
The homelab AppProject allowlist had no azhi repository, so an Application
pointing at ToyoGlobal/azhi-infra fails admission with 'repository not
permitted'. Adding the URL is the whole gate.
The Application itself targets the repository default branch
(harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That
overlay renders postgres and the API into namespace azi-riotpiao.
The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR
pull credential and azhi-riotpiao-app-env carries the database credentials,
the JWT secrets and the AES key.
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
cluster-maintenance tracks main (targetRevision: main), so the cluster never
had the CronJob or its Secret
This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
would have reclaimed nothing
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.
The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:
- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).
Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
Exit code 143 proved the previous job killed itself: the script's own cmdline contains the
string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver')
and skipping $$ removes that failure mode. Staggered 2 minutes apart.
Static pods survive mirror-pod deletion, so the first bootstrap job had no effect.
These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself,
staggered by 100s so only one control plane restarts at a time.
The first job could not pull its image; this one deletes the apiserver pods via the
in-cluster API with the mounted service-account token instead of a kubectl image.
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.
Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.
Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
## Why
`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).
The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:
```
minio tenant pod not ready (1/2)
-> headless Service minio-cluster-hl has no endpoints
-> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
-> Vault S3 storage init fails at startup
-> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```
## Control checks (so this is not a network story)
| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |
## Change
Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.
## Caveat
This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.
## After merge (ArgoCD, not kubectl)
The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:
```
kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds)
```
Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53
Co-authored-by: rock <[email protected]>
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes#51
Previous comments described llm_ttft_seconds / llm_itl_seconds /
llm_tokens_total as "streaming-only". That was wrong, and optimistic.
They are dead code. internal/proxy/llm_metrics.go defines
LLMMetricsCapture, which is the only thing that calls RecordTTFT /
RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites
anywhere in the repo, verified by grep. The exporter skips keys with no
samples, so those three series render as HELP-only stubs and can never be
graphed, streaming or not.
No behaviour change here; only the comments and the dashboard generator's
header are corrected so the next person does not build on a false
assumption. The gateway latency panels are left in place so the gap stays
visible — they populate unchanged once the capture wrapper is wired into
the streaming proxy path.
Series that DO work from the gateway, confirmed live with 3 series (one
per replica): queue_full_requests_total, warmup_requests_delayed_total,
vllm_prefix_cache_hit_tokens_total.
Both new ServiceMonitors sat permanently OutOfSync. Cause: `action: replace`
is the API server default for relabeling rules, so the live object carried it
while the manifests omitted it.
Rather than write out the default to paper over drift, removed the rules that
were never needed:
- vllm.yaml: the namespace/pod/service relabelings were redundant —
Kubernetes service discovery already supplies all three. Only `app` has to
be derived (TEI panels aggregate by it and SD does not promote arbitrary
pod labels). Kept that one, with an explicit action.
- api-gateway.yaml: the metricRelabelings block was a no-op — it mapped
__address__ onto instance, which is already the default. Removed. The
series inventory it was documenting moved up into the file header, which
is where it belonged anyway.
New Grafana dashboard (llm-observability) with 27 panels across 5 rows:
TTFT, inter-token latency, batching/scheduling, KV cache, throughput.
Both metric sources are shown side by side because they answer different
questions and neither is complete alone:
vLLM 0.11.0 (reasoning) — engine truth: kv_cache_usage_perc,
num_requests_running/waiting, num_preemptions_total, prefill vs
decode time, prefix cache, per-request histogram quantiles.
TEI 1.8.2 (embeddings, reranker) — te_batch_next_size, te_queue_size,
te_embed_duration, te_embed_queue_duration.
api-gateway — the ONLY per-request latency source covering Ollama
models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics.
It is derived from the SSE byte stream, so it is streaming-only and
counts SSE chunks rather than true tokens. vLLM's own histograms are
authoritative for reasoning.
All 44 panel queries were executed against live Prometheus before commit —
all return success. vLLM/TEI panels have live series; gateway panels show
no data until the netpol fix in homelab-frontend PR #62 merges.
Also fixes a dead alert. alerts/cag-kv-cache-rules.yaml derived
vllm:prefix_cache_hit_rate from vllm:cache_query_hit_total and
vllm:cache_query_miss_total. Those metric names do not exist in vLLM
0.11.0 — the real series are vllm:prefix_cache_hits_total and
vllm:prefix_cache_queries_total. The recording rule produced no data, so
VLLMPrefixCacheHitRateLow could never fire.
Not included: real GPU hardware metrics (SM utilisation, framebuffer,
power, temperature). Only nvidia-device-plugin is deployed, which exposes
none of them; DCGM exporter is required. vllm:kv_cache_usage_perc and
preemption rate are the only GPU-pressure proxies available today.
Both metric sources existed but neither was scraped:
- servicemonitors/vllm.yaml was orphaned (absent from kustomization
resources:) and would not have matched anyway — it used 'port: http'
but KServe-generated Services in llm-serving have UNNAMED ports.
Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it),
which avoids editing the InferenceServices and therefore avoids a
predictor restart / weights reload.
- api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total,
queue_full_requests_total, warmup_requests_delayed_total and
vllm_prefix_cache_hit_tokens_total, but the Deployment's
prometheus.io/scrape annotation is ignored by kube-prometheus-stack
(no additionalScrapeConfigs). Added a ServiceMonitor for it. This is
the only per-request latency source covering the two Ollama models,
which expose no metrics endpoint at all.
Ollama services are excluded from the vllm ServiceMonitor selector to
avoid permanently-DOWN targets.
Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate
derived from vllm:cache_query_hit_total/miss, which were never scraped.
Those alerts were dead; verify the metric names after the first scrape.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
GET /user/sign_up returned 200 — anyone reaching forgejo.riotpiao.com
could create an account. Add [service] DISABLE_REGISTRATION: true.
Admin-created accounts (UI/API) are unaffected.
32 degraded + 3 faulted volumes from replica rebuild failures.
cp-1/cp-2 over-committed; worker-1 had 602Gi free but scheduling
disabled. Enable it with 400Gi reserved for LLM weights.
- Add worker-1 to longhorn-nodes.yaml (allowScheduling: true)
- Add worker-1 disk config to add-disks-job (storageReserved: 400Gi,
evictionRequested: false)
rock
rock