Previous comments described llm_ttft_seconds / llm_itl_seconds /
llm_tokens_total as "streaming-only". That was wrong, and optimistic.
They are dead code. internal/proxy/llm_metrics.go defines
LLMMetricsCapture, which is the only thing that calls RecordTTFT /
RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites
anywhere in the repo, verified by grep. The exporter skips keys with no
samples, so those three series render as HELP-only stubs and can never be
graphed, streaming or not.
No behaviour change here; only the comments and the dashboard generator's
header are corrected so the next person does not build on a false
assumption. The gateway latency panels are left in place so the gap stays
visible — they populate unchanged once the capture wrapper is wired into
the streaming proxy path.
Series that DO work from the gateway, confirmed live with 3 series (one
per replica): queue_full_requests_total, warmup_requests_delayed_total,
vllm_prefix_cache_hit_tokens_total.
Both new ServiceMonitors sat permanently OutOfSync. Cause: `action: replace`
is the API server default for relabeling rules, so the live object carried it
while the manifests omitted it.
Rather than write out the default to paper over drift, removed the rules that
were never needed:
- vllm.yaml: the namespace/pod/service relabelings were redundant —
Kubernetes service discovery already supplies all three. Only `app` has to
be derived (TEI panels aggregate by it and SD does not promote arbitrary
pod labels). Kept that one, with an explicit action.
- api-gateway.yaml: the metricRelabelings block was a no-op — it mapped
__address__ onto instance, which is already the default. Removed. The
series inventory it was documenting moved up into the file header, which
is where it belonged anyway.
New Grafana dashboard (llm-observability) with 27 panels across 5 rows:
TTFT, inter-token latency, batching/scheduling, KV cache, throughput.
Both metric sources are shown side by side because they answer different
questions and neither is complete alone:
vLLM 0.11.0 (reasoning) — engine truth: kv_cache_usage_perc,
num_requests_running/waiting, num_preemptions_total, prefill vs
decode time, prefix cache, per-request histogram quantiles.
TEI 1.8.2 (embeddings, reranker) — te_batch_next_size, te_queue_size,
te_embed_duration, te_embed_queue_duration.
api-gateway — the ONLY per-request latency source covering Ollama
models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics.
It is derived from the SSE byte stream, so it is streaming-only and
counts SSE chunks rather than true tokens. vLLM's own histograms are
authoritative for reasoning.
All 44 panel queries were executed against live Prometheus before commit —
all return success. vLLM/TEI panels have live series; gateway panels show
no data until the netpol fix in homelab-frontend PR #62 merges.
Also fixes a dead alert. alerts/cag-kv-cache-rules.yaml derived
vllm:prefix_cache_hit_rate from vllm:cache_query_hit_total and
vllm:cache_query_miss_total. Those metric names do not exist in vLLM
0.11.0 — the real series are vllm:prefix_cache_hits_total and
vllm:prefix_cache_queries_total. The recording rule produced no data, so
VLLMPrefixCacheHitRateLow could never fire.
Not included: real GPU hardware metrics (SM utilisation, framebuffer,
power, temperature). Only nvidia-device-plugin is deployed, which exposes
none of them; DCGM exporter is required. vllm:kv_cache_usage_perc and
preemption rate are the only GPU-pressure proxies available today.
Both metric sources existed but neither was scraped:
- servicemonitors/vllm.yaml was orphaned (absent from kustomization
resources:) and would not have matched anyway — it used 'port: http'
but KServe-generated Services in llm-serving have UNNAMED ports.
Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it),
which avoids editing the InferenceServices and therefore avoids a
predictor restart / weights reload.
- api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total,
queue_full_requests_total, warmup_requests_delayed_total and
vllm_prefix_cache_hit_tokens_total, but the Deployment's
prometheus.io/scrape annotation is ignored by kube-prometheus-stack
(no additionalScrapeConfigs). Added a ServiceMonitor for it. This is
the only per-request latency source covering the two Ollama models,
which expose no metrics endpoint at all.
Ollama services are excluded from the vllm ServiceMonitor selector to
avoid permanently-DOWN targets.
Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate
derived from vllm:cache_query_hit_total/miss, which were never scraped.
Those alerts were dead; verify the metric names after the first scrape.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
32 degraded + 3 faulted volumes from replica rebuild failures.
cp-1/cp-2 over-committed; worker-1 had 602Gi free but scheduling
disabled. Enable it with 400Gi reserved for LLM weights.
- Add worker-1 to longhorn-nodes.yaml (allowScheduling: true)
- Add worker-1 disk config to add-disks-job (storageReserved: 400Gi,
evictionRequested: false)
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
- Add MINIO_BROWSER_REDIRECT_URL pointing to public URL
- Add MINIO_BROWSER_LOGIN_ANIMATION=off
- Enables Console to show OIDC/Authentik SSO button on login page
- Change MINIO_IDENTITY_OPENID_CLAIM_NAME from 'policy' to 'groups'
- Add MINIO_IDENTITY_OPENID_CLAIM_PREFIX (empty) to use raw group names
- Update scopes to 'openid,profile,email,groups'
- Enables Authentik group-based access (homelab-admins, minio-admins, etc)
- Users in homelab-admins group will get admin access to MinIO
Re-enable skip_org_role_sync=false to sync Admin role from grafana-admins group.
User creation worked with skip_org_role_sync=true, now restore role sync.
Set skip_org_role_sync=true to allow user creation on first OAuth signin.
When false, Grafana tries to sync org roles before user exists, causing creation to fail.
Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
- stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m
- paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)
Standard K8s NetworkPolicy ipBlock rules don't work under Cilium for API server access. Use Cilium-native kube-apiserver entity instead.
Tested: runner pod can now reach K8s API server.
Required by: homelab-frontend PR #25 (Tekton CI).
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #49
Co-authored-by: poimen <[email protected]>
CI runner needs kubectl access to create Tekton PipelineRuns for integration testing.
## Root Cause
The runner egress NetworkPolicy blocks `192.168.1.0/24` (LAN). The K8s API server runs on control-plane nodes in that subnet (`192.168.1.166:6443`). kubectl from inside the DinD container times out.
## Fix
Allow TCP port 6443 to `192.168.1.0/24` — scoped to control-plane API server only.
## Required By
homelab-frontend PR #25 (Tekton integration testing) — CI creates PipelineRuns via kubectl.
---------
Co-authored-by: rock <[email protected]>
Reviewed-on: #48
Co-authored-by: poimen <[email protected]>
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
Job containers spawned by the runner run inside dind. With TCP+TLS
(tcp://localhost:2376), localhost inside those containers doesn't
reach the dind sidecar. Unix socket at /run/docker.sock works because
both runner and dind share the /run emptyDir.
Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead
of only listening on TLS TCP.
Previous image (golang:1.26-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change golang runner image from golang:1.26-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, Go toolchain, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the golang runner pod.
Note: node:22-bookworm runner already has Node.js, no change needed.
Previous image (rust:1.83-bookworm) lacks Node.js, causing GitHub Actions
to fail with: 'exec: "node": executable file not found'
Solution:
- Change rust runner image from rust:1.83-bookworm to docker:27-cli
- docker:27-cli includes: Node.js, git, docker CLI, full dev tools
- Verified tag exists: docker manifest inspect docker:27-cli ✓
This allows actions/checkout@v4 and other GitHub Actions to run properly
on the rust runner pod.
- ClusterRoleBinding doesn't use spec: wrapper (unlike Deployment/StatefulSet)
- roleRef and subjects go at top level with metadata
- Fixes: 'strict decoding error: unknown field "spec"'
- Reverted to node:22-bookworm (no custom image)
- Removed Dockerfile.node (no CI to build it)
- Docker install step in riotpiao workflow is already the workaround
- Removed .gitea/workflows/build-runner-node.yml
- Homelab is GitOps only, not a buildable artifact
- Runner images managed via direct Dockerfile edits + manual pushes
rock
poimenandrock