Both metric sources existed but neither was scraped:
- servicemonitors/vllm.yaml was orphaned (absent from kustomization
resources:) and would not have matched anyway — it used 'port: http'
but KServe-generated Services in llm-serving have UNNAMED ports.
Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it),
which avoids editing the InferenceServices and therefore avoids a
predictor restart / weights reload.
- api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total,
queue_full_requests_total, warmup_requests_delayed_total and
vllm_prefix_cache_hit_tokens_total, but the Deployment's
prometheus.io/scrape annotation is ignored by kube-prometheus-stack
(no additionalScrapeConfigs). Added a ServiceMonitor for it. This is
the only per-request latency source covering the two Ollama models,
which expose no metrics endpoint at all.
Ollama services are excluded from the vllm ServiceMonitor selector to
avoid permanently-DOWN targets.
Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate
derived from vllm:cache_query_hit_total/miss, which were never scraped.
Those alerts were dead; verify the metric names after the first scrape.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
rock
Story Crater Bot