rock
6ae36981bb
Both metric sources existed but neither was scraped: - servicemonitors/vllm.yaml was orphaned (absent from kustomization resources:) and would not have matched anyway — it used 'port: http' but KServe-generated Services in llm-serving have UNNAMED ports. Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it), which avoids editing the InferenceServices and therefore avoids a predictor restart / weights reload. - api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total, queue_full_requests_total, warmup_requests_delayed_total and vllm_prefix_cache_hit_tokens_total, but the Deployment's prometheus.io/scrape annotation is ignored by kube-prometheus-stack (no additionalScrapeConfigs). Added a ServiceMonitor for it. This is the only per-request latency source covering the two Ollama models, which expose no metrics endpoint at all. Ollama services are excluded from the vllm ServiceMonitor selector to avoid permanently-DOWN targets. Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate derived from vllm:cache_query_hit_total/miss, which were never scraped. Those alerts were dead; verify the metric names after the first scrape.
47 lines
1.8 KiB
YAML
47 lines
1.8 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: vllm
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/name: vllm
|
|
app.kubernetes.io/part-of: llm-serving
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- llm-serving
|
|
selector:
|
|
# Only the three engines that actually serve /metrics.
|
|
# ornith-predictor and qwen-cpu are Ollama, which has no Prometheus
|
|
# endpoint (/metrics returns 404) — selecting them would add permanently
|
|
# DOWN targets. Ollama-model visibility comes from the api-gateway
|
|
# ServiceMonitor instead (llm_ttft_seconds / llm_itl_seconds are
|
|
# measured at the proxy, so they cover every model including Ollama).
|
|
matchExpressions:
|
|
- key: app.kubernetes.io/name
|
|
operator: In
|
|
values:
|
|
- llm-reasoning # vLLM 0.11.0 — full engine metrics
|
|
- llm-embeddings # TEI 1.8.2 — CPU
|
|
- llm-reranker # TEI 1.8.2 — CPU
|
|
endpoints:
|
|
# targetPort, not port: the KServe-generated Services on llm-serving have
|
|
# UNNAMED ports, so `port: <name>` can never match. Naming them would mean
|
|
# editing the InferenceServices (predictor restart = full weights reload,
|
|
# tens of seconds on the vLLM slot), so scrape the pod port directly.
|
|
# prometheus-operator v0.94.1 supports targetPort on ServiceMonitor.
|
|
- targetPort: 8080
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
path: /metrics
|
|
scheme: http
|
|
relabelings:
|
|
- sourceLabels: [__meta_kubernetes_namespace]
|
|
targetLabel: namespace
|
|
- sourceLabels: [__meta_kubernetes_pod_name]
|
|
targetLabel: pod
|
|
- sourceLabels: [__meta_kubernetes_service_name]
|
|
targetLabel: service
|
|
- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
|
targetLabel: app
|