rock
6ae36981bb
Both metric sources existed but neither was scraped: - servicemonitors/vllm.yaml was orphaned (absent from kustomization resources:) and would not have matched anyway — it used 'port: http' but KServe-generated Services in llm-serving have UNNAMED ports. Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it), which avoids editing the InferenceServices and therefore avoids a predictor restart / weights reload. - api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total, queue_full_requests_total, warmup_requests_delayed_total and vllm_prefix_cache_hit_tokens_total, but the Deployment's prometheus.io/scrape annotation is ignored by kube-prometheus-stack (no additionalScrapeConfigs). Added a ServiceMonitor for it. This is the only per-request latency source covering the two Ollama models, which expose no metrics endpoint at all. Ollama services are excluded from the vllm ServiceMonitor selector to avoid permanently-DOWN targets. Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate derived from vllm:cache_query_hit_total/miss, which were never scraped. Those alerts were dead; verify the metric names after the first scrape.
49 lines
2.0 KiB
YAML
49 lines
2.0 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: api-gateway
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/name: api-gateway
|
|
app.kubernetes.io/part-of: api
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- api
|
|
selector:
|
|
matchLabels:
|
|
app: api-gateway
|
|
endpoints:
|
|
# api-gateway Service port IS named `http` (unlike the KServe ones), so
|
|
# the name form works here.
|
|
- port: http
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
path: /metrics
|
|
scheme: http
|
|
|
|
# The gateway is the only source of per-request LLM latency that covers
|
|
# EVERY model, including the two Ollama ones that expose no metrics at
|
|
# all. Series it produces (internal/observability/prometheus.go):
|
|
# llm_ttft_seconds histogram time-to-first-token
|
|
# llm_itl_seconds histogram inter-token latency
|
|
# llm_tokens_total counter tokens generated
|
|
# queue_full_requests_total counter backpressure rejects
|
|
# warmup_requests_delayed_total counter vLLM slot contention
|
|
# vllm_prefix_cache_hit_tokens_total counter prefix-cache hits
|
|
# request_duration_seconds histogram full request
|
|
#
|
|
# The Deployment already carried prometheus.io/scrape annotations, but
|
|
# kube-prometheus-stack ignores pod annotations (no additionalScrape-
|
|
# Configs, serviceMonitorSelectorNilUsesHelmValues: false) — so none of
|
|
# the above was being collected. This ServiceMonitor is what actually
|
|
# wires it up.
|
|
metricRelabelings:
|
|
# 3 gateway replicas each build their own in-process histogram. Without
|
|
# this the label set is otherwise identical, so per-instance series
|
|
# would collide conceptually in sum()-by aggregations; instance is
|
|
# retained deliberately so per-replica skew stays visible.
|
|
- sourceLabels: [__address__]
|
|
targetLabel: instance
|
|
replacement: $1
|