rock
c36aa56b52
Both new ServiceMonitors sat permanently OutOfSync. Cause: is the API server default for relabeling rules, so the live object carried it while the manifests omitted it. Rather than write out the default to paper over drift, removed the rules that were never needed: - vllm.yaml: the namespace/pod/service relabelings were redundant — Kubernetes service discovery already supplies all three. Only has to be derived (TEI panels aggregate by it and SD does not promote arbitrary pod labels). Kept that one, with an explicit action. - api-gateway.yaml: the metricRelabelings block was a no-op — it mapped __address__ onto instance, which is already the default. Removed. The series inventory it was documenting moved up into the file header, which is where it belonged anyway.
50 lines
2.1 KiB
YAML
50 lines
2.1 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: vllm
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/name: vllm
|
|
app.kubernetes.io/part-of: llm-serving
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- llm-serving
|
|
selector:
|
|
# Only the three engines that actually serve /metrics.
|
|
# ornith-predictor and qwen-cpu are Ollama, which has no Prometheus
|
|
# endpoint (/metrics returns 404) — selecting them would add permanently
|
|
# DOWN targets. Ollama-model visibility comes from the api-gateway
|
|
# ServiceMonitor instead (llm_ttft_seconds / llm_itl_seconds are
|
|
# measured at the proxy, so they cover every model including Ollama).
|
|
matchExpressions:
|
|
- key: app.kubernetes.io/name
|
|
operator: In
|
|
values:
|
|
- llm-reasoning # vLLM 0.11.0 — full engine metrics
|
|
- llm-embeddings # TEI 1.8.2 — CPU
|
|
- llm-reranker # TEI 1.8.2 — CPU
|
|
endpoints:
|
|
# targetPort, not port: the KServe-generated Services on llm-serving have
|
|
# UNNAMED ports, so `port: <name>` can never match. Naming them would mean
|
|
# editing the InferenceServices (predictor restart = full weights reload,
|
|
# tens of seconds on the vLLM slot), so scrape the pod port directly.
|
|
# prometheus-operator v0.94.1 supports targetPort on ServiceMonitor.
|
|
- targetPort: 8080
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
path: /metrics
|
|
scheme: http
|
|
relabelings:
|
|
# Prometheus's Kubernetes service discovery already supplies the
|
|
# namespace, pod and service labels; the only one that needs deriving
|
|
# is `app`, since the TEI panels aggregate by it (by (app)) and SD does
|
|
# not promote arbitrary pod labels.
|
|
#
|
|
# `action: replace` is written explicitly. It is the API server's
|
|
# default for relabeling rules, so omitting it leaves the live object
|
|
# diverged from this manifest and ArgoCD reports permanent OutOfSync.
|
|
- action: replace
|
|
sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
|
targetLabel: app
|