rock
6ae36981bb
Both metric sources existed but neither was scraped: - servicemonitors/vllm.yaml was orphaned (absent from kustomization resources:) and would not have matched anyway — it used 'port: http' but KServe-generated Services in llm-serving have UNNAMED ports. Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it), which avoids editing the InferenceServices and therefore avoids a predictor restart / weights reload. - api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total, queue_full_requests_total, warmup_requests_delayed_total and vllm_prefix_cache_hit_tokens_total, but the Deployment's prometheus.io/scrape annotation is ignored by kube-prometheus-stack (no additionalScrapeConfigs). Added a ServiceMonitor for it. This is the only per-request latency source covering the two Ollama models, which expose no metrics endpoint at all. Ollama services are excluded from the vllm ServiceMonitor selector to avoid permanently-DOWN targets. Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate derived from vllm:cache_query_hit_total/miss, which were never scraped. Those alerts were dead; verify the metric names after the first scrape.
27 lines
1003 B
YAML
27 lines
1003 B
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# Single source for cluster monitoring config (alerts, servicemonitors, dashboards).
|
|
# NO top-level `namespace:` transformer — PrometheusRules deliberately live in
|
|
# per-app namespaces (forgejo, iam, argocd, storage, ...); a transformer would
|
|
# wrongly rewrite them all to one ns.
|
|
resources:
|
|
- alerts/ingress-alerts.yaml
|
|
- alerts/svc-argocd-rules.yaml
|
|
- alerts/svc-authentik-rules.yaml
|
|
- alerts/svc-forgejo-rules.yaml
|
|
- alerts/svc-grafana-rules.yaml
|
|
- alerts/svc-minio-rules.yaml
|
|
- alerts/svc-vault-rules.yaml
|
|
- alerts/cluster-alerts.yaml
|
|
- alerts/api-gateway-alerts.yaml
|
|
- alerts/cag-kv-cache-rules.yaml
|
|
- servicemonitors/argocd.yaml
|
|
- servicemonitors/authentik.yaml
|
|
- servicemonitors/forgejo.yaml
|
|
- servicemonitors/minio.yaml
|
|
- servicemonitors/ingress-nginx.yaml
|
|
- servicemonitors/vllm.yaml
|
|
- servicemonitors/api-gateway.yaml
|
|
- dashboards/cluster-infrastructure.yaml
|
|
- dashboards/api-gateway.yaml
|