homelab/k8s/infra/monitoring/servicemonitors/api-gateway.yaml

# WARNING — three of the series documented below are DEAD CODE as of
# 2026-09-29 and will never produce data:
#
#   llm_ttft_seconds, llm_itl_seconds, llm_tokens_total
#
# They are declared and exported in internal/observability/prometheus.go, but
# the only thing that would ever record samples is LLMMetricsCapture
# (internal/proxy/llm_metrics.go) — and NewLLMMetricsCapture() has ZERO call
# sites. Verified: `grep -rn NewLLMMetricsCapture` returns only the definition,
# and RecordTTFT/RecordITL/RecordTokens are called from nowhere else. The
# exporter skips keys with no samples, so these render as HELP-only stubs.
#
# The gateway latency panels on the llm-observability dashboard depend on that
# wiring. Until it lands, use vLLM's own histograms as the latency source for
# `reasoning`; the two Ollama models (ornith:35b, qwen2.5:3b-instruct) have no
# latency visibility at all, because Ollama exposes no metrics endpoint either.
#
# These three ARE recorded and do work (3 series, one per replica):
#   queue_full_requests_total           counter    backpressure rejects
#   warmup_requests_delayed_total       counter    vLLM slot contention
#   vllm_prefix_cache_hit_tokens_total  counter    prefix-cache hits
#   request_duration_seconds            histogram  full request
#
# Also note the Deployment already carried prometheus.io/scrape annotations,
# but kube-prometheus-stack ignores pod annotations (no additionalScrapeConfigs,
# serviceMonitorSelectorNilUsesHelmValues: false) — so nothing at all was being
# collected before this ServiceMonitor existed.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-gateway
  namespace: monitoring
  labels:
    app.kubernetes.io/name: api-gateway
    app.kubernetes.io/part-of: api
spec:
  namespaceSelector:
    matchNames:
      - api
  selector:
    matchLabels:
      app: api-gateway
  endpoints:
    # api-gateway Service port IS named `http` (unlike the KServe ones), so
    # the name form works here.
    - port: http
      interval: 30s
      scrapeTimeout: 10s
      path: /metrics
      scheme: http