homelab/k8s/infra/monitoring/servicemonitors/api-gateway.yaml

# History, recorded so nobody re-derives it the hard way: on 2026-09-29 the
# three latency series below recorded nothing. internal/proxy/llm_metrics.go
# declared a capture wrapper (LLMMetricsCapture) that was never constructed
# anywhere, so the exporter emitted HELP-only stubs and the dashboard panels
# stayed empty.
#
# homelab-frontend#63 fixed exactly that. The wrapper is now
# ResponseWriterWrapper, constructed by NewResponseWriterWrapper
# (internal/proxy/llm_metrics.go:38) and installed on the streaming proxy
# request path. Verified live on 2026-09-29:
#   llm_ttft_seconds  count=7  sum=598.212s
#   llm_itl_seconds   count=16116
#   llm_tokens_total  3 series
#
# These derived-from-SSE series cover EVERY model including the two Ollama ones
# (ornith:35b, qwen2.5:3b-instruct), which expose no metrics endpoint at all.
# They are proxy-side: TTFT is request-start to first byte and ITL is the gap
# between SSE writes, so both are granular to SSE write boundaries rather than
# to the engine's own token cadence. llm_tokens_total counts `data:` events,
# and the documented assumption in internal/proxy/llm_metrics.go is that these
# upstreams emit one such event per token — so it is a token count only as far
# as that assumption holds. vLLM's own histograms remain authoritative.
#
# Also recorded and working (3 series, one per replica):
#   queue_full_requests_total           counter    backpressure rejects
#   warmup_requests_delayed_total       counter    vLLM slot contention
#   vllm_prefix_cache_hit_tokens_total  counter    prefix-cache hits
#
# NOT working, do not build on it: request_duration_seconds is declared and
# exported but has 0 samples, and its only writer Metrics.RecordRequest has
# zero call sites outside _test.go — verified by ast-grep, not assumed. It is
# the same dead-code shape this branch already corrected for the latency
# series, so it is listed here rather than left to be rediscovered.
#
# Also note the Deployment already carried prometheus.io/scrape annotations,
# but kube-prometheus-stack ignores pod annotations (no additionalScrapeConfigs,
# serviceMonitorSelectorNilUsesHelmValues: false) — so nothing at all was being
# collected before this ServiceMonitor existed.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-gateway
  namespace: monitoring
  labels:
    app.kubernetes.io/name: api-gateway
    app.kubernetes.io/part-of: api
spec:
  namespaceSelector:
    matchNames:
      - api
  selector:
    matchLabels:
      app: api-gateway
  endpoints:
    # api-gateway Service port IS named `http` (unlike the KServe ones), so
    # the name form works here.
    - port: http
      interval: 30s
      scrapeTimeout: 10s
      path: /metrics
      scheme: http