rock
3c3b243790
Previous comments described llm_ttft_seconds / llm_itl_seconds / llm_tokens_total as "streaming-only". That was wrong, and optimistic. They are dead code. internal/proxy/llm_metrics.go defines LLMMetricsCapture, which is the only thing that calls RecordTTFT / RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites anywhere in the repo, verified by grep. The exporter skips keys with no samples, so those three series render as HELP-only stubs and can never be graphed, streaming or not. No behaviour change here; only the comments and the dashboard generator's header are corrected so the next person does not build on a false assumption. The gateway latency panels are left in place so the gap stays visible — they populate unchanged once the capture wrapper is wired into the streaming proxy path. Series that DO work from the gateway, confirmed live with 3 series (one per replica): queue_full_requests_total, warmup_requests_delayed_total, vllm_prefix_cache_hit_tokens_total.
52 lines
2.1 KiB
YAML
52 lines
2.1 KiB
YAML
# WARNING — three of the series documented below are DEAD CODE as of
|
|
# 2026-09-29 and will never produce data:
|
|
#
|
|
# llm_ttft_seconds, llm_itl_seconds, llm_tokens_total
|
|
#
|
|
# They are declared and exported in internal/observability/prometheus.go, but
|
|
# the only thing that would ever record samples is LLMMetricsCapture
|
|
# (internal/proxy/llm_metrics.go) — and NewLLMMetricsCapture() has ZERO call
|
|
# sites. Verified: `grep -rn NewLLMMetricsCapture` returns only the definition,
|
|
# and RecordTTFT/RecordITL/RecordTokens are called from nowhere else. The
|
|
# exporter skips keys with no samples, so these render as HELP-only stubs.
|
|
#
|
|
# The gateway latency panels on the llm-observability dashboard depend on that
|
|
# wiring. Until it lands, use vLLM's own histograms as the latency source for
|
|
# `reasoning`; the two Ollama models (ornith:35b, qwen2.5:3b-instruct) have no
|
|
# latency visibility at all, because Ollama exposes no metrics endpoint either.
|
|
#
|
|
# These three ARE recorded and do work (3 series, one per replica):
|
|
# queue_full_requests_total counter backpressure rejects
|
|
# warmup_requests_delayed_total counter vLLM slot contention
|
|
# vllm_prefix_cache_hit_tokens_total counter prefix-cache hits
|
|
# request_duration_seconds histogram full request
|
|
#
|
|
# Also note the Deployment already carried prometheus.io/scrape annotations,
|
|
# but kube-prometheus-stack ignores pod annotations (no additionalScrapeConfigs,
|
|
# serviceMonitorSelectorNilUsesHelmValues: false) — so nothing at all was being
|
|
# collected before this ServiceMonitor existed.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: api-gateway
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/name: api-gateway
|
|
app.kubernetes.io/part-of: api
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- api
|
|
selector:
|
|
matchLabels:
|
|
app: api-gateway
|
|
endpoints:
|
|
# api-gateway Service port IS named `http` (unlike the KServe ones), so
|
|
# the name form works here.
|
|
- port: http
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
path: /metrics
|
|
scheme: http
|
|
|