docs(monitoring): correct the gateway latency-metric claim

Previous comments described llm_ttft_seconds / llm_itl_seconds / llm_tokens_total as "streaming-only". That was wrong, and optimistic. They are dead code. internal/proxy/llm_metrics.go defines LLMMetricsCapture, which is the only thing that calls RecordTTFT / RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites anywhere in the repo, verified by grep. The exporter skips keys with no samples, so those three series render as HELP-only stubs and can never be graphed, streaming or not. No behaviour change here; only the comments and the dashboard generator's header are corrected so the next person does not build on a false assumption. The gateway latency panels are left in place so the gap stays visible — they populate unchanged once the capture wrapper is wired into the streaming proxy path. Series that DO work from the gateway, confirmed live with 3 series (one per replica): queue_full_requests_total, warmup_requests_delayed_total, vllm_prefix_cache_hit_tokens_total.
2026-09-29 22:47:38 +08:00
parent 420b5d6da6
commit 3c3b243790
2 changed files with 39 additions and 24 deletions
@@ -274,13 +274,23 @@ def build_api_gateway():
 #   vLLM + TEI (llm-serving)  — engine-truth, but only for the three engines
 #                               that expose /metrics. Ollama (ornith:35b,
 #                               qwen2.5:3b-instruct) exposes nothing at all.
-#   api-gateway (/metrics)    — proxy-side, covers EVERY model including the
-#                               Ollama ones, but is derived from the SSE byte
-#                               stream: TTFT = request-start to first byte,
-#                               ITL = gap between SSE writes carrying "data: ".
-#                               It is therefore streaming-only and counts SSE
-#                               chunks, not true tokens. Treat vLLM's own
-#                               histograms as authoritative for `reasoning`.
+#   api-gateway (/metrics)    — intended to cover EVERY model including the
+#                               Ollama ones, but see the dead-metrics note
+#                               below: its latency series are not recorded.
+#
+# NOTE — llm_ttft_seconds, llm_itl_seconds and llm_tokens_total are DEAD as of
+# 2026-09-29. The only recorder is LLMMetricsCapture
+# (internal/proxy/llm_metrics.go), and NewLLMMetricsCapture() has ZERO call
+# sites in the whole repo, so the exporter emits HELP-only stubs and any panel
+# querying them stays empty forever. Verified by grep, not assumed.
+#
+# The gateway latency panels are kept deliberately so the gap is visible — they
+# light up unchanged once the capture wrapper is wired into the streaming proxy
+# path. Until then vLLM's own histograms are the latency source for `reasoning`,
+# and the two Ollama models have no latency visibility from anywhere.
+#
+# Panels backed by queue_full_requests_total, warmup_requests_delayed_total and
+# vllm_prefix_cache_hit_tokens_total DO work today (3 series, one per replica).
 #
 # All metric names below were verified against a live scrape, not assumed:
 #   vLLM 0.11.0 exposes prefix_cache_hits_total / prefix_cache_queries_total,