fix(proxy): wire LLM latency metrics into the streaming response path #63

2026-09-29T14:50:55Z

rock commented

2026-09-29 14:50:55 +00:00

NewLLMMetricsCapture/ResponseWriterWrapper had zero call sites, so llm_ttft_seconds, llm_itl_seconds and llm_tokens_total were exported as HELP/TYPE stubs with no samples (verified against the live /metrics).

These are the only per-request LLM latency series covering the Ollama models (ornith:35b, qwen2.5:3b-instruct), which expose no /metrics of their own — so those models had zero latency visibility.

Change

Wire the wrapper around the response writer in ServeHTTP for model-routed requests, and delete the unused io.WriteCloser duplicate that made the gap hard to spot (two near-identical implementations, neither constructed).

Correctness details

Only SSE responses are measured. A non-streaming completion is written in one Write, which would otherwise be recorded as a TTFT sample equal to the whole request latency, with no ITL samples — plausible-looking but meaningless numbers.
requestStart is captured at the top of ServeHTTP so TTFT measures from request acceptance, not from when the upstream response is copied.
Token count skips the data: [DONE] terminator.
Unwrap() keeps http.ResponseController (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (#33) is unaffected.

Tests

Test	Guards
`TestResponseWriterWrapperRecordsSSEMetrics`	TTFT=1, ITL>0, tokens=2 (terminator excluded)
`TestResponseWriterWrapperIgnoresNonStreaming`	JSON response records nothing
`TestResponseWriterWrapperPreservesStreaming`	bytes + Flush pass through
`TestHandlerRecordsLLMMetricsForStreaming`	end-to-end through `ServeHTTP`

The last one fails if the wrapper is ever wired nowhere again — verified by disabling the wiring locally and watching it fail.

Verification needed post-merge

Confirm samples appear:

curl -s https://api.riotpiao.com/metrics | grep -E "llm_(ttft|itl)_seconds_count|llm_tokens_total"

`NewLLMMetricsCapture`/`ResponseWriterWrapper` had **zero call sites**, so `llm_ttft_seconds`, `llm_itl_seconds` and `llm_tokens_total` were exported as HELP/TYPE stubs with no samples (verified against the live `/metrics`). These are the only per-request LLM latency series covering the Ollama models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no `/metrics` of their own — so those models had zero latency visibility. ## Change Wire the wrapper around the response writer in `ServeHTTP` for model-routed requests, and delete the unused `io.WriteCloser` duplicate that made the gap hard to spot (two near-identical implementations, neither constructed). ## Correctness details - **Only SSE responses are measured.** A non-streaming completion is written in one `Write`, which would otherwise be recorded as a TTFT sample equal to the whole request latency, with no ITL samples — plausible-looking but meaningless numbers. - **`requestStart` is captured at the top of `ServeHTTP`** so TTFT measures from request acceptance, not from when the upstream response is copied. - **Token count skips the `data: [DONE]` terminator.** - **`Unwrap()`** keeps `http.ResponseController` (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (#33) is unaffected. ## Tests | Test | Guards | |---|---| | `TestResponseWriterWrapperRecordsSSEMetrics` | TTFT=1, ITL>0, tokens=2 (terminator excluded) | | `TestResponseWriterWrapperIgnoresNonStreaming` | JSON response records nothing | | `TestResponseWriterWrapperPreservesStreaming` | bytes + Flush pass through | | `TestHandlerRecordsLLMMetricsForStreaming` | end-to-end through `ServeHTTP` | The last one fails if the wrapper is ever wired nowhere again — verified by disabling the wiring locally and watching it fail. ## Verification needed post-merge Confirm samples appear: ``` curl -s https://api.riotpiao.com/metrics | grep -E "llm_(ttft|itl)_seconds_count|llm_tokens_total" ```

rock added 1 commit 2026-09-29 14:50:57 +00:00

fix(proxy): wire LLM latency metrics into the streaming response path

CI / CI (pull_request) Successful in 27m12s

Details

aac502ea1f

NewLLMMetricsCapture/ResponseWriterWrapper had zero call sites, so
llm_ttft_seconds, llm_itl_seconds and llm_tokens_total were exported as
HELP/TYPE stubs with no samples. These are the only per-request LLM latency
series covering the Ollama models (ornith:35b, qwen2.5:3b-instruct), which
expose no /metrics of their own.

Wire the wrapper around the response writer in ServeHTTP for model-routed
requests, and delete the unused io.WriteCloser duplicate that made the gap
hard to spot (two near-identical implementations, neither constructed).

Correctness details:
- Only SSE responses are measured. A non-streaming completion is written in
  one Write, which would otherwise be recorded as a TTFT sample equal to the
  whole request latency with no ITL samples.
- requestStart is captured at the top of ServeHTTP so TTFT measures from
  request acceptance, not from when the upstream response is copied.
- Token count skips the `data: [DONE]` terminator.
- Unwrap() keeps http.ResponseController (Flush/Hijack/deadlines) working
  through the wrapper, so SSE unbuffering (issue #33) is unaffected.

Tests: unit coverage for SSE vs non-streaming, wrapper transparency, and an
end-to-end handler test that fails if the wrapper is ever wired nowhere again.

rock merged commit aac68386d8 into main

2026-09-29 15:46:41 +00:00

rock deleted branch fix/llm-metrics-wiring

2026-09-29 15:46:41 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#63