2 Commits

Include renames

Author	SHA1	Message	Date
rock	aac68386d8	fix(proxy): wire LLM latency metrics into the streaming response path (#63) ... `NewLLMMetricsCapture`/`ResponseWriterWrapper` had **zero call sites**, so `llm_ttft_seconds`, `llm_itl_seconds` and `llm_tokens_total` were exported as HELP/TYPE stubs with no samples (verified against the live `/metrics`). These are the only per-request LLM latency series covering the Ollama models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no `/metrics` of their own — so those models had zero latency visibility. ## Change Wire the wrapper around the response writer in `ServeHTTP` for model-routed requests, and delete the unused `io.WriteCloser` duplicate that made the gap hard to spot (two near-identical implementations, neither constructed). ## Correctness details - **Only SSE responses are measured.** A non-streaming completion is written in one `Write`, which would otherwise be recorded as a TTFT sample equal to the whole request latency, with no ITL samples — plausible-looking but meaningless numbers. - **`requestStart` is captured at the top of `ServeHTTP`** so TTFT measures from request acceptance, not from when the upstream response is copied. - **Token count skips the `data: [DONE]` terminator.** - **`Unwrap()`** keeps `http.ResponseController` (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (#33) is unaffected. ## Tests | Test | Guards | |---|---| | `TestResponseWriterWrapperRecordsSSEMetrics` | TTFT=1, ITL>0, tokens=2 (terminator excluded) | | `TestResponseWriterWrapperIgnoresNonStreaming` | JSON response records nothing | | `TestResponseWriterWrapperPreservesStreaming` | bytes + Flush pass through | | `TestHandlerRecordsLLMMetricsForStreaming` | end-to-end through `ServeHTTP` | The last one fails if the wrapper is ever wired nowhere again — verified by disabling the wiring locally and watching it fail. ## Verification needed post-merge Confirm samples appear: ``` curl -s https://api.riotpiao.com/metrics | grep -E "llm_(ttft|itl)_seconds_count|llm_tokens_total" ```	2026-09-29 15:46:39 +00:00
rockandpoimen	d439536ca9	Add TTFT & ITL Metrics for LLM Inference (#28) ... Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total Closes #31 #32 #33 --------- Co-authored-by: poimen <[email protected]> Reviewed-on: #28	2026-09-15 08:53:58 +00:00