2 Commits

Include renames

Author	SHA1	Message	Date
Admin Bot	966eebe9fe	fix(proxy): wire LLM latency metrics into the streaming response path ... NewLLMMetricsCapture/ResponseWriterWrapper had zero call sites, so llm_ttft_seconds, llm_itl_seconds and llm_tokens_total were exported as HELP/TYPE stubs with no samples. These are the only per-request LLM latency series covering the Ollama models (ornith:35b, qwen2.5:3b-instruct), which expose no /metrics of their own. Wire the wrapper around the response writer in ServeHTTP for model-routed requests, and delete the unused io.WriteCloser duplicate that made the gap hard to spot (two near-identical implementations, neither constructed). Correctness details: - Only SSE responses are measured. A non-streaming completion is written in one Write, which would otherwise be recorded as a TTFT sample equal to the whole request latency with no ITL samples. - requestStart is captured at the top of ServeHTTP so TTFT measures from request acceptance, not from when the upstream response is copied. - Token count skips the `data: [DONE]` terminator. - Unwrap() keeps http.ResponseController (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (issue #33) is unaffected. Tests: unit coverage for SSE vs non-streaming, wrapper transparency, and an end-to-end handler test that fails if the wrapper is ever wired nowhere again.	2026-09-29 22:50:26 +08:00
rockandpoimen	d439536ca9	Add TTFT & ITL Metrics for LLM Inference (#28) ... Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total Closes #31 #32 #33 --------- Co-authored-by: poimen <[email protected]> Reviewed-on: #28	2026-09-15 08:53:58 +00:00