`NewLLMMetricsCapture`/`ResponseWriterWrapper` had **zero call sites**, so `llm_ttft_seconds`, `llm_itl_seconds` and `llm_tokens_total` were exported as HELP/TYPE stubs with no samples (verified against the live `/metrics`).
These are the only per-request LLM latency series covering the Ollama models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no `/metrics` of their own — so those models had zero latency visibility.
## Change
Wire the wrapper around the response writer in `ServeHTTP` for model-routed requests, and delete the unused `io.WriteCloser` duplicate that made the gap hard to spot (two near-identical implementations, neither constructed).
## Correctness details
- **Only SSE responses are measured.** A non-streaming completion is written in one `Write`, which would otherwise be recorded as a TTFT sample equal to the whole request latency, with no ITL samples — plausible-looking but meaningless numbers.
- **`requestStart` is captured at the top of `ServeHTTP`** so TTFT measures from request acceptance, not from when the upstream response is copied.
- **Token count skips the `data: [DONE]` terminator.**
- **`Unwrap()`** keeps `http.ResponseController` (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (#33) is unaffected.
## Tests
| Test | Guards |
|---|---|
| `TestResponseWriterWrapperRecordsSSEMetrics` | TTFT=1, ITL>0, tokens=2 (terminator excluded) |
| `TestResponseWriterWrapperIgnoresNonStreaming` | JSON response records nothing |
| `TestResponseWriterWrapperPreservesStreaming` | bytes + Flush pass through |
| `TestHandlerRecordsLLMMetricsForStreaming` | end-to-end through `ServeHTTP` |
The last one fails if the wrapper is ever wired nowhere again — verified by disabling the wiring locally and watching it fail.
## Verification needed post-merge
Confirm samples appear:
```
curl -s https://api.riotpiao.com/metrics | grep -E "llm_(ttft|itl)_seconds_count|llm_tokens_total"
```
- Add internal/tracing package with OTel tracer initialization
- HTTP middleware for server-side tracing (request/response attributes)
- Transport wrapper for client-side upstream call tracing
- Update proxy to use tracing transport
- Add OTEL_* env vars to k8s deployment
Traces flow: api-gateway -> otel-collector -> tempo -> grafana
- 2.3: unknown model errors (400 + RFC 9457 problem+json with valid_models)
- 2.5: GET /v1/models endpoint (derived from config, not hardcoded)
- 2.6: POST /v1/embeddings passthrough (body-based dispatch, no rewrite)
- 2.7: POST /v1/rerank with path rewrite (/v1/rerank → /rerank)
- wire proxy.Handler in main.go (was using dummy handler)
- 140+ tests passing, race detector clean
- all requests: client → nginx → gateway → upstreams
- ready for config deployment to go live
Baseline for the Kong replacement on api.riotpiao.com. Brings the working
tree under version control for the first time: gateway source, the task
board that drives the agent runs, test fixtures, and K8s manifests.
Anchor the gateway ignore rule to the repo root. Unanchored, "gateway"
also matched the cmd/gateway/ source directory, so the program entrypoint
was excluded from every commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
rock
