homelab-frontend

riotpiao-poimen/homelab-frontend

Fork 0

Author	SHA1	Message	Date
Admin Bot	aac502ea1f	fix(proxy): wire LLM latency metrics into the streaming response path CI / CI (pull_request) Successful in 27m12s Details NewLLMMetricsCapture/ResponseWriterWrapper had zero call sites, so llm_ttft_seconds, llm_itl_seconds and llm_tokens_total were exported as HELP/TYPE stubs with no samples. These are the only per-request LLM latency series covering the Ollama models (ornith:35b, qwen2.5:3b-instruct), which expose no /metrics of their own. Wire the wrapper around the response writer in ServeHTTP for model-routed requests, and delete the unused io.WriteCloser duplicate that made the gap hard to spot (two near-identical implementations, neither constructed). Correctness details: - Only SSE responses are measured. A non-streaming completion is written in one Write, which would otherwise be recorded as a TTFT sample equal to the whole request latency with no ITL samples. - requestStart is captured at the top of ServeHTTP so TTFT measures from request acceptance, not from when the upstream response is copied. - Token count skips the `data: [DONE]` terminator. - Unwrap() keeps http.ResponseController (Flush/Hijack/deadlines) working through the wrapper, so SSE unbuffering (issue #33) is unaffected. Tests: unit coverage for SSE vs non-streaming, wrapper transparency, and an end-to-end handler test that fails if the wrapper is ever wired nowhere again.	2026-09-29 22:50:39 +08:00
rockandAdmin Bot	28753d11ed	fix(netpol): allow Prometheus scrape of gateway /metrics (#62 ) CI / CI (push) Successful in 32m48s Details ## Problem The new `api-gateway` ServiceMonitor discovers all 3 replicas, but every scrape fails with `context deadline exceeded`: ``` down api-gateway api-gateway-6986b7c889-pmt4b err=Get "http://10.244.0.180:8080/metrics": context deadline down api-gateway api-gateway-6986b7c889-c28dt err=Get "http://10.244.1.1:8080/metrics": context deadline down api-gateway api-gateway-6986b7c889-552gl err=Get "http://10.244.2.61:8080/metrics": context deadline ``` ## Cause `network-policy.yaml` allowed Prometheus ingress with: ```yaml namespaceSelector: matchLabels: name: monitoring ``` Only `ingress-nginx` carries a plain `name:` label in this cluster. Every other namespace has just `kubernetes.io/metadata.name`, so this rule matched nothing and silently denied the scrape. (The rule directly above it — `name: ingress-nginx` — works only because that namespace is the documented exception.) ## Fix Select on `kubernetes.io/metadata.name: monitoring`. ## Impact Unblocks gateway metrics: `llm_ttft_seconds`, `llm_itl_seconds`, `llm_tokens_total`, `queue_full_requests_total`, `warmup_requests_delayed_total`, `vllm_prefix_cache_hit_tokens_total`. These are the only per-request LLM latency series that cover the two Ollama models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no metrics endpoint of their own. --------- Co-authored-by: Admin Bot <[email protected]> Reviewed-on: #62	2026-09-29 14:40:31 +00:00

Author

SHA1

Message

Date

Admin Bot

aac502ea1f

fix(proxy): wire LLM latency metrics into the streaming response path

CI / CI (pull_request) Successful in 27m12s

Details

NewLLMMetricsCapture/ResponseWriterWrapper had zero call sites, so
llm_ttft_seconds, llm_itl_seconds and llm_tokens_total were exported as
HELP/TYPE stubs with no samples. These are the only per-request LLM latency
series covering the Ollama models (ornith:35b, qwen2.5:3b-instruct), which
expose no /metrics of their own.

Wire the wrapper around the response writer in ServeHTTP for model-routed
requests, and delete the unused io.WriteCloser duplicate that made the gap
hard to spot (two near-identical implementations, neither constructed).

Correctness details:
- Only SSE responses are measured. A non-streaming completion is written in
  one Write, which would otherwise be recorded as a TTFT sample equal to the
  whole request latency with no ITL samples.
- requestStart is captured at the top of ServeHTTP so TTFT measures from
  request acceptance, not from when the upstream response is copied.
- Token count skips the `data: [DONE]` terminator.
- Unwrap() keeps http.ResponseController (Flush/Hijack/deadlines) working
  through the wrapper, so SSE unbuffering (issue #33) is unaffected.

Tests: unit coverage for SSE vs non-streaming, wrapper transparency, and an
end-to-end handler test that fails if the wrapper is ever wired nowhere again.

2026-09-29 22:50:39 +08:00

rockandAdmin Bot

28753d11ed

fix(netpol): allow Prometheus scrape of gateway /metrics (#62 )

CI / CI (push) Successful in 32m48s

Details

## Problem

The new `api-gateway` ServiceMonitor discovers all 3 replicas, but every scrape
fails with `context deadline exceeded`:

```
down   api-gateway   api-gateway-6986b7c889-pmt4b   err=Get "http://10.244.0.180:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-c28dt   err=Get "http://10.244.1.1:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-552gl   err=Get "http://10.244.2.61:8080/metrics": context deadline
```

## Cause

`network-policy.yaml` allowed Prometheus ingress with:

```yaml
namespaceSelector:
  matchLabels:
    name: monitoring
```

Only `ingress-nginx` carries a plain `name:` label in this cluster. Every other
namespace has just `kubernetes.io/metadata.name`, so this rule matched **nothing**
and silently denied the scrape. (The rule directly above it — `name: ingress-nginx`
— works only because that namespace is the documented exception.)

## Fix

Select on `kubernetes.io/metadata.name: monitoring`.

## Impact

Unblocks gateway metrics: `llm_ttft_seconds`, `llm_itl_seconds`,
`llm_tokens_total`, `queue_full_requests_total`,
`warmup_requests_delayed_total`, `vllm_prefix_cache_hit_tokens_total`.

These are the only per-request LLM latency series that cover the two Ollama
models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no metrics endpoint
of their own.

---------

Co-authored-by: Admin Bot <[email protected]>
Reviewed-on: #62

2026-09-29 14:40:31 +00:00

Compare commits

Diff Content Not Available