fix(netpol): allow Prometheus scrape of gateway /metrics #62

2026-09-29T14:08:48Z

rock commented

2026-09-29 14:08:48 +00:00

Problem

The new api-gateway ServiceMonitor discovers all 3 replicas, but every scrape
fails with context deadline exceeded:

down   api-gateway   api-gateway-6986b7c889-pmt4b   err=Get "http://10.244.0.180:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-c28dt   err=Get "http://10.244.1.1:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-552gl   err=Get "http://10.244.2.61:8080/metrics": context deadline

Cause

network-policy.yaml allowed Prometheus ingress with:

namespaceSelector:
  matchLabels:
    name: monitoring

Only ingress-nginx carries a plain name: label in this cluster. Every other
namespace has just kubernetes.io/metadata.name, so this rule matched nothing
and silently denied the scrape. (The rule directly above it — name: ingress-nginx
— works only because that namespace is the documented exception.)

Fix

Select on kubernetes.io/metadata.name: monitoring.

Impact

Unblocks gateway metrics: llm_ttft_seconds, llm_itl_seconds,
llm_tokens_total, queue_full_requests_total,
warmup_requests_delayed_total, vllm_prefix_cache_hit_tokens_total.

These are the only per-request LLM latency series that cover the two Ollama
models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics endpoint
of their own.

## Problem The new `api-gateway` ServiceMonitor discovers all 3 replicas, but every scrape fails with `context deadline exceeded`: ``` down api-gateway api-gateway-6986b7c889-pmt4b err=Get "http://10.244.0.180:8080/metrics": context deadline down api-gateway api-gateway-6986b7c889-c28dt err=Get "http://10.244.1.1:8080/metrics": context deadline down api-gateway api-gateway-6986b7c889-552gl err=Get "http://10.244.2.61:8080/metrics": context deadline ``` ## Cause `network-policy.yaml` allowed Prometheus ingress with: ```yaml namespaceSelector: matchLabels: name: monitoring ``` Only `ingress-nginx` carries a plain `name:` label in this cluster. Every other namespace has just `kubernetes.io/metadata.name`, so this rule matched **nothing** and silently denied the scrape. (The rule directly above it — `name: ingress-nginx` — works only because that namespace is the documented exception.) ## Fix Select on `kubernetes.io/metadata.name: monitoring`. ## Impact Unblocks gateway metrics: `llm_ttft_seconds`, `llm_itl_seconds`, `llm_tokens_total`, `queue_full_requests_total`, `warmup_requests_delayed_total`, `vllm_prefix_cache_hit_tokens_total`. These are the only per-request LLM latency series that cover the two Ollama models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no metrics endpoint of their own.

rock added 1 commit 2026-09-29 14:08:49 +00:00

fix(netpol): allow Prometheus scrape of gateway /metrics

CI / CI (pull_request) Successful in 27m49s

Details

c3a11bccec

Selector used a plain `name: monitoring` label, but only ingress-nginx
carries that label in this cluster — every other namespace has just
kubernetes.io/metadata.name. The rule matched nothing, so the new
api-gateway ServiceMonitor discovered all 3 replicas but every scrape
died with context deadline exceeded.

Switch to kubernetes.io/metadata.name: monitoring.

rock merged commit 28753d11ed into main

2026-09-29 14:40:33 +00:00

rock deleted branch fix/prometheus-scrape-netpol

2026-09-29 14:40:34 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#62