homelab-frontend/tasks/5.1-prometheus-parity.md

# 5.1 — Prometheus parity with the retiring Kong plugin (GREEN)

Phase: 5 — Observability
Stage: RED

Kong runs a cluster-wide `prometheus` plugin today. It is deleted at teardown. If the
gateway does not carry equivalent signal, observability REGRESSES at cutover and
nobody notices until an incident.

- [ ] A Prometheus text-format endpoint is exposed and scrapeable without authentication from inside the cluster
- [ ] Request rate is observable, labelled by route and by upstream
- [ ] Request latency is observable as a distribution, not a mean, labelled by route and upstream
- [ ] Response status is observable, labelled by route, upstream, and status class
- [ ] Bandwidth in both directions is observable per route and upstream
- [ ] Upstream health is observable — whether each configured upstream is currently reachable and answering
- [ ] Label values are drawn from the configured route and upstream names, never from raw request paths or user input, so cardinality cannot be driven by a caller
- [ ] Streaming responses record their full byte count and full duration, not the time to first byte
- [ ] The metrics endpoint is not reachable through the public route surface

Write the assertions against the metric families first; they must fail because the
families are absent, not because the endpoint 404s.

## Verify

```bash
curl -s localhost:8080/metrics | grep -E '^# TYPE' | awk '{print $3}' | sort -u
# expected: includes counter and histogram families covering requests, duration, bytes, upstream health

curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"reasoning","messages":[]}' >/dev/null
curl -s localhost:8080/metrics | grep 'upstream="reasoning-predictor"' | head
# expected: request, duration and byte samples all carry route and upstream labels
```