homelab-frontend/tasks/5.1-prometheus-parity.md

5.1 — Prometheus parity with the retiring Kong plugin (GREEN)

Phase: 5 — Observability Stage: RED

Kong runs a cluster-wide prometheus plugin today. It is deleted at teardown. If the gateway does not carry equivalent signal, observability REGRESSES at cutover and nobody notices until an incident.

• A Prometheus text-format endpoint is exposed and scrapeable without authentication from inside the cluster
• Request rate is observable, labelled by route and by upstream
• Request latency is observable as a distribution, not a mean, labelled by route and upstream
• Response status is observable, labelled by route, upstream, and status class
• Bandwidth in both directions is observable per route and upstream
• Upstream health is observable — whether each configured upstream is currently reachable and answering
• Label values are drawn from the configured route and upstream names, never from raw request paths or user input, so cardinality cannot be driven by a caller
• Streaming responses record their full byte count and full duration, not the time to first byte
• The metrics endpoint is not reachable through the public route surface

Write the assertions against the metric families first; they must fail because the families are absent, not because the endpoint 404s.

Verify

curl -s localhost:8080/metrics | grep -E '^# TYPE' | awk '{print $3}' | sort -u
# expected: includes counter and histogram families covering requests, duration, bytes, upstream health

curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"reasoning","messages":[]}' >/dev/null
curl -s localhost:8080/metrics | grep 'upstream="reasoning-predictor"' | head
# expected: request, duration and byte samples all carry route and upstream labels