31 lines
1.6 KiB
Markdown
31 lines
1.6 KiB
Markdown
|
# 5.3 — ServiceMonitor for gateway scraping (GREEN)
|
||
|
|
|
||
|
|
Phase: 5 — Observability
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [5.2](5.2-gateway-metrics.md)
|
||
|
|
|
||
|
|
Exposing metrics is not the same as having them collected. Kong's plugin was picked
|
||
|
|
up cluster-wide; the gateway must be explicitly registered or the dashboards go
|
||
|
|
blank at cutover.
|
||
|
|
|
||
|
|
- [ ] A ServiceMonitor selects the gateway Service and is committed to git, applied by Argo — never `kubectl apply`
|
||
|
|
- [ ] It carries whatever label the cluster's Prometheus uses to select ServiceMonitors, verified against the running Prometheus rather than assumed
|
||
|
|
- [ ] The scraped port is a named port on the gateway Service, matched by name not number
|
||
|
|
- [ ] Scrape interval and timeout are explicit
|
||
|
|
- [ ] The gateway appears as an `up == 1` target in Prometheus, in the expected namespace
|
||
|
|
- [ ] Metric labels identify the gateway pod and namespace, so 2 replicas are distinguishable
|
||
|
|
- [ ] Scraping works while the gateway is still unexposed to public traffic — this lands before cutover, not after
|
||
|
|
- [ ] Note: `prometheus-operated.monitoring` is headless with ClusterIP `None`, so any egress policy that touches it needs pod selectors, not a ClusterIP
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
kubectl -n monitoring get servicemonitor -l release --show-labels | grep -i frontend
|
||
|
|
# expected: the gateway ServiceMonitor exists and carries the selector label Prometheus uses
|
||
|
|
|
||
|
|
# Query Prometheus for the gateway target
|
||
|
|
curl -s 'http://localhost:9090/api/v1/query?query=up{job=~".*homelab-frontend.*"}' \
|
||
|
|
| python3 -c "import json,sys;r=json.load(sys.stdin)['data']['result'];print(len(r),[x['value'][1] for x in r])"
|
||
|
|
# expected: 2 targets, both reporting "1"
|
||
|
|
```
|