266 lines
6.4 KiB
Markdown
266 lines
6.4 KiB
Markdown
# Monitoring: Prometheus, Grafana & Loki
|
|
|
|
**Prometheus:** `prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090`
|
|
**Grafana:** `https://grafana.riotpiao.com`
|
|
**Loki:** `loki.logging.svc.cluster.local:3100`
|
|
**Namespaces:** `monitoring`, `logging`
|
|
|
|
## When to Use
|
|
|
|
- **Metrics** — CPU, memory, latency, request rate, custom business metrics (Prometheus)
|
|
- **Logs** — Pod logs, kernel logs, trace events (Loki)
|
|
- **Dashboards** — Real-time visualization (Grafana)
|
|
- **Alerts** — Page on high error rate, latency spike, or resource exhaustion
|
|
|
|
## Quick Start
|
|
|
|
**1. Access Grafana:**
|
|
```bash
|
|
# Browser: https://grafana.riotpiao.com
|
|
# Login: admin / GRAFANA_ADMIN_PASSWORD (from .env)
|
|
# Or via Authentik SSO
|
|
|
|
# Port-forward (if no external access)
|
|
make pf-grafana # localhost:3000
|
|
```
|
|
|
|
**2. View dashboards:**
|
|
```
|
|
Grafana → Dashboards → Homelab folder
|
|
- Service Availability (uptime probes + cert expiry)
|
|
- Latency & Golden Signals (request rate/error %/duration)
|
|
- Hardware Statistics (per-node CPU/mem/disk)
|
|
- Kube-Controller Health (API server metrics)
|
|
- Service Internals (per-service deep-dive)
|
|
```
|
|
|
|
**3. Add Prometheus data source:**
|
|
```
|
|
Grafana → Settings → Data Sources → Add
|
|
- Type: Prometheus
|
|
- URL: http://prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090
|
|
- Save & test
|
|
```
|
|
|
|
**4. Create dashboard:**
|
|
```json
|
|
{
|
|
"dashboard": {
|
|
"title": "My Service",
|
|
"panels": [
|
|
{
|
|
"targets": [
|
|
{
|
|
"expr": "rate(http_requests_total{service='myapp'}[5m])"
|
|
}
|
|
],
|
|
"title": "Request Rate"
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
## Instrumentation Patterns
|
|
|
|
**Go (Prometheus):**
|
|
```go
|
|
import "github.com/prometheus/client_golang/prometheus"
|
|
|
|
// Counter (monotonic increase)
|
|
requestsTotal := prometheus.NewCounterVec(
|
|
prometheus.CounterOpts{
|
|
Name: "http_requests_total",
|
|
Help: "Total requests",
|
|
},
|
|
[]string{"method", "status"},
|
|
)
|
|
|
|
// Gauge (can go up or down)
|
|
activeConnections := prometheus.NewGauge(prometheus.GaugeOpts{
|
|
Name: "active_connections",
|
|
})
|
|
|
|
// Histogram (latency buckets)
|
|
requestDuration := prometheus.NewHistogramVec(
|
|
prometheus.HistogramOpts{
|
|
Name: "http_request_duration_seconds",
|
|
Buckets: []float64{.001, .01, .1, 1},
|
|
},
|
|
[]string{"method", "path"},
|
|
)
|
|
|
|
// Record metric
|
|
requestsTotal.WithLabelValues("POST", "200").Inc()
|
|
requestDuration.WithLabelValues("POST", "/api/users").Observe(0.042)
|
|
```
|
|
|
|
**Node.js (prom-client):**
|
|
```javascript
|
|
const prometheus = require('prom-client');
|
|
|
|
const httpRequestDuration = new prometheus.Histogram({
|
|
name: 'http_request_duration_seconds',
|
|
help: 'Duration of HTTP requests in seconds',
|
|
labelNames: ['method', 'path', 'status'],
|
|
buckets: [0.001, 0.01, 0.1, 1],
|
|
});
|
|
|
|
// Middleware
|
|
app.use((req, res, next) => {
|
|
const start = Date.now();
|
|
res.on('finish', () => {
|
|
const duration = (Date.now() - start) / 1000;
|
|
httpRequestDuration.labels(req.method, req.path, res.statusCode).observe(duration);
|
|
});
|
|
next();
|
|
});
|
|
|
|
// Expose metrics
|
|
app.get('/metrics', (req, res) => {
|
|
res.set('Content-Type', prometheus.register.contentType);
|
|
res.end(prometheus.register.metrics());
|
|
});
|
|
```
|
|
|
|
## ServiceMonitor (Auto-Scrape)
|
|
|
|
**Prometheus automatically discovers ServiceMonitor resources:**
|
|
```yaml
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: myapp
|
|
namespace: myapp-ns
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app: myapp
|
|
endpoints:
|
|
- port: metrics
|
|
interval: 30s
|
|
path: /metrics
|
|
```
|
|
|
|
**Deploy & verify:**
|
|
```bash
|
|
kubectl apply -f myapp-servicemonitor.yaml
|
|
|
|
# Check Prometheus targets
|
|
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
|
# http://localhost:9090/targets → should show myapp
|
|
```
|
|
|
|
## PrometheusRule (Alerting)
|
|
|
|
**Define alert rules:**
|
|
```yaml
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: myapp-alerts
|
|
namespace: myapp-ns
|
|
spec:
|
|
groups:
|
|
- name: myapp
|
|
rules:
|
|
- alert: HighErrorRate
|
|
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05 # > 5% error rate
|
|
for: 5m
|
|
annotations:
|
|
summary: "High error rate for {{ $labels.service }}"
|
|
|
|
- alert: HighLatency
|
|
expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1 # p99 > 1s
|
|
for: 10m
|
|
annotations:
|
|
summary: "High latency for {{ $labels.path }}"
|
|
```
|
|
|
|
## Logging (Loki)
|
|
|
|
**View pod logs in Grafana:**
|
|
```
|
|
Grafana → Explore → Loki
|
|
Query: {namespace="story-crater-backend"} | json | level="error"
|
|
|
|
Label filters:
|
|
- namespace
|
|
- pod_name
|
|
- container
|
|
- level (error, warn, info)
|
|
```
|
|
|
|
**Log retention:**
|
|
- 10-day default (configurable)
|
|
- Older logs deleted automatically
|
|
- Chunks stored in MinIO (s3://loki-chunks)
|
|
|
|
## Monitoring Checklist
|
|
|
|
**For every service:**
|
|
- [ ] Export `/metrics` (Prometheus format)
|
|
- [ ] Add ServiceMonitor resource
|
|
- [ ] Add PrometheusRule (errors, latency)
|
|
- [ ] Create Grafana dashboard (6 rows: Availability, Resources, Domain, Logs, SLO, Related)
|
|
- [ ] Set up Slack/PagerDuty integration (if on-call)
|
|
|
|
## Key Queries
|
|
|
|
**Request rate by status:**
|
|
```promql
|
|
sum(rate(http_requests_total[5m])) by (status)
|
|
```
|
|
|
|
**Error rate %:**
|
|
```promql
|
|
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
|
|
```
|
|
|
|
**P99 latency:**
|
|
```promql
|
|
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
|
|
```
|
|
|
|
**Memory usage %:**
|
|
```promql
|
|
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
**Prometheus can't scrape ServiceMonitor:**
|
|
```bash
|
|
# Check ServiceMonitor exists
|
|
k get servicemonitor -A | grep myapp
|
|
|
|
# Verify Service has correct labels
|
|
k get svc -A -o json | jq '.items[] | select(.metadata.name=="myapp") | .metadata.labels'
|
|
|
|
# Check Prometheus targets
|
|
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
|
# http://localhost:9090/targets → look for "Down" targets with error
|
|
```
|
|
|
|
**Metrics missing from Grafana:**
|
|
```bash
|
|
# Verify pod is exporting metrics
|
|
k port-forward -n myapp-ns pod/myapp-0 8080:8080
|
|
curl http://localhost:8080/metrics | grep my_metric_name
|
|
|
|
# Check query syntax in Grafana
|
|
# Hover over query → "Query Inspector" to see response
|
|
```
|
|
|
|
**Alert not firing:**
|
|
```bash
|
|
# Check PrometheusRule is loaded
|
|
k get prometheusrule -n myapp-ns
|
|
|
|
# Verify query in Prometheus
|
|
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
|
# http://localhost:9090 → Graph tab → paste query
|
|
```
|
|
|
|
See `/TROUBLESHOOTING.md` for full incident guide.
|