132 lines
5.3 KiB
Markdown
132 lines
5.3 KiB
Markdown
# SLA: API Gateway & Platform Services
|
|
|
|
## API Gateway (api.riotpiao.com)
|
|
|
|
### Availability
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| 99.9% uptime | `probe_success{instance=~".*api.riotpiao.com.*"}` | `APIGatewayProbeDown` fires after 2m down |
|
|
| Monthly budget: 43.8 min downtime | 7-day SLO: `avg_over_time(probe_success[7d]) * 100` | |
|
|
| Zero ready pods = critical | `sum(kube_pod_status_ready{namespace="api"}) == 0` | `APIGatewayDown` fires after 1m |
|
|
|
|
### Latency
|
|
|
|
Baselines measured from 200-request canary run against live cluster.
|
|
SLA set at ~2x measured p99 for headroom.
|
|
|
|
| Endpoint | Measured p50 | Measured p99 | SLA (p95) | SLA (p99) | Alert |
|
|
|----------|-------------|-------------|-----------|-----------|-------|
|
|
| LLM Chat (qwen) | 514ms | 609ms | <1s | <2s | `APIGatewayLatencyHigh` |
|
|
| LLM Chat (reasoning) | 300ms | 328ms | <1s | <2s | `APIGatewayLatencyHigh` |
|
|
| LLM Chat (ornith:35b) | 1.2s | 1.2s | <3s | <5s | `APIGatewayLatencyCritical` |
|
|
| LLM Chat (streaming) | 569ms | 628ms | <1s | <2s | `APIGatewayLatencyHigh` |
|
|
| Embeddings | 189ms | 287ms | <500ms | <1s | `APIGatewayLatencyHigh` |
|
|
| Rerank | 106ms | 218ms | <500ms | <1s | `APIGatewayLatencyHigh` |
|
|
| Models list | 68ms | 277ms | <300ms | <500ms | `APIGatewayLatencyHigh` |
|
|
| Auth rejection | 69ms | 87ms | <200ms | <500ms | (no alert, expected fast) |
|
|
|
|
### Error Rate
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| 5xx < 1% | `nginx_ingress_controller_requests{status=~"5.."}` / total | `APIGateway5xxErrorRate` fires after 5m >1% |
|
|
| Total errors < 10% | 4xx + 5xx / total | `APIGatewayHighErrorRate` fires after 10m >10% |
|
|
|
|
---
|
|
|
|
## LLM Serving (llm-serving namespace)
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| All predictors running | replicas ready == desired per deployment | `LLMPredictorDown` fires after 5m |
|
|
| Zero LLM pods = critical | `sum(ready{namespace="llm-serving"}) == 0` | `LLMServingDown` fires after 2m |
|
|
| No restart storms | restart count in 15m | `LLMPredictorRestarted` on any restart |
|
|
|
|
---
|
|
|
|
## Cluster Infrastructure
|
|
|
|
### Node Health
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| All nodes Ready | `kube_node_status_condition` | `NodeNotReady` fires after 2m |
|
|
| CPU < 90% sustained | `node_cpu_seconds_total` | `NodeHighCPU` fires after 15m |
|
|
| Memory < 90% sustained | `node_memory_MemAvailable_bytes` | `NodeHighMemory` fires after 15m |
|
|
| Disk < 85% | `node_filesystem_avail_bytes` | `NodeDiskFull` fires after 5m (critical) |
|
|
|
|
### Pod Health
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| No pods pending > 10m | `kube_pod_status_phase{phase="Pending"}` | `PodStuckPending` |
|
|
| No CrashLoopBackOff > 5m | `kube_pod_container_status_waiting_reason` | `PodCrashLooping` (critical) |
|
|
| OOMKilled < 3/hour | `kube_pod_container_status_last_terminated_reason` | `OOMKilledSpike` |
|
|
| No restart storms | >5 restarts in 15m | `ContainerRestartStorm` |
|
|
|
|
### Jobs
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| No failed jobs | `kube_job_status_failed > 0` | `JobFailed` fires after 5m |
|
|
| No stuck jobs > 2h | `kube_job_status_active` + age | `JobStuckRunning` |
|
|
| CronJobs on schedule | last_schedule vs next_schedule | `CronJobMissedSchedule` fires after 10m |
|
|
|
|
### Storage
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| Longhorn drives healthy | `longhorn_disk_health` | `LonghornDriveOffline` fires after 5m (critical) |
|
|
|
|
### DNS
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| CoreDNS SERVFAIL < 0.5/s | `coredns_dns_responses_total{rcode="SERVFAIL"}` | `CoreDNSErrorSpike` fires after 5m |
|
|
|
|
### Probes
|
|
|
|
| Target | Measurement | Alert |
|
|
|--------|------------|-------|
|
|
| All service probes passing | `probe_success` | `ServiceProbeDown` fires after 3m (critical) |
|
|
| Probe latency < 2s | `probe_duration_seconds` | `ServiceProbeSlow` fires after 5m |
|
|
| Certs valid > 14 days | `certmanager_certificate_expiration_timestamp_seconds` | `CertificateExpiringSoon` |
|
|
|
|
---
|
|
|
|
## Alert Severity Levels
|
|
|
|
| Severity | Meaning | Response Time |
|
|
|----------|---------|--------------|
|
|
| **critical** | Service down or data loss risk. Immediate impact on users. | Investigate within 15 min |
|
|
| **warning** | Degraded performance or resource pressure. No immediate outage. | Investigate within 4 hours |
|
|
|
|
### Critical Alerts (require immediate action)
|
|
|
|
- `APIGatewayDown` — zero gateway pods
|
|
- `LLMServingDown` — zero LLM pods
|
|
- `NodeNotReady` — node lost
|
|
- `PodCrashLooping` — service crashing repeatedly
|
|
- `NodeDiskFull` — disk > 85%
|
|
- `LonghornDriveOffline` — storage unhealthy
|
|
- `ServiceProbeDown` — external service unreachable
|
|
- `APIGateway5xxErrorRate` — 5xx > 1%
|
|
- `APIGatewayLatencyCritical` — p99 > 5s
|
|
|
|
---
|
|
|
|
## Current Alert Status
|
|
|
|
Alerts firing after deployment:
|
|
|
|
| Alert | State | Root Cause |
|
|
|-------|-------|-----------|
|
|
| `APIGatewayProbeDown` | pending | Blackbox probe for api-gateway not yet active (pod restart needed) |
|
|
| `PodStuckPending` | pending | `sms/macos-bluebubbles` pending 22d (scheduling constraint) |
|
|
| `PodCrashLooping` | pending | `iam/authentik-provision` job in Error state |
|
|
| `DeploymentReplicasUnavailable` | pending | Same root causes above |
|
|
| `ServiceProbeDown` | pending | api-gateway probe target not in blackbox yet |
|
|
|
|
None are false positives. All reflect real cluster state.
|