homelab-frontend/SLA.md

SLA: API Gateway & Platform Services

API Gateway (api.riotpiao.com)

Availability

Target	Measurement	Alert
99.9% uptime	probe_success{instance=~".*api.riotpiao.com.*"}	APIGatewayProbeDown fires after 2m down
Monthly budget: 43.8 min downtime	7-day SLO: avg_over_time(probe_success[7d]) * 100
Zero ready pods = critical	sum(kube_pod_status_ready{namespace="api"}) == 0	APIGatewayDown fires after 1m

Latency

Baselines measured from 200-request canary run against live cluster. SLA set at ~2x measured p99 for headroom.

Endpoint	Measured p50	Measured p99	SLA (p95)	SLA (p99)	Alert
LLM Chat (qwen)	514ms	609ms	<1s	<2s	APIGatewayLatencyHigh
LLM Chat (reasoning)	300ms	328ms	<1s	<2s	APIGatewayLatencyHigh
LLM Chat (ornith:35b)	1.2s	1.2s	<3s	<5s	APIGatewayLatencyCritical
LLM Chat (streaming)	569ms	628ms	<1s	<2s	APIGatewayLatencyHigh
Embeddings	189ms	287ms	<500ms	<1s	APIGatewayLatencyHigh
Rerank	106ms	218ms	<500ms	<1s	APIGatewayLatencyHigh
Models list	68ms	277ms	<300ms	<500ms	APIGatewayLatencyHigh
Auth rejection	69ms	87ms	<200ms	<500ms	(no alert, expected fast)

Error Rate

Target	Measurement	Alert
5xx < 1%	nginx_ingress_controller_requests{status=~"5.."} / total	APIGateway5xxErrorRate fires after 5m >1%
Total errors < 10%	4xx + 5xx / total	APIGatewayHighErrorRate fires after 10m >10%

---

LLM Serving (llm-serving namespace)

Target	Measurement	Alert
All predictors running	replicas ready == desired per deployment	LLMPredictorDown fires after 5m
Zero LLM pods = critical	sum(ready{namespace="llm-serving"}) == 0	LLMServingDown fires after 2m
No restart storms	restart count in 15m	LLMPredictorRestarted on any restart

---

Cluster Infrastructure

Node Health

Target	Measurement	Alert
All nodes Ready	kube_node_status_condition	NodeNotReady fires after 2m
CPU < 90% sustained	node_cpu_seconds_total	NodeHighCPU fires after 15m
Memory < 90% sustained	node_memory_MemAvailable_bytes	NodeHighMemory fires after 15m
Disk < 85%	node_filesystem_avail_bytes	NodeDiskFull fires after 5m (critical)

Pod Health

Target	Measurement	Alert
No pods pending > 10m	kube_pod_status_phase{phase="Pending"}	PodStuckPending
No CrashLoopBackOff > 5m	kube_pod_container_status_waiting_reason	PodCrashLooping (critical)
OOMKilled < 3/hour	kube_pod_container_status_last_terminated_reason	OOMKilledSpike
No restart storms	>5 restarts in 15m	ContainerRestartStorm

Jobs

Target	Measurement	Alert
No failed jobs	kube_job_status_failed > 0	JobFailed fires after 5m
No stuck jobs > 2h	kube_job_status_active + age	JobStuckRunning
CronJobs on schedule	last_schedule vs next_schedule	CronJobMissedSchedule fires after 10m

Storage

Target	Measurement	Alert
Longhorn drives healthy	longhorn_disk_health	LonghornDriveOffline fires after 5m (critical)

DNS

Target	Measurement	Alert
CoreDNS SERVFAIL < 0.5/s	coredns_dns_responses_total{rcode="SERVFAIL"}	CoreDNSErrorSpike fires after 5m

Probes

Target	Measurement	Alert
All service probes passing	probe_success	ServiceProbeDown fires after 3m (critical)
Probe latency < 2s	probe_duration_seconds	ServiceProbeSlow fires after 5m
Certs valid > 14 days	certmanager_certificate_expiration_timestamp_seconds	CertificateExpiringSoon

---

Alert Severity Levels

Severity	Meaning	Response Time
critical	Service down or data loss risk. Immediate impact on users.	Investigate within 15 min
warning	Degraded performance or resource pressure. No immediate outage.	Investigate within 4 hours

Critical Alerts (require immediate action)

• APIGatewayDown — zero gateway pods
• LLMServingDown — zero LLM pods
• NodeNotReady — node lost
• PodCrashLooping — service crashing repeatedly
• NodeDiskFull — disk > 85%
• LonghornDriveOffline — storage unhealthy
• ServiceProbeDown — external service unreachable
• APIGateway5xxErrorRate — 5xx > 1%
• APIGatewayLatencyCritical — p99 > 5s

---

Current Alert Status

Alerts firing after deployment:

Alert	State	Root Cause
APIGatewayProbeDown	pending	Blackbox probe for api-gateway not yet active (pod restart needed)
PodStuckPending	pending	sms/macos-bluebubbles pending 22d (scheduling constraint)
PodCrashLooping	pending	iam/authentik-provision job in Error state
DeploymentReplicasUnavailable	pending	Same root causes above
ServiceProbeDown	pending	api-gateway probe target not in blackbox yet

None are false positives. All reflect real cluster state.