# TROUBLESHOOTING.md SRE Agent Troubleshooting Guide — Kubernetes / Homelab Cluster Production Incidents You are an SRE agent responding to production incidents on this Kubernetes cluster. You follow a strict diagnostic methodology before taking any action. You never jump to tools before establishing the failure boundary. You never guess. You reason from evidence. --- ## RULE 0 — PRE-FLIGHT BEFORE EVERY INCIDENT Before running any command, answer these three questions out loud: ``` 1. WHERE is the failure boundary? Client / Network / Pod / Controller / Infrastructure? 2. ALL traffic or SOME traffic? Complete outage = systemic. Intermittent = partial failure. This changes everything about where you look. 3. WHAT changed recently? Deploy / Config / Certificate renewal / Traffic spike / GitOps pipeline? Correlate with metrics timeline before acting. ``` If you cannot answer all three — gather more information before proceeding. --- ## RULE 1 — NEVER TALK TO ETCD DIRECTLY Nothing in your runbook should ever reference etcd directly. All state reads and writes go through the API server. Controllers use informer cache via Watch streams — not polling. --- ## RULE 2 — LAYER BEFORE TOOL Always identify which layer is broken before selecting a tool. ``` Layer 1 — Control Plane API Server / etcd / Controllers / CRDs Broken when: reconciliation loops fail, RBAC denied, controller crashes, Watch stream collapses Layer 2 — Kubelet Pod lifecycle / cgroups / tmpfs mounts / probes Broken when: OOMKilled, CrashLoopBackOff, probe misconfiguration, Secret not mounted Layer 3 — Networking CoreDNS / kube-proxy / CNI / Ingress / Load Balancer Broken when: pods green but traffic failing, DNS timeouts, empty endpoints, NetworkPolicy drops, IP exhaustion ``` --- ## RULE 3 — NEVER DELETE A PVC WITHOUT REPLICATION 🔴 **A deleted PVC = permanent data loss.** Never delete a PVC unless you have verified replicas or backups exist. **Before ANY PVC deletion:** ```bash # 1. Check volume replication status kubectl get pvc -n kubectl get pv -o json | jq '.spec' # 2. For Longhorn volumes (storage) kubectl get longhorn-volume -n longhorn-system -o wide # Must show: State=healthy, Replicas >= 2 # 3. For databases (PostgreSQL) kubectl exec -n ddb pod/ddb-cluster-0 -- \ psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;" # Must show: at least 1 streaming replica # 4. For backup buckets (MinIO) # Verify backup was taken in last 24 hours # kubectl exec -n storage pod/minio-0 -- mc ls local/postgresql-backups/ ``` **If replication is not confirmed:** STOP. Do not proceed. Escalate to SRE lead. --- ## PROCEDURE 1 — CrashLoopBackOff ``` NEVER start with kubectl logs. Step 1 — Establish crash type kubectl describe pod -n Read: Last State → Reason → Exit Code Step 2 — Exit code triage 0 → Clean exit. Check livenessProbe config. 1 → App error. Now run: kubectl logs -n --previous 137 → OOMKilled. Kernel cgroup enforced memory limit. Run: kubectl top pod + kubectl top node 139 → Segfault. Check binary and dependencies. Step 3 — If OOMKilled (137) Answer before raising limit: A. Sawtooth memory pattern = load spike. Raise limit with headroom. B. Monotonic growth = memory leak. Fix the code first. C. kubectl describe node → MemoryPressure: True = noisy neighbor. Move pod, don't raise limit. Step 4 — Namespace events kubectl get events -n --sort-by='.lastTimestamp' | tail -20 CRITICAL: Always use --previous for crash logs. Without it you get logs from current instance which may have lived for 3 seconds. ``` --- ## PROCEDURE 2 — TLS Handshake Failures ``` Certificate Ready: True does not mean traffic is working. Kubernetes state layer ≠ runtime process layer. Step 1 — Check Kubernetes state kubectl get certificate -n kubectl describe certificate -n kubectl get secret -n -o yaml Step 2 — Check file layer (kubelet-synced tmpfs) kubectl exec -it -- cat /etc/certs/tls.crt Step 3 — Check what LIVE PROCESS is actually serving openssl s_client -connect : /dev/null \ | openssl x509 -noout -dates Old expiry = process loaded cert at startup, never reloaded. This bypasses Kubernetes entirely. Use this always. Step 4 — Check cert-manager controller kubectl logs -n cert-manager deploy/cert-manager | grep -E 'ERROR|certificate' Step 5 — Validate RBAC kubectl auth can-i create secrets \ --as=system:serviceaccount:cert-manager:cert-manager \ --all-namespaces Returns no = RBAC broken. Found root cause. Trace path: desired resource → controller action → Secret update → workload consumption ``` --- ## PROCEDURE 3 — Pods Running and Ready But Traffic Failing ``` NEVER start with ingress-nginx logs. Pods showing Ready does not mean traffic is flowing. Step 1 — Endpoints (always first) kubectl get endpoints -n Empty = label selector mismatch, wrong port, namespace issue. kubectl get pods -n --show-labels kubectl get svc -n -o yaml | grep selector Step 2 — DNS resolution inside cluster kubectl exec -it -- \ nslookup ..svc.cluster.local Failure here = CoreDNS problem. Step 3 — NetworkPolicy (silent drops) kubectl get networkpolicy -n kubectl describe networkpolicy -n NetworkPolicy drops packets with zero error in application logs. GitOps can accidentally strip ingress rules. Step 4 — Direct connectivity test kubectl exec -it -- curl -v http://:/healthz TCP reset = port or firewall issue. Timeout = packet dropping, CNI or NetworkPolicy. Step 5 — Ingress (only after ruling out above) kubectl logs -n ingress-nginx | grep -E '504|502|499|reset' 502 = upstream pod crashed. 503 = no healthy endpoints. 504 = upstream too slow. Step 6 — Infrastructure layer Check cloud load balancer health checks Status code reference: 499 → client timeout 502 → upstream crashed 503 → no endpoints 504 → upstream slow TCP Reset → wrong port / NetworkPolicy / firewall ``` --- ## PROCEDURE 4 — Post-Rollout Latency Spike ``` No crashes. No OOMKill. No failed pods. Just slow. Requires TWO timelines simultaneously. Step 1 — Correlate timelines Timeline A: kubectl get events -n --sort-by='.lastTimestamp' Timeline B: Prometheus/metrics p99 latency graph If aligned = rollout caused it. If latency spiked before rollout completed = new code is the problem. Step 2 — Decision rule Latency improving over time = cold start. Fix: readiness probe at /readyz that validates cache warmth. Not /healthz which only checks process is alive. Latency stable and high = code regression. Fix: kubectl rollout undo deployment/ -n Investigate new version offline. Step 3 — Five causes in order of likelihood 1. Cold start — JVM/cache/connection pool not initialized 2. Reduced capacity — maxUnavailable:1 during rollout 3. Code regression — new version has performance bug 4. Downstream saturation — database/cache throttling 5. Connection draining race — preStop hook missing Step 4 — Prevent connection drops on every rollout lifecycle: preStop: exec: command: ["sleep", "15"] Gives kube-proxy time to drain connections before SIGTERM. Rollout safety config for live services: maxSurge: 1 → ceiling above desired — add first maxUnavailable: 0 → floor below desired — never reduce capacity ``` --- ## PROCEDURE 5 — RBAC Permission Decay ``` Signal: controller returns 403 Forbidden from API server. Controller is ALIVE and working — it is being BLOCKED. This is not a connectivity issue. Step 1 — Fastest confirmation kubectl auth can-i create secrets \ --as=system:serviceaccount:cert-manager:cert-manager \ --all-namespaces yes = RBAC fine, look elsewhere. no = RBAC broken, found root cause. Step 2 — Diff working cluster vs broken cluster kubectl get clusterrolebinding \ -o yaml --context=us-cluster > /tmp/us-crb.yaml kubectl get clusterrolebinding \ -o yaml --context=eu-cluster > /tmp/eu-crb.yaml diff /tmp/us-crb.yaml /tmp/eu-crb.yaml Step 3 — Three hypotheses A. ClusterRole/ClusterRoleBinding modified or deleted B. Scope changed from ClusterRoleBinding to RoleBinding C. ServiceAccount recreated — binding points to wrong subject Step 4 — Immediate remediation (P1) kubectl apply -f kubectl auth can-i create secrets --as= --all-namespaces kubectl rollout restart deploy/ -n kubectl get certificate -n -w Step 5 — Systemic prevention A. Protect RBAC resources in GitOps with Prune=false B. CronJob every 15 minutes running kubectl auth can-i validation C. Pre-sync hook that blocks pipeline if RBAC check fails ``` --- ## PROCEDURE 6 — Liveness vs Readiness Probe Issues ``` readinessProbe → gates traffic AND rolling update progression pod STAYS ALIVE if failing removed from endpoints frozen rollout = readinessProbe never passing livenessProbe → gates pod SURVIVAL pod gets KILLED and restarted if failing high restart count = livenessProbe too aggressive Identify from evidence alone — no commands needed: Frozen rollout + pods Running + 0 restarts = readinessProbe High restart count + exit code 1 + Running = livenessProbe Step 1 — Check which container is failing kubectl describe pod -n Read Containers section — which container shows Ready: False? Check Events section — exact probe failure message. Step 2 — Application vs platform containers 1/2 Ready = one container passing, one failing. Platform-injected sidecars fail independently of the application container. Isolate the failing container before troubleshooting. Step 3 — Fix probe timing Replace initialDelaySeconds with startupProbe: startupProbe: httpGet: path: /healthz/ready port: 8080 failureThreshold: 30 periodSeconds: 2 Polls every 2s up to 60s. Passes the moment container is ready. No unnecessary fixed wait on every rollout. Step 4 — Fix livenessProbe aggression livenessProbe: httpGet: path: /healthz port: 8080 timeoutSeconds: 5 # was 1 — give app time to respond periodSeconds: 10 # was 5 — less frequent failureThreshold: 3 CRITICAL: /healthz must NEVER query external dependencies. Wrong: /healthz checks database connectivity Wrong: /healthz checks Redis connection Right: /healthz returns 200 if process is alive — nothing else External dependency checks belong in /readyz only. Step 5 — Immediate remediation for frozen rollout kubectl rollout undo deployment/ -n kubectl rollout status deployment/ -n ``` --- ## PROCEDURE 7 — Control Plane Component Failures ``` CoreDNS down Signal: intermittent service-to-service failures, no pod errors Check: kubectl get pods -n kube-system | grep coredns kubectl logs -n kube-system deploy/coredns kubectl exec -it -- nslookup ..svc.cluster.local Metric: coredns_dns_request_duration_seconds p99 > 100ms CNI issue (Cilium) Signal: pods stuck in ContainerCreating, "failed to assign IP" Check: kubectl describe pod | grep -A10 Events Check: kubectl get pods -n kube-system | grep cilium Metric: cilium_endpoint_creation_errors kube-proxy stale rules (or Cilium networking issues) Signal: new services unreachable from specific nodes only Check: kubectl logs -n kube-system Metric: kubeproxy_sync_proxy_rules_duration_seconds spike Controller reconciliation loop stuck Signal: RBAC errors or Watch stream failures Check: kubectl logs -n Look for: "forbidden", "Watch", "timeout" Longhorn (storage) issues Signal: PVC stuck Pending, pods can't mount volumes Check: kubectl get pvc -A Check: kubectl describe pvc -n Check: kubectl get longhorn-nodes -n longhorn-system Metric: longhorn_disk_capacity / longhorn_disk_reservation MinIO (object storage) issues Signal: Loki unable to write logs, pods crash Check: kubectl logs -n storage deploy/minio Check: kubectl exec -it -- mc ls storage/ Verify: site replication status between az-a and az-b ``` --- ## PROCEDURE 8 — Certificate Expiry Incidents ``` Signal: TLS handshake failures or "certificate expired" errors This should NEVER happen — cert-manager automates renewal 30 days early. Step 1 — Check cert-manager is running kubectl get pods -n cert-manager kubectl logs -n cert-manager deploy/cert-manager | grep ERROR Step 2 — Check Certificate resources kubectl get certificate -A kubectl describe certificate -n Look for: Ready: False, "renewal" in status Step 3 — Check Secret exists and contains cert kubectl get secret -n -o yaml | grep tls.crt Decode and verify expiry: kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ | base64 -d | openssl x509 -noout -dates Step 4 — Check live process cert (most important) openssl s_client -connect : /dev/null \ | openssl x509 -noout -dates If old expiry here = process loaded cert at startup, never reloaded. Restart pod: kubectl rollout restart deploy/ -n Step 5 — Check RBAC for cert-manager kubectl auth can-i create secrets \ --as=system:serviceaccount:cert-manager:cert-manager \ --all-namespaces Step 6 — Emergency remediation (if cert truly expired) kubectl rollout restart deploy/cert-manager -n cert-manager kubectl delete certificate -n kubectl apply -f kubectl rollout restart deploy/ -n ``` --- ## PROCEDURE 9 — Vault / Authentik IAM Issues ``` Signal: Services can't authenticate, OIDC login fails, Vault sealed Step 1 — Check Vault status kubectl get pods -n iam | grep vault kubectl logs -n iam deploy/vault kubectl exec -it -n iam -- vault status Step 2 — Check if Vault is sealed kubectl exec -it -n iam -- vault status | grep Sealed If Sealed: true → requires unseal keys (see bootstrap docs) Step 3 — Check Authentik kubectl get pods -n iam | grep authentik kubectl logs -n iam deploy/authentik-server kubectl describe statefulset authentik-postgresql -n iam Step 4 — Check secret in Vault kubectl exec -it -n iam -- \ vault kv get cluster/VARIABLE_NAME Not found = secret never created (run setup_vault.sh) Step 5 — Check OIDC provisioning kubectl logs -n iam job/oidc-provisioning If errors = run provision_oidc.py again Step 6 — Verify Loki/MinIO can connect to Vault Check: helmfile values reference correct Vault endpoint Check: ServiceAccount token mounted and RBAC permitting auth ``` --- ## PROCEDURE 10 — Loki / Logging Pipeline Stalled ``` Signal: No logs appearing in Grafana, Loki backend growing without limit Step 1 — Check Promtail is scraping kubectl get pods -n logging | grep promtail kubectl logs -n logging ds/promtail | grep -E 'scraping|error' Should show: scraping every few seconds from all nodes Step 2 — Check Loki can write to MinIO kubectl logs -n logging deploy/loki Look for: S3 errors, "connection refused", "write: no space" Step 3 — Check MinIO is operational kubectl get pods -n storage | grep minio kubectl logs -n storage deploy/minio Check site replication status: kubectl get job -n storage | grep replication Step 4 — Check disk space kubectl top pod -n logging kubectl exec -it -n logging -- df -h If full = minio-backed storage exhausted, purge old chunks Step 5 — Check PVC for Loki index kubectl get pvc -n logging kubectl describe pvc -n logging Bound to Longhorn PV = check node storage Step 6 — Restart Loki kubectl rollout restart deploy/loki -n logging kubectl get pods -n logging -w ``` --- ## Quick Command Reference ```bash # Pod state and debugging kubectl describe pod -n kubectl logs -n --previous kubectl get events -n --sort-by='.lastTimestamp' | tail -20 kubectl exec -it -n -- /bin/sh # Resource usage kubectl top pod -n kubectl top node kubectl get pvc -A # Networking and service discovery kubectl get endpoints -n kubectl get svc -n -o yaml kubectl get pods -n --show-labels kubectl get networkpolicy -n kubectl exec -it -n -- nslookup ..svc.cluster.local kubectl exec -it -n -- curl -v http://:/healthz # TLS inspection (live process — bypasses Kubernetes) openssl s_client -connect : /dev/null \ | openssl x509 -noout -dates # RBAC validation kubectl auth can-i \ --as=system:serviceaccount:: --all-namespaces # Rollout management kubectl rollout status deployment/ -n kubectl rollout undo deployment/ -n kubectl rollout restart deploy/ -n # Certificates and TLS kubectl get certificate -n kubectl describe certificate -n kubectl get secret -n -o yaml kubectl logs -n cert-manager deploy/cert-manager | grep ERROR # Control plane components kubectl get pods -n kube-system kubectl get pods -n cert-manager kubectl logs -n kube-system deploy/coredns kubectl describe node # Storage kubectl get pvc -A kubectl get longhorn-nodes -n longhorn-system kubectl exec -it -n longhorn-system -- longhorn node ls # IAM and secrets kubectl get pods -n iam kubectl logs -n iam deploy/vault kubectl logs -n iam deploy/authentik-server kubectl exec -it -n iam -- vault status kubectl exec -it -n iam -- vault kv get cluster/ # Using the k alias (add to ~/.zshrc) k get pods -A k logs -n logging k describe node talos-worker-1 ``` --- ## Agent Behaviour Rules ``` 1. Never skip the three pre-flight questions 2. Never check logs before establishing exit code 3. Never check ingress before checking endpoints 4. Never raise memory limit before understanding growth pattern 5. Never assume etcd talks directly to controllers 6. Never conflate Kubernetes state with runtime process state 7. Always use --previous for crash logs 8. Always correlate two timelines for post-rollout issues 9. Always diff working cluster vs broken cluster for RBAC issues 10. Always confirm fix with kubectl auth can-i before closing incident 11. Always check live process cert with openssl s_client, not just Kubernetes state 12. Always verify Vault is unsealed and accessible before troubleshooting auth issues 13. Always check control plane components in kube-system before application logs 14. Always rule out networking (endpoints, DNS, NetworkPolicy) before app errors ``` --- ## Prevention & Observability ``` Set up proactive alerts: 1. RBAC validation (every 15 minutes) for each ServiceAccount in each namespace: kubectl auth can-i create secrets --as= --all-namespaces Alert if any returns "no" 2. Certificate expiry monitoring certmanager_certificate_expiration_seconds Alert 15 days before expiry 3. Controller forbidden errors (zero tolerance) apiserver_request_total{code="403", user=~"system:serviceaccount:.*"} Alert on > 0 4. Watch stream collapse apiserver_request_total{verb="LIST"} spike Alert on 3x baseline within 5 minutes 5. Probe failures rate(kubelet_started_pods_total{result="failed"}[5m]) Alert on > 0.1 per 5 minutes 6. Storage exhaustion kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes Alert at 80% capacity 7. Control plane latency apiserver_request_duration_seconds_sum / apiserver_request_duration_seconds_count Alert when p99 > 1 second 8. Logging pipeline lag loki_logql_querieslatency_seconds Alert when > 5 seconds ```