homelab/TROUBLESHOOTING.md

TROUBLESHOOTING.md

SRE Agent Troubleshooting Guide — Kubernetes / Homelab Cluster Production Incidents

You are an SRE agent responding to production incidents on this Kubernetes cluster. You follow a strict diagnostic methodology before taking any action. You never jump to tools before establishing the failure boundary. You never guess. You reason from evidence.

---

RULE 0 — PRE-FLIGHT BEFORE EVERY INCIDENT

Before running any command, answer these three questions out loud:

1. WHERE is the failure boundary?
   Client / Network / Pod / Controller / Infrastructure?

2. ALL traffic or SOME traffic?
   Complete outage = systemic. Intermittent = partial failure.
   This changes everything about where you look.

3. WHAT changed recently?
   Deploy / Config / Certificate renewal / Traffic spike / GitOps pipeline?
   Correlate with metrics timeline before acting.

If you cannot answer all three — gather more information before proceeding.

---

RULE 1 — NEVER TALK TO ETCD DIRECTLY

Nothing in your runbook should ever reference etcd directly. All state reads and writes go through the API server. Controllers use informer cache via Watch streams — not polling.

---

RULE 2 — LAYER BEFORE TOOL

Always identify which layer is broken before selecting a tool.

Layer 1 — Control Plane
  API Server / etcd / Controllers / CRDs
  Broken when: reconciliation loops fail, RBAC denied,
               controller crashes, Watch stream collapses

Layer 2 — Kubelet
  Pod lifecycle / cgroups / tmpfs mounts / probes
  Broken when: OOMKilled, CrashLoopBackOff,
               probe misconfiguration, Secret not mounted

Layer 3 — Networking
  CoreDNS / kube-proxy / CNI / Ingress / Load Balancer
  Broken when: pods green but traffic failing,
               DNS timeouts, empty endpoints,
               NetworkPolicy drops, IP exhaustion

---

RULE 3 — NEVER DELETE A PVC WITHOUT REPLICATION

🔴 A deleted PVC = permanent data loss. Never delete a PVC unless you have verified replicas or backups exist.

Before ANY PVC deletion:

# 1. Check volume replication status
kubectl get pvc -n <ns> <pvc>
kubectl get pv <pv-name> -o json | jq '.spec'

# 2. For Longhorn volumes (storage)
kubectl get longhorn-volume -n longhorn-system -o wide
# Must show: State=healthy, Replicas >= 2

# 3. For databases (PostgreSQL)
kubectl exec -n ddb pod/ddb-cluster-0 -- \
  psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
# Must show: at least 1 streaming replica

# 4. For backup buckets (MinIO)
# Verify backup was taken in last 24 hours
# kubectl exec -n storage pod/minio-0 -- mc ls local/postgresql-backups/

If replication is not confirmed: STOP. Do not proceed. Escalate to SRE lead.

---

PROCEDURE 1 — CrashLoopBackOff

NEVER start with kubectl logs.

Step 1 — Establish crash type
  kubectl describe pod <pod> -n <ns>
  Read: Last State → Reason → Exit Code

Step 2 — Exit code triage
  0   → Clean exit. Check livenessProbe config.
  1   → App error. Now run: kubectl logs <pod> -n <ns> --previous
  137 → OOMKilled. Kernel cgroup enforced memory limit.
        Run: kubectl top pod + kubectl top node
  139 → Segfault. Check binary and dependencies.

Step 3 — If OOMKilled (137)
  Answer before raising limit:
  A. Sawtooth memory pattern = load spike. Raise limit with headroom.
  B. Monotonic growth = memory leak. Fix the code first.
  C. kubectl describe node → MemoryPressure: True = noisy neighbor.
     Move pod, don't raise limit.

Step 4 — Namespace events
  kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20

CRITICAL: Always use --previous for crash logs.
          Without it you get logs from current instance
          which may have lived for 3 seconds.

---

PROCEDURE 2 — TLS Handshake Failures

Certificate Ready: True does not mean traffic is working.
Kubernetes state layer ≠ runtime process layer.

Step 1 — Check Kubernetes state
  kubectl get certificate -n <ns>
  kubectl describe certificate <name> -n <ns>
  kubectl get secret <tls-secret> -n <ns> -o yaml

Step 2 — Check file layer (kubelet-synced tmpfs)
  kubectl exec -it <pod> -- cat /etc/certs/tls.crt

Step 3 — Check what LIVE PROCESS is actually serving
  openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
    | openssl x509 -noout -dates
  Old expiry = process loaded cert at startup, never reloaded.
  This bypasses Kubernetes entirely. Use this always.

Step 4 — Check cert-manager controller
  kubectl logs -n cert-manager deploy/cert-manager | grep -E 'ERROR|certificate'

Step 5 — Validate RBAC
  kubectl auth can-i create secrets \
    --as=system:serviceaccount:cert-manager:cert-manager \
    --all-namespaces
  Returns no = RBAC broken. Found root cause.

Trace path:
  desired resource → controller action → Secret update → workload consumption

---

PROCEDURE 3 — Pods Running and Ready But Traffic Failing

NEVER start with ingress-nginx logs.
Pods showing Ready does not mean traffic is flowing.

Step 1 — Endpoints (always first)
  kubectl get endpoints <svc> -n <ns>
  Empty = label selector mismatch, wrong port, namespace issue.
  kubectl get pods -n <ns> --show-labels
  kubectl get svc <svc> -n <ns> -o yaml | grep selector

Step 2 — DNS resolution inside cluster
  kubectl exec -it <pod> -- \
    nslookup <svc>.<ns>.svc.cluster.local
  Failure here = CoreDNS problem.

Step 3 — NetworkPolicy (silent drops)
  kubectl get networkpolicy -n <ns>
  kubectl describe networkpolicy <name> -n <ns>
  NetworkPolicy drops packets with zero error in application logs.
  GitOps can accidentally strip ingress rules.

Step 4 — Direct connectivity test
  kubectl exec -it <pod> -- curl -v http://<svc>:<port>/healthz
  TCP reset = port or firewall issue.
  Timeout = packet dropping, CNI or NetworkPolicy.

Step 5 — Ingress (only after ruling out above)
  kubectl logs -n ingress-nginx <pod> | grep -E '504|502|499|reset'
  502 = upstream pod crashed.
  503 = no healthy endpoints.
  504 = upstream too slow.

Step 6 — Infrastructure layer
  Check cloud load balancer health checks

Status code reference:
  499 → client timeout
  502 → upstream crashed
  503 → no endpoints
  504 → upstream slow
  TCP Reset → wrong port / NetworkPolicy / firewall

---

PROCEDURE 4 — Post-Rollout Latency Spike

No crashes. No OOMKill. No failed pods. Just slow.
Requires TWO timelines simultaneously.

Step 1 — Correlate timelines
  Timeline A: kubectl get events -n <ns> --sort-by='.lastTimestamp'
  Timeline B: Prometheus/metrics p99 latency graph
  If aligned = rollout caused it.
  If latency spiked before rollout completed = new code is the problem.

Step 2 — Decision rule
  Latency improving over time = cold start.
    Fix: readiness probe at /readyz that validates cache warmth.
    Not /healthz which only checks process is alive.

  Latency stable and high = code regression.
    Fix: kubectl rollout undo deployment/<name> -n <ns>
    Investigate new version offline.

Step 3 — Five causes in order of likelihood
  1. Cold start — JVM/cache/connection pool not initialized
  2. Reduced capacity — maxUnavailable:1 during rollout
  3. Code regression — new version has performance bug
  4. Downstream saturation — database/cache throttling
  5. Connection draining race — preStop hook missing

Step 4 — Prevent connection drops on every rollout
  lifecycle:
    preStop:
      exec:
        command: ["sleep", "15"]
  Gives kube-proxy time to drain connections before SIGTERM.

Rollout safety config for live services:
  maxSurge: 1        → ceiling above desired — add first
  maxUnavailable: 0  → floor below desired — never reduce capacity

---

PROCEDURE 5 — RBAC Permission Decay

Signal: controller returns 403 Forbidden from API server.
Controller is ALIVE and working — it is being BLOCKED.
This is not a connectivity issue.

Step 1 — Fastest confirmation
  kubectl auth can-i create secrets \
    --as=system:serviceaccount:cert-manager:cert-manager \
    --all-namespaces
  yes = RBAC fine, look elsewhere.
  no  = RBAC broken, found root cause.

Step 2 — Diff working cluster vs broken cluster
  kubectl get clusterrolebinding <name> \
    -o yaml --context=us-cluster > /tmp/us-crb.yaml
  kubectl get clusterrolebinding <name> \
    -o yaml --context=eu-cluster > /tmp/eu-crb.yaml
  diff /tmp/us-crb.yaml /tmp/eu-crb.yaml

Step 3 — Three hypotheses
  A. ClusterRole/ClusterRoleBinding modified or deleted
  B. Scope changed from ClusterRoleBinding to RoleBinding
  C. ServiceAccount recreated — binding points to wrong subject

Step 4 — Immediate remediation (P1)
  kubectl apply -f <backup-rbac-config>
  kubectl auth can-i create secrets --as=<sa> --all-namespaces
  kubectl rollout restart deploy/<controller> -n <ns>
  kubectl get certificate -n <ns> -w

Step 5 — Systemic prevention
  A. Protect RBAC resources in GitOps with Prune=false
  B. CronJob every 15 minutes running kubectl auth can-i validation
  C. Pre-sync hook that blocks pipeline if RBAC check fails

---

PROCEDURE 6 — Liveness vs Readiness Probe Issues

readinessProbe  → gates traffic AND rolling update progression
                  pod STAYS ALIVE if failing
                  removed from endpoints
                  frozen rollout = readinessProbe never passing

livenessProbe   → gates pod SURVIVAL
                  pod gets KILLED and restarted if failing
                  high restart count = livenessProbe too aggressive

Identify from evidence alone — no commands needed:
  Frozen rollout + pods Running + 0 restarts = readinessProbe
  High restart count + exit code 1 + Running = livenessProbe

Step 1 — Check which container is failing
  kubectl describe pod <pod> -n <ns>
  Read Containers section — which container shows Ready: False?
  Check Events section — exact probe failure message.

Step 2 — Application vs platform containers
  1/2 Ready = one container passing, one failing.
  Platform-injected sidecars fail independently
  of the application container.
  Isolate the failing container before troubleshooting.

Step 3 — Fix probe timing
  Replace initialDelaySeconds with startupProbe:
  startupProbe:
    httpGet:
      path: /healthz/ready
      port: 8080
    failureThreshold: 30
    periodSeconds: 2
  Polls every 2s up to 60s. Passes the moment container is ready.
  No unnecessary fixed wait on every rollout.

Step 4 — Fix livenessProbe aggression
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8080
    timeoutSeconds: 5    # was 1 — give app time to respond
    periodSeconds: 10    # was 5 — less frequent
    failureThreshold: 3

CRITICAL: /healthz must NEVER query external dependencies.
  Wrong: /healthz checks database connectivity
  Wrong: /healthz checks Redis connection
  Right: /healthz returns 200 if process is alive — nothing else
  External dependency checks belong in /readyz only.

Step 5 — Immediate remediation for frozen rollout
  kubectl rollout undo deployment/<name> -n <ns>
  kubectl rollout status deployment/<name> -n <ns>

---

PROCEDURE 7 — Control Plane Component Failures

CoreDNS down
  Signal: intermittent service-to-service failures, no pod errors
  Check: kubectl get pods -n kube-system | grep coredns
         kubectl logs -n kube-system deploy/coredns
         kubectl exec -it <pod> -- nslookup <svc>.<ns>.svc.cluster.local
  Metric: coredns_dns_request_duration_seconds p99 > 100ms

CNI issue (Cilium)
  Signal: pods stuck in ContainerCreating, "failed to assign IP"
  Check: kubectl describe pod <stuck-pod> | grep -A10 Events
  Check: kubectl get pods -n kube-system | grep cilium
  Metric: cilium_endpoint_creation_errors

kube-proxy stale rules (or Cilium networking issues)
  Signal: new services unreachable from specific nodes only
  Check: kubectl logs -n kube-system <kube-proxy-or-cilium-pod>
  Metric: kubeproxy_sync_proxy_rules_duration_seconds spike

Controller reconciliation loop stuck
  Signal: RBAC errors or Watch stream failures
  Check: kubectl logs -n <ns> <controller-pod>
  Look for: "forbidden", "Watch", "timeout"

Longhorn (storage) issues
  Signal: PVC stuck Pending, pods can't mount volumes
  Check: kubectl get pvc -A
  Check: kubectl describe pvc <name> -n <ns>
  Check: kubectl get longhorn-nodes -n longhorn-system
  Metric: longhorn_disk_capacity / longhorn_disk_reservation

MinIO (object storage) issues
  Signal: Loki unable to write logs, pods crash
  Check: kubectl logs -n storage deploy/minio
  Check: kubectl exec -it <minio-pod> -- mc ls storage/
  Verify: site replication status between az-a and az-b

---

PROCEDURE 8 — Certificate Expiry Incidents

Signal: TLS handshake failures or "certificate expired" errors
This should NEVER happen — cert-manager automates renewal 30 days early.

Step 1 — Check cert-manager is running
  kubectl get pods -n cert-manager
  kubectl logs -n cert-manager deploy/cert-manager | grep ERROR

Step 2 — Check Certificate resources
  kubectl get certificate -A
  kubectl describe certificate <name> -n <ns>
  Look for: Ready: False, "renewal" in status

Step 3 — Check Secret exists and contains cert
  kubectl get secret <tls-secret> -n <ns> -o yaml | grep tls.crt
  Decode and verify expiry:
  kubectl get secret <tls-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -dates

Step 4 — Check live process cert (most important)
  openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
    | openssl x509 -noout -dates
  If old expiry here = process loaded cert at startup, never reloaded.
  Restart pod: kubectl rollout restart deploy/<name> -n <ns>

Step 5 — Check RBAC for cert-manager
  kubectl auth can-i create secrets \
    --as=system:serviceaccount:cert-manager:cert-manager \
    --all-namespaces

Step 6 — Emergency remediation (if cert truly expired)
  kubectl rollout restart deploy/cert-manager -n cert-manager
  kubectl delete certificate <name> -n <ns>
  kubectl apply -f <certificate-yaml>
  kubectl rollout restart deploy/<dependent-app> -n <ns>

---

PROCEDURE 9 — Vault / Authentik IAM Issues

Signal: Services can't authenticate, OIDC login fails, Vault sealed

Step 1 — Check Vault status
  kubectl get pods -n iam | grep vault
  kubectl logs -n iam deploy/vault
  kubectl exec -it <vault-pod> -n iam -- vault status

Step 2 — Check if Vault is sealed
  kubectl exec -it <vault-pod> -n iam -- vault status | grep Sealed
  If Sealed: true → requires unseal keys (see bootstrap docs)

Step 3 — Check Authentik
  kubectl get pods -n iam | grep authentik
  kubectl logs -n iam deploy/authentik-server
  kubectl describe statefulset authentik-postgresql -n iam

Step 4 — Check secret in Vault
  kubectl exec -it <vault-pod> -n iam -- \
    vault kv get cluster/VARIABLE_NAME
  Not found = secret never created (run setup_vault.sh)

Step 5 — Check OIDC provisioning
  kubectl logs -n iam job/oidc-provisioning
  If errors = run provision_oidc.py again

Step 6 — Verify Loki/MinIO can connect to Vault
  Check: helmfile values reference correct Vault endpoint
  Check: ServiceAccount token mounted and RBAC permitting auth

---

PROCEDURE 10 — Loki / Logging Pipeline Stalled

Signal: No logs appearing in Grafana, Loki backend growing without limit

Step 1 — Check Promtail is scraping
  kubectl get pods -n logging | grep promtail
  kubectl logs -n logging ds/promtail | grep -E 'scraping|error'
  Should show: scraping every few seconds from all nodes

Step 2 — Check Loki can write to MinIO
  kubectl logs -n logging deploy/loki
  Look for: S3 errors, "connection refused", "write: no space"

Step 3 — Check MinIO is operational
  kubectl get pods -n storage | grep minio
  kubectl logs -n storage deploy/minio
  Check site replication status:
  kubectl get job -n storage | grep replication

Step 4 — Check disk space
  kubectl top pod -n logging
  kubectl exec -it <loki-pod> -n logging -- df -h
  If full = minio-backed storage exhausted, purge old chunks

Step 5 — Check PVC for Loki index
  kubectl get pvc -n logging
  kubectl describe pvc <loki-pvc> -n logging
  Bound to Longhorn PV = check node storage

Step 6 — Restart Loki
  kubectl rollout restart deploy/loki -n logging
  kubectl get pods -n logging -w

---

Quick Command Reference

# Pod state and debugging
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
kubectl exec -it <pod> -n <ns> -- /bin/sh

# Resource usage
kubectl top pod <pod> -n <ns>
kubectl top node
kubectl get pvc -A

# Networking and service discovery
kubectl get endpoints <svc> -n <ns>
kubectl get svc <svc> -n <ns> -o yaml
kubectl get pods -n <ns> --show-labels
kubectl get networkpolicy -n <ns>
kubectl exec -it <pod> -n <ns> -- nslookup <svc>.<ns>.svc.cluster.local
kubectl exec -it <pod> -n <ns> -- curl -v http://<svc>:<port>/healthz

# TLS inspection (live process — bypasses Kubernetes)
openssl s_client -connect <ip>:<port> </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

# RBAC validation
kubectl auth can-i <verb> <resource> \
  --as=system:serviceaccount:<ns>:<sa> --all-namespaces

# Rollout management
kubectl rollout status deployment/<name> -n <ns>
kubectl rollout undo deployment/<name> -n <ns>
kubectl rollout restart deploy/<name> -n <ns>

# Certificates and TLS
kubectl get certificate -n <ns>
kubectl describe certificate <name> -n <ns>
kubectl get secret <tls-secret> -n <ns> -o yaml
kubectl logs -n cert-manager deploy/cert-manager | grep ERROR

# Control plane components
kubectl get pods -n kube-system
kubectl get pods -n cert-manager
kubectl logs -n kube-system deploy/coredns
kubectl describe node <node-name>

# Storage
kubectl get pvc -A
kubectl get longhorn-nodes -n longhorn-system
kubectl exec -it <longhorn-pod> -n longhorn-system -- longhorn node ls

# IAM and secrets
kubectl get pods -n iam
kubectl logs -n iam deploy/vault
kubectl logs -n iam deploy/authentik-server
kubectl exec -it <vault-pod> -n iam -- vault status
kubectl exec -it <vault-pod> -n iam -- vault kv get cluster/<KEY>

# Using the k alias (add to ~/.zshrc)
k get pods -A
k logs -n logging <pod>
k describe node talos-worker-1

---

Agent Behaviour Rules

1. Never skip the three pre-flight questions
2. Never check logs before establishing exit code
3. Never check ingress before checking endpoints
4. Never raise memory limit before understanding growth pattern
5. Never assume etcd talks directly to controllers
6. Never conflate Kubernetes state with runtime process state
7. Always use --previous for crash logs
8. Always correlate two timelines for post-rollout issues
9. Always diff working cluster vs broken cluster for RBAC issues
10. Always confirm fix with kubectl auth can-i before closing incident
11. Always check live process cert with openssl s_client, not just Kubernetes state
12. Always verify Vault is unsealed and accessible before troubleshooting auth issues
13. Always check control plane components in kube-system before application logs
14. Always rule out networking (endpoints, DNS, NetworkPolicy) before app errors

---

Prevention & Observability

Set up proactive alerts:

1. RBAC validation (every 15 minutes)
   for each ServiceAccount in each namespace:
     kubectl auth can-i create secrets --as=<sa> --all-namespaces
   Alert if any returns "no"

2. Certificate expiry monitoring
   certmanager_certificate_expiration_seconds
   Alert 15 days before expiry

3. Controller forbidden errors (zero tolerance)
   apiserver_request_total{code="403", user=~"system:serviceaccount:.*"}
   Alert on > 0

4. Watch stream collapse
   apiserver_request_total{verb="LIST"} spike
   Alert on 3x baseline within 5 minutes

5. Probe failures
   rate(kubelet_started_pods_total{result="failed"}[5m])
   Alert on > 0.1 per 5 minutes

6. Storage exhaustion
   kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
   Alert at 80% capacity

7. Control plane latency
   apiserver_request_duration_seconds_sum / apiserver_request_duration_seconds_count
   Alert when p99 > 1 second

8. Logging pipeline lag
   loki_logql_querieslatency_seconds
   Alert when > 5 seconds