Story Crater Bot
5b00616f41
- README: cluster architecture, quick start, use cases - USAGE: stack topology, custom CLI reference - TROUBLESHOOTING: operational safety rules - .env.example: configuration template - Makefile: build shortcuts
631 lines
20 KiB
Markdown
631 lines
20 KiB
Markdown
|
|
|
|
# TROUBLESHOOTING.md
|
|
|
|
SRE Agent Troubleshooting Guide — Kubernetes / Homelab Cluster Production Incidents
|
|
|
|
You are an SRE agent responding to production incidents on this Kubernetes cluster. You follow a strict diagnostic methodology before taking any action. You never jump to tools before establishing the failure boundary. You never guess. You reason from evidence.
|
|
|
|
---
|
|
|
|
## RULE 0 — PRE-FLIGHT BEFORE EVERY INCIDENT
|
|
|
|
Before running any command, answer these three questions out loud:
|
|
|
|
```
|
|
1. WHERE is the failure boundary?
|
|
Client / Network / Pod / Controller / Infrastructure?
|
|
|
|
2. ALL traffic or SOME traffic?
|
|
Complete outage = systemic. Intermittent = partial failure.
|
|
This changes everything about where you look.
|
|
|
|
3. WHAT changed recently?
|
|
Deploy / Config / Certificate renewal / Traffic spike / GitOps pipeline?
|
|
Correlate with metrics timeline before acting.
|
|
```
|
|
|
|
If you cannot answer all three — gather more information before proceeding.
|
|
|
|
---
|
|
|
|
## RULE 1 — NEVER TALK TO ETCD DIRECTLY
|
|
|
|
Nothing in your runbook should ever reference etcd directly.
|
|
All state reads and writes go through the API server.
|
|
Controllers use informer cache via Watch streams — not polling.
|
|
|
|
---
|
|
|
|
## RULE 2 — LAYER BEFORE TOOL
|
|
|
|
Always identify which layer is broken before selecting a tool.
|
|
|
|
```
|
|
Layer 1 — Control Plane
|
|
API Server / etcd / Controllers / CRDs
|
|
Broken when: reconciliation loops fail, RBAC denied,
|
|
controller crashes, Watch stream collapses
|
|
|
|
Layer 2 — Kubelet
|
|
Pod lifecycle / cgroups / tmpfs mounts / probes
|
|
Broken when: OOMKilled, CrashLoopBackOff,
|
|
probe misconfiguration, Secret not mounted
|
|
|
|
Layer 3 — Networking
|
|
CoreDNS / kube-proxy / CNI / Ingress / Load Balancer
|
|
Broken when: pods green but traffic failing,
|
|
DNS timeouts, empty endpoints,
|
|
NetworkPolicy drops, IP exhaustion
|
|
```
|
|
|
|
---
|
|
|
|
## RULE 3 — NEVER DELETE A PVC WITHOUT REPLICATION
|
|
|
|
🔴 **A deleted PVC = permanent data loss.** Never delete a PVC unless you have verified replicas or backups exist.
|
|
|
|
**Before ANY PVC deletion:**
|
|
|
|
```bash
|
|
# 1. Check volume replication status
|
|
kubectl get pvc -n <ns> <pvc>
|
|
kubectl get pv <pv-name> -o json | jq '.spec'
|
|
|
|
# 2. For Longhorn volumes (storage)
|
|
kubectl get longhorn-volume -n longhorn-system -o wide
|
|
# Must show: State=healthy, Replicas >= 2
|
|
|
|
# 3. For databases (PostgreSQL)
|
|
kubectl exec -n ddb pod/ddb-cluster-0 -- \
|
|
psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
|
|
# Must show: at least 1 streaming replica
|
|
|
|
# 4. For backup buckets (MinIO)
|
|
# Verify backup was taken in last 24 hours
|
|
# kubectl exec -n storage pod/minio-0 -- mc ls local/postgresql-backups/
|
|
```
|
|
|
|
**If replication is not confirmed:** STOP. Do not proceed. Escalate to SRE lead.
|
|
|
|
---
|
|
|
|
## PROCEDURE 1 — CrashLoopBackOff
|
|
|
|
```
|
|
NEVER start with kubectl logs.
|
|
|
|
Step 1 — Establish crash type
|
|
kubectl describe pod <pod> -n <ns>
|
|
Read: Last State → Reason → Exit Code
|
|
|
|
Step 2 — Exit code triage
|
|
0 → Clean exit. Check livenessProbe config.
|
|
1 → App error. Now run: kubectl logs <pod> -n <ns> --previous
|
|
137 → OOMKilled. Kernel cgroup enforced memory limit.
|
|
Run: kubectl top pod + kubectl top node
|
|
139 → Segfault. Check binary and dependencies.
|
|
|
|
Step 3 — If OOMKilled (137)
|
|
Answer before raising limit:
|
|
A. Sawtooth memory pattern = load spike. Raise limit with headroom.
|
|
B. Monotonic growth = memory leak. Fix the code first.
|
|
C. kubectl describe node → MemoryPressure: True = noisy neighbor.
|
|
Move pod, don't raise limit.
|
|
|
|
Step 4 — Namespace events
|
|
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
|
|
|
|
CRITICAL: Always use --previous for crash logs.
|
|
Without it you get logs from current instance
|
|
which may have lived for 3 seconds.
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 2 — TLS Handshake Failures
|
|
|
|
```
|
|
Certificate Ready: True does not mean traffic is working.
|
|
Kubernetes state layer ≠ runtime process layer.
|
|
|
|
Step 1 — Check Kubernetes state
|
|
kubectl get certificate -n <ns>
|
|
kubectl describe certificate <name> -n <ns>
|
|
kubectl get secret <tls-secret> -n <ns> -o yaml
|
|
|
|
Step 2 — Check file layer (kubelet-synced tmpfs)
|
|
kubectl exec -it <pod> -- cat /etc/certs/tls.crt
|
|
|
|
Step 3 — Check what LIVE PROCESS is actually serving
|
|
openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
|
|
| openssl x509 -noout -dates
|
|
Old expiry = process loaded cert at startup, never reloaded.
|
|
This bypasses Kubernetes entirely. Use this always.
|
|
|
|
Step 4 — Check cert-manager controller
|
|
kubectl logs -n cert-manager deploy/cert-manager | grep -E 'ERROR|certificate'
|
|
|
|
Step 5 — Validate RBAC
|
|
kubectl auth can-i create secrets \
|
|
--as=system:serviceaccount:cert-manager:cert-manager \
|
|
--all-namespaces
|
|
Returns no = RBAC broken. Found root cause.
|
|
|
|
Trace path:
|
|
desired resource → controller action → Secret update → workload consumption
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 3 — Pods Running and Ready But Traffic Failing
|
|
|
|
```
|
|
NEVER start with ingress-nginx logs.
|
|
Pods showing Ready does not mean traffic is flowing.
|
|
|
|
Step 1 — Endpoints (always first)
|
|
kubectl get endpoints <svc> -n <ns>
|
|
Empty = label selector mismatch, wrong port, namespace issue.
|
|
kubectl get pods -n <ns> --show-labels
|
|
kubectl get svc <svc> -n <ns> -o yaml | grep selector
|
|
|
|
Step 2 — DNS resolution inside cluster
|
|
kubectl exec -it <pod> -- \
|
|
nslookup <svc>.<ns>.svc.cluster.local
|
|
Failure here = CoreDNS problem.
|
|
|
|
Step 3 — NetworkPolicy (silent drops)
|
|
kubectl get networkpolicy -n <ns>
|
|
kubectl describe networkpolicy <name> -n <ns>
|
|
NetworkPolicy drops packets with zero error in application logs.
|
|
GitOps can accidentally strip ingress rules.
|
|
|
|
Step 4 — Direct connectivity test
|
|
kubectl exec -it <pod> -- curl -v http://<svc>:<port>/healthz
|
|
TCP reset = port or firewall issue.
|
|
Timeout = packet dropping, CNI or NetworkPolicy.
|
|
|
|
Step 5 — Ingress (only after ruling out above)
|
|
kubectl logs -n ingress-nginx <pod> | grep -E '504|502|499|reset'
|
|
502 = upstream pod crashed.
|
|
503 = no healthy endpoints.
|
|
504 = upstream too slow.
|
|
|
|
Step 6 — Infrastructure layer
|
|
Check cloud load balancer health checks
|
|
|
|
Status code reference:
|
|
499 → client timeout
|
|
502 → upstream crashed
|
|
503 → no endpoints
|
|
504 → upstream slow
|
|
TCP Reset → wrong port / NetworkPolicy / firewall
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 4 — Post-Rollout Latency Spike
|
|
|
|
```
|
|
No crashes. No OOMKill. No failed pods. Just slow.
|
|
Requires TWO timelines simultaneously.
|
|
|
|
Step 1 — Correlate timelines
|
|
Timeline A: kubectl get events -n <ns> --sort-by='.lastTimestamp'
|
|
Timeline B: Prometheus/metrics p99 latency graph
|
|
If aligned = rollout caused it.
|
|
If latency spiked before rollout completed = new code is the problem.
|
|
|
|
Step 2 — Decision rule
|
|
Latency improving over time = cold start.
|
|
Fix: readiness probe at /readyz that validates cache warmth.
|
|
Not /healthz which only checks process is alive.
|
|
|
|
Latency stable and high = code regression.
|
|
Fix: kubectl rollout undo deployment/<name> -n <ns>
|
|
Investigate new version offline.
|
|
|
|
Step 3 — Five causes in order of likelihood
|
|
1. Cold start — JVM/cache/connection pool not initialized
|
|
2. Reduced capacity — maxUnavailable:1 during rollout
|
|
3. Code regression — new version has performance bug
|
|
4. Downstream saturation — database/cache throttling
|
|
5. Connection draining race — preStop hook missing
|
|
|
|
Step 4 — Prevent connection drops on every rollout
|
|
lifecycle:
|
|
preStop:
|
|
exec:
|
|
command: ["sleep", "15"]
|
|
Gives kube-proxy time to drain connections before SIGTERM.
|
|
|
|
Rollout safety config for live services:
|
|
maxSurge: 1 → ceiling above desired — add first
|
|
maxUnavailable: 0 → floor below desired — never reduce capacity
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 5 — RBAC Permission Decay
|
|
|
|
```
|
|
Signal: controller returns 403 Forbidden from API server.
|
|
Controller is ALIVE and working — it is being BLOCKED.
|
|
This is not a connectivity issue.
|
|
|
|
Step 1 — Fastest confirmation
|
|
kubectl auth can-i create secrets \
|
|
--as=system:serviceaccount:cert-manager:cert-manager \
|
|
--all-namespaces
|
|
yes = RBAC fine, look elsewhere.
|
|
no = RBAC broken, found root cause.
|
|
|
|
Step 2 — Diff working cluster vs broken cluster
|
|
kubectl get clusterrolebinding <name> \
|
|
-o yaml --context=us-cluster > /tmp/us-crb.yaml
|
|
kubectl get clusterrolebinding <name> \
|
|
-o yaml --context=eu-cluster > /tmp/eu-crb.yaml
|
|
diff /tmp/us-crb.yaml /tmp/eu-crb.yaml
|
|
|
|
Step 3 — Three hypotheses
|
|
A. ClusterRole/ClusterRoleBinding modified or deleted
|
|
B. Scope changed from ClusterRoleBinding to RoleBinding
|
|
C. ServiceAccount recreated — binding points to wrong subject
|
|
|
|
Step 4 — Immediate remediation (P1)
|
|
kubectl apply -f <backup-rbac-config>
|
|
kubectl auth can-i create secrets --as=<sa> --all-namespaces
|
|
kubectl rollout restart deploy/<controller> -n <ns>
|
|
kubectl get certificate -n <ns> -w
|
|
|
|
Step 5 — Systemic prevention
|
|
A. Protect RBAC resources in GitOps with Prune=false
|
|
B. CronJob every 15 minutes running kubectl auth can-i validation
|
|
C. Pre-sync hook that blocks pipeline if RBAC check fails
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 6 — Liveness vs Readiness Probe Issues
|
|
|
|
```
|
|
readinessProbe → gates traffic AND rolling update progression
|
|
pod STAYS ALIVE if failing
|
|
removed from endpoints
|
|
frozen rollout = readinessProbe never passing
|
|
|
|
livenessProbe → gates pod SURVIVAL
|
|
pod gets KILLED and restarted if failing
|
|
high restart count = livenessProbe too aggressive
|
|
|
|
Identify from evidence alone — no commands needed:
|
|
Frozen rollout + pods Running + 0 restarts = readinessProbe
|
|
High restart count + exit code 1 + Running = livenessProbe
|
|
|
|
Step 1 — Check which container is failing
|
|
kubectl describe pod <pod> -n <ns>
|
|
Read Containers section — which container shows Ready: False?
|
|
Check Events section — exact probe failure message.
|
|
|
|
Step 2 — Application vs platform containers
|
|
1/2 Ready = one container passing, one failing.
|
|
Platform-injected sidecars fail independently
|
|
of the application container.
|
|
Isolate the failing container before troubleshooting.
|
|
|
|
Step 3 — Fix probe timing
|
|
Replace initialDelaySeconds with startupProbe:
|
|
startupProbe:
|
|
httpGet:
|
|
path: /healthz/ready
|
|
port: 8080
|
|
failureThreshold: 30
|
|
periodSeconds: 2
|
|
Polls every 2s up to 60s. Passes the moment container is ready.
|
|
No unnecessary fixed wait on every rollout.
|
|
|
|
Step 4 — Fix livenessProbe aggression
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /healthz
|
|
port: 8080
|
|
timeoutSeconds: 5 # was 1 — give app time to respond
|
|
periodSeconds: 10 # was 5 — less frequent
|
|
failureThreshold: 3
|
|
|
|
CRITICAL: /healthz must NEVER query external dependencies.
|
|
Wrong: /healthz checks database connectivity
|
|
Wrong: /healthz checks Redis connection
|
|
Right: /healthz returns 200 if process is alive — nothing else
|
|
External dependency checks belong in /readyz only.
|
|
|
|
Step 5 — Immediate remediation for frozen rollout
|
|
kubectl rollout undo deployment/<name> -n <ns>
|
|
kubectl rollout status deployment/<name> -n <ns>
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 7 — Control Plane Component Failures
|
|
|
|
```
|
|
CoreDNS down
|
|
Signal: intermittent service-to-service failures, no pod errors
|
|
Check: kubectl get pods -n kube-system | grep coredns
|
|
kubectl logs -n kube-system deploy/coredns
|
|
kubectl exec -it <pod> -- nslookup <svc>.<ns>.svc.cluster.local
|
|
Metric: coredns_dns_request_duration_seconds p99 > 100ms
|
|
|
|
CNI issue (Cilium)
|
|
Signal: pods stuck in ContainerCreating, "failed to assign IP"
|
|
Check: kubectl describe pod <stuck-pod> | grep -A10 Events
|
|
Check: kubectl get pods -n kube-system | grep cilium
|
|
Metric: cilium_endpoint_creation_errors
|
|
|
|
kube-proxy stale rules (or Cilium networking issues)
|
|
Signal: new services unreachable from specific nodes only
|
|
Check: kubectl logs -n kube-system <kube-proxy-or-cilium-pod>
|
|
Metric: kubeproxy_sync_proxy_rules_duration_seconds spike
|
|
|
|
Controller reconciliation loop stuck
|
|
Signal: RBAC errors or Watch stream failures
|
|
Check: kubectl logs -n <ns> <controller-pod>
|
|
Look for: "forbidden", "Watch", "timeout"
|
|
|
|
Longhorn (storage) issues
|
|
Signal: PVC stuck Pending, pods can't mount volumes
|
|
Check: kubectl get pvc -A
|
|
Check: kubectl describe pvc <name> -n <ns>
|
|
Check: kubectl get longhorn-nodes -n longhorn-system
|
|
Metric: longhorn_disk_capacity / longhorn_disk_reservation
|
|
|
|
MinIO (object storage) issues
|
|
Signal: Loki unable to write logs, pods crash
|
|
Check: kubectl logs -n storage deploy/minio
|
|
Check: kubectl exec -it <minio-pod> -- mc ls storage/
|
|
Verify: site replication status between az-a and az-b
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 8 — Certificate Expiry Incidents
|
|
|
|
```
|
|
Signal: TLS handshake failures or "certificate expired" errors
|
|
This should NEVER happen — cert-manager automates renewal 30 days early.
|
|
|
|
Step 1 — Check cert-manager is running
|
|
kubectl get pods -n cert-manager
|
|
kubectl logs -n cert-manager deploy/cert-manager | grep ERROR
|
|
|
|
Step 2 — Check Certificate resources
|
|
kubectl get certificate -A
|
|
kubectl describe certificate <name> -n <ns>
|
|
Look for: Ready: False, "renewal" in status
|
|
|
|
Step 3 — Check Secret exists and contains cert
|
|
kubectl get secret <tls-secret> -n <ns> -o yaml | grep tls.crt
|
|
Decode and verify expiry:
|
|
kubectl get secret <tls-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' \
|
|
| base64 -d | openssl x509 -noout -dates
|
|
|
|
Step 4 — Check live process cert (most important)
|
|
openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
|
|
| openssl x509 -noout -dates
|
|
If old expiry here = process loaded cert at startup, never reloaded.
|
|
Restart pod: kubectl rollout restart deploy/<name> -n <ns>
|
|
|
|
Step 5 — Check RBAC for cert-manager
|
|
kubectl auth can-i create secrets \
|
|
--as=system:serviceaccount:cert-manager:cert-manager \
|
|
--all-namespaces
|
|
|
|
Step 6 — Emergency remediation (if cert truly expired)
|
|
kubectl rollout restart deploy/cert-manager -n cert-manager
|
|
kubectl delete certificate <name> -n <ns>
|
|
kubectl apply -f <certificate-yaml>
|
|
kubectl rollout restart deploy/<dependent-app> -n <ns>
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 9 — Vault / Authentik IAM Issues
|
|
|
|
```
|
|
Signal: Services can't authenticate, OIDC login fails, Vault sealed
|
|
|
|
Step 1 — Check Vault status
|
|
kubectl get pods -n iam | grep vault
|
|
kubectl logs -n iam deploy/vault
|
|
kubectl exec -it <vault-pod> -n iam -- vault status
|
|
|
|
Step 2 — Check if Vault is sealed
|
|
kubectl exec -it <vault-pod> -n iam -- vault status | grep Sealed
|
|
If Sealed: true → requires unseal keys (see bootstrap docs)
|
|
|
|
Step 3 — Check Authentik
|
|
kubectl get pods -n iam | grep authentik
|
|
kubectl logs -n iam deploy/authentik-server
|
|
kubectl describe statefulset authentik-postgresql -n iam
|
|
|
|
Step 4 — Check secret in Vault
|
|
kubectl exec -it <vault-pod> -n iam -- \
|
|
vault kv get cluster/VARIABLE_NAME
|
|
Not found = secret never created (run setup_vault.sh)
|
|
|
|
Step 5 — Check OIDC provisioning
|
|
kubectl logs -n iam job/oidc-provisioning
|
|
If errors = run provision_oidc.py again
|
|
|
|
Step 6 — Verify Loki/MinIO can connect to Vault
|
|
Check: helmfile values reference correct Vault endpoint
|
|
Check: ServiceAccount token mounted and RBAC permitting auth
|
|
```
|
|
|
|
---
|
|
|
|
## PROCEDURE 10 — Loki / Logging Pipeline Stalled
|
|
|
|
```
|
|
Signal: No logs appearing in Grafana, Loki backend growing without limit
|
|
|
|
Step 1 — Check Promtail is scraping
|
|
kubectl get pods -n logging | grep promtail
|
|
kubectl logs -n logging ds/promtail | grep -E 'scraping|error'
|
|
Should show: scraping every few seconds from all nodes
|
|
|
|
Step 2 — Check Loki can write to MinIO
|
|
kubectl logs -n logging deploy/loki
|
|
Look for: S3 errors, "connection refused", "write: no space"
|
|
|
|
Step 3 — Check MinIO is operational
|
|
kubectl get pods -n storage | grep minio
|
|
kubectl logs -n storage deploy/minio
|
|
Check site replication status:
|
|
kubectl get job -n storage | grep replication
|
|
|
|
Step 4 — Check disk space
|
|
kubectl top pod -n logging
|
|
kubectl exec -it <loki-pod> -n logging -- df -h
|
|
If full = minio-backed storage exhausted, purge old chunks
|
|
|
|
Step 5 — Check PVC for Loki index
|
|
kubectl get pvc -n logging
|
|
kubectl describe pvc <loki-pvc> -n logging
|
|
Bound to Longhorn PV = check node storage
|
|
|
|
Step 6 — Restart Loki
|
|
kubectl rollout restart deploy/loki -n logging
|
|
kubectl get pods -n logging -w
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Command Reference
|
|
|
|
```bash
|
|
# Pod state and debugging
|
|
kubectl describe pod <pod> -n <ns>
|
|
kubectl logs <pod> -n <ns> --previous
|
|
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
|
|
kubectl exec -it <pod> -n <ns> -- /bin/sh
|
|
|
|
# Resource usage
|
|
kubectl top pod <pod> -n <ns>
|
|
kubectl top node
|
|
kubectl get pvc -A
|
|
|
|
# Networking and service discovery
|
|
kubectl get endpoints <svc> -n <ns>
|
|
kubectl get svc <svc> -n <ns> -o yaml
|
|
kubectl get pods -n <ns> --show-labels
|
|
kubectl get networkpolicy -n <ns>
|
|
kubectl exec -it <pod> -n <ns> -- nslookup <svc>.<ns>.svc.cluster.local
|
|
kubectl exec -it <pod> -n <ns> -- curl -v http://<svc>:<port>/healthz
|
|
|
|
# TLS inspection (live process — bypasses Kubernetes)
|
|
openssl s_client -connect <ip>:<port> </dev/null 2>/dev/null \
|
|
| openssl x509 -noout -dates
|
|
|
|
# RBAC validation
|
|
kubectl auth can-i <verb> <resource> \
|
|
--as=system:serviceaccount:<ns>:<sa> --all-namespaces
|
|
|
|
# Rollout management
|
|
kubectl rollout status deployment/<name> -n <ns>
|
|
kubectl rollout undo deployment/<name> -n <ns>
|
|
kubectl rollout restart deploy/<name> -n <ns>
|
|
|
|
# Certificates and TLS
|
|
kubectl get certificate -n <ns>
|
|
kubectl describe certificate <name> -n <ns>
|
|
kubectl get secret <tls-secret> -n <ns> -o yaml
|
|
kubectl logs -n cert-manager deploy/cert-manager | grep ERROR
|
|
|
|
# Control plane components
|
|
kubectl get pods -n kube-system
|
|
kubectl get pods -n cert-manager
|
|
kubectl logs -n kube-system deploy/coredns
|
|
kubectl describe node <node-name>
|
|
|
|
# Storage
|
|
kubectl get pvc -A
|
|
kubectl get longhorn-nodes -n longhorn-system
|
|
kubectl exec -it <longhorn-pod> -n longhorn-system -- longhorn node ls
|
|
|
|
# IAM and secrets
|
|
kubectl get pods -n iam
|
|
kubectl logs -n iam deploy/vault
|
|
kubectl logs -n iam deploy/authentik-server
|
|
kubectl exec -it <vault-pod> -n iam -- vault status
|
|
kubectl exec -it <vault-pod> -n iam -- vault kv get cluster/<KEY>
|
|
|
|
# Using the k alias (add to ~/.zshrc)
|
|
k get pods -A
|
|
k logs -n logging <pod>
|
|
k describe node talos-worker-1
|
|
```
|
|
|
|
---
|
|
|
|
## Agent Behaviour Rules
|
|
|
|
```
|
|
1. Never skip the three pre-flight questions
|
|
2. Never check logs before establishing exit code
|
|
3. Never check ingress before checking endpoints
|
|
4. Never raise memory limit before understanding growth pattern
|
|
5. Never assume etcd talks directly to controllers
|
|
6. Never conflate Kubernetes state with runtime process state
|
|
7. Always use --previous for crash logs
|
|
8. Always correlate two timelines for post-rollout issues
|
|
9. Always diff working cluster vs broken cluster for RBAC issues
|
|
10. Always confirm fix with kubectl auth can-i before closing incident
|
|
11. Always check live process cert with openssl s_client, not just Kubernetes state
|
|
12. Always verify Vault is unsealed and accessible before troubleshooting auth issues
|
|
13. Always check control plane components in kube-system before application logs
|
|
14. Always rule out networking (endpoints, DNS, NetworkPolicy) before app errors
|
|
```
|
|
|
|
---
|
|
|
|
## Prevention & Observability
|
|
|
|
```
|
|
Set up proactive alerts:
|
|
|
|
1. RBAC validation (every 15 minutes)
|
|
for each ServiceAccount in each namespace:
|
|
kubectl auth can-i create secrets --as=<sa> --all-namespaces
|
|
Alert if any returns "no"
|
|
|
|
2. Certificate expiry monitoring
|
|
certmanager_certificate_expiration_seconds
|
|
Alert 15 days before expiry
|
|
|
|
3. Controller forbidden errors (zero tolerance)
|
|
apiserver_request_total{code="403", user=~"system:serviceaccount:.*"}
|
|
Alert on > 0
|
|
|
|
4. Watch stream collapse
|
|
apiserver_request_total{verb="LIST"} spike
|
|
Alert on 3x baseline within 5 minutes
|
|
|
|
5. Probe failures
|
|
rate(kubelet_started_pods_total{result="failed"}[5m])
|
|
Alert on > 0.1 per 5 minutes
|
|
|
|
6. Storage exhaustion
|
|
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
|
Alert at 80% capacity
|
|
|
|
7. Control plane latency
|
|
apiserver_request_duration_seconds_sum / apiserver_request_duration_seconds_count
|
|
Alert when p99 > 1 second
|
|
|
|
8. Logging pipeline lag
|
|
loki_logql_querieslatency_seconds
|
|
Alert when > 5 seconds
|
|
```
|