docs: add foundation docs and cluster configuration templates
- README: cluster architecture, quick start, use cases - USAGE: stack topology, custom CLI reference - TROUBLESHOOTING: operational safety rules - .env.example: configuration template - Makefile: build shortcuts
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,630 @@
|
||||
|
||||
|
||||
# TROUBLESHOOTING.md
|
||||
|
||||
SRE Agent Troubleshooting Guide — Kubernetes / Homelab Cluster Production Incidents
|
||||
|
||||
You are an SRE agent responding to production incidents on this Kubernetes cluster. You follow a strict diagnostic methodology before taking any action. You never jump to tools before establishing the failure boundary. You never guess. You reason from evidence.
|
||||
|
||||
---
|
||||
|
||||
## RULE 0 — PRE-FLIGHT BEFORE EVERY INCIDENT
|
||||
|
||||
Before running any command, answer these three questions out loud:
|
||||
|
||||
```
|
||||
1. WHERE is the failure boundary?
|
||||
Client / Network / Pod / Controller / Infrastructure?
|
||||
|
||||
2. ALL traffic or SOME traffic?
|
||||
Complete outage = systemic. Intermittent = partial failure.
|
||||
This changes everything about where you look.
|
||||
|
||||
3. WHAT changed recently?
|
||||
Deploy / Config / Certificate renewal / Traffic spike / GitOps pipeline?
|
||||
Correlate with metrics timeline before acting.
|
||||
```
|
||||
|
||||
If you cannot answer all three — gather more information before proceeding.
|
||||
|
||||
---
|
||||
|
||||
## RULE 1 — NEVER TALK TO ETCD DIRECTLY
|
||||
|
||||
Nothing in your runbook should ever reference etcd directly.
|
||||
All state reads and writes go through the API server.
|
||||
Controllers use informer cache via Watch streams — not polling.
|
||||
|
||||
---
|
||||
|
||||
## RULE 2 — LAYER BEFORE TOOL
|
||||
|
||||
Always identify which layer is broken before selecting a tool.
|
||||
|
||||
```
|
||||
Layer 1 — Control Plane
|
||||
API Server / etcd / Controllers / CRDs
|
||||
Broken when: reconciliation loops fail, RBAC denied,
|
||||
controller crashes, Watch stream collapses
|
||||
|
||||
Layer 2 — Kubelet
|
||||
Pod lifecycle / cgroups / tmpfs mounts / probes
|
||||
Broken when: OOMKilled, CrashLoopBackOff,
|
||||
probe misconfiguration, Secret not mounted
|
||||
|
||||
Layer 3 — Networking
|
||||
CoreDNS / kube-proxy / CNI / Ingress / Load Balancer
|
||||
Broken when: pods green but traffic failing,
|
||||
DNS timeouts, empty endpoints,
|
||||
NetworkPolicy drops, IP exhaustion
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## RULE 3 — NEVER DELETE A PVC WITHOUT REPLICATION
|
||||
|
||||
🔴 **A deleted PVC = permanent data loss.** Never delete a PVC unless you have verified replicas or backups exist.
|
||||
|
||||
**Before ANY PVC deletion:**
|
||||
|
||||
```bash
|
||||
# 1. Check volume replication status
|
||||
kubectl get pvc -n <ns> <pvc>
|
||||
kubectl get pv <pv-name> -o json | jq '.spec'
|
||||
|
||||
# 2. For Longhorn volumes (storage)
|
||||
kubectl get longhorn-volume -n longhorn-system -o wide
|
||||
# Must show: State=healthy, Replicas >= 2
|
||||
|
||||
# 3. For databases (PostgreSQL)
|
||||
kubectl exec -n ddb pod/ddb-cluster-0 -- \
|
||||
psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
|
||||
# Must show: at least 1 streaming replica
|
||||
|
||||
# 4. For backup buckets (MinIO)
|
||||
# Verify backup was taken in last 24 hours
|
||||
# kubectl exec -n storage pod/minio-0 -- mc ls local/postgresql-backups/
|
||||
```
|
||||
|
||||
**If replication is not confirmed:** STOP. Do not proceed. Escalate to SRE lead.
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 1 — CrashLoopBackOff
|
||||
|
||||
```
|
||||
NEVER start with kubectl logs.
|
||||
|
||||
Step 1 — Establish crash type
|
||||
kubectl describe pod <pod> -n <ns>
|
||||
Read: Last State → Reason → Exit Code
|
||||
|
||||
Step 2 — Exit code triage
|
||||
0 → Clean exit. Check livenessProbe config.
|
||||
1 → App error. Now run: kubectl logs <pod> -n <ns> --previous
|
||||
137 → OOMKilled. Kernel cgroup enforced memory limit.
|
||||
Run: kubectl top pod + kubectl top node
|
||||
139 → Segfault. Check binary and dependencies.
|
||||
|
||||
Step 3 — If OOMKilled (137)
|
||||
Answer before raising limit:
|
||||
A. Sawtooth memory pattern = load spike. Raise limit with headroom.
|
||||
B. Monotonic growth = memory leak. Fix the code first.
|
||||
C. kubectl describe node → MemoryPressure: True = noisy neighbor.
|
||||
Move pod, don't raise limit.
|
||||
|
||||
Step 4 — Namespace events
|
||||
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
|
||||
|
||||
CRITICAL: Always use --previous for crash logs.
|
||||
Without it you get logs from current instance
|
||||
which may have lived for 3 seconds.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 2 — TLS Handshake Failures
|
||||
|
||||
```
|
||||
Certificate Ready: True does not mean traffic is working.
|
||||
Kubernetes state layer ≠ runtime process layer.
|
||||
|
||||
Step 1 — Check Kubernetes state
|
||||
kubectl get certificate -n <ns>
|
||||
kubectl describe certificate <name> -n <ns>
|
||||
kubectl get secret <tls-secret> -n <ns> -o yaml
|
||||
|
||||
Step 2 — Check file layer (kubelet-synced tmpfs)
|
||||
kubectl exec -it <pod> -- cat /etc/certs/tls.crt
|
||||
|
||||
Step 3 — Check what LIVE PROCESS is actually serving
|
||||
openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -dates
|
||||
Old expiry = process loaded cert at startup, never reloaded.
|
||||
This bypasses Kubernetes entirely. Use this always.
|
||||
|
||||
Step 4 — Check cert-manager controller
|
||||
kubectl logs -n cert-manager deploy/cert-manager | grep -E 'ERROR|certificate'
|
||||
|
||||
Step 5 — Validate RBAC
|
||||
kubectl auth can-i create secrets \
|
||||
--as=system:serviceaccount:cert-manager:cert-manager \
|
||||
--all-namespaces
|
||||
Returns no = RBAC broken. Found root cause.
|
||||
|
||||
Trace path:
|
||||
desired resource → controller action → Secret update → workload consumption
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 3 — Pods Running and Ready But Traffic Failing
|
||||
|
||||
```
|
||||
NEVER start with ingress-nginx logs.
|
||||
Pods showing Ready does not mean traffic is flowing.
|
||||
|
||||
Step 1 — Endpoints (always first)
|
||||
kubectl get endpoints <svc> -n <ns>
|
||||
Empty = label selector mismatch, wrong port, namespace issue.
|
||||
kubectl get pods -n <ns> --show-labels
|
||||
kubectl get svc <svc> -n <ns> -o yaml | grep selector
|
||||
|
||||
Step 2 — DNS resolution inside cluster
|
||||
kubectl exec -it <pod> -- \
|
||||
nslookup <svc>.<ns>.svc.cluster.local
|
||||
Failure here = CoreDNS problem.
|
||||
|
||||
Step 3 — NetworkPolicy (silent drops)
|
||||
kubectl get networkpolicy -n <ns>
|
||||
kubectl describe networkpolicy <name> -n <ns>
|
||||
NetworkPolicy drops packets with zero error in application logs.
|
||||
GitOps can accidentally strip ingress rules.
|
||||
|
||||
Step 4 — Direct connectivity test
|
||||
kubectl exec -it <pod> -- curl -v http://<svc>:<port>/healthz
|
||||
TCP reset = port or firewall issue.
|
||||
Timeout = packet dropping, CNI or NetworkPolicy.
|
||||
|
||||
Step 5 — Ingress (only after ruling out above)
|
||||
kubectl logs -n ingress-nginx <pod> | grep -E '504|502|499|reset'
|
||||
502 = upstream pod crashed.
|
||||
503 = no healthy endpoints.
|
||||
504 = upstream too slow.
|
||||
|
||||
Step 6 — Infrastructure layer
|
||||
Check cloud load balancer health checks
|
||||
|
||||
Status code reference:
|
||||
499 → client timeout
|
||||
502 → upstream crashed
|
||||
503 → no endpoints
|
||||
504 → upstream slow
|
||||
TCP Reset → wrong port / NetworkPolicy / firewall
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 4 — Post-Rollout Latency Spike
|
||||
|
||||
```
|
||||
No crashes. No OOMKill. No failed pods. Just slow.
|
||||
Requires TWO timelines simultaneously.
|
||||
|
||||
Step 1 — Correlate timelines
|
||||
Timeline A: kubectl get events -n <ns> --sort-by='.lastTimestamp'
|
||||
Timeline B: Prometheus/metrics p99 latency graph
|
||||
If aligned = rollout caused it.
|
||||
If latency spiked before rollout completed = new code is the problem.
|
||||
|
||||
Step 2 — Decision rule
|
||||
Latency improving over time = cold start.
|
||||
Fix: readiness probe at /readyz that validates cache warmth.
|
||||
Not /healthz which only checks process is alive.
|
||||
|
||||
Latency stable and high = code regression.
|
||||
Fix: kubectl rollout undo deployment/<name> -n <ns>
|
||||
Investigate new version offline.
|
||||
|
||||
Step 3 — Five causes in order of likelihood
|
||||
1. Cold start — JVM/cache/connection pool not initialized
|
||||
2. Reduced capacity — maxUnavailable:1 during rollout
|
||||
3. Code regression — new version has performance bug
|
||||
4. Downstream saturation — database/cache throttling
|
||||
5. Connection draining race — preStop hook missing
|
||||
|
||||
Step 4 — Prevent connection drops on every rollout
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["sleep", "15"]
|
||||
Gives kube-proxy time to drain connections before SIGTERM.
|
||||
|
||||
Rollout safety config for live services:
|
||||
maxSurge: 1 → ceiling above desired — add first
|
||||
maxUnavailable: 0 → floor below desired — never reduce capacity
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 5 — RBAC Permission Decay
|
||||
|
||||
```
|
||||
Signal: controller returns 403 Forbidden from API server.
|
||||
Controller is ALIVE and working — it is being BLOCKED.
|
||||
This is not a connectivity issue.
|
||||
|
||||
Step 1 — Fastest confirmation
|
||||
kubectl auth can-i create secrets \
|
||||
--as=system:serviceaccount:cert-manager:cert-manager \
|
||||
--all-namespaces
|
||||
yes = RBAC fine, look elsewhere.
|
||||
no = RBAC broken, found root cause.
|
||||
|
||||
Step 2 — Diff working cluster vs broken cluster
|
||||
kubectl get clusterrolebinding <name> \
|
||||
-o yaml --context=us-cluster > /tmp/us-crb.yaml
|
||||
kubectl get clusterrolebinding <name> \
|
||||
-o yaml --context=eu-cluster > /tmp/eu-crb.yaml
|
||||
diff /tmp/us-crb.yaml /tmp/eu-crb.yaml
|
||||
|
||||
Step 3 — Three hypotheses
|
||||
A. ClusterRole/ClusterRoleBinding modified or deleted
|
||||
B. Scope changed from ClusterRoleBinding to RoleBinding
|
||||
C. ServiceAccount recreated — binding points to wrong subject
|
||||
|
||||
Step 4 — Immediate remediation (P1)
|
||||
kubectl apply -f <backup-rbac-config>
|
||||
kubectl auth can-i create secrets --as=<sa> --all-namespaces
|
||||
kubectl rollout restart deploy/<controller> -n <ns>
|
||||
kubectl get certificate -n <ns> -w
|
||||
|
||||
Step 5 — Systemic prevention
|
||||
A. Protect RBAC resources in GitOps with Prune=false
|
||||
B. CronJob every 15 minutes running kubectl auth can-i validation
|
||||
C. Pre-sync hook that blocks pipeline if RBAC check fails
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 6 — Liveness vs Readiness Probe Issues
|
||||
|
||||
```
|
||||
readinessProbe → gates traffic AND rolling update progression
|
||||
pod STAYS ALIVE if failing
|
||||
removed from endpoints
|
||||
frozen rollout = readinessProbe never passing
|
||||
|
||||
livenessProbe → gates pod SURVIVAL
|
||||
pod gets KILLED and restarted if failing
|
||||
high restart count = livenessProbe too aggressive
|
||||
|
||||
Identify from evidence alone — no commands needed:
|
||||
Frozen rollout + pods Running + 0 restarts = readinessProbe
|
||||
High restart count + exit code 1 + Running = livenessProbe
|
||||
|
||||
Step 1 — Check which container is failing
|
||||
kubectl describe pod <pod> -n <ns>
|
||||
Read Containers section — which container shows Ready: False?
|
||||
Check Events section — exact probe failure message.
|
||||
|
||||
Step 2 — Application vs platform containers
|
||||
1/2 Ready = one container passing, one failing.
|
||||
Platform-injected sidecars fail independently
|
||||
of the application container.
|
||||
Isolate the failing container before troubleshooting.
|
||||
|
||||
Step 3 — Fix probe timing
|
||||
Replace initialDelaySeconds with startupProbe:
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /healthz/ready
|
||||
port: 8080
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
Polls every 2s up to 60s. Passes the moment container is ready.
|
||||
No unnecessary fixed wait on every rollout.
|
||||
|
||||
Step 4 — Fix livenessProbe aggression
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
timeoutSeconds: 5 # was 1 — give app time to respond
|
||||
periodSeconds: 10 # was 5 — less frequent
|
||||
failureThreshold: 3
|
||||
|
||||
CRITICAL: /healthz must NEVER query external dependencies.
|
||||
Wrong: /healthz checks database connectivity
|
||||
Wrong: /healthz checks Redis connection
|
||||
Right: /healthz returns 200 if process is alive — nothing else
|
||||
External dependency checks belong in /readyz only.
|
||||
|
||||
Step 5 — Immediate remediation for frozen rollout
|
||||
kubectl rollout undo deployment/<name> -n <ns>
|
||||
kubectl rollout status deployment/<name> -n <ns>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 7 — Control Plane Component Failures
|
||||
|
||||
```
|
||||
CoreDNS down
|
||||
Signal: intermittent service-to-service failures, no pod errors
|
||||
Check: kubectl get pods -n kube-system | grep coredns
|
||||
kubectl logs -n kube-system deploy/coredns
|
||||
kubectl exec -it <pod> -- nslookup <svc>.<ns>.svc.cluster.local
|
||||
Metric: coredns_dns_request_duration_seconds p99 > 100ms
|
||||
|
||||
CNI issue (Cilium)
|
||||
Signal: pods stuck in ContainerCreating, "failed to assign IP"
|
||||
Check: kubectl describe pod <stuck-pod> | grep -A10 Events
|
||||
Check: kubectl get pods -n kube-system | grep cilium
|
||||
Metric: cilium_endpoint_creation_errors
|
||||
|
||||
kube-proxy stale rules (or Cilium networking issues)
|
||||
Signal: new services unreachable from specific nodes only
|
||||
Check: kubectl logs -n kube-system <kube-proxy-or-cilium-pod>
|
||||
Metric: kubeproxy_sync_proxy_rules_duration_seconds spike
|
||||
|
||||
Controller reconciliation loop stuck
|
||||
Signal: RBAC errors or Watch stream failures
|
||||
Check: kubectl logs -n <ns> <controller-pod>
|
||||
Look for: "forbidden", "Watch", "timeout"
|
||||
|
||||
Longhorn (storage) issues
|
||||
Signal: PVC stuck Pending, pods can't mount volumes
|
||||
Check: kubectl get pvc -A
|
||||
Check: kubectl describe pvc <name> -n <ns>
|
||||
Check: kubectl get longhorn-nodes -n longhorn-system
|
||||
Metric: longhorn_disk_capacity / longhorn_disk_reservation
|
||||
|
||||
MinIO (object storage) issues
|
||||
Signal: Loki unable to write logs, pods crash
|
||||
Check: kubectl logs -n storage deploy/minio
|
||||
Check: kubectl exec -it <minio-pod> -- mc ls storage/
|
||||
Verify: site replication status between az-a and az-b
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 8 — Certificate Expiry Incidents
|
||||
|
||||
```
|
||||
Signal: TLS handshake failures or "certificate expired" errors
|
||||
This should NEVER happen — cert-manager automates renewal 30 days early.
|
||||
|
||||
Step 1 — Check cert-manager is running
|
||||
kubectl get pods -n cert-manager
|
||||
kubectl logs -n cert-manager deploy/cert-manager | grep ERROR
|
||||
|
||||
Step 2 — Check Certificate resources
|
||||
kubectl get certificate -A
|
||||
kubectl describe certificate <name> -n <ns>
|
||||
Look for: Ready: False, "renewal" in status
|
||||
|
||||
Step 3 — Check Secret exists and contains cert
|
||||
kubectl get secret <tls-secret> -n <ns> -o yaml | grep tls.crt
|
||||
Decode and verify expiry:
|
||||
kubectl get secret <tls-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' \
|
||||
| base64 -d | openssl x509 -noout -dates
|
||||
|
||||
Step 4 — Check live process cert (most important)
|
||||
openssl s_client -connect <pod-ip>:<port> </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -dates
|
||||
If old expiry here = process loaded cert at startup, never reloaded.
|
||||
Restart pod: kubectl rollout restart deploy/<name> -n <ns>
|
||||
|
||||
Step 5 — Check RBAC for cert-manager
|
||||
kubectl auth can-i create secrets \
|
||||
--as=system:serviceaccount:cert-manager:cert-manager \
|
||||
--all-namespaces
|
||||
|
||||
Step 6 — Emergency remediation (if cert truly expired)
|
||||
kubectl rollout restart deploy/cert-manager -n cert-manager
|
||||
kubectl delete certificate <name> -n <ns>
|
||||
kubectl apply -f <certificate-yaml>
|
||||
kubectl rollout restart deploy/<dependent-app> -n <ns>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 9 — Vault / Authentik IAM Issues
|
||||
|
||||
```
|
||||
Signal: Services can't authenticate, OIDC login fails, Vault sealed
|
||||
|
||||
Step 1 — Check Vault status
|
||||
kubectl get pods -n iam | grep vault
|
||||
kubectl logs -n iam deploy/vault
|
||||
kubectl exec -it <vault-pod> -n iam -- vault status
|
||||
|
||||
Step 2 — Check if Vault is sealed
|
||||
kubectl exec -it <vault-pod> -n iam -- vault status | grep Sealed
|
||||
If Sealed: true → requires unseal keys (see bootstrap docs)
|
||||
|
||||
Step 3 — Check Authentik
|
||||
kubectl get pods -n iam | grep authentik
|
||||
kubectl logs -n iam deploy/authentik-server
|
||||
kubectl describe statefulset authentik-postgresql -n iam
|
||||
|
||||
Step 4 — Check secret in Vault
|
||||
kubectl exec -it <vault-pod> -n iam -- \
|
||||
vault kv get cluster/VARIABLE_NAME
|
||||
Not found = secret never created (run setup_vault.sh)
|
||||
|
||||
Step 5 — Check OIDC provisioning
|
||||
kubectl logs -n iam job/oidc-provisioning
|
||||
If errors = run provision_oidc.py again
|
||||
|
||||
Step 6 — Verify Loki/MinIO can connect to Vault
|
||||
Check: helmfile values reference correct Vault endpoint
|
||||
Check: ServiceAccount token mounted and RBAC permitting auth
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PROCEDURE 10 — Loki / Logging Pipeline Stalled
|
||||
|
||||
```
|
||||
Signal: No logs appearing in Grafana, Loki backend growing without limit
|
||||
|
||||
Step 1 — Check Promtail is scraping
|
||||
kubectl get pods -n logging | grep promtail
|
||||
kubectl logs -n logging ds/promtail | grep -E 'scraping|error'
|
||||
Should show: scraping every few seconds from all nodes
|
||||
|
||||
Step 2 — Check Loki can write to MinIO
|
||||
kubectl logs -n logging deploy/loki
|
||||
Look for: S3 errors, "connection refused", "write: no space"
|
||||
|
||||
Step 3 — Check MinIO is operational
|
||||
kubectl get pods -n storage | grep minio
|
||||
kubectl logs -n storage deploy/minio
|
||||
Check site replication status:
|
||||
kubectl get job -n storage | grep replication
|
||||
|
||||
Step 4 — Check disk space
|
||||
kubectl top pod -n logging
|
||||
kubectl exec -it <loki-pod> -n logging -- df -h
|
||||
If full = minio-backed storage exhausted, purge old chunks
|
||||
|
||||
Step 5 — Check PVC for Loki index
|
||||
kubectl get pvc -n logging
|
||||
kubectl describe pvc <loki-pvc> -n logging
|
||||
Bound to Longhorn PV = check node storage
|
||||
|
||||
Step 6 — Restart Loki
|
||||
kubectl rollout restart deploy/loki -n logging
|
||||
kubectl get pods -n logging -w
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
```bash
|
||||
# Pod state and debugging
|
||||
kubectl describe pod <pod> -n <ns>
|
||||
kubectl logs <pod> -n <ns> --previous
|
||||
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20
|
||||
kubectl exec -it <pod> -n <ns> -- /bin/sh
|
||||
|
||||
# Resource usage
|
||||
kubectl top pod <pod> -n <ns>
|
||||
kubectl top node
|
||||
kubectl get pvc -A
|
||||
|
||||
# Networking and service discovery
|
||||
kubectl get endpoints <svc> -n <ns>
|
||||
kubectl get svc <svc> -n <ns> -o yaml
|
||||
kubectl get pods -n <ns> --show-labels
|
||||
kubectl get networkpolicy -n <ns>
|
||||
kubectl exec -it <pod> -n <ns> -- nslookup <svc>.<ns>.svc.cluster.local
|
||||
kubectl exec -it <pod> -n <ns> -- curl -v http://<svc>:<port>/healthz
|
||||
|
||||
# TLS inspection (live process — bypasses Kubernetes)
|
||||
openssl s_client -connect <ip>:<port> </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -dates
|
||||
|
||||
# RBAC validation
|
||||
kubectl auth can-i <verb> <resource> \
|
||||
--as=system:serviceaccount:<ns>:<sa> --all-namespaces
|
||||
|
||||
# Rollout management
|
||||
kubectl rollout status deployment/<name> -n <ns>
|
||||
kubectl rollout undo deployment/<name> -n <ns>
|
||||
kubectl rollout restart deploy/<name> -n <ns>
|
||||
|
||||
# Certificates and TLS
|
||||
kubectl get certificate -n <ns>
|
||||
kubectl describe certificate <name> -n <ns>
|
||||
kubectl get secret <tls-secret> -n <ns> -o yaml
|
||||
kubectl logs -n cert-manager deploy/cert-manager | grep ERROR
|
||||
|
||||
# Control plane components
|
||||
kubectl get pods -n kube-system
|
||||
kubectl get pods -n cert-manager
|
||||
kubectl logs -n kube-system deploy/coredns
|
||||
kubectl describe node <node-name>
|
||||
|
||||
# Storage
|
||||
kubectl get pvc -A
|
||||
kubectl get longhorn-nodes -n longhorn-system
|
||||
kubectl exec -it <longhorn-pod> -n longhorn-system -- longhorn node ls
|
||||
|
||||
# IAM and secrets
|
||||
kubectl get pods -n iam
|
||||
kubectl logs -n iam deploy/vault
|
||||
kubectl logs -n iam deploy/authentik-server
|
||||
kubectl exec -it <vault-pod> -n iam -- vault status
|
||||
kubectl exec -it <vault-pod> -n iam -- vault kv get cluster/<KEY>
|
||||
|
||||
# Using the k alias (add to ~/.zshrc)
|
||||
k get pods -A
|
||||
k logs -n logging <pod>
|
||||
k describe node talos-worker-1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agent Behaviour Rules
|
||||
|
||||
```
|
||||
1. Never skip the three pre-flight questions
|
||||
2. Never check logs before establishing exit code
|
||||
3. Never check ingress before checking endpoints
|
||||
4. Never raise memory limit before understanding growth pattern
|
||||
5. Never assume etcd talks directly to controllers
|
||||
6. Never conflate Kubernetes state with runtime process state
|
||||
7. Always use --previous for crash logs
|
||||
8. Always correlate two timelines for post-rollout issues
|
||||
9. Always diff working cluster vs broken cluster for RBAC issues
|
||||
10. Always confirm fix with kubectl auth can-i before closing incident
|
||||
11. Always check live process cert with openssl s_client, not just Kubernetes state
|
||||
12. Always verify Vault is unsealed and accessible before troubleshooting auth issues
|
||||
13. Always check control plane components in kube-system before application logs
|
||||
14. Always rule out networking (endpoints, DNS, NetworkPolicy) before app errors
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prevention & Observability
|
||||
|
||||
```
|
||||
Set up proactive alerts:
|
||||
|
||||
1. RBAC validation (every 15 minutes)
|
||||
for each ServiceAccount in each namespace:
|
||||
kubectl auth can-i create secrets --as=<sa> --all-namespaces
|
||||
Alert if any returns "no"
|
||||
|
||||
2. Certificate expiry monitoring
|
||||
certmanager_certificate_expiration_seconds
|
||||
Alert 15 days before expiry
|
||||
|
||||
3. Controller forbidden errors (zero tolerance)
|
||||
apiserver_request_total{code="403", user=~"system:serviceaccount:.*"}
|
||||
Alert on > 0
|
||||
|
||||
4. Watch stream collapse
|
||||
apiserver_request_total{verb="LIST"} spike
|
||||
Alert on 3x baseline within 5 minutes
|
||||
|
||||
5. Probe failures
|
||||
rate(kubelet_started_pods_total{result="failed"}[5m])
|
||||
Alert on > 0.1 per 5 minutes
|
||||
|
||||
6. Storage exhaustion
|
||||
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
|
||||
Alert at 80% capacity
|
||||
|
||||
7. Control plane latency
|
||||
apiserver_request_duration_seconds_sum / apiserver_request_duration_seconds_count
|
||||
Alert when p99 > 1 second
|
||||
|
||||
8. Logging pipeline lag
|
||||
loki_logql_querieslatency_seconds
|
||||
Alert when > 5 seconds
|
||||
```
|
||||
Reference in New Issue
Block a user