homelab/project-usage/infra-troubleshooting.md

Troubleshooting: Homelab Cluster Operations

Troubleshooting procedures and operational gotchas for homelab cluster. Cross-reference root TROUBLESHOOTING.md for full SRE diagnostic methodology (layer-before-tool, control-plane/kubelet/networking); this doc is the quick cheatsheet + gotchas specific to this repo.

---

Quick Patterns

Pod stuck in CrashLoopBackOff

k logs -n <ns> <pod> --tail=50
k describe pod -n <ns> <pod> | grep -A 10 Events

Service unreachable (503)

# Check endpoints exist
k get endpoints -n <ns> <svc>
k get pods -n <ns> -o wide

# Test connectivity
k exec -it <pod> -- curl http://<svc>.<ns>.svc.cluster.local:8080

Helm release stuck

helmfile status | grep -E "FAILED|UNKNOWN|PENDING"
helm status <release> -n <ns> --show-resources
k logs -n <ns> deploy/<app> | head -100

---

Hard Rules (Never Violate)

🔴 NEVER delete a PVC unless there are replicas or backups. A PVC deletion = permanent data loss. Verify replication status first.

# Before ANY PVC delete:
k get pvc -n <ns> <pvc>
k get pv <pvc-backing-pv> -o json | jq '.spec'  # check replication config

# For Longhorn: verify replicas >= 2
k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'

# For PostgreSQL: verify standby replicas are healthy
k exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c "SELECT * FROM pg_stat_replication;"

---

Project Gotchas

• Field name = variable name: In Vault, use core put cluster/KEY KEY="value" (never value=)
• vsource expansion: .env values must be empty (KEY=) to fetch from Vault; hardcoded values pass through
• Helmfile template syntax: Use {{ env "VAR" }} not ${VAR} (shell syntax, not Go template)
• CNPG initialization: Wait for cluster to be ready before creating databases (use postInitApplicationSQL, not helmfile hooks)
• MinIO credentials: Use Secret volumes, never --env flag (exposes in kubectl describe)
• Temporal namespace registration: never manually temporal operator namespace create for a namespace a Queue's temporal.io/namespace label references — queue-operator registers it automatically. See ~/workplace/kmsvc-manage/CLAUDE.md ("Temporal Namespace Registration") for the full rule and why.
• kubectl hangs / connection refused: you're probably off-LAN. kubectl config use-context admin@homelab-cluster-1 (WireGuard path, 10.6.0.1:6443) — the default context (admin@homelab-cluster, 192.168.1.213:6443) only works on-LAN. See README.md "kubectl Context".
• No wildcard DNS: *.riotpiao.homelab.com isn't a real DNS zone — every hostname is a manual /etc/hosts line (10.6.0.1 for WireGuard, the Cilium LB-IPAM IP for LAN). Adding an Ingress host: rule doesn't make it resolvable; you must also add the /etc/hosts line, on every client machine that needs it.
• gRPC through nginx ingress: add nginx.ingress.kubernetes.io/backend-protocol: "GRPC" to proxy h2c to a plaintext-gRPC backend (see temporal-grpc Ingress in k8s/ingress/ingress.yaml). TLS still terminates at nginx via the wildcard cert — clients need --tls, not client certs.
• Soft podAntiAffinity doesn't rebalance existing pods. preferredDuringSchedulingIgnoredDuringExecution only applies at scheduling time — pods that landed on the same node before the constraint existed (or before other nodes were Ready) stay there forever. Fixing the affinity config in values.yaml/CRD only affects future pod creation; existing skew needs kubectl delete pod <name> (one at a time, verify healthy before the next) to force a reschedule under the now-correct constraint. Hit this with Cassandra, the Kafka KafkaNodePool, and CNPG's ddb-cluster all stacking onto talos-cp-1.
• Cassandra/StatefulSet pod deleted+recreated with no PVC (ephemeral storage) can crash-loop on rejoin: Other bootstrapping/leaving/moving nodes detected or A node required to move the data consistently is down (/<old-ip>) — the ring still holds a stale gossip entry for the deleted pod's old IP. Fix: kubectl exec <a live cassandra pod> -- nodetool assassinate <stale-ip> from a healthy node, then let the crash-looping pod's next restart retry.
• helm upgrade failing with conflict ... using v1: .data.<field> after a manual kubectl apply patch: you (or an agent) hand-patched a resource Helm manages, and kubectl apply's default client-side-apply field manager now owns that field. Reclaim it once: kubectl get <resource> -o yaml | kubectl apply -f - --server-side --field-manager=helm --force-conflicts, then retry the plain helm upgrade (no --force needed).
• ~/.config/talos/secrets.toml still uses the legacy talos name in its path: Even though the CLI binary is core, the config file it reads is ~/.config/talos/secrets.toml (not ~/.config/core/...). This is real CLI behavior, not a doc bug — don't try to rename the path, just be aware if you're troubleshooting Vault access (core get/core put failing) and checking whether ~/.config/talos/secrets.toml exists or is readable.