# Troubleshooting: Homelab Cluster Operations

Troubleshooting procedures and operational gotchas for homelab cluster. Cross-reference root `TROUBLESHOOTING.md` for full SRE diagnostic methodology (layer-before-tool, control-plane/kubelet/networking); this doc is the quick cheatsheet + gotchas specific to this repo.

---

## Quick Patterns

### Pod stuck in CrashLoopBackOff
```bash
k logs -n <ns> <pod> --tail=50
k describe pod -n <ns> <pod> | grep -A 10 Events
```

### Service unreachable (503)
```bash
# Check endpoints exist
k get endpoints -n <ns> <svc>
k get pods -n <ns> -o wide

# Test connectivity
k exec -it <pod> -- curl http://<svc>.<ns>.svc.cluster.local:8080
```

### Helm release stuck
```bash
helmfile status | grep -E "FAILED|UNKNOWN|PENDING"
helm status <release> -n <ns> --show-resources
k logs -n <ns> deploy/<app> | head -100
```

---

## Hard Rules (Never Violate)

🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.

```bash
# Before ANY PVC delete:
k get pvc -n <ns> <pvc>
k get pv <pvc-backing-pv> -o json | jq '.spec'  # check replication config

# For Longhorn: verify replicas >= 2
k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'

# For PostgreSQL: verify standby replicas are healthy
k exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c "SELECT * FROM pg_stat_replication;"
```

---

## Project Gotchas

- **Field name = variable name:** In Vault, use `core put cluster/KEY KEY="value"` (never `value=`)
- **vsource expansion:** `.env` values must be empty (`KEY=`) to fetch from Vault; hardcoded values pass through
- **Helmfile template syntax:** Use `{{ env "VAR" }}` not `${VAR}` (shell syntax, not Go template)
- **CNPG initialization:** Wait for cluster to be ready before creating databases (use `postInitApplicationSQL`, not helmfile hooks)
- **MinIO credentials:** Use Secret volumes, never `--env` flag (exposes in `kubectl describe`)
- **Temporal namespace registration:** never manually `temporal operator namespace create` for a namespace a Queue's `temporal.io/namespace` label references — `queue-operator` registers it automatically. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why.
- **kubectl hangs / connection refused:** you're probably off-LAN. `kubectl config use-context admin@homelab-cluster-1` (WireGuard path, `10.6.0.1:6443`) — the default context (`admin@homelab-cluster`, `192.168.1.213:6443`) only works on-LAN. See README.md "kubectl Context".
- **No wildcard DNS:** `*.riotpiao.homelab.com` isn't a real DNS zone — every hostname is a manual `/etc/hosts` line (`10.6.0.1` for WireGuard, the Cilium LB-IPAM IP for LAN). Adding an Ingress `host:` rule doesn't make it resolvable; you must also add the `/etc/hosts` line, on every client machine that needs it.
- **gRPC through nginx ingress:** add `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` to proxy h2c to a plaintext-gRPC backend (see `temporal-grpc` Ingress in `k8s/ingress/ingress.yaml`). TLS still terminates at nginx via the wildcard cert — clients need `--tls`, not client certs.
- **Soft podAntiAffinity doesn't rebalance existing pods.** `preferredDuringSchedulingIgnoredDuringExecution` only applies at scheduling time — pods that landed on the same node before the constraint existed (or before other nodes were `Ready`) stay there forever. Fixing the affinity config in values.yaml/CRD only affects *future* pod creation; existing skew needs `kubectl delete pod <name>` (one at a time, verify healthy before the next) to force a reschedule under the now-correct constraint. Hit this with Cassandra, the Kafka `KafkaNodePool`, and CNPG's `ddb-cluster` all stacking onto `talos-cp-1`.
- **Cassandra/StatefulSet pod deleted+recreated with no PVC (ephemeral storage) can crash-loop on rejoin:** `Other bootstrapping/leaving/moving nodes detected` or `A node required to move the data consistently is down (/<old-ip>)` — the ring still holds a stale gossip entry for the deleted pod's old IP. Fix: `kubectl exec <a live cassandra pod> -- nodetool assassinate <stale-ip>` from a healthy node, then let the crash-looping pod's next restart retry.
- **`helm upgrade` failing with `conflict ... using v1: .data.<field>` after a manual `kubectl apply` patch:** you (or an agent) hand-patched a resource Helm manages, and `kubectl apply`'s default client-side-apply field manager now owns that field. Reclaim it once: `kubectl get <resource> -o yaml | kubectl apply -f - --server-side --field-manager=helm --force-conflicts`, then retry the plain `helm upgrade` (no `--force` needed).
- **`~/.config/talos/secrets.toml` still uses the legacy `talos` name in its path:** Even though the CLI binary is `core`, the config file it reads is `~/.config/talos/secrets.toml` (not `~/.config/core/...`). This is real CLI behavior, not a doc bug — don't try to rename the path, just be aware if you're troubleshooting Vault access (`core get`/`core put` failing) and checking whether `~/.config/talos/secrets.toml` exists or is readable.