Cluster Recovery Plan

Context: Control-plane node (talos-cp-1, 192.168.1.213) had a corrupted machine CA in terraform.tfvars, causing a "broken key size" TLS error. During troubleshooting, an apply-config run with install.wipe: true (before the fix) wiped the OS/etcd disk (sda). Longhorn storage disks (sdb/sdc/sdd, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is NotReady (no CNI yet).

Phase 1 — Get CNI up (blocks everything else)

Cluster config has cni: name: none (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint:

helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set k8sServiceHost=localhost \
  --set k8sServicePort=7445

Verify: node flips to Ready, cilium status reports OK.

Phase 2 — Bootstrap ArgoCD (app-of-apps)

If ArgoCD's own manifests + root Application live in git (k8s/argocd/), recovery is a 3-step process:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# wait for argocd-server to be Ready
kubectl apply -f k8s/argocd/root-app.yaml   # app-of-apps entrypoint

Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs.

Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall)

Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica."

Confirm disk path via longhorn-disk.cfg (already confirmed: /var/lib/longhorn-disk{1,2,3})
Locate replicas/<volume-name>/ directories, read volume.meta for size
Launch a temporary longhornio/longhorn-engine container against the replica directory to expose it as a block device
Mount and copy data out manually, per volume

Not automatic — budget real time, one volume at a time.

Phase 4 — WireGuard / router path

DuckDNS (riotpiao.duckdns.org) resolves correctly to current public IP (24.114.42.68, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from off-LAN — needs router-side verification:

Confirm port-forward rule is UDP 51820 → 192.168.1.213:51820 exactly
If double-NAT, forward must be on the outermost internet-facing router
Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding)

Separately — LAN-side symptom: tunnel also fails to connect from the same LAN as the server, via the public DDNS hostname. Root cause: NAT hairpin/loopback not supported by the router — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug.

Fixes, in order of robustness:

Enable NAT hairpinning/loopback in router settings, if supported
Split-DNS: resolve the DDNS hostname to the LAN IP (192.168.1.213) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS)
Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at 192.168.1.213:51820 while on LAN; switch back to the DDNS hostname when away

Phase 5 — Redeploy DuckDNS updater

The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a CronJob running curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip=" every few minutes.

Recommended order: Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency).

5.4 KiB Raw Blame History