# Cluster Recovery Plan

**Context:** Control-plane node (`talos-cp-1`, 192.168.1.213) had a corrupted machine CA in `terraform.tfvars`, causing a "broken key size" TLS error. During troubleshooting, an `apply-config` run with `install.wipe: true` (before the fix) wiped the OS/etcd disk (`sda`). Longhorn storage disks (`sdb`/`sdc`/`sdd`, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is `NotReady` (no CNI yet).

## Phase 1 — Get CNI up (blocks everything else)

Cluster config has `cni: name: none` (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint:

```bash
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set k8sServiceHost=localhost \
  --set k8sServicePort=7445
```

Verify: node flips to `Ready`, `cilium status` reports OK.

## Phase 2 — Bootstrap ArgoCD (app-of-apps)

If ArgoCD's own manifests + root Application live in git (`k8s/argocd/`), recovery is a 3-step process:

```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# wait for argocd-server to be Ready
kubectl apply -f k8s/argocd/root-app.yaml   # app-of-apps entrypoint
```

Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs.

## Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall)

Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica."

1. Confirm disk path via `longhorn-disk.cfg` (already confirmed: `/var/lib/longhorn-disk{1,2,3}`)
2. Locate `replicas/<volume-name>/` directories, read `volume.meta` for size
3. Launch a temporary `longhornio/longhorn-engine` container against the replica directory to expose it as a block device
4. Mount and copy data out manually, per volume

Not automatic — budget real time, one volume at a time.

## Phase 4 — WireGuard / router path

DuckDNS (`riotpiao.duckdns.org`) resolves correctly to current public IP (`24.114.42.68`, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from **off-LAN** — needs router-side verification:
- Confirm port-forward rule is **UDP 51820 → 192.168.1.213:51820** exactly
- If double-NAT, forward must be on the outermost internet-facing router
- Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding)

**Separately — LAN-side symptom:** tunnel also fails to connect **from the same LAN** as the server, via the public DDNS hostname. Root cause: **NAT hairpin/loopback not supported by the router** — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug.

Fixes, in order of robustness:
1. Enable NAT hairpinning/loopback in router settings, if supported
2. Split-DNS: resolve the DDNS hostname to the LAN IP (`192.168.1.213`) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS)
3. Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at `192.168.1.213:51820` while on LAN; switch back to the DDNS hostname when away

## Phase 5 — Redeploy DuckDNS updater

The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a `CronJob` running `curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip="` every few minutes.

---

**Recommended order:** Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency).

## Sources

- [How to Install Cilium on Talos Linux Step by Step](https://oneuptime.com/blog/post/2026-03-03-install-cilium-on-talos-linux-step-by-step/view)
- [Deploy Cilium CNI - Sidero Documentation](https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium)
- [Cluster Bootstrapping - Argo CD](https://argo-cd.readthedocs.io/en/latest/operator-manual/cluster-bootstrapping/)
- [How to Bootstrap an Entire Cluster with ArgoCD App-of-Apps](https://oneuptime.com/blog/post/2026-02-26-argocd-bootstrap-cluster-app-of-apps/view)
- [Restoring Data from an Orphaned Replica Directory - Longhorn KB](https://longhorn.io/kb/restoring-data-from-an-orphaned-replica-directory/)
- [Port Forwarding for WireGuard](https://portforward.com/wireguard/)
- [Guide Wireguard-portforwarding - SNBForums](https://www.snbforums.com/threads/guide-wireguard-portforwarding.89737/)
- [Help with hairpin NAT for wireguard on ubuntu server - Ubiquiti Community](https://community.ui.com/questions/Help-with-hairpin-NAT-for-wireguard-on-ubuntu-server/12d4cbe7-f1f5-448d-9d77-ea8226c18f16)
- [Wireguard VPN on Pi4 - cannot connect client - Raspberry Pi Forums](https://forums.raspberrypi.com/viewtopic.php?t=339692)