82 lines
5.4 KiB
Markdown
82 lines
5.4 KiB
Markdown
# Cluster Recovery Plan
|
|
|
|
**Context:** Control-plane node (`talos-cp-1`, 192.168.1.213) had a corrupted machine CA in `terraform.tfvars`, causing a "broken key size" TLS error. During troubleshooting, an `apply-config` run with `install.wipe: true` (before the fix) wiped the OS/etcd disk (`sda`). Longhorn storage disks (`sdb`/`sdc`/`sdd`, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is `NotReady` (no CNI yet).
|
|
|
|
## Phase 1 — Get CNI up (blocks everything else)
|
|
|
|
Cluster config has `cni: name: none` (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint:
|
|
|
|
```bash
|
|
helm repo add cilium https://helm.cilium.io/
|
|
helm repo update
|
|
helm install cilium cilium/cilium --namespace kube-system \
|
|
--set ipam.mode=kubernetes \
|
|
--set kubeProxyReplacement=true \
|
|
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
|
|
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
|
|
--set cgroup.autoMount.enabled=false \
|
|
--set cgroup.hostRoot=/sys/fs/cgroup \
|
|
--set k8sServiceHost=localhost \
|
|
--set k8sServicePort=7445
|
|
```
|
|
|
|
Verify: node flips to `Ready`, `cilium status` reports OK.
|
|
|
|
## Phase 2 — Bootstrap ArgoCD (app-of-apps)
|
|
|
|
If ArgoCD's own manifests + root Application live in git (`k8s/argocd/`), recovery is a 3-step process:
|
|
|
|
```bash
|
|
kubectl create namespace argocd
|
|
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
|
|
# wait for argocd-server to be Ready
|
|
kubectl apply -f k8s/argocd/root-app.yaml # app-of-apps entrypoint
|
|
```
|
|
|
|
Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs.
|
|
|
|
## Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall)
|
|
|
|
Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica."
|
|
|
|
1. Confirm disk path via `longhorn-disk.cfg` (already confirmed: `/var/lib/longhorn-disk{1,2,3}`)
|
|
2. Locate `replicas/<volume-name>/` directories, read `volume.meta` for size
|
|
3. Launch a temporary `longhornio/longhorn-engine` container against the replica directory to expose it as a block device
|
|
4. Mount and copy data out manually, per volume
|
|
|
|
Not automatic — budget real time, one volume at a time.
|
|
|
|
## Phase 4 — WireGuard / router path
|
|
|
|
DuckDNS (`riotpiao.duckdns.org`) resolves correctly to current public IP (`24.114.42.68`, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from **off-LAN** — needs router-side verification:
|
|
- Confirm port-forward rule is **UDP 51820 → 192.168.1.213:51820** exactly
|
|
- If double-NAT, forward must be on the outermost internet-facing router
|
|
- Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding)
|
|
|
|
**Separately — LAN-side symptom:** tunnel also fails to connect **from the same LAN** as the server, via the public DDNS hostname. Root cause: **NAT hairpin/loopback not supported by the router** — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug.
|
|
|
|
Fixes, in order of robustness:
|
|
1. Enable NAT hairpinning/loopback in router settings, if supported
|
|
2. Split-DNS: resolve the DDNS hostname to the LAN IP (`192.168.1.213`) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS)
|
|
3. Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at `192.168.1.213:51820` while on LAN; switch back to the DDNS hostname when away
|
|
|
|
## Phase 5 — Redeploy DuckDNS updater
|
|
|
|
The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a `CronJob` running `curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip="` every few minutes.
|
|
|
|
---
|
|
|
|
**Recommended order:** Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency).
|
|
|
|
## Sources
|
|
|
|
- [How to Install Cilium on Talos Linux Step by Step](https://oneuptime.com/blog/post/2026-03-03-install-cilium-on-talos-linux-step-by-step/view)
|
|
- [Deploy Cilium CNI - Sidero Documentation](https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium)
|
|
- [Cluster Bootstrapping - Argo CD](https://argo-cd.readthedocs.io/en/latest/operator-manual/cluster-bootstrapping/)
|
|
- [How to Bootstrap an Entire Cluster with ArgoCD App-of-Apps](https://oneuptime.com/blog/post/2026-02-26-argocd-bootstrap-cluster-app-of-apps/view)
|
|
- [Restoring Data from an Orphaned Replica Directory - Longhorn KB](https://longhorn.io/kb/restoring-data-from-an-orphaned-replica-directory/)
|
|
- [Port Forwarding for WireGuard](https://portforward.com/wireguard/)
|
|
- [Guide Wireguard-portforwarding - SNBForums](https://www.snbforums.com/threads/guide-wireguard-portforwarding.89737/)
|
|
- [Help with hairpin NAT for wireguard on ubuntu server - Ubiquiti Community](https://community.ui.com/questions/Help-with-hairpin-NAT-for-wireguard-on-ubuntu-server/12d4cbe7-f1f5-448d-9d77-ea8226c18f16)
|
|
- [Wireguard VPN on Pi4 - cannot connect client - Raspberry Pi Forums](https://forums.raspberrypi.com/viewtopic.php?t=339692)
|