# Storage Architecture Clarification **Issue:** CLAUDE.md contradicts actual Longhorn configuration manifests. --- ## 🚨 Contradiction Found ### CLAUDE.md States: ``` | Node | Storage | |------|---------| | talos-cp-1 (.213) | sole Longhorn node | | talos-cp-2 (.163) | none | | talos-cp-3 (.166) | none | "Only talos-cp-1 runs workloads and holds storage → stateful services are single-instance." ``` ### Actual Longhorn Manifests Show: **1. Explicit Node CRDs for ALL 3 nodes:** ```yaml # k8s/infrastructure/longhorn/longhorn-nodes.yaml --- apiVersion: longhorn.io/v1beta2 kind: Node metadata: name: talos-cp-2 spec: allowScheduling: true # ← Storage enabled! disks: default-disk: allowScheduling: true path: /var/lib/longhorn --- apiVersion: longhorn.io/v1beta2 kind: Node metadata: name: talos-cp-3 spec: allowScheduling: true # ← Storage enabled! disks: default-disk: allowScheduling: true path: /var/lib/longhorn ``` **2. Taint toleration for control-plane:** ```yaml # longhorn-taint-toleration.yaml value: "node-role.kubernetes.io/control-plane:NoSchedule" # Allows Longhorn DaemonSet on ALL control-plane nodes ``` **3. StorageClass with 3 replicas:** ```yaml # longhorn-wffc-storageclass.yaml parameters: numberOfReplicas: "3" # ← 3-way replication! volumeBindingMode: WaitForFirstConsumer ``` **4. PostSync job to expand existing volumes:** ```yaml # expand-replicas-job.yaml # Patches ALL volumes from 1 → 3 replicas ``` --- ## 🔍 What's the Truth? **Need to verify cluster state:** ```bash # Check Longhorn nodes kubectl get nodes.longhorn.io -n longhorn-system -o wide # Expected output (if 3-node setup is actually working): # NAME READY ALLOWSCHEDULING SCHEDULABLE AGE # talos-cp-1 True true true Xd # talos-cp-2 True true true Xd # talos-cp-3 True true true Xd # Check actual replica counts kubectl get volumes.longhorn.io -n longhorn-system \ -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas,STATE:.status.state' # Check DDB PVCs kubectl get pvc -n ddb kubectl describe pvc -n ddb | grep -A 5 "Volumes:" ``` --- ## 📊 Two Possible Scenarios ### **Scenario A: 3-Node Replication is Active** ✅ **If the Longhorn manifests are actually deployed:** ``` Storage Architecture: ┌─────────────────────────────────────────────────┐ │ DDB PVC (10Gi, Longhorn) │ ├─────────────────────────────────────────────────┤ │ Replica 1: talos-cp-1:/var/lib/longhorn │ │ Replica 2: talos-cp-2:/var/lib/longhorn │ │ Replica 3: talos-cp-3:/var/lib/longhorn │ └─────────────────────────────────────────────────┘ DDB PostgreSQL Pods: ┌──────────────┬──────────────┬──────────────┐ │ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│ │ (cp-1) │ (cp-2) │ (cp-3) │ │ Primary │ Replica │ Replica │ └──────────────┴──────────────┴──────────────┘ ↓ ↓ ↓ Reads all 3 Longhorn replicas locally (dataLocality: best-effort) Failure Scenarios: ❌ cp-1 fails → Replica 2 & 3 still available ❌ cp-2 fails → Replica 1 & 3 still available ❌ cp-3 fails → Replica 1 & 2 still available ✅ Data survives ANY single node failure ``` **This is TRUE HA storage!** ✅ ### **Scenario B: CLAUDE.md is Correct** ❌ **If Longhorn manifests are NOT actually deployed:** ``` Storage Architecture: ┌─────────────────────────────────────────────────┐ │ DDB PVC (10Gi, Longhorn) │ ├─────────────────────────────────────────────────┤ │ Replica 1: talos-cp-1:/var/lib/longhorn │ │ (NO replicas on cp-2, cp-3) │ └─────────────────────────────────────────────────┘ DDB PostgreSQL Pods: ┌──────────────┬──────────────┬──────────────┐ │ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│ │ (cp-1) │ (cp-2) │ (cp-3) │ │ Primary │ Replica │ Replica │ └──────────────┴──────────────┴──────────────┘ ↓ ↓ ↓ ALL pods must read from cp-1 over network (single point of failure) Failure Scenarios: ❌ cp-1 disk fails → PERMANENT DATA LOSS ❌ cp-1 node fails → All PVCs inaccessible ❌ NO HA for storage at all ``` **This is NOT HA storage!** ❌ --- ## 🎯 Action Required: Verify Cluster State **Run these commands to determine which scenario is true:** ```bash # 1. Check if Longhorn Node CRs exist kubectl get nodes.longhorn.io -n longhorn-system # 2. Check if taint toleration is set kubectl get setting taint-toleration -n longhorn-system -o yaml # 3. Check actual volume replica counts kubectl get volumes.longhorn.io -n longhorn-system \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.numberOfReplicas}{"\n"}{end}' # 4. Check DDB PVC details kubectl get pvc -n ddb -o yaml | grep -A 10 "volumeName:" # 5. Check Longhorn DaemonSet pods kubectl get pods -n longhorn-system -o wide | grep longhorn-manager # Should show pods on ALL 3 nodes if 3-node setup is active ``` --- ## 🔧 If Scenario B (Single Node) is True **You need to deploy the Longhorn HA configuration:** ```bash # Apply the Longhorn HA manifests kubectl apply -k k8s/infrastructure/longhorn/ # This will: # 1. Create Node CRs for cp-2, cp-3 # 2. Set taint toleration # 3. Create 3-replica StorageClass # 4. Run PostSync job to expand existing volumes # Verify expansion happened kubectl get job longhorn-expand-replicas -n longhorn-system kubectl logs job/longhorn-expand-replicas -n longhorn-system ``` --- ## 📝 Corrected Documentation **If 3-node replication IS active, update CLAUDE.md:** ```diff | Node | IP | Zone | Scheduling | Storage | |------|----|----|-----------|---------| -| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node | -| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none | -| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none | +| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | Longhorn (replica 1/3) | +| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | Longhorn (replica 2/3) | +| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | Longhorn (replica 3/3) | -holds storage → stateful services are single-instance. +holds storage → stateful services are HA (3-replica volumes). ``` **And update the hard rule:** ```diff -🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage -node — all replicas are pinned to that node name. Renaming orphans its Longhorn -node CR and faults every volume (permanent data loss). +🔴 **NEVER rename ANY control-plane node.** Longhorn volumes have 3 replicas +pinned to specific node names (talos-cp-1, talos-cp-2, talos-cp-3). Renaming +ANY node orphans its Longhorn Node CR and degrades all volumes. Loss of 2+ nodes +simultaneously = permanent data loss. ``` --- ## 🎯 Impact on DDB Configuration **If 3-node replication is active:** ### DDB cluster is actually HA! ✅ ``` Compute HA: 3 PostgreSQL pods across 3 nodes ✅ Storage HA: 3 Longhorn replicas across 3 nodes ✅ Network HA: 3 etcd members, Cilium IPAM ✅ Failure tolerance: - 1 node failure: Cluster continues (2/3 quorum) - 1 disk failure: Data intact (2/3 replicas) - 2 nodes fail: ❌ Etcd loses quorum, data degrades ``` **Current DDB config is acceptable if 3-replica storage is confirmed.** ### If Single-Node Storage: **You MUST either:** 1. **Deploy Longhorn 3-node config** (recommended) ```bash kubectl apply -k k8s/infrastructure/longhorn/ ``` 2. **Reduce DDB to 1 instance** (match storage reality) ```yaml # k8s/data/cluster/ddb-cluster.yaml instances: 1 # Single instance if single-node storage ``` 3. **Add external backup** (mitigate single-node risk) ```yaml # DDB backup to MinIO (see DDB-REVIEW.md) backup: barmanObjectStore: destinationPath: s3://ddb-backups/ ``` --- ## ✅ Recommended Actions (Priority Order) 1. **[ ] Verify cluster state** (run commands above) 2. **[ ] If single-node storage:** Deploy Longhorn 3-node config 3. **[ ] Wait for replicas to expand** (watch Longhorn UI) 4. **[ ] Update CLAUDE.md** with correct topology 5. **[ ] Update DDB-REVIEW.md** to reflect actual HA status 6. **[ ] Update GITOPS-REBUILD-PLAN.md** storage section 7. **[ ] Document findings** in TROUBLESHOOTING.md --- **Next:** Please run the verification commands and report back: - Are there 3 Longhorn nodes? - What are the actual replica counts on DDB PVCs? - Is the expand-replicas job present/completed?