homelab/STORAGE-ARCHITECTURE-CLARIFICATION.md

# Storage Architecture Clarification

**Issue:** CLAUDE.md contradicts actual Longhorn configuration manifests.

---

## 🚨 Contradiction Found

### CLAUDE.md States:
```
| Node | Storage |
|------|---------|
| talos-cp-1 (.213) | sole Longhorn node |
| talos-cp-2 (.163) | none |
| talos-cp-3 (.166) | none |

"Only talos-cp-1 runs workloads and holds storage → 
 stateful services are single-instance."
```

### Actual Longhorn Manifests Show:

**1. Explicit Node CRDs for ALL 3 nodes:**
```yaml
# k8s/infrastructure/longhorn/longhorn-nodes.yaml
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: talos-cp-2
spec:
  allowScheduling: true  # ← Storage enabled!
  disks:
    default-disk:
      allowScheduling: true
      path: /var/lib/longhorn
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: talos-cp-3
spec:
  allowScheduling: true  # ← Storage enabled!
  disks:
    default-disk:
      allowScheduling: true
      path: /var/lib/longhorn
```

**2. Taint toleration for control-plane:**
```yaml
# longhorn-taint-toleration.yaml
value: "node-role.kubernetes.io/control-plane:NoSchedule"
# Allows Longhorn DaemonSet on ALL control-plane nodes
```

**3. StorageClass with 3 replicas:**
```yaml
# longhorn-wffc-storageclass.yaml
parameters:
  numberOfReplicas: "3"  # ← 3-way replication!
  volumeBindingMode: WaitForFirstConsumer
```

**4. PostSync job to expand existing volumes:**
```yaml
# expand-replicas-job.yaml
# Patches ALL volumes from 1 → 3 replicas
```

---

## 🔍 What's the Truth?

**Need to verify cluster state:**

```bash
# Check Longhorn nodes
kubectl get nodes.longhorn.io -n longhorn-system -o wide

# Expected output (if 3-node setup is actually working):
# NAME          READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
# talos-cp-1    True    true              true          Xd
# talos-cp-2    True    true              true          Xd
# talos-cp-3    True    true              true          Xd

# Check actual replica counts
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas,STATE:.status.state'

# Check DDB PVCs
kubectl get pvc -n ddb
kubectl describe pvc <pvc-name> -n ddb | grep -A 5 "Volumes:"
```

---

## 📊 Two Possible Scenarios

### **Scenario A: 3-Node Replication is Active** ✅

**If the Longhorn manifests are actually deployed:**

```
Storage Architecture:
┌─────────────────────────────────────────────────┐
│ DDB PVC (10Gi, Longhorn)                       │
├─────────────────────────────────────────────────┤
│ Replica 1: talos-cp-1:/var/lib/longhorn        │
│ Replica 2: talos-cp-2:/var/lib/longhorn        │
│ Replica 3: talos-cp-3:/var/lib/longhorn        │
└─────────────────────────────────────────────────┘

DDB PostgreSQL Pods:
┌──────────────┬──────────────┬──────────────┐
│ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│
│ (cp-1)       │ (cp-2)       │ (cp-3)       │
│ Primary      │ Replica      │ Replica      │
└──────────────┴──────────────┴──────────────┘
        ↓              ↓              ↓
   Reads all 3 Longhorn replicas locally
   (dataLocality: best-effort)

Failure Scenarios:
❌ cp-1 fails → Replica 2 & 3 still available
❌ cp-2 fails → Replica 1 & 3 still available
❌ cp-3 fails → Replica 1 & 2 still available
✅ Data survives ANY single node failure
```

**This is TRUE HA storage!** ✅

### **Scenario B: CLAUDE.md is Correct** ❌

**If Longhorn manifests are NOT actually deployed:**

```
Storage Architecture:
┌─────────────────────────────────────────────────┐
│ DDB PVC (10Gi, Longhorn)                       │
├─────────────────────────────────────────────────┤
│ Replica 1: talos-cp-1:/var/lib/longhorn        │
│ (NO replicas on cp-2, cp-3)                     │
└─────────────────────────────────────────────────┘

DDB PostgreSQL Pods:
┌──────────────┬──────────────┬──────────────┐
│ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│
│ (cp-1)       │ (cp-2)       │ (cp-3)       │
│ Primary      │ Replica      │ Replica      │
└──────────────┴──────────────┴──────────────┘
        ↓              ↓              ↓
   ALL pods must read from cp-1 over network
   (single point of failure)

Failure Scenarios:
❌ cp-1 disk fails → PERMANENT DATA LOSS
❌ cp-1 node fails → All PVCs inaccessible
❌ NO HA for storage at all
```

**This is NOT HA storage!** ❌

---

## 🎯 Action Required: Verify Cluster State

**Run these commands to determine which scenario is true:**

```bash
# 1. Check if Longhorn Node CRs exist
kubectl get nodes.longhorn.io -n longhorn-system

# 2. Check if taint toleration is set
kubectl get setting taint-toleration -n longhorn-system -o yaml

# 3. Check actual volume replica counts
kubectl get volumes.longhorn.io -n longhorn-system \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.numberOfReplicas}{"\n"}{end}'

# 4. Check DDB PVC details
kubectl get pvc -n ddb -o yaml | grep -A 10 "volumeName:"

# 5. Check Longhorn DaemonSet pods
kubectl get pods -n longhorn-system -o wide | grep longhorn-manager
# Should show pods on ALL 3 nodes if 3-node setup is active
```

---

## 🔧 If Scenario B (Single Node) is True

**You need to deploy the Longhorn HA configuration:**

```bash
# Apply the Longhorn HA manifests
kubectl apply -k k8s/infrastructure/longhorn/

# This will:
# 1. Create Node CRs for cp-2, cp-3
# 2. Set taint toleration
# 3. Create 3-replica StorageClass
# 4. Run PostSync job to expand existing volumes

# Verify expansion happened
kubectl get job longhorn-expand-replicas -n longhorn-system
kubectl logs job/longhorn-expand-replicas -n longhorn-system
```

---

## 📝 Corrected Documentation

**If 3-node replication IS active, update CLAUDE.md:**

```diff
 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
-| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
-| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
-| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | Longhorn (replica 1/3) |
+| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | Longhorn (replica 2/3) |
+| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | Longhorn (replica 3/3) |

-holds storage → stateful services are single-instance.
+holds storage → stateful services are HA (3-replica volumes).
```

**And update the hard rule:**

```diff
-🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
-node — all replicas are pinned to that node name. Renaming orphans its Longhorn
-node CR and faults every volume (permanent data loss).
+🔴 **NEVER rename ANY control-plane node.** Longhorn volumes have 3 replicas
+pinned to specific node names (talos-cp-1, talos-cp-2, talos-cp-3). Renaming
+ANY node orphans its Longhorn Node CR and degrades all volumes. Loss of 2+ nodes
+simultaneously = permanent data loss.
```

---

## 🎯 Impact on DDB Configuration

**If 3-node replication is active:**

### DDB cluster is actually HA! ✅

```
Compute HA: 3 PostgreSQL pods across 3 nodes ✅
Storage HA: 3 Longhorn replicas across 3 nodes ✅
Network HA: 3 etcd members, Cilium IPAM ✅

Failure tolerance:
- 1 node failure: Cluster continues (2/3 quorum)
- 1 disk failure: Data intact (2/3 replicas)
- 2 nodes fail: ❌ Etcd loses quorum, data degrades
```

**Current DDB config is acceptable if 3-replica storage is confirmed.**

### If Single-Node Storage:

**You MUST either:**

1. **Deploy Longhorn 3-node config** (recommended)
   ```bash
   kubectl apply -k k8s/infrastructure/longhorn/
   ```

2. **Reduce DDB to 1 instance** (match storage reality)
   ```yaml
   # k8s/data/cluster/ddb-cluster.yaml
   instances: 1  # Single instance if single-node storage
   ```

3. **Add external backup** (mitigate single-node risk)
   ```yaml
   # DDB backup to MinIO (see DDB-REVIEW.md)
   backup:
     barmanObjectStore:
       destinationPath: s3://ddb-backups/
   ```

---

## ✅ Recommended Actions (Priority Order)

1. **[ ] Verify cluster state** (run commands above)
2. **[ ] If single-node storage:** Deploy Longhorn 3-node config
3. **[ ] Wait for replicas to expand** (watch Longhorn UI)
4. **[ ] Update CLAUDE.md** with correct topology
5. **[ ] Update DDB-REVIEW.md** to reflect actual HA status
6. **[ ] Update GITOPS-REBUILD-PLAN.md** storage section
7. **[ ] Document findings** in TROUBLESHOOTING.md

---

**Next:** Please run the verification commands and report back:
- Are there 3 Longhorn nodes?
- What are the actual replica counts on DDB PVCs?
- Is the expand-replicas job present/completed?