homelab/STORAGE-ARCHITECTURE-CLARIFICATION.md

Storage Architecture Clarification

Issue: CLAUDE.md contradicts actual Longhorn configuration manifests.

---

🚨 Contradiction Found

CLAUDE.md States:

| Node | Storage |
|------|---------|
| talos-cp-1 (.213) | sole Longhorn node |
| talos-cp-2 (.163) | none |
| talos-cp-3 (.166) | none |

"Only talos-cp-1 runs workloads and holds storage → 
 stateful services are single-instance."

Actual Longhorn Manifests Show:

1. Explicit Node CRDs for ALL 3 nodes:

# k8s/infrastructure/longhorn/longhorn-nodes.yaml
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: talos-cp-2
spec:
  allowScheduling: true  # ← Storage enabled!
  disks:
    default-disk:
      allowScheduling: true
      path: /var/lib/longhorn
---
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: talos-cp-3
spec:
  allowScheduling: true  # ← Storage enabled!
  disks:
    default-disk:
      allowScheduling: true
      path: /var/lib/longhorn

2. Taint toleration for control-plane:

# longhorn-taint-toleration.yaml
value: "node-role.kubernetes.io/control-plane:NoSchedule"
# Allows Longhorn DaemonSet on ALL control-plane nodes

3. StorageClass with 3 replicas:

# longhorn-wffc-storageclass.yaml
parameters:
  numberOfReplicas: "3"  # ← 3-way replication!
  volumeBindingMode: WaitForFirstConsumer

4. PostSync job to expand existing volumes:

# expand-replicas-job.yaml
# Patches ALL volumes from 1 → 3 replicas

---

🔍 What's the Truth?

Need to verify cluster state:

# Check Longhorn nodes
kubectl get nodes.longhorn.io -n longhorn-system -o wide

# Expected output (if 3-node setup is actually working):
# NAME          READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
# talos-cp-1    True    true              true          Xd
# talos-cp-2    True    true              true          Xd
# talos-cp-3    True    true              true          Xd

# Check actual replica counts
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas,STATE:.status.state'

# Check DDB PVCs
kubectl get pvc -n ddb
kubectl describe pvc <pvc-name> -n ddb | grep -A 5 "Volumes:"

---

📊 Two Possible Scenarios

Scenario A: 3-Node Replication is Active ✅

If the Longhorn manifests are actually deployed:

Storage Architecture:
┌─────────────────────────────────────────────────┐
│ DDB PVC (10Gi, Longhorn)                       │
├─────────────────────────────────────────────────┤
│ Replica 1: talos-cp-1:/var/lib/longhorn        │
│ Replica 2: talos-cp-2:/var/lib/longhorn        │
│ Replica 3: talos-cp-3:/var/lib/longhorn        │
└─────────────────────────────────────────────────┘

DDB PostgreSQL Pods:
┌──────────────┬──────────────┬──────────────┐
│ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│
│ (cp-1)       │ (cp-2)       │ (cp-3)       │
│ Primary      │ Replica      │ Replica      │
└──────────────┴──────────────┴──────────────┘
        ↓              ↓              ↓
   Reads all 3 Longhorn replicas locally
   (dataLocality: best-effort)

Failure Scenarios:
❌ cp-1 fails → Replica 2 & 3 still available
❌ cp-2 fails → Replica 1 & 3 still available
❌ cp-3 fails → Replica 1 & 2 still available
✅ Data survives ANY single node failure

This is TRUE HA storage! ✅

Scenario B: CLAUDE.md is Correct ❌

If Longhorn manifests are NOT actually deployed:

Storage Architecture:
┌─────────────────────────────────────────────────┐
│ DDB PVC (10Gi, Longhorn)                       │
├─────────────────────────────────────────────────┤
│ Replica 1: talos-cp-1:/var/lib/longhorn        │
│ (NO replicas on cp-2, cp-3)                     │
└─────────────────────────────────────────────────┘

DDB PostgreSQL Pods:
┌──────────────┬──────────────┬──────────────┐
│ ddb-cluster-1│ ddb-cluster-2│ ddb-cluster-3│
│ (cp-1)       │ (cp-2)       │ (cp-3)       │
│ Primary      │ Replica      │ Replica      │
└──────────────┴──────────────┴──────────────┘
        ↓              ↓              ↓
   ALL pods must read from cp-1 over network
   (single point of failure)

Failure Scenarios:
❌ cp-1 disk fails → PERMANENT DATA LOSS
❌ cp-1 node fails → All PVCs inaccessible
❌ NO HA for storage at all

This is NOT HA storage! ❌

---

🎯 Action Required: Verify Cluster State

Run these commands to determine which scenario is true:

# 1. Check if Longhorn Node CRs exist
kubectl get nodes.longhorn.io -n longhorn-system

# 2. Check if taint toleration is set
kubectl get setting taint-toleration -n longhorn-system -o yaml

# 3. Check actual volume replica counts
kubectl get volumes.longhorn.io -n longhorn-system \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.numberOfReplicas}{"\n"}{end}'

# 4. Check DDB PVC details
kubectl get pvc -n ddb -o yaml | grep -A 10 "volumeName:"

# 5. Check Longhorn DaemonSet pods
kubectl get pods -n longhorn-system -o wide | grep longhorn-manager
# Should show pods on ALL 3 nodes if 3-node setup is active

---

🔧 If Scenario B (Single Node) is True

You need to deploy the Longhorn HA configuration:

# Apply the Longhorn HA manifests
kubectl apply -k k8s/infrastructure/longhorn/

# This will:
# 1. Create Node CRs for cp-2, cp-3
# 2. Set taint toleration
# 3. Create 3-replica StorageClass
# 4. Run PostSync job to expand existing volumes

# Verify expansion happened
kubectl get job longhorn-expand-replicas -n longhorn-system
kubectl logs job/longhorn-expand-replicas -n longhorn-system

---

📝 Corrected Documentation

If 3-node replication IS active, update CLAUDE.md:

 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
-| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
-| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
-| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | Longhorn (replica 1/3) |
+| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | Longhorn (replica 2/3) |
+| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | Longhorn (replica 3/3) |

-holds storage → stateful services are single-instance.
+holds storage → stateful services are HA (3-replica volumes).

And update the hard rule:

-🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
-node — all replicas are pinned to that node name. Renaming orphans its Longhorn
-node CR and faults every volume (permanent data loss).
+🔴 **NEVER rename ANY control-plane node.** Longhorn volumes have 3 replicas
+pinned to specific node names (talos-cp-1, talos-cp-2, talos-cp-3). Renaming
+ANY node orphans its Longhorn Node CR and degrades all volumes. Loss of 2+ nodes
+simultaneously = permanent data loss.

---

🎯 Impact on DDB Configuration

If 3-node replication is active:

DDB cluster is actually HA! ✅

Compute HA: 3 PostgreSQL pods across 3 nodes ✅
Storage HA: 3 Longhorn replicas across 3 nodes ✅
Network HA: 3 etcd members, Cilium IPAM ✅

Failure tolerance:
- 1 node failure: Cluster continues (2/3 quorum)
- 1 disk failure: Data intact (2/3 replicas)
- 2 nodes fail: ❌ Etcd loses quorum, data degrades

Current DDB config is acceptable if 3-replica storage is confirmed.

If Single-Node Storage:

You MUST either:

1. Deploy Longhorn 3-node config (recommended)

   kubectl apply -k k8s/infrastructure/longhorn/
   

2. Reduce DDB to 1 instance (match storage reality)

   # k8s/data/cluster/ddb-cluster.yaml
   instances: 1  # Single instance if single-node storage
   

3. Add external backup (mitigate single-node risk)

   # DDB backup to MinIO (see DDB-REVIEW.md)
   backup:
     barmanObjectStore:
       destinationPath: s3://ddb-backups/
   

---

✅ Recommended Actions (Priority Order)

1. [ ] Verify cluster state (run commands above)
2. [ ] If single-node storage: Deploy Longhorn 3-node config
3. [ ] Wait for replicas to expand (watch Longhorn UI)
4. [ ] Update CLAUDE.md with correct topology
5. [ ] Update DDB-REVIEW.md to reflect actual HA status
6. [ ] Update GITOPS-REBUILD-PLAN.md storage section
7. [ ] Document findings in TROUBLESHOOTING.md

---

Next: Please run the verification commands and report back:

• Are there 3 Longhorn nodes?
• What are the actual replica counts on DDB PVCs?
• Is the expand-replicas job present/completed?