Story Crater Bot
f656338a15
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
122 lines
3.8 KiB
Markdown
122 lines
3.8 KiB
Markdown
# Storage Migration Analysis - storageClassName Update
|
||
|
||
## ❌ Problem: Field is Immutable
|
||
|
||
Kubernetes does not allow changing `spec.storageClassName` on existing PVCs.
|
||
|
||
```
|
||
Error: spec is immutable after creation except resources.requests
|
||
and volumeAttributesClassName for bound claims
|
||
```
|
||
|
||
## 📊 Current Situation
|
||
|
||
**PVCs using old StorageClass names (7 total):**
|
||
|
||
| Namespace | PVC | Old StorageClass | Volume | Replicas | Status |
|
||
|-----------|-----|------------------|--------|----------|--------|
|
||
| dashboard | portainer | longhorn-wffc | pvc-cb87... | 3 | ✅ Working |
|
||
| monitoring | prometheus-...| longhorn-wffc | pvc-03237... | 3 | ✅ Working |
|
||
| sqs | data-kmsvc-pool-0 | longhorn-kafka | pvc-0362a... | 3 | ✅ Working |
|
||
| sqs | data-kmsvc-pool-1 | longhorn-kafka | pvc-55190... | 3 | ✅ Working |
|
||
| sqs | data-kmsvc-pool-2 | longhorn-kafka | pvc-79f42... | 3 | ✅ Working |
|
||
| sqs | redis-replicas-1 | longhorn-wffc | pvc-a4625... | 3 | ✅ Working |
|
||
| sqs | redis-replicas-2 | longhorn-wffc | pvc-e330f... | 3 | ✅ Working |
|
||
|
||
**All volumes have 3 replicas and work perfectly.**
|
||
|
||
## 🤔 Is Migration Necessary?
|
||
|
||
**Functional Impact:** NONE
|
||
- ✅ All PVCs are Bound
|
||
- ✅ All volumes have 3 replicas
|
||
- ✅ Applications work normally
|
||
- ✅ New PVCs will use unified `longhorn` StorageClass automatically
|
||
|
||
**Cosmetic Issue Only:**
|
||
- PVC metadata shows old StorageClass name
|
||
- Doesn't affect functionality at all
|
||
- Old StorageClasses already deleted from cluster
|
||
|
||
**My Recommendation:** **DON'T MIGRATE** - not worth the risk/effort
|
||
|
||
## 🔧 If You REALLY Want to Migrate...
|
||
|
||
### Option 1: Live Migration (Complex, Risky)
|
||
|
||
For each PVC:
|
||
1. Create new PVC with correct StorageClass
|
||
2. Use a data copy tool (rsync pod, Velero, snapshot)
|
||
3. Scale down application
|
||
4. Copy data from old volume to new volume
|
||
5. Update application to use new PVC
|
||
6. Test
|
||
7. Delete old PVC
|
||
|
||
**Downtime:** Yes (per application)
|
||
**Risk:** Medium (data copy could fail)
|
||
**Effort:** ~30 min per PVC × 7 = 3.5 hours
|
||
|
||
### Option 2: Snapshot & Restore (Cleaner, Requires Longhorn Snapshots)
|
||
|
||
For each PVC:
|
||
1. Create Longhorn snapshot of volume
|
||
2. Create new PVC from snapshot (with correct StorageClass)
|
||
3. Scale down application
|
||
4. Update application to use new PVC
|
||
5. Scale up, test
|
||
6. Delete old PVC
|
||
|
||
**Downtime:** Yes (per application)
|
||
**Risk:** Low (snapshots are atomic)
|
||
**Effort:** ~20 min per PVC × 7 = 2.5 hours
|
||
|
||
### Option 3: Recreate StatefulSet/Deployment (Simplest for some)
|
||
|
||
For StatefulSets (Kafka, Redis):
|
||
1. Backup data externally
|
||
2. Delete StatefulSet (with --cascade=orphan to keep pods)
|
||
3. Delete PVCs
|
||
4. Recreate StatefulSet (will create new PVCs with default StorageClass)
|
||
5. Restore data
|
||
|
||
**Downtime:** Yes
|
||
**Risk:** High (data loss if backup fails)
|
||
**Effort:** Variable
|
||
|
||
## ✅ My Strong Recommendation
|
||
|
||
**DO NOTHING.**
|
||
|
||
Here's why:
|
||
1. The storageClassName field in PVC spec is **metadata only** after creation
|
||
2. The actual volume-to-PVC binding is independent
|
||
3. All volumes already have 3 replicas ✅
|
||
4. All applications work perfectly ✅
|
||
5. New PVCs will use `longhorn` automatically ✅
|
||
6. Migration has downtime + risk for ZERO functional benefit
|
||
|
||
**Over time, as you replace/recreate applications, PVCs will naturally migrate to the new StorageClass.**
|
||
|
||
### Natural Migration Path
|
||
|
||
When you eventually need to:
|
||
- Upgrade an application (Helm chart update)
|
||
- Resize a volume
|
||
- Move to a different namespace
|
||
- Rebuild the cluster
|
||
|
||
...THEN recreate the PVC with the correct StorageClass. No rush.
|
||
|
||
## 📋 If You Still Want to Proceed
|
||
|
||
I can create detailed step-by-step migration scripts for each application, but I need confirmation that you understand:
|
||
|
||
1. ⚠️ Downtime required for each application
|
||
2. ⚠️ Risk of data loss if migration fails
|
||
3. ⚠️ 2-3 hours of work for cosmetic benefit only
|
||
4. ✅ Current setup works perfectly as-is
|
||
|
||
**Do you want me to proceed with migration scripts?**
|
||
|