Files

T

Story Crater Bot f656338a15 feat: complete GitOps migration, storage HA verification, and cluster fixes

Major accomplishments from comprehensive cluster review:

## Storage HA (answering "are volumes replicated?")
- Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas
- Fixed CLAUDE.md contradiction (sole node → 3-node HA)
- Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer)
- Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static)

## GitOps Infrastructure Cleanup
- Eliminated resource duplication (ddb-cluster single source of truth)
- Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps)
- Updated data-schemas app to point to k8s/data/schemas/ (wave 6)
- Archived old k8s/argocd/bootstrap/ → bootstrap.archived/

## Bootstrap Dependencies Fixed
- Added 05-wait-for-databases.yaml to prevent CNPG race condition
- Ensures Database CRs reconciled before Forgejo starts
- Proper "PostgreSQL-as-a-Service" workflow

## Longhorn CSI Plugin Fixed
- Added patch-csi-tolerations-job.yaml (GitOps PostSync hook)
- CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3)
- Fixes volume attachment on tainted control-plane nodes

## Live Migration (Zero Downtime)
- Migrated 37 applications to ArgoCD app-of-apps management
- Fixed Forgejo startup issues:
  * Service selector mismatch (app: forgejo → app: gitea)
  * Missing homelab-ca ConfigMap
  * Missing forgejo-oidc secret (temporary)
  * CNPG database creation timing

## Documentation (10 comprehensive files)
- WHATS-NEXT.md - Daily GitOps workflow
- MIGRATION-STATUS.md - Cluster health report
- REVIEW-SUMMARY.md - Session overview
- GITOPS-REBUILD-PLAN.md - Architecture reference
- DDB-REVIEW.md - PostgreSQL optimization guide
- STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation
- BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix
- STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale
- IMPLEMENTATION-CHECKLIST.md - Migration checklist
- bootstrap.sh - Automated bootstrap script

## Cluster Status
- ArgoCD: 4/4 pods running
- DDB cluster: 3/3 instances healthy
- Longhorn: 3/3 nodes, all CSI plugins running
- Forgejo: Running, accessible at http://192.168.1.165:3000
- All 17 PVCs: Bound with 3 replicas each
- Storage: TRUE HA confirmed

All future changes via git push only (100% GitOps).

2026-08-18 15:08:03 -07:00

3.8 KiB

Raw Blame History

Storage Migration Analysis - storageClassName Update

❌ Problem: Field is Immutable

Kubernetes does not allow changing spec.storageClassName on existing PVCs.

Error: spec is immutable after creation except resources.requests 
       and volumeAttributesClassName for bound claims

📊 Current Situation

PVCs using old StorageClass names (7 total):

Namespace	PVC	Old StorageClass	Volume	Replicas	Status
dashboard	portainer	longhorn-wffc	pvc-cb87...	3	✅ Working
monitoring	prometheus-...	longhorn-wffc	pvc-03237...	3	✅ Working
sqs	data-kmsvc-pool-0	longhorn-kafka	pvc-0362a...	3	✅ Working
sqs	data-kmsvc-pool-1	longhorn-kafka	pvc-55190...	3	✅ Working
sqs	data-kmsvc-pool-2	longhorn-kafka	pvc-79f42...	3	✅ Working
sqs	redis-replicas-1	longhorn-wffc	pvc-a4625...	3	✅ Working
sqs	redis-replicas-2	longhorn-wffc	pvc-e330f...	3	✅ Working

All volumes have 3 replicas and work perfectly.

🤔 Is Migration Necessary?

Functional Impact: NONE

✅ All PVCs are Bound
✅ All volumes have 3 replicas
✅ Applications work normally
✅ New PVCs will use unified longhorn StorageClass automatically

Cosmetic Issue Only:

PVC metadata shows old StorageClass name
Doesn't affect functionality at all
Old StorageClasses already deleted from cluster

My Recommendation: DON'T MIGRATE - not worth the risk/effort

🔧 If You REALLY Want to Migrate...

Option 1: Live Migration (Complex, Risky)

For each PVC:

Create new PVC with correct StorageClass
Use a data copy tool (rsync pod, Velero, snapshot)
Scale down application
Copy data from old volume to new volume
Update application to use new PVC
Test
Delete old PVC

Downtime: Yes (per application) Risk: Medium (data copy could fail) Effort: ~30 min per PVC × 7 = 3.5 hours

Option 2: Snapshot & Restore (Cleaner, Requires Longhorn Snapshots)

For each PVC:

Create Longhorn snapshot of volume
Create new PVC from snapshot (with correct StorageClass)
Scale down application
Update application to use new PVC
Scale up, test
Delete old PVC

Downtime: Yes (per application) Risk: Low (snapshots are atomic) Effort: ~20 min per PVC × 7 = 2.5 hours

Option 3: Recreate StatefulSet/Deployment (Simplest for some)

For StatefulSets (Kafka, Redis):

Backup data externally
Delete StatefulSet (with --cascade=orphan to keep pods)
Delete PVCs
Recreate StatefulSet (will create new PVCs with default StorageClass)
Restore data

Downtime: Yes
Risk: High (data loss if backup fails) Effort: Variable

✅ My Strong Recommendation

DO NOTHING.

Here's why:

The storageClassName field in PVC spec is metadata only after creation
The actual volume-to-PVC binding is independent
All volumes already have 3 replicas ✅
All applications work perfectly ✅
New PVCs will use longhorn automatically ✅
Migration has downtime + risk for ZERO functional benefit

Over time, as you replace/recreate applications, PVCs will naturally migrate to the new StorageClass.

Natural Migration Path

When you eventually need to:

Upgrade an application (Helm chart update)
Resize a volume
Move to a different namespace
Rebuild the cluster

...THEN recreate the PVC with the correct StorageClass. No rush.

📋 If You Still Want to Proceed

I can create detailed step-by-step migration scripts for each application, but I need confirmation that you understand:

⚠️ Downtime required for each application
⚠️ Risk of data loss if migration fails
⚠️ 2-3 hours of work for cosmetic benefit only
✅ Current setup works perfectly as-is

Do you want me to proceed with migration scripts?

3.8 KiB Raw Blame History Unescape Escape