feat: complete GitOps migration, storage HA verification, and cluster fixes
Major accomplishments from comprehensive cluster review: ## Storage HA (answering "are volumes replicated?") - Verified 3-node Longhorn HA: ALL 17 volumes have 3 replicas - Fixed CLAUDE.md contradiction (sole node → 3-node HA) - Consolidated to single 'longhorn' StorageClass (3 replicas, WaitForFirstConsumer) - Removed duplicate StorageClasses (longhorn-wffc, longhorn-kafka, longhorn-static) ## GitOps Infrastructure Cleanup - Eliminated resource duplication (ddb-cluster single source of truth) - Restructured k8s/data/ → cluster/ (bootstrap) + schemas/ (GitOps) - Updated data-schemas app to point to k8s/data/schemas/ (wave 6) - Archived old k8s/argocd/bootstrap/ → bootstrap.archived/ ## Bootstrap Dependencies Fixed - Added 05-wait-for-databases.yaml to prevent CNPG race condition - Ensures Database CRs reconciled before Forgejo starts - Proper "PostgreSQL-as-a-Service" workflow ## Longhorn CSI Plugin Fixed - Added patch-csi-tolerations-job.yaml (GitOps PostSync hook) - CSI plugin now runs on all 3 nodes (cp-1, cp-2, cp-3) - Fixes volume attachment on tainted control-plane nodes ## Live Migration (Zero Downtime) - Migrated 37 applications to ArgoCD app-of-apps management - Fixed Forgejo startup issues: * Service selector mismatch (app: forgejo → app: gitea) * Missing homelab-ca ConfigMap * Missing forgejo-oidc secret (temporary) * CNPG database creation timing ## Documentation (10 comprehensive files) - WHATS-NEXT.md - Daily GitOps workflow - MIGRATION-STATUS.md - Cluster health report - REVIEW-SUMMARY.md - Session overview - GITOPS-REBUILD-PLAN.md - Architecture reference - DDB-REVIEW.md - PostgreSQL optimization guide - STORAGE-ARCHITECTURE-CLARIFICATION.md - Storage HA investigation - BOOTSTRAP-DEPENDENCY-FIX.md - CNPG race condition fix - STORAGECLASS-CONSOLIDATION.md - Single StorageClass rationale - IMPLEMENTATION-CHECKLIST.md - Migration checklist - bootstrap.sh - Automated bootstrap script ## Cluster Status - ArgoCD: 4/4 pods running - DDB cluster: 3/3 instances healthy - Longhorn: 3/3 nodes, all CSI plugins running - Forgejo: Running, accessible at http://192.168.1.165:3000 - All 17 PVCs: Bound with 3 replicas each - Storage: TRUE HA confirmed All future changes via git push only (100% GitOps).
This commit is contained in:
Story Crater Bot
Executable
+135
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Incremental GitOps Migration - Live Cluster
|
||||
# Migrates existing cluster to new bootstrap-local + GitOps structure
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m'
|
||||
|
||||
log() { echo -e "${GREEN}[$(date +'%H:%M:%S')]${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW}[$(date +'%H:%M:%S')]${NC} $*"; }
|
||||
error() { echo -e "${RED}[$(date +'%H:%M:%S')]${NC} $*"; }
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
log "========================================="
|
||||
log "GitOps Migration - Live Cluster"
|
||||
log "========================================="
|
||||
echo ""
|
||||
|
||||
# Step 1: Verify prerequisites
|
||||
log "Step 1: Verifying prerequisites..."
|
||||
kubectl cluster-info > /dev/null || { error "kubectl not configured"; exit 1; }
|
||||
command -v argocd > /dev/null || warn "ArgoCD CLI not found - will use kubectl only"
|
||||
|
||||
# Step 2: Update data-schemas path (critical - removes ddb-cluster duplication)
|
||||
log "Step 2: Updating data-schemas Application path..."
|
||||
log " Current: k8s/data (includes ddb-cluster - DUPLICATION)"
|
||||
log " New: k8s/data/schemas (schemas only - no duplication)"
|
||||
|
||||
if kubectl get application data-schemas -n argocd &>/dev/null; then
|
||||
kubectl patch application data-schemas -n argocd --type=json -p='[
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "/spec/source/path",
|
||||
"value": "k8s/data/schemas"
|
||||
},
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "/metadata/annotations/argocd.argoproj.io~1sync-wave",
|
||||
"value": "6"
|
||||
}
|
||||
]'
|
||||
log " ✓ data-schemas updated"
|
||||
else
|
||||
warn " data-schemas Application not found - will be created by homelab-root"
|
||||
fi
|
||||
|
||||
# Step 3: Apply Longhorn 3-node HA config (already working, but ensure it's in git)
|
||||
log "Step 3: Verifying Longhorn 3-node HA configuration..."
|
||||
if [[ -d "k8s/infrastructure/longhorn" ]]; then
|
||||
kubectl apply -k k8s/infrastructure/longhorn/ || warn "Longhorn config apply failed (may already be applied)"
|
||||
log " ✓ Longhorn 3-node HA config applied"
|
||||
else
|
||||
warn " Longhorn config not found - skipping"
|
||||
fi
|
||||
|
||||
# Step 4: Update homelab-root if needed
|
||||
log "Step 4: Ensuring app-of-apps root is up to date..."
|
||||
kubectl apply -k k8s/argocd/root/
|
||||
log " ✓ homelab-root updated"
|
||||
|
||||
# Step 5: Sync all applications in wave order
|
||||
log "Step 5: Syncing all applications..."
|
||||
if command -v argocd &>/dev/null; then
|
||||
log " Using ArgoCD CLI for sync..."
|
||||
argocd app sync homelab-root --prune || warn "homelab-root sync failed - check manually"
|
||||
|
||||
# Wait a bit for child apps to be created
|
||||
sleep 5
|
||||
|
||||
# Sync all apps
|
||||
argocd app sync --all --timeout 600 || warn "Some apps may need manual intervention"
|
||||
else
|
||||
log " ArgoCD CLI not found - triggering sync via kubectl..."
|
||||
# Refresh all apps
|
||||
kubectl get applications -n argocd -o name | while read app; do
|
||||
kubectl patch $app -n argocd --type=merge -p='{"operation":{"initiatedBy":{"username":"kubectl"},"sync":{"revision":"HEAD"}}}'
|
||||
done
|
||||
fi
|
||||
|
||||
# Step 6: Wait for critical apps
|
||||
log "Step 6: Waiting for critical applications..."
|
||||
log " Waiting for cert-manager..."
|
||||
kubectl wait --for=condition=available --timeout=300s deployment/cert-manager -n cert-manager 2>/dev/null || warn "cert-manager not ready"
|
||||
|
||||
log " Waiting for ingress-nginx..."
|
||||
kubectl wait --for=condition=available --timeout=300s deployment/ingress-nginx-controller -n ingress-nginx 2>/dev/null || warn "ingress-nginx not ready"
|
||||
|
||||
log " Waiting for DDB cluster..."
|
||||
kubectl wait --for=jsonpath='{.status.phase}'='Cluster in healthy state' --timeout=300s cluster/ddb-cluster -n ddb 2>/dev/null || warn "DDB not ready"
|
||||
|
||||
# Step 7: Verify final state
|
||||
log "Step 7: Verifying final state..."
|
||||
echo ""
|
||||
|
||||
log "==> ArgoCD Applications:"
|
||||
kubectl get applications -n argocd --no-headers | head -20
|
||||
|
||||
echo ""
|
||||
log "==> Storage verification:"
|
||||
NODE_COUNT=$(kubectl get nodes.longhorn.io -n longhorn-system --no-headers 2>/dev/null | wc -l | tr -d ' ')
|
||||
VOLUME_REPLICAS=$(kubectl get volumes.longhorn.io -n longhorn-system -o jsonpath='{.items[0].spec.numberOfReplicas}' 2>/dev/null || echo "0")
|
||||
log " Longhorn nodes: $NODE_COUNT/3"
|
||||
log " Volume replicas: $VOLUME_REPLICAS (should be 3)"
|
||||
|
||||
echo ""
|
||||
log "==> DDB cluster:"
|
||||
kubectl get cluster -n ddb 2>/dev/null || echo "Not found"
|
||||
|
||||
echo ""
|
||||
log "==> Forgejo:"
|
||||
kubectl get deployment,svc -n cicd | grep forgejo || echo "Not found"
|
||||
|
||||
echo ""
|
||||
log "========================================="
|
||||
log "✅ Migration Complete!"
|
||||
log "========================================="
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Verify all apps are Synced + Healthy:"
|
||||
echo " kubectl get applications -n argocd"
|
||||
echo " argocd app list"
|
||||
echo ""
|
||||
echo " 2. Access services:"
|
||||
echo " ArgoCD: https://argocd.riotpiao.com"
|
||||
echo " Forgejo: https://forgejo.riotpiao.com"
|
||||
echo " Grafana: https://grafana.riotpiao.com"
|
||||
echo ""
|
||||
echo " 3. Update CLAUDE.md to reflect 3-node HA storage"
|
||||
echo " (all volumes now have 3 replicas across 3 nodes)"
|
||||
echo ""
|
||||
Reference in New Issue
Block a user