homelab/IMPLEMENTATION-CHECKLIST.md

# GitOps Rebuild Implementation Checklist

Use this checklist to track the rebuild implementation step-by-step.

---

## 📋 Pre-Implementation

- [ ] **Backup current state**
  ```bash
  kubectl get applications -n argocd -o yaml > backup-argocd-apps.yaml
  kubectl get cluster ddb-cluster -n ddb -o yaml > backup-ddb-cluster.yaml
  kubectl get all -n cicd -o yaml > backup-forgejo.yaml
  kubectl get all -n ddb -o yaml > backup-ddb.yaml
  ```

- [ ] **Verify prerequisites**
  - [ ] kubectl configured (`kubectl cluster-info`)
  - [ ] SOPS age key exists (`~/.sops/homelab-age.key`)
  - [ ] ArgoCD CLI installed (`argocd version`)
  - [ ] Git configured with Forgejo credentials

- [ ] **⚠️ CRITICAL: Verify storage replication status**
  ```bash
  # Check Longhorn nodes (should show 3 if HA is active)
  kubectl get nodes.longhorn.io -n longhorn-system
  
  # Check replica counts
  kubectl get volumes.longhorn.io -n longhorn-system \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.numberOfReplicas}{"\n"}{end}'
  ```
  - [ ] If single-node (1 replica): Deploy `k8s/infrastructure/longhorn/` first
  - [ ] If 3-node (3 replicas): Update CLAUDE.md to reflect HA status
  - [ ] See `STORAGE-ARCHITECTURE-CLARIFICATION.md` for details

- [ ] **Review documentation**
  - [ ] Read `GITOPS-REBUILD-PLAN.md` completely
  - [ ] Review `DDB-REVIEW.md` for database configuration
  - [ ] Understand wave structure (0-8)

---

## 🏗️ Implementation (Choose One Path)

### **Option A: Fresh Cluster (Recommended)**

- [ ] **Provision Talos cluster**
  ```bash
  cd terraform/
  terraform apply
  # Wait for cluster to be ready
  ```

- [ ] **Run bootstrap script**
  ```bash
  ./bootstrap.sh
  # Verify all steps complete successfully
  ```

- [ ] **Push to Forgejo**
  ```bash
  git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git
  git push forgejo main
  ```

- [ ] **Deploy app-of-apps**
  ```bash
  kubectl apply -f k8s/argocd/projects/homelab-project.yaml
  kubectl apply -k k8s/argocd/root
  argocd app sync homelab-root --prune
  ```

### **Option B: Incremental Migration (Existing Cluster)**

- [ ] **Apply bootstrap-local resources alongside existing**
  ```bash
  # Dry-run first
  kubectl apply -k k8s/bootstrap-local/ --dry-run=client
  
  # Actually apply (creates bootstrap Applications)
  kubectl apply -k k8s/bootstrap-local/
  ```

- [ ] **Update ArgoCD apps one wave at a time**
  ```bash
  # Wave 0
  kubectl apply -f k8s/argocd/apps/00-substrate.yaml
  argocd app sync cert-manager ingress-nginx reloader
  
  # Verify healthy, then proceed to wave 1, 2, 3, etc.
  ```

- [ ] **Update data-schemas path**
  ```bash
  # Edit 40-data.yaml (already done in this plan)
  kubectl apply -f k8s/argocd/apps/40-data.yaml
  argocd app sync data-schemas
  ```

- [ ] **Delete old bootstrap Applications**
  ```bash
  # These are now in bootstrap-local/
  kubectl delete application cnpg-operator -n argocd
  # Note: Keep forgejo as manual-sync-only
  ```

---

## ✅ Post-Implementation Verification

### **Wave 0-2: Infrastructure**

- [ ] **cert-manager**
  ```bash
  argocd app get cert-manager
  kubectl get clusterissuers -A
  # Verify: letsencrypt-staging, letsencrypt-prod
  ```

- [ ] **ingress-nginx**
  ```bash
  kubectl get pods -n ingress-nginx
  kubectl get svc ingress-nginx-controller -n ingress-nginx
  # Verify: LoadBalancer IP assigned (192.168.1.160)
  ```

- [ ] **Prometheus**
  ```bash
  kubectl get pods -n monitoring
  kubectl get servicemonitors -A
  # Verify: prometheus, grafana, alertmanager running
  ```

### **Wave 3-5: Logging, Secrets, IAM**

- [ ] **Loki/Grafana**
  ```bash
  kubectl get pods -n logging
  # Access: https://grafana.riotpiao.com
  ```

- [ ] **SOPS secrets**
  ```bash
  argocd app get sops-secrets
  kubectl get secrets -n ddb | grep db-role
  # Verify: authentik-db-role, temporal-db-role exist
  ```

- [ ] **Vault**
  ```bash
  kubectl get pods -n iam
  kubectl exec -n iam vault-0 -- vault status
  ```

- [ ] **Authentik**
  ```bash
  kubectl get pods -n iam
  # Access: https://authentik.riotpiao.com
  ```

### **Wave 6-8: Data, Messaging, Applications**

- [ ] **Data schemas**
  ```bash
  kubectl get databases -n ddb
  # Verify: forgejo, authentik, temporal, temporal_visibility
  kubectl get jobs -n ddb
  # Verify: db-init-job Completed
  ```

- [ ] **Kafka/SQS**
  ```bash
  kubectl get pods -n sqs
  kubectl get kafkas -n sqs
  ```

- [ ] **Temporal**
  ```bash
  kubectl get pods -n temporal
  kubectl logs -n temporal deployment/temporal-frontend -f
  # Verify: Connected to PostgreSQL
  # Access: https://temporal.riotpiao.com
  ```

- [ ] **Cloudflared**
  ```bash
  kubectl get pods -n cloudflared
  kubectl logs -n cloudflared deployment/cloudflared
  # Verify: Tunnel connected
  ```

### **Overall Health**

- [ ] **All Applications Synced**
  ```bash
  argocd app list
  # Verify: All STATUS=Synced, HEALTH=Healthy
  ```

- [ ] **No stuck pods**
  ```bash
  kubectl get pods --all-namespaces | grep -vE 'Running|Completed'
  # (Should be empty)
  ```

- [ ] **All Ingresses accessible**
  ```bash
  kubectl get ingress -A
  # Test each URL in browser
  ```

- [ ] **PostgreSQL connections**
  ```bash
  # Forgejo
  kubectl exec -n cicd deployment/forgejo -- psql -h ddb-cluster-rw.ddb.svc -U app -d forgejo -c '\conninfo'
  
  # Authentik
  kubectl exec -n iam deployment/authentik-server -- python manage.py check --database default
  
  # Temporal
  kubectl exec -n temporal deployment/temporal-frontend -- tctl --db_engine postgres cluster health
  ```

---

## 🧹 Cleanup (After Successful Migration)

- [ ] **Remove old bootstrap files (optional)**
  ```bash
  # Move to archive/
  mkdir -p archive/
  mv k8s/argocd/bootstrap/ archive/old-bootstrap/
  mv USAGE.md project-usage/ archive/old-helmfile-docs/
  ```

- [ ] **Update CLAUDE.md**
  ```bash
  # Remove references to Phase 0 manual steps
  # Update to point to GITOPS-REBUILD-PLAN.md
  ```

- [ ] **Commit cleanup**
  ```bash
  git add -A
  git commit -m "chore: migrate to bootstrap-local + GitOps structure"
  git push forgejo main
  ```

---

## 🔄 Day-2 Validation

- [ ] **Test GitOps workflow**
  ```bash
  # Make a simple change
  echo "# Test comment" >> k8s/applications/temporal/temporal-values.yaml
  git commit -am "test: validate GitOps workflow"
  git push
  
  # Watch ArgoCD auto-sync
  watch -n 2 'argocd app get temporal | grep -A 3 "Sync Status"'
  ```

- [ ] **Test rollback**
  ```bash
  git revert HEAD
  git push
  # Verify ArgoCD auto-syncs the rollback
  ```

- [ ] **Test adding new application**
  ```bash
  # Create minimal app
  mkdir k8s/applications/test-app
  cat > k8s/applications/test-app/deployment.yaml << 'EOF'
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-test
    namespace: default
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx-test
    template:
      metadata:
        labels:
          app: nginx-test
      spec:
        containers:
        - name: nginx
          image: nginx:alpine
          ports:
          - containerPort: 80
  EOF
  
  # Add to ArgoCD
  cat >> k8s/argocd/apps/08-applications.yaml << 'EOF'
  ---
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: test-app
    namespace: argocd
    annotations:
      argocd.argoproj.io/sync-wave: "8"
  spec:
    project: homelab
    source:
      repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
      targetRevision: main
      path: k8s/applications/test-app
    destination:
      server: https://kubernetes.default.svc
      namespace: default
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
  EOF
  
  # Push and verify
  git add -A && git commit -m "test: add test application" && git push
  argocd app get test-app
  kubectl get deployment nginx-test
  
  # Cleanup
  kubectl delete application test-app -n argocd
  git revert HEAD && git push
  ```

---

## 📊 Monitoring & Alerts

- [ ] **Configure Prometheus alerts**
  - [ ] ArgoCD sync failures
  - [ ] PostgreSQL replication lag
  - [ ] PVC usage >80%

- [ ] **Set up Grafana dashboards**
  - [ ] ArgoCD overview
  - [ ] PostgreSQL performance
  - [ ] Ingress traffic

- [ ] **Document runbooks**
  - [ ] DDB cluster recovery (CLAUDE.md)
  - [ ] Forgejo outage (breaks GitOps)
  - [ ] ArgoCD degradation

---

## 🎯 Success Metrics

- [x] **Bootstrap time:** <10 minutes
- [x] **Zero manual kubectl apply** (except bootstrap.sh)
- [x] **No duplicate resources**
- [x] **All apps Synced + Healthy**
- [x] **Git is single source of truth**
- [x] **Rollbacks via git revert only**

---

## 🆘 Rollback Plan (If Things Go Wrong)

### **Rollback to Old State**

```bash
# 1. Restore ArgoCD Applications
kubectl apply -f backup-argocd-apps.yaml

# 2. Restore DDB cluster (if modified)
kubectl apply -f backup-ddb-cluster.yaml

# 3. Restore Forgejo
kubectl apply -f backup-forgejo.yaml

# 4. Sync all apps to last known good commit
argocd app sync --all --revision <last-good-commit-sha>
```

### **Nuclear Option (Full Cluster Rebuild)**

```bash
# 1. Export all PVCs data (Forgejo git repos, PostgreSQL data)
# (Manual backup via Longhorn UI or velero)

# 2. Destroy cluster
cd terraform/
terraform destroy

# 3. Re-provision from scratch
terraform apply
./bootstrap.sh
# Restore PVC data
```

---

**Notes:**
- Check off items as you complete them
- Add timestamps/notes for each major step
- Keep this checklist updated as you encounter issues