homelab/GITOPS-REBUILD-PLAN.md

# GitOps Infrastructure Rebuild Plan

**Goal:** Bootstrap cluster from local source + GitOps-managed future state with zero resource duplication.

---

## 📊 Current vs Proposed Architecture

### Current Issues ❌

1. **Circular dependencies** - CNPG operator, ddb-cluster, Forgejo scattered between bootstrap/, data/, argocd/bootstrap/
2. **Resource duplication** - ddb-cluster.yaml in k8s/data/ deployed by wave 4 app, but should be bootstrap-only
3. **Manual bootstrap steps** - Scattered across BOOTSTRAP.md, requires copying secrets between namespaces
4. **SOPS secrets** - db-role-secrets.enc.yaml excluded from kustomization (out-of-band)
5. **No single source of truth** - Same resources defined in multiple places

### Proposed Architecture ✅

```
┌─────────────────────────────────────────────────────────────┐
│ Phase 0: Local Bootstrap (One-time, k8s/bootstrap-local/)  │
│ ─────────────────────────────────────────────────────────── │
│ • ArgoCD + SOPS plugin                                      │
│ • CloudNativePG operator                                    │
│ • ddb-cluster (PostgreSQL 3 replicas)                       │
│ • Forgejo + dependencies (DB, Redis)                        │
│                                                              │
│ Script: ./bootstrap.sh (automated)                          │
└─────────────────────────────────────────────────────────────┘
                              ↓
                    git push to Forgejo
                              ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 1: GitOps (100% automated, ArgoCD app-of-apps)        │
│ ─────────────────────────────────────────────────────────── │
│ Wave 0: Substrate (cert-manager, nginx, reloader)           │
│ Wave 1: Networking (Cilium policies, CoreDNS)               │
│ Wave 2: Storage/Observability (MinIO, Longhorn, Prometheus) │
│ Wave 3: Logging (Loki, Grafana, Promtail)                   │
│ Wave 4: Secrets (ALL *.enc.yaml via SOPS)                   │
│ Wave 5: IAM (Vault, Authentik, Forgejo runner)              │
│ Wave 6: Data Schemas (DB init, per-app databases)           │
│ Wave 7: Messaging (Kafka, Redis, SQS)                       │
│ Wave 8: Applications (Temporal, Portainer, etc.)            │
│                                                              │
│ All changes: git commit → push → ArgoCD auto-sync           │
└─────────────────────────────────────────────────────────────┘
```

---

## 🗂️ Directory Structure (Single Source of Truth)

```
homelab/
├── bootstrap.sh                        # Automated bootstrap script
├── GITOPS-REBUILD-PLAN.md             # This file
├── DDB-REVIEW.md                      # DDB configuration review
│
├── k8s/
│   ├── bootstrap-local/               # 🔴 Phase 0: Apply once, never touched by ArgoCD
│   │   ├── kustomization.yaml         # Orchestrates 00-04
│   │   ├── 00-namespaces.yaml         # All namespaces with PodSecurity labels
│   │   ├── 01-argocd.yaml             # ArgoCD ConfigMaps
│   │   ├── 02-cnpg-operator.yaml      # CNPG operator Application
│   │   ├── 03-ddb-bootstrap.yaml      # PostgreSQL cluster + Forgejo DB + Redis
│   │   └── 04-forgejo.yaml            # Forgejo Application (inline values)
│   │
│   ├── argocd/
│   │   ├── root/
│   │   │   ├── kustomization.yaml
│   │   │   └── homelab-root.yaml      # 🟢 App-of-apps (Phase 1 entry point)
│   │   ├── projects/
│   │   │   └── homelab-project.yaml
│   │   └── apps/                      # Wave-based applications
│   │       ├── 00-substrate.yaml      # cert-manager, nginx, reloader
│   │       ├── 01-networking.yaml     # RENAMED from 05-networking
│   │       ├── 02-storage-obs.yaml    # RENAMED from 10-storage-observability
│   │       ├── 03-logging.yaml        # RENAMED from 20-logging
│   │       ├── 04-secrets.yaml        # RENAMED from 00-secrets (after logging for Grafana)
│   │       ├── 05-iam.yaml            # RENAMED from 30-security
│   │       ├── 06-data.yaml           # RENAMED from 40-data (schemas ONLY, not cluster)
│   │       ├── 07-messaging.yaml      # RENAMED from 50-messaging
│   │       └── 08-applications.yaml   # RENAMED from 60-applications
│   │
│   ├── data/
│   │   ├── cluster/                   # 🔴 Bootstrap-only (duplicated in bootstrap-local/)
│   │   │   ├── ddb-cluster.yaml       # NOT deployed by ArgoCD
│   │   │   ├── forgejo-database.yaml  # NOT deployed by ArgoCD
│   │   │   └── kustomization.yaml     # Reference only
│   │   └── schemas/                   # 🟢 GitOps-managed (wave 6)
│   │       ├── authentik-database.yaml
│   │       ├── temporal-database.yaml
│   │       ├── temporal-visibility-database.yaml
│   │       ├── schemas.yaml
│   │       ├── db-init-job.yaml
│   │       └── kustomization.yaml
│   │
│   ├── security/
│   │   ├── sops-secrets/
│   │   │   └── (all *.enc.yaml handled by wave 4 SOPS app)
│   │   └── ...
│   │
│   └── [other directories unchanged]
│
└── terraform/                         # Talos configs ONLY (no k8s resources)
    ├── main.tf
    ├── variables.tf
    └── provider.tf
```

---

## 🚀 Execution Plan

### **Step 1: Prepare Local Environment**

```bash
# 1. Ensure prerequisites
export KUBECONFIG=~/.kube/homelab-config
kubectl cluster-info  # Verify cluster reachable

# 2. Verify SOPS age key exists
ls ~/.sops/homelab-age.key

# 3. Install ArgoCD CLI (if not present)
brew install argocd  # macOS
# or: curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
```

### **Step 2: Backup Current State (Safety)**

```bash
# Export all current resources for rollback
kubectl get applications -n argocd -o yaml > backup-argocd-apps.yaml
kubectl get cluster ddb-cluster -n ddb -o yaml > backup-ddb-cluster.yaml
kubectl get all -n cicd -o yaml > backup-forgejo.yaml
```

### **Step 3: Clean Shutdown (Optional - for fresh rebuild)**

⚠️ **ONLY if doing a complete rebuild**. Skip if incrementally migrating.

```bash
# Delete all ArgoCD Applications (keeps resources, just removes ArgoCD management)
kubectl delete applications --all -n argocd

# Optionally delete the entire cluster and re-provision (nuclear option)
# talosctl reset --nodes <nodes> --graceful --reboot
```

### **Step 4: Run Bootstrap Script**

```bash
# Execute automated bootstrap (creates ArgoCD, CNPG, DDB, Forgejo)
./bootstrap.sh

# Expected output:
# ✅ ArgoCD installed
# ✅ SOPS age secret created
# ✅ CNPG operator ready
# ✅ DDB cluster healthy
# ✅ Forgejo ready at https://forgejo.riotpiao.com
```

### **Step 5: Push to Forgejo**

```bash
# Add Forgejo remote (if not already added)
git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git

# Push entire repo
git push forgejo main

# Verify via Forgejo UI: https://forgejo.riotpiao.com
```

### **Step 6: Deploy App-of-Apps Root**

```bash
# Apply homelab-project + homelab-root
kubectl apply -f k8s/argocd/projects/homelab-project.yaml
kubectl apply -k k8s/argocd/root

# Sync everything (waves 0-8)
argocd app sync homelab-root --prune

# Watch the sync (Ctrl+C to exit)
watch -n 2 'kubectl get applications -n argocd'
```

### **Step 7: Verify All Applications Healthy**

```bash
# Check ArgoCD Applications
argocd app list

# Expected output (all Synced + Healthy):
# NAME                    CLUSTER                         NAMESPACE       PROJECT  STATUS  HEALTH
# homelab-root            https://kubernetes.default.svc  argocd          homelab  Synced  Healthy
# cert-manager            https://kubernetes.default.svc  cert-manager    homelab  Synced  Healthy
# ingress-nginx           https://kubernetes.default.svc  ingress-nginx   homelab  Synced  Healthy
# data-schemas            https://kubernetes.default.svc  ddb             homelab  Synced  Healthy
# temporal                https://kubernetes.default.svc  temporal        homelab  Synced  Healthy
# ... (all other apps)

# Check all pods
kubectl get pods --all-namespaces | grep -v Running
# (should be empty or only Completed jobs)
```

---

## 🔐 DDB Cluster - ArgoCD Application

**Status:** DDB cluster is **bootstrap-only** (NOT managed by ArgoCD).

**Reason:** Circular dependency - Forgejo (which hosts the git repo) depends on ddb-cluster → ddb-cluster cannot be in the git repo Forgejo hosts.

⚠️ **CRITICAL: Storage HA Status Unknown**
- CLAUDE.md claims single-node storage (cp-1 only) = NO HA
- Longhorn manifests show 3-node config (cp-1, cp-2, cp-3) = HA
- **VERIFY BEFORE PROCEEDING:** See `STORAGE-ARCHITECTURE-CLARIFICATION.md`
- If single-node: Deploy `k8s/infrastructure/longhorn/` first

**Current approach (recommended):**
- ddb-cluster deployed via `k8s/bootstrap-local/03-ddb-bootstrap.yaml`
- Applied once during bootstrap, never touched afterward
- Schema changes (Database CRs) are GitOps-managed via wave 6 `data-schemas` app

**Alternative (if Forgejo is decoupled):**
If you later move Forgejo to an external git host (GitHub, GitLab), you could create:

```yaml
# k8s/argocd/apps/03-database.yaml (NOT currently used)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ddb-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/data/cluster
  destination:
    server: https://kubernetes.default.svc
    namespace: ddb
  syncPolicy:
    automated:
      prune: false  # Safety: never auto-delete database
      selfHeal: true
```

**DDB Configuration Review:** See `DDB-REVIEW.md` for:
- Current vs recommended configuration
- Resource limits, connection pooling, backups
- Monitoring and performance tuning

---

## 📋 Resource Mapping (No Duplication)

| Resource | Bootstrap (k8s/bootstrap-local/) | GitOps (ArgoCD apps/) | Terraform |
|----------|----------------------------------|-----------------------|-----------|
| ArgoCD | ✅ 01-argocd.yaml | ❌ Not managed | ❌ |
| CNPG operator | ✅ 02-cnpg-operator.yaml (as Application) | ❌ | ❌ |
| ddb-cluster | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| forgejo-database | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| Forgejo | ✅ 04-forgejo.yaml | ❌ (manual sync only) | ❌ |
| Forgejo Redis | ✅ 03-ddb-bootstrap.yaml | ❌ | ❌ |
| cert-manager | ❌ | ✅ Wave 0 (00-substrate) | ❌ |
| ingress-nginx | ❌ | ✅ Wave 0 (00-substrate) | ❌ |
| Prometheus | ❌ | ✅ Wave 2 (02-storage-obs) | ❌ |
| Loki | ❌ | ✅ Wave 3 (03-logging) | ❌ |
| SOPS secrets | ❌ | ✅ Wave 4 (04-secrets) | ❌ |
| Authentik | ❌ | ✅ Wave 5 (05-iam) | ❌ |
| authentik-database | ❌ | ✅ Wave 6 (06-data schemas) | ❌ |
| temporal-database | ❌ | ✅ Wave 6 (06-data schemas) | ❌ |
| Temporal | ❌ | ✅ Wave 8 (08-applications) | ❌ |
| Talos configs | ❌ | ❌ | ✅ terraform/ |

**Rule:** Each resource has exactly ONE source of truth. No overlap.

---

## 🧪 Testing the Rebuild

### **Dry-Run (Safe)**

```bash
# 1. Test bootstrap script without actually applying
./bootstrap.sh --dry-run  # (modify script to support this flag)

# 2. Test kustomization builds
kubectl kustomize k8s/bootstrap-local/
kubectl kustomize k8s/data/schemas/

# 3. Test ArgoCD app rendering
argocd app diff homelab-root --local k8s/argocd/apps/
```

### **Incremental Migration (Safer than Full Rebuild)**

If cluster is currently running, migrate incrementally:

```bash
# 1. Add bootstrap-local/ resources alongside existing (won't conflict)
kubectl apply -k k8s/bootstrap-local/ --dry-run=client

# 2. Update ArgoCD apps one wave at a time
kubectl apply -f k8s/argocd/apps/00-substrate.yaml
argocd app sync cert-manager ingress-nginx reloader
# (verify healthy, then proceed to next wave)

# 3. Delete old bootstrap Applications
kubectl delete application cnpg-operator -n argocd  # Now in bootstrap-local
kubectl delete application forgejo -n argocd        # Now manual-sync-only
```

---

## 🎯 Success Criteria

✅ **Bootstrap completes in <10 minutes** (`./bootstrap.sh`)  
✅ **All namespaces pre-created with correct PodSecurity labels**  
✅ **ddb-cluster healthy (3/3 replicas)**  
✅ **Forgejo accessible at https://forgejo.riotpiao.com**  
✅ **ArgoCD app-of-apps sync completes (all waves)**  
✅ **All Applications show Synced + Healthy**  
✅ **No duplicate resources** (each resource managed by exactly one mechanism)  
✅ **SOPS secrets decrypted successfully** (all *.enc.yaml)  
✅ **Temporal connects to PostgreSQL** (via temporal-database)  
✅ **Authentik connects to PostgreSQL** (via authentik-database)  
✅ **Future changes via git push only** (no manual kubectl apply)  

---

## 🔄 Day-2 Operations (Post-Bootstrap)

### **Making Changes**

```bash
# 1. Edit manifests locally
vim k8s/applications/temporal/temporal-values.yaml

# 2. Commit + push
git add -A
git commit -m "fix(temporal): increase replicas to 3"
git push forgejo main

# 3. ArgoCD auto-syncs within 3 minutes (or manual)
argocd app sync temporal
```

### **Adding a New Service**

```bash
# 1. Create manifests
mkdir k8s/applications/myapp
cat > k8s/applications/myapp/kustomization.yaml << EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
resources:
  - deployment.yaml
  - service.yaml
EOF

# 2. Create ArgoCD Application
cat > k8s/argocd/apps/08-applications.yaml << 'EOF'
# (append to existing file)
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/applications/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

# 3. Push
git add -A && git commit -m "feat(myapp): add new application" && git push
# ArgoCD picks it up automatically
```

### **Updating DDB Cluster (Bootstrap Resources)**

Since ddb-cluster is bootstrap-only, changes require manual apply:

```bash
# 1. Edit k8s/bootstrap-local/03-ddb-bootstrap.yaml
vim k8s/bootstrap-local/03-ddb-bootstrap.yaml

# 2. Apply changes (will patch existing cluster)
kubectl apply -f k8s/bootstrap-local/03-ddb-bootstrap.yaml

# 3. Commit to git (for record-keeping)
git add k8s/bootstrap-local/03-ddb-bootstrap.yaml
git commit -m "chore(ddb): increase shared_buffers to 512MB"
git push
```

---

## 📚 Related Documents

- **CLAUDE.md** - Cluster topology, hard rules, GitOps gotchas
- **DDB-REVIEW.md** - PostgreSQL configuration review and recommendations
- **USAGE.md** - (STALE - helmfile era, ignore deployment procedures)
- **TROUBLESHOOTING.md** - Generic Kubernetes debugging methodology
- **k8s/argocd/bootstrap/BOOTSTRAP.md** - (OLD - replaced by this plan)

---

## ❓ FAQ

**Q: Why is ddb-cluster not managed by ArgoCD?**  
A: Circular dependency - Forgejo needs DDB, ArgoCD syncs from Forgejo. Breaking the cycle requires one manual bootstrap step.

**Q: Can I move Forgejo to ArgoCD management?**  
A: Only if you move the git repo to an external host (GitHub, GitLab). The `04-forgejo.yaml` Application already exists, just change `syncPolicy.automated` to enable it.

**Q: What if bootstrap.sh fails halfway?**  
A: Re-run it. All commands are idempotent (`kubectl apply`, `--dry-run=client -o yaml | kubectl apply`).

**Q: How do I rollback a bad GitOps change?**  
A: `git revert <commit>` → `git push`. ArgoCD auto-syncs the rollback.

**Q: Why rename wave numbers (00 → 0, 05 → 1, etc.)?**  
A: Cleaner sequential naming (0-8). Old helmfile convention used gaps (00, 05, 10, ...) which are unnecessary in pure ArgoCD.

**Q: Can I delete k8s/argocd/bootstrap/ after migration?**  
A: Yes, once bootstrap-local/ is working. Keep BOOTSTRAP.md for historical reference if needed.

---

**Last Updated:** 2025-01-XX  
**Status:** Ready for implementation