homelab/GITOPS-REBUILD-PLAN.md

GitOps Infrastructure Rebuild Plan

Goal: Bootstrap cluster from local source + GitOps-managed future state with zero resource duplication.

---

📊 Current vs Proposed Architecture

Current Issues ❌

1. Circular dependencies - CNPG operator, ddb-cluster, Forgejo scattered between bootstrap/, data/, argocd/bootstrap/
2. Resource duplication - ddb-cluster.yaml in k8s/data/ deployed by wave 4 app, but should be bootstrap-only
3. Manual bootstrap steps - Scattered across BOOTSTRAP.md, requires copying secrets between namespaces
4. SOPS secrets - db-role-secrets.enc.yaml excluded from kustomization (out-of-band)
5. No single source of truth - Same resources defined in multiple places

Proposed Architecture ✅

┌─────────────────────────────────────────────────────────────┐
│ Phase 0: Local Bootstrap (One-time, k8s/bootstrap-local/)  │
│ ─────────────────────────────────────────────────────────── │
│ • ArgoCD + SOPS plugin                                      │
│ • CloudNativePG operator                                    │
│ • ddb-cluster (PostgreSQL 3 replicas)                       │
│ • Forgejo + dependencies (DB, Redis)                        │
│                                                              │
│ Script: ./bootstrap.sh (automated)                          │
└─────────────────────────────────────────────────────────────┘
                              ↓
                    git push to Forgejo
                              ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 1: GitOps (100% automated, ArgoCD app-of-apps)        │
│ ─────────────────────────────────────────────────────────── │
│ Wave 0: Substrate (cert-manager, nginx, reloader)           │
│ Wave 1: Networking (Cilium policies, CoreDNS)               │
│ Wave 2: Storage/Observability (MinIO, Longhorn, Prometheus) │
│ Wave 3: Logging (Loki, Grafana, Promtail)                   │
│ Wave 4: Secrets (ALL *.enc.yaml via SOPS)                   │
│ Wave 5: IAM (Vault, Authentik, Forgejo runner)              │
│ Wave 6: Data Schemas (DB init, per-app databases)           │
│ Wave 7: Messaging (Kafka, Redis, SQS)                       │
│ Wave 8: Applications (Temporal, Portainer, etc.)            │
│                                                              │
│ All changes: git commit → push → ArgoCD auto-sync           │
└─────────────────────────────────────────────────────────────┘

---

🗂️ Directory Structure (Single Source of Truth)

homelab/
├── bootstrap.sh                        # Automated bootstrap script
├── GITOPS-REBUILD-PLAN.md             # This file
├── DDB-REVIEW.md                      # DDB configuration review
│
├── k8s/
│   ├── bootstrap-local/               # 🔴 Phase 0: Apply once, never touched by ArgoCD
│   │   ├── kustomization.yaml         # Orchestrates 00-04
│   │   ├── 00-namespaces.yaml         # All namespaces with PodSecurity labels
│   │   ├── 01-argocd.yaml             # ArgoCD ConfigMaps
│   │   ├── 02-cnpg-operator.yaml      # CNPG operator Application
│   │   ├── 03-ddb-bootstrap.yaml      # PostgreSQL cluster + Forgejo DB + Redis
│   │   └── 04-forgejo.yaml            # Forgejo Application (inline values)
│   │
│   ├── argocd/
│   │   ├── root/
│   │   │   ├── kustomization.yaml
│   │   │   └── homelab-root.yaml      # 🟢 App-of-apps (Phase 1 entry point)
│   │   ├── projects/
│   │   │   └── homelab-project.yaml
│   │   └── apps/                      # Wave-based applications
│   │       ├── 00-substrate.yaml      # cert-manager, nginx, reloader
│   │       ├── 01-networking.yaml     # RENAMED from 05-networking
│   │       ├── 02-storage-obs.yaml    # RENAMED from 10-storage-observability
│   │       ├── 03-logging.yaml        # RENAMED from 20-logging
│   │       ├── 04-secrets.yaml        # RENAMED from 00-secrets (after logging for Grafana)
│   │       ├── 05-iam.yaml            # RENAMED from 30-security
│   │       ├── 06-data.yaml           # RENAMED from 40-data (schemas ONLY, not cluster)
│   │       ├── 07-messaging.yaml      # RENAMED from 50-messaging
│   │       └── 08-applications.yaml   # RENAMED from 60-applications
│   │
│   ├── data/
│   │   ├── cluster/                   # 🔴 Bootstrap-only (duplicated in bootstrap-local/)
│   │   │   ├── ddb-cluster.yaml       # NOT deployed by ArgoCD
│   │   │   ├── forgejo-database.yaml  # NOT deployed by ArgoCD
│   │   │   └── kustomization.yaml     # Reference only
│   │   └── schemas/                   # 🟢 GitOps-managed (wave 6)
│   │       ├── authentik-database.yaml
│   │       ├── temporal-database.yaml
│   │       ├── temporal-visibility-database.yaml
│   │       ├── schemas.yaml
│   │       ├── db-init-job.yaml
│   │       └── kustomization.yaml
│   │
│   ├── security/
│   │   ├── sops-secrets/
│   │   │   └── (all *.enc.yaml handled by wave 4 SOPS app)
│   │   └── ...
│   │
│   └── [other directories unchanged]
│
└── terraform/                         # Talos configs ONLY (no k8s resources)
    ├── main.tf
    ├── variables.tf
    └── provider.tf

---

🚀 Execution Plan

Step 1: Prepare Local Environment

# 1. Ensure prerequisites
export KUBECONFIG=~/.kube/homelab-config
kubectl cluster-info  # Verify cluster reachable

# 2. Verify SOPS age key exists
ls ~/.sops/homelab-age.key

# 3. Install ArgoCD CLI (if not present)
brew install argocd  # macOS
# or: curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

Step 2: Backup Current State (Safety)

# Export all current resources for rollback
kubectl get applications -n argocd -o yaml > backup-argocd-apps.yaml
kubectl get cluster ddb-cluster -n ddb -o yaml > backup-ddb-cluster.yaml
kubectl get all -n cicd -o yaml > backup-forgejo.yaml

Step 3: Clean Shutdown (Optional - for fresh rebuild)

⚠️ ONLY if doing a complete rebuild. Skip if incrementally migrating.

# Delete all ArgoCD Applications (keeps resources, just removes ArgoCD management)
kubectl delete applications --all -n argocd

# Optionally delete the entire cluster and re-provision (nuclear option)
# talosctl reset --nodes <nodes> --graceful --reboot

Step 4: Run Bootstrap Script

# Execute automated bootstrap (creates ArgoCD, CNPG, DDB, Forgejo)
./bootstrap.sh

# Expected output:
# ✅ ArgoCD installed
# ✅ SOPS age secret created
# ✅ CNPG operator ready
# ✅ DDB cluster healthy
# ✅ Forgejo ready at https://forgejo.riotpiao.com

Step 5: Push to Forgejo

# Add Forgejo remote (if not already added)
git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git

# Push entire repo
git push forgejo main

# Verify via Forgejo UI: https://forgejo.riotpiao.com

Step 6: Deploy App-of-Apps Root

# Apply homelab-project + homelab-root
kubectl apply -f k8s/argocd/projects/homelab-project.yaml
kubectl apply -k k8s/argocd/root

# Sync everything (waves 0-8)
argocd app sync homelab-root --prune

# Watch the sync (Ctrl+C to exit)
watch -n 2 'kubectl get applications -n argocd'

Step 7: Verify All Applications Healthy

# Check ArgoCD Applications
argocd app list

# Expected output (all Synced + Healthy):
# NAME                    CLUSTER                         NAMESPACE       PROJECT  STATUS  HEALTH
# homelab-root            https://kubernetes.default.svc  argocd          homelab  Synced  Healthy
# cert-manager            https://kubernetes.default.svc  cert-manager    homelab  Synced  Healthy
# ingress-nginx           https://kubernetes.default.svc  ingress-nginx   homelab  Synced  Healthy
# data-schemas            https://kubernetes.default.svc  ddb             homelab  Synced  Healthy
# temporal                https://kubernetes.default.svc  temporal        homelab  Synced  Healthy
# ... (all other apps)

# Check all pods
kubectl get pods --all-namespaces | grep -v Running
# (should be empty or only Completed jobs)

---

🔐 DDB Cluster - ArgoCD Application

Status: DDB cluster is bootstrap-only (NOT managed by ArgoCD).

Reason: Circular dependency - Forgejo (which hosts the git repo) depends on ddb-cluster → ddb-cluster cannot be in the git repo Forgejo hosts.

⚠️ CRITICAL: Storage HA Status Unknown

• CLAUDE.md claims single-node storage (cp-1 only) = NO HA
• Longhorn manifests show 3-node config (cp-1, cp-2, cp-3) = HA
• VERIFY BEFORE PROCEEDING: See STORAGE-ARCHITECTURE-CLARIFICATION.md
• If single-node: Deploy k8s/infrastructure/longhorn/ first

Current approach (recommended):

• ddb-cluster deployed via k8s/bootstrap-local/03-ddb-bootstrap.yaml
• Applied once during bootstrap, never touched afterward
• Schema changes (Database CRs) are GitOps-managed via wave 6 data-schemas app

Alternative (if Forgejo is decoupled): If you later move Forgejo to an external git host (GitHub, GitLab), you could create:

# k8s/argocd/apps/03-database.yaml (NOT currently used)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ddb-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/data/cluster
  destination:
    server: https://kubernetes.default.svc
    namespace: ddb
  syncPolicy:
    automated:
      prune: false  # Safety: never auto-delete database
      selfHeal: true

DDB Configuration Review: See DDB-REVIEW.md for:

• Current vs recommended configuration
• Resource limits, connection pooling, backups
• Monitoring and performance tuning

---

📋 Resource Mapping (No Duplication)

Resource	Bootstrap (k8s/bootstrap-local/)	GitOps (ArgoCD apps/)	Terraform
ArgoCD	✅ 01-argocd.yaml	❌ Not managed	❌
CNPG operator	✅ 02-cnpg-operator.yaml (as Application)	❌	❌
ddb-cluster	✅ 03-ddb-bootstrap.yaml	❌	❌
forgejo-database	✅ 03-ddb-bootstrap.yaml	❌	❌
Forgejo	✅ 04-forgejo.yaml	❌ (manual sync only)	❌
Forgejo Redis	✅ 03-ddb-bootstrap.yaml	❌	❌
cert-manager	❌	✅ Wave 0 (00-substrate)	❌
ingress-nginx	❌	✅ Wave 0 (00-substrate)	❌
Prometheus	❌	✅ Wave 2 (02-storage-obs)	❌
Loki	❌	✅ Wave 3 (03-logging)	❌
SOPS secrets	❌	✅ Wave 4 (04-secrets)	❌
Authentik	❌	✅ Wave 5 (05-iam)	❌
authentik-database	❌	✅ Wave 6 (06-data schemas)	❌
temporal-database	❌	✅ Wave 6 (06-data schemas)	❌
Temporal	❌	✅ Wave 8 (08-applications)	❌
Talos configs	❌	❌	✅ terraform/

Rule: Each resource has exactly ONE source of truth. No overlap.

---

🧪 Testing the Rebuild

Dry-Run (Safe)

# 1. Test bootstrap script without actually applying
./bootstrap.sh --dry-run  # (modify script to support this flag)

# 2. Test kustomization builds
kubectl kustomize k8s/bootstrap-local/
kubectl kustomize k8s/data/schemas/

# 3. Test ArgoCD app rendering
argocd app diff homelab-root --local k8s/argocd/apps/

Incremental Migration (Safer than Full Rebuild)

If cluster is currently running, migrate incrementally:

# 1. Add bootstrap-local/ resources alongside existing (won't conflict)
kubectl apply -k k8s/bootstrap-local/ --dry-run=client

# 2. Update ArgoCD apps one wave at a time
kubectl apply -f k8s/argocd/apps/00-substrate.yaml
argocd app sync cert-manager ingress-nginx reloader
# (verify healthy, then proceed to next wave)

# 3. Delete old bootstrap Applications
kubectl delete application cnpg-operator -n argocd  # Now in bootstrap-local
kubectl delete application forgejo -n argocd        # Now manual-sync-only

---

🎯 Success Criteria

✅ Bootstrap completes in <10 minutes (./bootstrap.sh)
✅ All namespaces pre-created with correct PodSecurity labels
✅ ddb-cluster healthy (3/3 replicas)
✅ Forgejo accessible at https://forgejo.riotpiao.com
✅ ArgoCD app-of-apps sync completes (all waves)
✅ All Applications show Synced + Healthy
✅ No duplicate resources (each resource managed by exactly one mechanism)
✅ SOPS secrets decrypted successfully (all *.enc.yaml)
✅ Temporal connects to PostgreSQL (via temporal-database)
✅ Authentik connects to PostgreSQL (via authentik-database)
✅ Future changes via git push only (no manual kubectl apply)

---

🔄 Day-2 Operations (Post-Bootstrap)

Making Changes

# 1. Edit manifests locally
vim k8s/applications/temporal/temporal-values.yaml

# 2. Commit + push
git add -A
git commit -m "fix(temporal): increase replicas to 3"
git push forgejo main

# 3. ArgoCD auto-syncs within 3 minutes (or manual)
argocd app sync temporal

Adding a New Service

# 1. Create manifests
mkdir k8s/applications/myapp
cat > k8s/applications/myapp/kustomization.yaml << EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp
resources:
  - deployment.yaml
  - service.yaml
EOF

# 2. Create ArgoCD Application
cat > k8s/argocd/apps/08-applications.yaml << 'EOF'
# (append to existing file)
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/applications/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

# 3. Push
git add -A && git commit -m "feat(myapp): add new application" && git push
# ArgoCD picks it up automatically

Updating DDB Cluster (Bootstrap Resources)

Since ddb-cluster is bootstrap-only, changes require manual apply:

# 1. Edit k8s/bootstrap-local/03-ddb-bootstrap.yaml
vim k8s/bootstrap-local/03-ddb-bootstrap.yaml

# 2. Apply changes (will patch existing cluster)
kubectl apply -f k8s/bootstrap-local/03-ddb-bootstrap.yaml

# 3. Commit to git (for record-keeping)
git add k8s/bootstrap-local/03-ddb-bootstrap.yaml
git commit -m "chore(ddb): increase shared_buffers to 512MB"
git push

---

📚 Related Documents

• CLAUDE.md - Cluster topology, hard rules, GitOps gotchas
• DDB-REVIEW.md - PostgreSQL configuration review and recommendations
• USAGE.md - (STALE - helmfile era, ignore deployment procedures)
• TROUBLESHOOTING.md - Generic Kubernetes debugging methodology
• k8s/argocd/bootstrap/BOOTSTRAP.md - (OLD - replaced by this plan)

---

❓ FAQ

Q: Why is ddb-cluster not managed by ArgoCD?
A: Circular dependency - Forgejo needs DDB, ArgoCD syncs from Forgejo. Breaking the cycle requires one manual bootstrap step.

Q: Can I move Forgejo to ArgoCD management?
A: Only if you move the git repo to an external host (GitHub, GitLab). The 04-forgejo.yaml Application already exists, just change syncPolicy.automated to enable it.

Q: What if bootstrap.sh fails halfway?
A: Re-run it. All commands are idempotent (kubectl apply, --dry-run=client -o yaml | kubectl apply).

Q: How do I rollback a bad GitOps change?
A: git revert <commit> → git push. ArgoCD auto-syncs the rollback.

Q: Why rename wave numbers (00 → 0, 05 → 1, etc.)?
A: Cleaner sequential naming (0-8). Old helmfile convention used gaps (00, 05, 10, ...) which are unnecessary in pure ArgoCD.

Q: Can I delete k8s/argocd/bootstrap/ after migration?
A: Yes, once bootstrap-local/ is working. Keep BOOTSTRAP.md for historical reference if needed.

---

Last Updated: 2025-01-XX
Status: Ready for implementation