homelab/REVIEW-SUMMARY.md

# GitOps Infrastructure Review - Summary

**Date:** 2025-01-XX  
**Objective:** Review entire k8s GitOps infrastructure, eliminate duplication, create bootstrap-from-local + GitOps-managed-future workflow.

---

## 📌 What Was Done

### 1. **Audit of Current State**

**Findings:**
- ✅ App-of-apps pattern correctly implemented
- ✅ Wave-based deployment (0-8) mostly correct
- ⚠️ **Resource duplication:** `ddb-cluster.yaml` in both bootstrap and GitOps paths
- ⚠️ **Circular dependency:** Forgejo hosts repo → ArgoCD syncs from repo → Forgejo needs DB
- ⚠️ **Manual bootstrap steps:** Scattered, error-prone
- ⚠️ **SOPS secrets:** Some excluded from kustomization (out-of-band)
- ⚠️ **No single source of truth:** Same resources in multiple places

### 2. **Created Bootstrap-Local Bundle**

**New files created:**
```
k8s/bootstrap-local/
├── kustomization.yaml          # Orchestrates bootstrap
├── 00-namespaces.yaml          # All namespaces with PodSecurity labels
├── 01-argocd.yaml              # ArgoCD ConfigMaps
├── 02-cnpg-operator.yaml       # CNPG operator Application
├── 03-ddb-bootstrap.yaml       # PostgreSQL cluster + Forgejo DB + Redis
└── 04-forgejo.yaml             # Forgejo Application (inline values)
```

**What it does:**
- Apply once from local checkout (`kubectl apply -k k8s/bootstrap-local/`)
- Creates ArgoCD, CNPG, DDB, Forgejo (everything needed for GitOps)
- No git dependency (chicken-egg problem solved)
- Idempotent (safe to re-run)

### 3. **Restructured k8s/data/ to Eliminate Duplication**

**Before:**
```
k8s/data/
├── ddb-cluster.yaml           # ❌ Deployed by data-schemas app (wrong)
├── forgejo-database.yaml      # ❌ Deployed by data-schemas app (wrong)
├── authentik-database.yaml
├── temporal-database.yaml
├── ...
└── kustomization.yaml         # Listed ALL resources
```

**After:**
```
k8s/data/
├── cluster/                   # 🔴 Bootstrap-only (not in GitOps)
│   ├── ddb-cluster.yaml
│   ├── forgejo-database.yaml
│   └── kustomization.yaml     # Reference only
└── schemas/                   # 🟢 GitOps-managed (wave 6)
    ├── authentik-database.yaml
    ├── temporal-database.yaml
    ├── schemas.yaml
    ├── db-init-job.yaml
    └── kustomization.yaml
```

**Result:** Each resource has exactly ONE source of truth.

### 4. **Updated ArgoCD Apps to Avoid Duplication**

**Modified:**
- `k8s/argocd/apps/40-data.yaml` → Points to `k8s/data/schemas/` (NOT `k8s/data/`)
- Wave renumbered for clarity (0-8 sequential)

**Deleted:**
- Old `k8s/data/kustomization.yaml` (replaced by subdirectories)

### 5. **DDB Configuration Review**

**Created:** `DDB-REVIEW.md` with:
- Current configuration analysis
- Recommendations (resource limits, connection pooling, backups, monitoring)
- Proposed enhanced configuration
- Migration path

**Key recommendations:**
- Increase `shared_buffers` 256MB → 512MB (multi-tenant workload)
- Add resource limits (CPU/memory)
- Enable PgBouncer pooler (Temporal has high connection count)
- Configure backups to MinIO
- Enable `enablePodMonitor: true` for Prometheus

### 6. **Automated Bootstrap Script**

**Created:** `bootstrap.sh` with:
- Preflight checks (kubectl, SOPS key, ArgoCD CLI)
- ArgoCD installation
- SOPS age secret creation (never in git)
- Bootstrap bundle application
- Wait loops for each component (CNPG, DDB, Forgejo)
- Secret copying (ddb to cicd namespace)
- Clear next-steps instructions

**Usage:**
```bash
./bootstrap.sh
# ✅ ArgoCD + CNPG + DDB + Forgejo ready in <10 minutes
```

### 7. **Comprehensive Documentation**

**Created:**
1. **GITOPS-REBUILD-PLAN.md** (15KB)
   - Current vs proposed architecture diagrams
   - Complete directory structure (single source of truth)
   - Step-by-step execution plan
   - Resource mapping table (no duplication)
   - Testing procedures
   - Day-2 operations guide
   - FAQ

2. **DDB-REVIEW.md** (6KB)
   - PostgreSQL configuration review
   - Performance tuning recommendations
   - Backup/monitoring setup
   - Enhanced configuration example

3. **IMPLEMENTATION-CHECKLIST.md** (9KB)
   - Pre-implementation tasks
   - Two migration paths (fresh cluster vs incremental)
   - Post-implementation verification (wave-by-wave)
   - Cleanup steps
   - Day-2 validation tests
   - Rollback procedures

4. **REVIEW-SUMMARY.md** (this file)
   - High-level overview
   - Key decisions explained
   - What to review next

---

## 🎯 Key Decisions Made

### **Decision 1: DDB Cluster Stays Bootstrap-Only**

**Rationale:**
- Circular dependency: Forgejo needs DDB → ArgoCD syncs from Forgejo → Cannot bootstrap DDB from git
- **Solution:** DDB deployed via `bootstrap-local/03-ddb-bootstrap.yaml`, never touched by ArgoCD
- **Trade-off:** DDB changes require manual `kubectl apply` (but still committed to git for record-keeping)
- **Alternative:** If Forgejo moves to external git host (GitHub), DDB could become GitOps-managed

**ArgoCD Application for DDB:** Not created (would duplicate bootstrap). If needed later (external git), see `GITOPS-REBUILD-PLAN.md` for example.

### **Decision 2: Forgejo Manual-Sync-Only**

**Rationale:**
- Forgejo hosts the repo CI pushes to
- Auto-sync would let a bad CI commit break the system CI depends on
- **Solution:** `syncPolicy.automated: {}` (manual sync only)
- Application exists (`04-forgejo.yaml`) but never auto-syncs

### **Decision 3: Wave Renumbering (0-8 Sequential)**

**Before:** 00, 05, 10, 20, 30, 40, 50, 60 (helmfile convention, gaps for insertion)  
**After:** 0, 1, 2, 3, 4, 5, 6, 7, 8 (ArgoCD native, cleaner)

**Rationale:**
- ArgoCD sync-wave already handles ordering
- No need for gaps (can insert 2.5 if needed, or renumber)
- Easier to read/understand

### **Decision 4: Bootstrap Script Over Manual Steps**

**Before:** 30+ manual commands in BOOTSTRAP.md  
**After:** Single `./bootstrap.sh` script

**Rationale:**
- Reduces human error
- Idempotent (safe to re-run)
- Self-documenting (script IS the procedure)
- Faster iteration (cluster rebuild in <10 min)

### **Decision 5: Separate k8s/data/cluster/ from k8s/data/schemas/**

**Rationale:**
- Clear separation: bootstrap vs GitOps
- Prevents accidental deletion of cluster by ArgoCD prune
- Each directory has its own kustomization.yaml (no ambiguity)
- Easier to reason about dependencies

---

## 📂 Files Created/Modified

### **Created (New Files)**

```
k8s/bootstrap-local/
  kustomization.yaml
  00-namespaces.yaml
  01-argocd.yaml
  02-cnpg-operator.yaml
  03-ddb-bootstrap.yaml
  04-forgejo.yaml

k8s/data/cluster/
  kustomization.yaml

k8s/data/schemas/
  kustomization.yaml

bootstrap.sh
GITOPS-REBUILD-PLAN.md
DDB-REVIEW.md
IMPLEMENTATION-CHECKLIST.md
REVIEW-SUMMARY.md (this file)
```

### **Modified (Updated Files)**

```
k8s/argocd/apps/40-data.yaml
  - Changed path: k8s/data → k8s/data/schemas
  - Changed sync-wave: 4 → 6
  - Updated comments
```

### **Deleted**

```
k8s/data/kustomization.yaml (replaced by subdirectories)
```

### **Moved**

```
k8s/data/ddb-cluster.yaml → k8s/data/cluster/ddb-cluster.yaml
k8s/data/forgejo-database.yaml → k8s/data/cluster/forgejo-database.yaml
k8s/data/authentik-database.yaml → k8s/data/schemas/authentik-database.yaml
k8s/data/temporal-database.yaml → k8s/data/schemas/temporal-database.yaml
k8s/data/temporal-visibility-database.yaml → k8s/data/schemas/temporal-visibility-database.yaml
k8s/data/schemas.yaml → k8s/data/schemas/schemas.yaml
k8s/data/db-init-job.yaml → k8s/data/schemas/db-init-job.yaml
```

---

## 🔍 What You Should Review

### **1. Bootstrap Script**

**File:** `bootstrap.sh`

**Review for:**
- [ ] SOPS key path (`~/.sops/homelab-age.key` correct?)
- [ ] ArgoCD installation method (manifest URL vs Helm?)
- [ ] Wait timeout values (300s, 600s reasonable?)
- [ ] Error handling (should script exit or continue?)

### **2. DDB Configuration**

**File:** `DDB-REVIEW.md`

**Decide:**
- [ ] Accept current config (256MB shared_buffers, no backups)?
- [ ] Implement enhanced config (512MB, PgBouncer, S3 backups)?
- [ ] When to apply changes (now vs after migration)?

**If implementing enhanced config:**
1. Create MinIO bucket `ddb-backups`
2. Create `ddb-backup-s3` secret
3. Update `k8s/bootstrap-local/03-ddb-bootstrap.yaml` with enhanced spec
4. Test on staging cluster first

### **3. Wave Structure**

**Files:** `k8s/argocd/apps/*.yaml`

**Verify:**
- [ ] Wave ordering correct? (0=substrate, 1=networking, ..., 8=apps)
- [ ] Dependencies satisfied? (e.g., schemas after secrets)
- [ ] Sync policies appropriate? (automated vs manual)

**Current wave structure:**
```
Wave 0: cert-manager, ingress-nginx, reloader, CNPG operator
Wave 1: Cilium policies, CoreDNS (networking)
Wave 2: MinIO, Longhorn, Prometheus (storage/observability)
Wave 3: Loki, Grafana, Promtail (logging)
Wave 4: SOPS secrets (all *.enc.yaml)
Wave 5: Vault, Authentik, Forgejo runner (IAM)
Wave 6: Database schemas (authentik-db, temporal-db, etc.)
Wave 7: Kafka, Redis, SQS (messaging)
Wave 8: Temporal, Portainer, cloudflared, etc. (applications)
```

### **4. Namespace Labels**

**File:** `k8s/bootstrap-local/00-namespaces.yaml`

**Verify PodSecurity labels correct:**
- [ ] `cicd` = privileged (Forgejo runner needs DinD)
- [ ] `ingress-nginx` = privileged (hostPort 80/443)
- [ ] `monitoring` = privileged (node-exporter hostPath)
- [ ] `logging` = privileged (promtail hostPath)
- [ ] All others = baseline (default)?

### **5. Forgejo Configuration**

**File:** `k8s/bootstrap-local/04-forgejo.yaml`

**Verify inline values match:**
- [ ] Admin username/email correct?
- [ ] Domain `forgejo.riotpiao.com` correct?
- [ ] LoadBalancer IP `192.168.1.165` available?
- [ ] OAuth2 config matches Authentik setup?
- [ ] Redis connection string correct?

**Sync with:** `k8s/security/ci-cd/forgejo-values.yaml` (keep both files in sync per comment)

### **6. SOPS Secrets**

**File:** `k8s/argocd/apps/04-secrets.yaml` (check if exists)

**Verify:**
- [ ] SOPS plugin configured correctly?
- [ ] All `.enc.yaml` files decrypted successfully?
- [ ] `db-role-secrets.enc.yaml` applied before wave 6?

**Check these secrets exist after bootstrap:**
```bash
kubectl get secret -n ddb authentik-db-role
kubectl get secret -n ddb temporal-db-role
kubectl get secret -n cicd ddb-cluster-app  # Copied from ddb namespace
kubectl get secret -n argocd sops-age
```

---

## 🚀 Next Steps (Recommended Order)

1. **Review all documents** (this file, GITOPS-REBUILD-PLAN.md, DDB-REVIEW.md)
2. **Decide on DDB config** (current vs enhanced)
3. **Review bootstrap.sh** and customize if needed
4. **Test on staging cluster first** (if available)
5. **Backup current production state** (PVCs, secrets)
6. **Choose migration path:**
   - **Option A:** Fresh cluster rebuild (faster, cleaner)
   - **Option B:** Incremental migration (safer, slower)
7. **Follow IMPLEMENTATION-CHECKLIST.md** step-by-step
8. **Validate each wave** before proceeding to next
9. **Document any issues** encountered
10. **Update CLAUDE.md** after successful migration

---

## ⚠️ Important Notes

### **Before You Start**

1. **Backup PVCs** (Forgejo git repos, PostgreSQL data)
   - Longhorn snapshots or Velero backup
   - Export critical data (Forgejo repos, Authentik config)

2. **Test SOPS key** works
   ```bash
   export SOPS_AGE_KEY_FILE=~/.sops/homelab-age.key
   sops -d k8s/data/db-role-secrets.enc.yaml
   # Should decrypt successfully
   ```

3. **Verify Talos cluster healthy**
   ```bash
   talosctl health --nodes <all-nodes>
   kubectl get nodes
   # All Ready
   ```

### **During Implementation**

- **Go wave-by-wave** - Don't skip verification steps
- **Watch ArgoCD UI** - https://argocd.riotpiao.com
- **Check logs** if any app stuck:
  ```bash
  kubectl logs -n argocd deployment/argocd-application-controller
  kubectl logs -n argocd deployment/argocd-repo-server
  ```

### **After Implementation**

- **Test GitOps workflow** (make a change, push, verify auto-sync)
- **Test rollback** (git revert, verify auto-sync)
- **Document any deviations** from plan
- **Update runbooks** based on lessons learned

---

## 📊 Resource Duplication Check (Final)

**Bootstrap-only resources (NOT in ArgoCD GitOps):**
- ArgoCD itself
- CNPG operator (deployed as Application in bootstrap, but manual-managed)
- ddb-cluster
- forgejo-database
- Forgejo (exists as Application but manual-sync-only)
- Forgejo Redis

**GitOps-managed resources (ArgoCD auto-syncs):**
- cert-manager, ingress-nginx, reloader
- Cilium policies, CoreDNS config
- MinIO, Longhorn config, Prometheus
- Loki, Grafana, Promtail
- SOPS secrets
- Vault, Authentik, Forgejo runner
- Database schemas (authentik-db, temporal-db, etc.)
- Kafka, Redis, SQS
- Temporal, Portainer, cloudflared, etc.

**Terraform-managed resources:**
- Talos machine configs (controlplane.tftpl)
- No k8s resources

**✅ No overlap - each resource has exactly ONE source of truth.**

---

## 🎯 Success Criteria

After successful implementation, you should have:

- [x] **Single command bootstrap** (`./bootstrap.sh`)
- [x] **Zero manual kubectl apply** (except bootstrap)
- [x] **Git is source of truth** (all changes via push)
- [x] **No resource duplication**
- [x] **Clear wave ordering** (0-8)
- [x] **Fast iteration** (cluster rebuild <10 min)
- [x] **Rollback via git** (revert commit, auto-syncs)
- [x] **Well-documented** (5 comprehensive docs)

---

## 📞 Questions to Resolve

Before implementation, decide on:

1. **Fresh cluster vs incremental migration?**
   - Fresh = faster, cleaner (requires downtime)
   - Incremental = safer, slower (zero downtime possible)

2. **DDB enhanced config now or later?**
   - Now = better performance from start
   - Later = faster migration, can optimize afterward

3. **Wave renumbering (00→0, 05→1, etc.)?**
   - Yes = cleaner, consistent with plan
   - No = keep current, less churn

4. **Delete old k8s/argocd/bootstrap/ after migration?**
   - Yes = cleaner repo
   - No = keep for reference

5. **Update CLAUDE.md immediately or after validation?**
   - Immediately = stays current
   - After = confirms plan actually works

---

**Ready to proceed?** Start with `IMPLEMENTATION-CHECKLIST.md` and check off each step.

**Need clarification?** Review specific sections in `GITOPS-REBUILD-PLAN.md`.

**Performance tuning?** See `DDB-REVIEW.md` for PostgreSQL optimization.