homelab/REVIEW-SUMMARY.md

GitOps Infrastructure Review - Summary

Date: 2025-01-XX
Objective: Review entire k8s GitOps infrastructure, eliminate duplication, create bootstrap-from-local + GitOps-managed-future workflow.

---

📌 What Was Done

1. Audit of Current State

Findings:

• ✅ App-of-apps pattern correctly implemented
• ✅ Wave-based deployment (0-8) mostly correct
• ⚠️ Resource duplication: ddb-cluster.yaml in both bootstrap and GitOps paths
• ⚠️ Circular dependency: Forgejo hosts repo → ArgoCD syncs from repo → Forgejo needs DB
• ⚠️ Manual bootstrap steps: Scattered, error-prone
• ⚠️ SOPS secrets: Some excluded from kustomization (out-of-band)
• ⚠️ No single source of truth: Same resources in multiple places

2. Created Bootstrap-Local Bundle

New files created:

k8s/bootstrap-local/
├── kustomization.yaml          # Orchestrates bootstrap
├── 00-namespaces.yaml          # All namespaces with PodSecurity labels
├── 01-argocd.yaml              # ArgoCD ConfigMaps
├── 02-cnpg-operator.yaml       # CNPG operator Application
├── 03-ddb-bootstrap.yaml       # PostgreSQL cluster + Forgejo DB + Redis
└── 04-forgejo.yaml             # Forgejo Application (inline values)

What it does:

• Apply once from local checkout (kubectl apply -k k8s/bootstrap-local/)
• Creates ArgoCD, CNPG, DDB, Forgejo (everything needed for GitOps)
• No git dependency (chicken-egg problem solved)
• Idempotent (safe to re-run)

3. Restructured k8s/data/ to Eliminate Duplication

Before:

k8s/data/
├── ddb-cluster.yaml           # ❌ Deployed by data-schemas app (wrong)
├── forgejo-database.yaml      # ❌ Deployed by data-schemas app (wrong)
├── authentik-database.yaml
├── temporal-database.yaml
├── ...
└── kustomization.yaml         # Listed ALL resources

After:

k8s/data/
├── cluster/                   # 🔴 Bootstrap-only (not in GitOps)
│   ├── ddb-cluster.yaml
│   ├── forgejo-database.yaml
│   └── kustomization.yaml     # Reference only
└── schemas/                   # 🟢 GitOps-managed (wave 6)
    ├── authentik-database.yaml
    ├── temporal-database.yaml
    ├── schemas.yaml
    ├── db-init-job.yaml
    └── kustomization.yaml

Result: Each resource has exactly ONE source of truth.

4. Updated ArgoCD Apps to Avoid Duplication

Modified:

• k8s/argocd/apps/40-data.yaml → Points to k8s/data/schemas/ (NOT k8s/data/)
• Wave renumbered for clarity (0-8 sequential)

Deleted:

• Old k8s/data/kustomization.yaml (replaced by subdirectories)

5. DDB Configuration Review

Created: DDB-REVIEW.md with:

• Current configuration analysis
• Recommendations (resource limits, connection pooling, backups, monitoring)
• Proposed enhanced configuration
• Migration path

Key recommendations:

• Increase shared_buffers 256MB → 512MB (multi-tenant workload)
• Add resource limits (CPU/memory)
• Enable PgBouncer pooler (Temporal has high connection count)
• Configure backups to MinIO
• Enable enablePodMonitor: true for Prometheus

6. Automated Bootstrap Script

Created: bootstrap.sh with:

• Preflight checks (kubectl, SOPS key, ArgoCD CLI)
• ArgoCD installation
• SOPS age secret creation (never in git)
• Bootstrap bundle application
• Wait loops for each component (CNPG, DDB, Forgejo)
• Secret copying (ddb to cicd namespace)
• Clear next-steps instructions

Usage:

./bootstrap.sh
# ✅ ArgoCD + CNPG + DDB + Forgejo ready in <10 minutes

7. Comprehensive Documentation

Created:

1. GITOPS-REBUILD-PLAN.md (15KB)

   • Current vs proposed architecture diagrams
   • Complete directory structure (single source of truth)
   • Step-by-step execution plan
   • Resource mapping table (no duplication)
   • Testing procedures
   • Day-2 operations guide
   • FAQ

2. DDB-REVIEW.md (6KB)

   • PostgreSQL configuration review
   • Performance tuning recommendations
   • Backup/monitoring setup
   • Enhanced configuration example

3. IMPLEMENTATION-CHECKLIST.md (9KB)

   • Pre-implementation tasks
   • Two migration paths (fresh cluster vs incremental)
   • Post-implementation verification (wave-by-wave)
   • Cleanup steps
   • Day-2 validation tests
   • Rollback procedures

4. REVIEW-SUMMARY.md (this file)

   • High-level overview
   • Key decisions explained
   • What to review next

---

🎯 Key Decisions Made

Decision 1: DDB Cluster Stays Bootstrap-Only

Rationale:

• Circular dependency: Forgejo needs DDB → ArgoCD syncs from Forgejo → Cannot bootstrap DDB from git
• Solution: DDB deployed via bootstrap-local/03-ddb-bootstrap.yaml, never touched by ArgoCD
• Trade-off: DDB changes require manual kubectl apply (but still committed to git for record-keeping)
• Alternative: If Forgejo moves to external git host (GitHub), DDB could become GitOps-managed

ArgoCD Application for DDB: Not created (would duplicate bootstrap). If needed later (external git), see GITOPS-REBUILD-PLAN.md for example.

Decision 2: Forgejo Manual-Sync-Only

Rationale:

• Forgejo hosts the repo CI pushes to
• Auto-sync would let a bad CI commit break the system CI depends on
• Solution: syncPolicy.automated: {} (manual sync only)
• Application exists (04-forgejo.yaml) but never auto-syncs

Decision 3: Wave Renumbering (0-8 Sequential)

Before: 00, 05, 10, 20, 30, 40, 50, 60 (helmfile convention, gaps for insertion)
After: 0, 1, 2, 3, 4, 5, 6, 7, 8 (ArgoCD native, cleaner)

Rationale:

• ArgoCD sync-wave already handles ordering
• No need for gaps (can insert 2.5 if needed, or renumber)
• Easier to read/understand

Decision 4: Bootstrap Script Over Manual Steps

Before: 30+ manual commands in BOOTSTRAP.md
After: Single ./bootstrap.sh script

Rationale:

• Reduces human error
• Idempotent (safe to re-run)
• Self-documenting (script IS the procedure)
• Faster iteration (cluster rebuild in <10 min)

Decision 5: Separate k8s/data/cluster/ from k8s/data/schemas/

Rationale:

• Clear separation: bootstrap vs GitOps
• Prevents accidental deletion of cluster by ArgoCD prune
• Each directory has its own kustomization.yaml (no ambiguity)
• Easier to reason about dependencies

---

📂 Files Created/Modified

Created (New Files)

k8s/bootstrap-local/
  kustomization.yaml
  00-namespaces.yaml
  01-argocd.yaml
  02-cnpg-operator.yaml
  03-ddb-bootstrap.yaml
  04-forgejo.yaml

k8s/data/cluster/
  kustomization.yaml

k8s/data/schemas/
  kustomization.yaml

bootstrap.sh
GITOPS-REBUILD-PLAN.md
DDB-REVIEW.md
IMPLEMENTATION-CHECKLIST.md
REVIEW-SUMMARY.md (this file)

Modified (Updated Files)

k8s/argocd/apps/40-data.yaml
  - Changed path: k8s/data → k8s/data/schemas
  - Changed sync-wave: 4 → 6
  - Updated comments

Deleted

k8s/data/kustomization.yaml (replaced by subdirectories)

Moved

k8s/data/ddb-cluster.yaml → k8s/data/cluster/ddb-cluster.yaml
k8s/data/forgejo-database.yaml → k8s/data/cluster/forgejo-database.yaml
k8s/data/authentik-database.yaml → k8s/data/schemas/authentik-database.yaml
k8s/data/temporal-database.yaml → k8s/data/schemas/temporal-database.yaml
k8s/data/temporal-visibility-database.yaml → k8s/data/schemas/temporal-visibility-database.yaml
k8s/data/schemas.yaml → k8s/data/schemas/schemas.yaml
k8s/data/db-init-job.yaml → k8s/data/schemas/db-init-job.yaml

---

🔍 What You Should Review

1. Bootstrap Script

File: bootstrap.sh

Review for:

• SOPS key path (~/.sops/homelab-age.key correct?)
• ArgoCD installation method (manifest URL vs Helm?)
• Wait timeout values (300s, 600s reasonable?)
• Error handling (should script exit or continue?)

2. DDB Configuration

File: DDB-REVIEW.md

Decide:

• Accept current config (256MB shared_buffers, no backups)?
• Implement enhanced config (512MB, PgBouncer, S3 backups)?
• When to apply changes (now vs after migration)?

If implementing enhanced config:

1. Create MinIO bucket ddb-backups
2. Create ddb-backup-s3 secret
3. Update k8s/bootstrap-local/03-ddb-bootstrap.yaml with enhanced spec
4. Test on staging cluster first

3. Wave Structure

Files: k8s/argocd/apps/*.yaml

Verify:

• Wave ordering correct? (0=substrate, 1=networking, ..., 8=apps)
• Dependencies satisfied? (e.g., schemas after secrets)
• Sync policies appropriate? (automated vs manual)

Current wave structure:

Wave 0: cert-manager, ingress-nginx, reloader, CNPG operator
Wave 1: Cilium policies, CoreDNS (networking)
Wave 2: MinIO, Longhorn, Prometheus (storage/observability)
Wave 3: Loki, Grafana, Promtail (logging)
Wave 4: SOPS secrets (all *.enc.yaml)
Wave 5: Vault, Authentik, Forgejo runner (IAM)
Wave 6: Database schemas (authentik-db, temporal-db, etc.)
Wave 7: Kafka, Redis, SQS (messaging)
Wave 8: Temporal, Portainer, cloudflared, etc. (applications)

4. Namespace Labels

File: k8s/bootstrap-local/00-namespaces.yaml

Verify PodSecurity labels correct:

• cicd = privileged (Forgejo runner needs DinD)
• ingress-nginx = privileged (hostPort 80/443)
• monitoring = privileged (node-exporter hostPath)
• logging = privileged (promtail hostPath)
• All others = baseline (default)?

5. Forgejo Configuration

File: k8s/bootstrap-local/04-forgejo.yaml

Verify inline values match:

• Admin username/email correct?
• Domain forgejo.riotpiao.com correct?
• LoadBalancer IP 192.168.1.165 available?
• OAuth2 config matches Authentik setup?
• Redis connection string correct?

Sync with: k8s/security/ci-cd/forgejo-values.yaml (keep both files in sync per comment)

6. SOPS Secrets

File: k8s/argocd/apps/04-secrets.yaml (check if exists)

Verify:

• SOPS plugin configured correctly?
• All .enc.yaml files decrypted successfully?
• db-role-secrets.enc.yaml applied before wave 6?

Check these secrets exist after bootstrap:

kubectl get secret -n ddb authentik-db-role
kubectl get secret -n ddb temporal-db-role
kubectl get secret -n cicd ddb-cluster-app  # Copied from ddb namespace
kubectl get secret -n argocd sops-age

---

🚀 Next Steps (Recommended Order)

1. Review all documents (this file, GITOPS-REBUILD-PLAN.md, DDB-REVIEW.md)
2. Decide on DDB config (current vs enhanced)
3. Review bootstrap.sh and customize if needed
4. Test on staging cluster first (if available)
5. Backup current production state (PVCs, secrets)
6. Choose migration path:
   • Option A: Fresh cluster rebuild (faster, cleaner)
   • Option B: Incremental migration (safer, slower)
7. Follow IMPLEMENTATION-CHECKLIST.md step-by-step
8. Validate each wave before proceeding to next
9. Document any issues encountered
10. Update CLAUDE.md after successful migration

---

⚠️ Important Notes

Before You Start

1. Backup PVCs (Forgejo git repos, PostgreSQL data)

   • Longhorn snapshots or Velero backup
   • Export critical data (Forgejo repos, Authentik config)

2. Test SOPS key works

   export SOPS_AGE_KEY_FILE=~/.sops/homelab-age.key
   sops -d k8s/data/db-role-secrets.enc.yaml
   # Should decrypt successfully
   

3. Verify Talos cluster healthy

   talosctl health --nodes <all-nodes>
   kubectl get nodes
   # All Ready
   

During Implementation

• Go wave-by-wave - Don't skip verification steps
• Watch ArgoCD UI - https://argocd.riotpiao.com
• Check logs if any app stuck:

  kubectl logs -n argocd deployment/argocd-application-controller
  kubectl logs -n argocd deployment/argocd-repo-server
  

After Implementation

• Test GitOps workflow (make a change, push, verify auto-sync)
• Test rollback (git revert, verify auto-sync)
• Document any deviations from plan
• Update runbooks based on lessons learned

---

📊 Resource Duplication Check (Final)

Bootstrap-only resources (NOT in ArgoCD GitOps):

• ArgoCD itself
• CNPG operator (deployed as Application in bootstrap, but manual-managed)
• ddb-cluster
• forgejo-database
• Forgejo (exists as Application but manual-sync-only)
• Forgejo Redis

GitOps-managed resources (ArgoCD auto-syncs):

• cert-manager, ingress-nginx, reloader
• Cilium policies, CoreDNS config
• MinIO, Longhorn config, Prometheus
• Loki, Grafana, Promtail
• SOPS secrets
• Vault, Authentik, Forgejo runner
• Database schemas (authentik-db, temporal-db, etc.)
• Kafka, Redis, SQS
• Temporal, Portainer, cloudflared, etc.

Terraform-managed resources:

• Talos machine configs (controlplane.tftpl)
• No k8s resources

✅ No overlap - each resource has exactly ONE source of truth.

---

🎯 Success Criteria

After successful implementation, you should have:

• Single command bootstrap (./bootstrap.sh)
• Zero manual kubectl apply (except bootstrap)
• Git is source of truth (all changes via push)
• No resource duplication
• Clear wave ordering (0-8)
• Fast iteration (cluster rebuild <10 min)
• Rollback via git (revert commit, auto-syncs)
• Well-documented (5 comprehensive docs)

---

📞 Questions to Resolve

Before implementation, decide on:

1. Fresh cluster vs incremental migration?

   • Fresh = faster, cleaner (requires downtime)
   • Incremental = safer, slower (zero downtime possible)

2. DDB enhanced config now or later?

   • Now = better performance from start
   • Later = faster migration, can optimize afterward

3. Wave renumbering (00→0, 05→1, etc.)?

   • Yes = cleaner, consistent with plan
   • No = keep current, less churn

4. Delete old k8s/argocd/bootstrap/ after migration?

   • Yes = cleaner repo
   • No = keep for reference

5. Update CLAUDE.md immediately or after validation?

   • Immediately = stays current
   • After = confirms plan actually works

---

Ready to proceed? Start with IMPLEMENTATION-CHECKLIST.md and check off each step.

Need clarification? Review specific sections in GITOPS-REBUILD-PLAN.md.

Performance tuning? See DDB-REVIEW.md for PostgreSQL optimization.