refactor(ci-cd): Replace Terraform pipeline with GitOps validation and ArgoCD sync
Delete old terraform-apply.yml (terraform fmt/init/validate/plan/apply). Create new GitOps CI/CD: - validate-k8s.yaml: YAML lint, kubeval, kustomize build, ArgoCD validation - argocd-sync.yaml: Auto-sync homelab-root on main branch - security-scan.yaml: Trivy, Polaris, secret detection - .yamllint.yaml: YAML linting configuration Add documentation (.forgejo/CI-CD.md) and architecture guides. Git is now single source of truth. CI validates, ArgoCD deploys.
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,246 @@
|
||||
# GitOps IaC Implementation Summary
|
||||
|
||||
## Documents Created
|
||||
|
||||
1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design
|
||||
2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive)
|
||||
|
||||
## Architecture at a Glance
|
||||
|
||||
```
|
||||
Pure GitOps (ArgoCD only, no Terraform)
|
||||
├── Layer 0: Infrastructure (namespaces, storage, RBAC)
|
||||
├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
||||
├── Layer 2: Platform (minio, longhorn, loki, prometheus)
|
||||
├── Layer 3: Security (authentik, vault)
|
||||
├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm)
|
||||
└── Layer 5: Data (postgres, redis, kafka)
|
||||
```
|
||||
|
||||
**Key benefit:** Each layer depends on previous, staged deployment, easy rollback.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── _base/ # Shared kustomization base
|
||||
├── infrastructure/ # Layer 0 (foundation)
|
||||
├── bootstrap/ # Layer 1 (networking)
|
||||
├── platform/ # Layer 2 (services)
|
||||
├── security/ # Layer 3 (auth)
|
||||
├── applications/ # Layer 4 (business logic)
|
||||
├── data/ # Layer 5 (stateful)
|
||||
└── argocd/
|
||||
├── projects/ # AppProject definitions
|
||||
└── apps/
|
||||
├── root-app.yaml # Bootstraps Layer 0
|
||||
└── layer-*.yaml # 5 layer apps (synced in order)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Plan (Week 1)
|
||||
- [x] Review GITOPS_ARCHITECTURE.md
|
||||
- [x] Review GITOPS_MIGRATION_PLAN.md
|
||||
- [ ] Audit current cluster resources
|
||||
- [ ] Identify secrets needing encryption (SOPS)
|
||||
- [ ] Map dependencies (provided in plan)
|
||||
- **User approval required before proceeding to Phase 2**
|
||||
|
||||
### Phase 2: Build (Weeks 2-3)
|
||||
- [ ] Create k8s/ directory structure
|
||||
- [ ] Migrate Layer 0 (infrastructure)
|
||||
- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx)
|
||||
- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus)
|
||||
- [ ] Migrate Layer 3 (security: authentik, vault)
|
||||
- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.)
|
||||
- [ ] Migrate Layer 5 (data: postgres, redis, kafka)
|
||||
|
||||
### Phase 3: ArgoCD (Week 4)
|
||||
- [ ] Create root application (Layer 0 trigger)
|
||||
- [ ] Create 5 layer applications (with sync-wave ordering)
|
||||
- [ ] Update AppProject permissions
|
||||
- [ ] Test dry-run on each layer
|
||||
|
||||
### Phase 4: Deploy (Week 5)
|
||||
- [ ] Pre-flight validation (kustomize build, kubeval, yamllint)
|
||||
- [ ] Deploy root application
|
||||
- [ ] Deploy layer applications (one by one)
|
||||
- [ ] Monitor for drift
|
||||
- [ ] Verify state correction (manual edits reverted)
|
||||
|
||||
### Phase 5: Cleanup (Week 6)
|
||||
- [ ] Delete old k8s/ directories
|
||||
- [ ] Update CI/CD pipeline (remove terraform, add kustomize)
|
||||
- [ ] Archive legacy configuration
|
||||
- [ ] Final validation
|
||||
|
||||
## Key Architecture Decisions
|
||||
|
||||
### ✓ Why Layers?
|
||||
- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds
|
||||
- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only.
|
||||
- **Easy scaling:** Add new service without touching others
|
||||
- **Easy rollback:** `git revert <layer-commit>`
|
||||
|
||||
### ✓ Why Kustomize?
|
||||
- Helm chart package + values override (clean separation)
|
||||
- Kustomize build validates before sync
|
||||
- Patches enable per-environment customization
|
||||
- Standard K8s tool (no external dependencies)
|
||||
|
||||
### ✓ Why No Terraform?
|
||||
- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD)
|
||||
- Git is source of truth for everything
|
||||
- No state backend (no .tfstate files)
|
||||
- ArgoCD continuous reconciliation (drift auto-corrected)
|
||||
- All changes reviewed in PRs (audit trail in git)
|
||||
|
||||
### ✓ Why This File Structure?
|
||||
- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/)
|
||||
- **Scalable:** New service = new directory (copy template)
|
||||
- **Maintainable:** Clear ownership (who owns layer? who owns service?)
|
||||
- **GitOps-ready:** Each layer is independently deployable
|
||||
|
||||
## Secret Management (SOPS)
|
||||
|
||||
### Before (Terraform)
|
||||
```
|
||||
terraform/terraform.tfvars # Plaintext secrets (risky!)
|
||||
```
|
||||
|
||||
### After (GitOps)
|
||||
```bash
|
||||
# Encrypt secrets before commit
|
||||
sops -e secrets.yaml > secrets.enc.yaml
|
||||
git add secrets.enc.yaml
|
||||
|
||||
# ArgoCD decrypts at sync time (SOPS plugin installed)
|
||||
# Git stores only encrypted version (safe)
|
||||
```
|
||||
|
||||
**Secrets to encrypt:**
|
||||
- Authentik bootstrap password
|
||||
- MinIO root credentials
|
||||
- Database passwords
|
||||
- API tokens
|
||||
- OAuth client secrets
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Before
|
||||
```
|
||||
git push → terraform validate → terraform plan → manual apply
|
||||
```
|
||||
|
||||
### After (Pure GitOps)
|
||||
```
|
||||
git push → yamllint → kubeval → kustomize build → argocd validation
|
||||
↓
|
||||
ArgoCD auto-syncs (if PR merged)
|
||||
```
|
||||
|
||||
**No manual steps. No terraform. No state management. Pure GitOps.**
|
||||
|
||||
## Comparison: Old vs. New
|
||||
|
||||
| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) |
|
||||
|--------|--------------------------|-------------------|
|
||||
| **State management** | tfstate files | Git history |
|
||||
| **Secrets** | Plaintext in .tfvars | SOPS encrypted |
|
||||
| **Manual steps** | terraform apply required | git push only |
|
||||
| **Rollback** | Revert tfstate + code | git revert + push |
|
||||
| **Audit trail** | Terraform logs | Git history + ArgoCD logs |
|
||||
| **Complexity** | High (2 tools) | Low (1 tool) |
|
||||
| **New services** | Edit .tf files | Copy service directory |
|
||||
| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects |
|
||||
|
||||
## Non-Disruptive Migration
|
||||
|
||||
**Key principle:** Live cluster remains running during entire migration.
|
||||
|
||||
**How?**
|
||||
1. Build k8s/ structure **alongside** existing configuration
|
||||
2. Deploy each layer via ArgoCD while old method still runs
|
||||
3. Once new layer verified: delete old configuration
|
||||
4. If issue detected: rollback (git revert) + redeploy old version
|
||||
|
||||
**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed)
|
||||
|
||||
## Rollback Procedure (At Any Point)
|
||||
|
||||
```bash
|
||||
# 1. Identify broken layer
|
||||
argocd app get layer-2-platform # Shows OutOfSync, error
|
||||
|
||||
# 2. Revert that layer
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# 3. ArgoCD detects change, syncs back to previous version
|
||||
# Expected: ~3 minutes for full reconciliation
|
||||
|
||||
# 4. Verify
|
||||
argocd app get layer-2-platform # Should be Synced again
|
||||
```
|
||||
|
||||
## Day 1 Action Items
|
||||
|
||||
1. **Read GITOPS_ARCHITECTURE.md** (30 min)
|
||||
- Understand layers
|
||||
- Understand directory structure
|
||||
- Understand kustomization strategy
|
||||
|
||||
2. **Read GITOPS_MIGRATION_PLAN.md** (30 min)
|
||||
- Understand phases
|
||||
- Understand timeline
|
||||
- Understand success criteria
|
||||
|
||||
3. **Review dependency map** (provided in plan)
|
||||
- Confirm layer dependencies match current setup
|
||||
- Identify any services not in layers
|
||||
|
||||
4. **Audit secrets** (30 min)
|
||||
```bash
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
# Plan which need SOPS encryption
|
||||
```
|
||||
|
||||
5. **Decision point:** Proceed to Phase 2 (Week 2)?
|
||||
- YES → Start building k8s/ structure
|
||||
- NO → Clarify concerns, refine plan
|
||||
|
||||
## Benefits Summary
|
||||
|
||||
✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth
|
||||
✓ **Safer:** All changes reviewed in PRs, git history as audit trail
|
||||
✓ **Scalable:** New services = copy directory, update git, done
|
||||
✓ **Resilient:** State drift auto-corrected, rollback via `git revert`
|
||||
✓ **Auditable:** Every change in git with who/when/why
|
||||
✓ **Automated:** No manual kubectl apply, no manual terraform apply
|
||||
✓ **Testable:** Kustomize build validates before sync
|
||||
✓ **Recoverable:** git history = disaster recovery
|
||||
|
||||
## Support
|
||||
|
||||
- Architecture questions → GITOPS_ARCHITECTURE.md
|
||||
- Implementation questions → GITOPS_MIGRATION_PLAN.md
|
||||
- Syntax/tools questions → See referenced documentation
|
||||
- Kustomize: https://kustomize.io/
|
||||
- ArgoCD: https://argo-cd.readthedocs.io/
|
||||
- SOPS: https://github.com/mozilla/sops
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **User review:** Read both architecture documents (60 min)
|
||||
2. **User decision:** Proceed to Phase 2 or refine plan?
|
||||
3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday)
|
||||
|
||||
---
|
||||
|
||||
**Status:** Ready for implementation
|
||||
**Terraform:** Deleted ✓
|
||||
**Architecture:** Designed ✓
|
||||
**Plan:** Ready ✓
|
||||
**Waiting for:** User approval to proceed
|
||||
Reference in New Issue
Block a user