refactor(ci-cd): Replace Terraform pipeline with GitOps validation and ArgoCD sync
Delete old terraform-apply.yml (terraform fmt/init/validate/plan/apply). Create new GitOps CI/CD: - validate-k8s.yaml: YAML lint, kubeval, kustomize build, ArgoCD validation - argocd-sync.yaml: Auto-sync homelab-root on main branch - security-scan.yaml: Trivy, Polaris, secret detection - .yamllint.yaml: YAML linting configuration Add documentation (.forgejo/CI-CD.md) and architecture guides. Git is now single source of truth. CI validates, ArgoCD deploys.
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,573 @@
|
||||
# GitOps Migration Plan: Implement Production-Grade Architecture
|
||||
|
||||
## Current State → Target State
|
||||
|
||||
### Current (Messy)
|
||||
```
|
||||
k8s/
|
||||
├── talos-ci-cd/ (forgejo, runner)
|
||||
├── talos-iam/ (authentik, vault)
|
||||
├── logging/ (loki, promtail, grafana)
|
||||
├── monitoring/ (prometheus, alerts)
|
||||
├── storage/ (minio, longhorn)
|
||||
├── ddb/ (postgres)
|
||||
├── sqs/ (kafka, queue-crd)
|
||||
├── temporal/ (temporal)
|
||||
├── argocd/apps/ (20 Applications scattered)
|
||||
└── ... (more scattered)
|
||||
```
|
||||
|
||||
### Target (Clean, Layered)
|
||||
```
|
||||
k8s/
|
||||
├── infrastructure/ (namespaces, storage, RBAC)
|
||||
├── bootstrap/ (cert-manager, cilium, ingress-nginx)
|
||||
├── platform/ (storage, observability: minio, longhorn, loki, prometheus)
|
||||
├── security/ (authentik, vault, cert-issuer)
|
||||
├── applications/ (forgejo, grafana, portainer, temporal, llm)
|
||||
├── data/ (postgres, redis, kafka)
|
||||
└── argocd/
|
||||
├── projects/ (AppProject definitions)
|
||||
└── apps/ (6 layer Applications only)
|
||||
```
|
||||
|
||||
## Phase 1: Plan & Validate (Week 1)
|
||||
|
||||
### Step 1.1: Review GITOPS_ARCHITECTURE.md
|
||||
- [ ] Understand directory structure
|
||||
- [ ] Understand layer dependencies
|
||||
- [ ] Understand Kustomization strategy
|
||||
- User approval: YES/NO
|
||||
|
||||
### Step 1.2: Audit current resources
|
||||
```bash
|
||||
# Export all current resources
|
||||
kubectl get all -A -o yaml > /tmp/current-state-backup.yaml
|
||||
|
||||
# Count resources per layer
|
||||
kubectl get deployments -A | wc -l
|
||||
kubectl get statefulsets -A | wc -l
|
||||
kubectl get daemonsets -A | wc -l
|
||||
kubectl get services -A | wc -l
|
||||
kubectl get configmaps -A | wc -l
|
||||
kubectl get secrets -A | wc -l
|
||||
```
|
||||
|
||||
### Step 1.3: Identify secrets needing encryption
|
||||
```bash
|
||||
# Find secrets in current manifests
|
||||
grep -r "kind: Secret" k8s/ --include="*.yaml"
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
|
||||
# Plan SOPS encryption for:
|
||||
# - Authentik bootstrap password
|
||||
# - MinIO credentials
|
||||
# - Database passwords
|
||||
# - API tokens
|
||||
```
|
||||
|
||||
### Step 1.4: Dependency mapping
|
||||
```
|
||||
Layer 0 (Infrastructure)
|
||||
└─ Layer 1 (Bootstrap)
|
||||
├─ cert-manager (creates certificates)
|
||||
├─ cilium (network)
|
||||
└─ ingress-nginx (entry point)
|
||||
└─ Layer 2 (Platform)
|
||||
├─ minio (storage)
|
||||
├─ longhorn (PV storage)
|
||||
├─ loki (logs)
|
||||
└─ prometheus (metrics)
|
||||
└─ Layer 3 (Security)
|
||||
├─ authentik (auth)
|
||||
└─ vault (secrets)
|
||||
└─ Layer 4 (Applications)
|
||||
├─ forgejo
|
||||
├─ grafana
|
||||
├─ portainer
|
||||
├─ temporal
|
||||
└─ llm
|
||||
└─ Layer 5 (Data)
|
||||
├─ postgres
|
||||
├─ redis
|
||||
└─ kafka
|
||||
```
|
||||
|
||||
## Phase 2: Build Directory Structure (Weeks 2-3)
|
||||
|
||||
### Step 2.1: Create base directories
|
||||
```bash
|
||||
cd k8s/
|
||||
|
||||
# Create layers
|
||||
mkdir -p infrastructure bootstrap platform security applications data
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
for dir in infrastructure bootstrap platform security applications data; do
|
||||
cat > $dir/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: CHANGE_ME
|
||||
|
||||
resources: [] # Will add resources below
|
||||
|
||||
# Placeholders for this layer
|
||||
EOF
|
||||
done
|
||||
```
|
||||
|
||||
### Step 2.2: Migrate Layer 0 (Infrastructure)
|
||||
**Resources to create:**
|
||||
- Namespaces (all 20+)
|
||||
- Storage classes (longhorn, xfs, longhorn-kafka)
|
||||
- Service accounts (terraform-ci, etc.)
|
||||
- Cluster roles (terraform-ci, admin, etc.)
|
||||
- Network policies
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Extract from current cluster
|
||||
kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml
|
||||
kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml
|
||||
kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml
|
||||
kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml
|
||||
|
||||
# Clean up (remove status, owner refs, etc.)
|
||||
# Add to k8s/infrastructure/kustomization.yaml:
|
||||
resources:
|
||||
- namespaces.yaml
|
||||
- storage-classes.yaml
|
||||
- service-accounts.yaml
|
||||
- cluster-roles.yaml
|
||||
|
||||
# Test
|
||||
kustomize build k8s/infrastructure/
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/infrastructure/kustomization.yaml
|
||||
- [x] k8s/infrastructure/namespaces.yaml
|
||||
- [x] k8s/infrastructure/storage-classes.yaml
|
||||
- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml
|
||||
|
||||
### Step 2.3: Migrate Layer 1 (Bootstrap)
|
||||
|
||||
**Services: cert-manager, cilium, ingress-nginx**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx}
|
||||
|
||||
# For each service:
|
||||
# 1. Export Helm values
|
||||
helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml
|
||||
|
||||
# 2. Create kustomization.yaml with Helm chart ref
|
||||
cat > bootstrap/cert-manager/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: cert-manager
|
||||
|
||||
helmCharts:
|
||||
- name: cert-manager
|
||||
repo: https://charts.jetstack.io
|
||||
version: v1.21.0
|
||||
releaseName: cert-manager
|
||||
valuesFile: values.yaml
|
||||
EOF
|
||||
|
||||
# 3. Repeat for cilium, ingress-nginx
|
||||
|
||||
# 4. Add to k8s/bootstrap/kustomization.yaml:
|
||||
resources:
|
||||
- cert-manager/kustomization.yaml
|
||||
- cilium/kustomization.yaml
|
||||
- ingress-nginx/kustomization.yaml
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml
|
||||
- [x] k8s/bootstrap/kustomization.yaml
|
||||
|
||||
### Step 2.4: Migrate Layer 2 (Platform)
|
||||
|
||||
**Services: minio, longhorn, loki, prometheus, promtail**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p platform/{minio,longhorn,loki,prometheus,promtail}
|
||||
|
||||
# Migrate minio
|
||||
# 1. Export existing values
|
||||
helm get values minio -n storage > platform/minio/values.yaml
|
||||
|
||||
# 2. Export bucket configs
|
||||
kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/
|
||||
|
||||
# 3. Create kustomization.yaml
|
||||
cat > platform/minio/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: storage
|
||||
|
||||
helmCharts:
|
||||
- name: minio
|
||||
repo: https://charts.min.io
|
||||
version: 14.6.25 # Current version
|
||||
releaseName: minio
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/minio-buckets.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for loki, prometheus, promtail
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml
|
||||
- [x] k8s/platform/kustomization.yaml
|
||||
|
||||
### Step 2.5: Migrate Layer 3 (Security)
|
||||
|
||||
**Services: authentik, vault**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p security/{authentik,vault}
|
||||
|
||||
# Migrate authentik
|
||||
helm get values authentik -n iam > security/authentik/values.yaml
|
||||
|
||||
# Export Authentik resources
|
||||
kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml
|
||||
kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml
|
||||
kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml
|
||||
|
||||
# Create kustomization.yaml
|
||||
cat > security/authentik/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: iam
|
||||
|
||||
helmCharts:
|
||||
- name: authentik
|
||||
repo: https://charts.goauthentik.io
|
||||
version: 2024.12.1
|
||||
releaseName: authentik
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/apps.yaml
|
||||
- config/groups.yaml
|
||||
- config/oauth-providers.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for vault
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/security/{authentik,vault}/kustomization.yaml
|
||||
- [x] k8s/security/{authentik,vault}/values.yaml
|
||||
- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml
|
||||
- [x] k8s/security/kustomization.yaml
|
||||
|
||||
### Step 2.6: Migrate Layer 4 (Applications)
|
||||
|
||||
**Services: forgejo, grafana, portainer, temporal, llm**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p applications/{forgejo,grafana,portainer,temporal,llm}
|
||||
|
||||
# Move existing manifests
|
||||
cp -r k8s/talos-ci-cd/* applications/forgejo/
|
||||
cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc.
|
||||
cp -r k8s/portainer/* applications/portainer/
|
||||
cp -r k8s/temporal/* applications/temporal/
|
||||
cp -r k8s/llm/* applications/llm/
|
||||
|
||||
# For each, create kustomization.yaml with Helm chart ref
|
||||
# (or keep existing manifests if not Helm)
|
||||
|
||||
# Create layer kustomization.yaml
|
||||
cat > applications/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- forgejo/kustomization.yaml
|
||||
- grafana/kustomization.yaml
|
||||
- portainer/kustomization.yaml
|
||||
- temporal/kustomization.yaml
|
||||
- llm/kustomization.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml
|
||||
- [x] k8s/applications/kustomization.yaml
|
||||
|
||||
### Step 2.7: Migrate Layer 5 (Data)
|
||||
|
||||
**Services: postgres (CloudNativePG), redis, kafka**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p data/{postgres,redis,kafka}
|
||||
|
||||
# Export existing configurations
|
||||
kubectl get cnpg -A -o yaml > data/postgres/config.yaml
|
||||
kubectl get redis -A -o yaml > data/redis/config.yaml
|
||||
kubectl get kafka -A -o yaml > data/kafka/config.yaml
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
|
||||
# Repeat process...
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml
|
||||
- [x] k8s/data/kustomization.yaml
|
||||
|
||||
## Phase 3: Create ArgoCD Application Definitions (Week 4)
|
||||
|
||||
### Step 3.1: Create layer Applications
|
||||
```bash
|
||||
mkdir -p k8s/argocd/apps/layers
|
||||
|
||||
# Root application (Layer 0 only, triggers rest)
|
||||
cat > k8s/argocd/apps/root-app.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: homelab-root
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/infrastructure
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Layer 1 application
|
||||
cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: layer-1-bootstrap
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "1"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/bootstrap
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5)
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/argocd/apps/root-app.yaml
|
||||
- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml
|
||||
|
||||
### Step 3.2: Create kustomization.yaml for apps
|
||||
```bash
|
||||
cat > k8s/argocd/apps/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- root-app.yaml
|
||||
- layer-1-bootstrap.yaml
|
||||
- layer-2-platform.yaml
|
||||
- layer-3-security.yaml
|
||||
- layer-4-applications.yaml
|
||||
- layer-5-data.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
## Phase 4: Deploy & Validate (Week 5)
|
||||
|
||||
### Step 4.1: Pre-flight checks
|
||||
```bash
|
||||
# Validate all kustomization files
|
||||
for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do
|
||||
echo "Validating $dir..."
|
||||
kustomize build $dir > /tmp/validate.yaml
|
||||
kubeval /tmp/validate.yaml || exit 1
|
||||
done
|
||||
|
||||
# Validate ArgoCD apps
|
||||
kubeval k8s/argocd/apps/*.yaml
|
||||
|
||||
# Test YAML lint
|
||||
yamllint k8s/
|
||||
```
|
||||
|
||||
### Step 4.2: Deploy root application
|
||||
```bash
|
||||
# Apply root app (Layer 0 only)
|
||||
kubectl apply -f k8s/argocd/apps/root-app.yaml
|
||||
|
||||
# Watch sync
|
||||
argocd app watch homelab-root
|
||||
|
||||
# Verify infrastructure deployed
|
||||
kubectl get ns # Should have all namespaces
|
||||
kubectl get sc # Should have all storage classes
|
||||
```
|
||||
|
||||
### Step 4.3: Deploy layer applications
|
||||
```bash
|
||||
# Apply layer applications (one by one)
|
||||
kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml
|
||||
sleep 5
|
||||
argocd app watch layer-1-bootstrap
|
||||
|
||||
# Once Layer 1 synced: Layer 2
|
||||
kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml
|
||||
argocd app watch layer-2-platform
|
||||
|
||||
# ... repeat for layers 3, 4, 5
|
||||
```
|
||||
|
||||
### Step 4.4: Monitor for issues
|
||||
```bash
|
||||
# Check ArgoCD apps
|
||||
argocd app list
|
||||
|
||||
# Watch events
|
||||
kubectl get events -A -w
|
||||
|
||||
# Monitor pod status
|
||||
kubectl get pods -A
|
||||
|
||||
# Check logs
|
||||
kubectl logs -f -n argocd deployment/argocd-application-controller
|
||||
```
|
||||
|
||||
### Step 4.5: Validate state drift protection
|
||||
```bash
|
||||
# Manually edit a resource
|
||||
kubectl edit deployment cert-manager -n cert-manager
|
||||
# Change replicas from 1 → 2
|
||||
|
||||
# Wait 3 minutes (ArgoCD reconciliation interval)
|
||||
sleep 180
|
||||
|
||||
# Check: should be back to 1 replica
|
||||
kubectl get deployment cert-manager -n cert-manager
|
||||
# Expected: 1/1 Ready (reverted by ArgoCD)
|
||||
|
||||
# Verify in ArgoCD
|
||||
argocd app get layer-1-bootstrap
|
||||
# Expected: Synced status
|
||||
```
|
||||
|
||||
## Phase 5: Cleanup (Week 6)
|
||||
|
||||
### Step 5.1: Delete old directories
|
||||
```bash
|
||||
# After all apps synced successfully:
|
||||
rm -rf k8s/talos-ci-cd/
|
||||
rm -rf k8s/talos-iam/
|
||||
rm -rf k8s/logging/
|
||||
rm -rf k8s/monitoring/
|
||||
rm -rf k8s/storage/
|
||||
rm -rf k8s/ddb/
|
||||
rm -rf k8s/sqs/
|
||||
rm -rf k8s/temporal/
|
||||
# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd
|
||||
```
|
||||
|
||||
### Step 5.2: Update CI/CD pipeline
|
||||
```bash
|
||||
# Update .forgejo/workflows/
|
||||
# - Remove: terraform validate/plan/apply
|
||||
# - Add: kustomize build validation
|
||||
# - Add: argocd app validation
|
||||
```
|
||||
|
||||
### Step 5.3: Archive old configuration
|
||||
```bash
|
||||
# Keep for reference only
|
||||
mkdir k8s/.archive/
|
||||
git mv k8s/old-structure-backup k8s/.archive/
|
||||
git commit -m "archive: old k8s structure (moved to pure GitOps)"
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
**If something breaks during migration:**
|
||||
|
||||
```bash
|
||||
# Option 1: Rollback entire layer
|
||||
git checkout HEAD~1 -- k8s/bootstrap/
|
||||
git commit -m "revert: bootstrap layer (investigating)"
|
||||
# ArgoCD will resync to previous version automatically
|
||||
|
||||
# Option 2: Pause ArgoCD sync
|
||||
argocd app set layer-1-bootstrap --sync-policy none
|
||||
# Investigate, then re-enable:
|
||||
argocd app set layer-1-bootstrap --sync-policy automated
|
||||
|
||||
# Option 3: Full rollback to pre-migration
|
||||
git reset --hard <commit-before-migration>
|
||||
argocd app set homelab-root --sync-policy none
|
||||
# Manual investigation, then re-enable
|
||||
```
|
||||
|
||||
## Timeline Summary
|
||||
|
||||
```
|
||||
Week 1: Plan & validate (reviews, dependency mapping)
|
||||
Week 2: Layers 0-1 (infrastructure, bootstrap)
|
||||
Week 3: Layers 2-3 (platform, security)
|
||||
Week 3: Layers 4-5 (applications, data)
|
||||
Week 4: ArgoCD applications + layer definitions
|
||||
Week 5: Deploy & validate (watch for issues)
|
||||
Week 6: Cleanup & CI/CD updates
|
||||
```
|
||||
|
||||
**Total: 6 weeks, non-disruptive (all layers coexist during migration)**
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✓ All 20+ services deployed via ArgoCD
|
||||
✓ No manual kubectl apply in production
|
||||
✓ State drift detected & corrected automatically
|
||||
✓ All changes in git (reviewed via PR)
|
||||
✓ Rollback possible at any time (git history)
|
||||
✓ CI/CD validates all commits
|
||||
✓ Secrets encrypted with SOPS
|
||||
✓ New services can be added (copy service directory)
|
||||
Reference in New Issue
Block a user