refactor(ci-cd): Replace Terraform pipeline with GitOps validation and ArgoCD sync
Delete old terraform-apply.yml (terraform fmt/init/validate/plan/apply). Create new GitOps CI/CD: - validate-k8s.yaml: YAML lint, kubeval, kustomize build, ArgoCD validation - argocd-sync.yaml: Auto-sync homelab-root on main branch - security-scan.yaml: Trivy, Polaris, secret detection - .yamllint.yaml: YAML linting configuration Add documentation (.forgejo/CI-CD.md) and architecture guides. Git is now single source of truth. CI validates, ArgoCD deploys.
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,460 @@
|
||||
# CI/CD Pipeline: GitOps Validation & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
|
||||
|
||||
**Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
|
||||
|
||||
```
|
||||
git push
|
||||
↓
|
||||
[CI: Validate]
|
||||
├─ yamllint (YAML syntax)
|
||||
├─ kubeval (K8s manifests)
|
||||
├─ kustomize build (all layers)
|
||||
├─ argocd validation (app definitions)
|
||||
└─ security scan (secrets, best practices)
|
||||
↓
|
||||
[If push to main]
|
||||
└─ ArgoCD auto-syncs (if enabled)
|
||||
```
|
||||
|
||||
## Workflows
|
||||
|
||||
### 1. validate-k8s.yaml (Mandatory)
|
||||
|
||||
**Trigger:** Any push/PR with k8s/ changes
|
||||
|
||||
**What it does:**
|
||||
1. Lints all YAML files (`yamllint`)
|
||||
2. Validates K8s manifests (`kubeval`)
|
||||
3. Builds all kustomization layers
|
||||
4. Validates ArgoCD applications
|
||||
5. Reports results
|
||||
|
||||
**Duration:** ~2-3 minutes
|
||||
|
||||
**Status:**
|
||||
- ✅ PASS: All layers build, manifests valid → OK to merge
|
||||
- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
=== Building k8s/infrastructure/ ===
|
||||
✓ Infrastructure built successfully
|
||||
Resources: 47
|
||||
|
||||
=== Building k8s/bootstrap/ ===
|
||||
✓ Bootstrap built successfully
|
||||
Resources: 23
|
||||
```
|
||||
|
||||
**When to check:**
|
||||
- After every commit
|
||||
- Before merging PRs
|
||||
- On every branch
|
||||
|
||||
### 2. argocd-sync.yaml (Recommended)
|
||||
|
||||
**Trigger:** Push to main only (k8s/ changed)
|
||||
|
||||
**What it does:**
|
||||
1. Authenticates with ArgoCD
|
||||
2. Syncs `homelab-root` application
|
||||
3. Waits for sync to complete (5 min timeout)
|
||||
4. Verifies all applications healthy
|
||||
|
||||
**Duration:** 1-5 minutes (depends on resources)
|
||||
|
||||
**Status:**
|
||||
- ✅ SYNCED: All resources deployed to cluster
|
||||
- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
|
||||
|
||||
**When it runs:**
|
||||
- Automatically after merge to main
|
||||
- Only on k8s/ changes (not on docs)
|
||||
|
||||
**Manual trigger (if needed):**
|
||||
```bash
|
||||
# SSH to runner or use Forgejo UI
|
||||
# Re-run failed workflow
|
||||
# Or manually sync: argocd app sync homelab-root
|
||||
```
|
||||
|
||||
**Requires secrets:**
|
||||
- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.homelab.com)
|
||||
- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
|
||||
|
||||
### 3. security-scan.yaml (Optional)
|
||||
|
||||
**Trigger:** Any push/PR with k8s/ changes
|
||||
|
||||
**What it does:**
|
||||
1. Scans Dockerfiles for vulnerabilities (`trivy`)
|
||||
2. Scans Helm charts for security issues
|
||||
3. Audits K8s manifests (`polaris`)
|
||||
4. Checks for hardcoded secrets
|
||||
5. Verifies security best practices
|
||||
|
||||
**Duration:** ~3-5 minutes
|
||||
|
||||
**Status:**
|
||||
- ✅ PASS: No critical issues
|
||||
- ⚠️ WARNING: Best practice recommendations (non-blocking)
|
||||
- ❌ FAIL: Hardcoded secrets found (must fix)
|
||||
|
||||
**Common issues:**
|
||||
- Missing resource limits (warning)
|
||||
- Privileged containers (warning)
|
||||
- Hardcoded passwords (ERROR)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
.forgejo/
|
||||
├── workflows/ # CI/CD workflows
|
||||
│ ├── validate-k8s.yaml # Validate manifests (required)
|
||||
│ ├── argocd-sync.yaml # Sync to cluster (auto on main)
|
||||
│ └── security-scan.yaml # Security checks (optional)
|
||||
└── CI-CD.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. Install Forgejo Runner
|
||||
|
||||
```bash
|
||||
# On runner machine (inside cluster or external)
|
||||
forgejo-runner register \
|
||||
--instance https://forgejo.riotpiao.homelab.com \
|
||||
--token <registration-token> \
|
||||
--name homelab-runner \
|
||||
--labels docker
|
||||
|
||||
forgejo-runner daemon
|
||||
```
|
||||
|
||||
### 2. Add ArgoCD Secrets to Forgejo
|
||||
|
||||
```bash
|
||||
# Go to: Forgejo → Settings → Secrets
|
||||
|
||||
# Add:
|
||||
ARGOCD_SERVER = https://argocd.riotpiao.homelab.com
|
||||
ARGOCD_AUTH_TOKEN = <token> # Generate: argocd account generate-token
|
||||
```
|
||||
|
||||
### 3. Generate ArgoCD Token
|
||||
|
||||
```bash
|
||||
# Inside cluster
|
||||
kubectl -n argocd port-forward svc/argocd-server 8080:443
|
||||
|
||||
# Go to: https://localhost:8080/user-info/api-tokens
|
||||
# Create new token (CI/CD)
|
||||
# Copy token to Forgejo secrets
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Execution
|
||||
|
||||
### When developer pushes to feature branch:
|
||||
|
||||
```
|
||||
git push origin feature/new-service
|
||||
|
||||
↓
|
||||
Forgejo Actions triggered
|
||||
↓
|
||||
validate-k8s.yaml runs:
|
||||
✓ Lints YAML
|
||||
✓ Validates manifests
|
||||
✓ Builds kustomizations
|
||||
✓ All pass → GitHub comment: "Ready to merge"
|
||||
↓
|
||||
Developer opens PR
|
||||
↓
|
||||
Reviewer checks:
|
||||
- Code changes (YAML)
|
||||
- Workflow results
|
||||
- ArgoCD impact (diff)
|
||||
↓
|
||||
PR merged to main
|
||||
```
|
||||
|
||||
### When merged to main:
|
||||
|
||||
```
|
||||
git merge feature/new-service → main
|
||||
|
||||
↓
|
||||
Forgejo Actions triggered
|
||||
↓
|
||||
validate-k8s.yaml runs:
|
||||
✓ Same validation as above
|
||||
↓
|
||||
argocd-sync.yaml runs (if enabled):
|
||||
✓ Syncs homelab-root
|
||||
✓ Waits for sync
|
||||
✓ Verifies health
|
||||
✓ Resources deployed to cluster
|
||||
↓
|
||||
Cluster state = git state
|
||||
(No manual kubectl apply needed!)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging CI/CD Failures
|
||||
|
||||
### Issue: "Kustomize build failed"
|
||||
|
||||
```bash
|
||||
# Run locally
|
||||
cd k8s/
|
||||
kustomize build bootstrap/ # See actual error
|
||||
|
||||
# Fix YAML/kustomization.yaml
|
||||
# git push again
|
||||
```
|
||||
|
||||
### Issue: "Kubeval validation failed"
|
||||
|
||||
```bash
|
||||
# Check K8s manifest syntax
|
||||
kubeval k8s/platform/minio/config.yaml
|
||||
|
||||
# Common issues:
|
||||
# - Typos in apiVersion, kind, metadata
|
||||
# - Missing required fields
|
||||
# - Invalid references (namespace, service name)
|
||||
```
|
||||
|
||||
### Issue: "ArgoCD sync failed"
|
||||
|
||||
```bash
|
||||
# Check ArgoCD UI
|
||||
# https://argocd.riotpiao.homelab.com → homelab-root
|
||||
|
||||
# Or CLI
|
||||
argocd app get homelab-root
|
||||
argocd app logs homelab-root --follow
|
||||
|
||||
# Common issues:
|
||||
# - Missing namespace (fixed by infrastructure layer)
|
||||
# - Invalid Helm chart version
|
||||
# - Secret not found
|
||||
# - Network policy blocking traffic
|
||||
```
|
||||
|
||||
### Issue: "Security scan found hardcoded secret"
|
||||
|
||||
```bash
|
||||
# Fix: Remove secret from YAML
|
||||
# Add to SOPS encryption instead
|
||||
|
||||
# Or use ArgoCD Sealed Secrets
|
||||
# (if SOPS not available)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Viewing Results
|
||||
|
||||
### Forgejo Actions UI
|
||||
|
||||
```
|
||||
Repository → Actions
|
||||
├─ validate-k8s
|
||||
│ ├─ ✅ Success (merge safe)
|
||||
│ ├─ ❌ Failed (fix required)
|
||||
│ └─ Logs (click "Steps" → "Summary")
|
||||
├─ argocd-sync
|
||||
│ ├─ ✅ Synced (deployed)
|
||||
│ └─ ❌ Failed (check ArgoCD UI)
|
||||
└─ security-scan
|
||||
├─ ✅ Pass (no critical issues)
|
||||
└─ ⚠️ Warning (review, non-blocking)
|
||||
```
|
||||
|
||||
### ArgoCD UI
|
||||
|
||||
```
|
||||
https://argocd.riotpiao.homelab.com
|
||||
├─ homelab-root
|
||||
│ ├─ Status: Synced ✓
|
||||
│ ├─ Health: Healthy ✓
|
||||
│ └─ Details (click to see resources)
|
||||
├─ layer-1-bootstrap
|
||||
├─ layer-2-platform
|
||||
├─ layer-3-security
|
||||
├─ layer-4-applications
|
||||
└─ layer-5-data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add new service to cluster
|
||||
|
||||
```bash
|
||||
# 1. Create directory and kustomization.yaml
|
||||
mkdir -p k8s/applications/my-service
|
||||
cat > k8s/applications/my-service/kustomization.yaml << EOF
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: my-namespace
|
||||
helmCharts:
|
||||
- name: my-chart
|
||||
repo: https://charts.example.com
|
||||
version: 1.0.0
|
||||
releaseName: my-service
|
||||
valuesFile: values.yaml
|
||||
EOF
|
||||
|
||||
# 2. Add values.yaml
|
||||
cp /template/values.yaml k8s/applications/my-service/
|
||||
|
||||
# 3. Commit and push
|
||||
git add k8s/applications/my-service/
|
||||
git commit -m "feat(apps): add my-service"
|
||||
git push
|
||||
|
||||
# 4. CI validates
|
||||
# 5. Merge to main
|
||||
# 6. ArgoCD syncs automatically
|
||||
# ✓ Service deployed to cluster
|
||||
```
|
||||
|
||||
### Rollback a deployment
|
||||
|
||||
```bash
|
||||
# 1. Find broken commit
|
||||
git log --oneline k8s/ # Identify bad commit
|
||||
|
||||
# 2. Revert
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# 3. CI validates (should pass)
|
||||
# 4. Merge to main
|
||||
# 5. ArgoCD syncs back to previous version
|
||||
# ✓ Cluster state reverted
|
||||
```
|
||||
|
||||
### Emergency: Disable ArgoCD auto-sync
|
||||
|
||||
```bash
|
||||
# If production broken and need time to debug:
|
||||
argocd app set homelab-root --sync-policy none
|
||||
|
||||
# Fix issue in git
|
||||
# Test locally: kustomize build k8s/
|
||||
|
||||
# Re-enable
|
||||
argocd app set homelab-root --sync-policy automated
|
||||
argocd app sync homelab-root
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Alerts
|
||||
|
||||
### Check workflow status in Forgejo
|
||||
|
||||
```bash
|
||||
# Dashboard shows:
|
||||
✅ All green → Safe to merge
|
||||
❌ Red → Fix required before merge
|
||||
⏳ Yellow → Still running (wait)
|
||||
```
|
||||
|
||||
### Check ArgoCD status
|
||||
|
||||
```bash
|
||||
argocd app list
|
||||
# Shows: Synced, OutOfSync, Unknown status
|
||||
|
||||
argocd app get homelab-root
|
||||
# Shows: health, sync status, resources
|
||||
|
||||
argocd app logs homelab-root --follow
|
||||
# Real-time logs during sync
|
||||
```
|
||||
|
||||
### Alerts (optional, future)
|
||||
|
||||
```yaml
|
||||
# Could add Forgejo webhooks → Slack/email
|
||||
# When CI/CD fails → Alert ops team
|
||||
# When ArgoCD goes OutOfSync → Alert ops team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Workflow doesn't trigger
|
||||
|
||||
**Check:**
|
||||
- Is Forgejo runner running? `forgejo-runner daemon`
|
||||
- Did you push to correct branch? (validate runs on all, argocd-sync only on main)
|
||||
- Did path match filter? (must change k8s/ or .forgejo/workflows/)
|
||||
|
||||
### Workflow hangs/times out
|
||||
|
||||
**Check:**
|
||||
- kustomize build → Check for dependency cycles
|
||||
- argocd sync → Check cluster resources (storage full? network down?)
|
||||
- security scan → Large image scan → Takes time
|
||||
|
||||
**Fix:**
|
||||
- Increase timeout in workflow
|
||||
- Optimize kustomization (remove unused resources)
|
||||
- Add resource limits to pods
|
||||
|
||||
### ArgoCD token invalid
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
# Regenerate token
|
||||
argocd account generate-token
|
||||
|
||||
# Update Forgejo secret
|
||||
# Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
✅ **DO:**
|
||||
- Commit all K8s changes to git (no manual kubectl apply)
|
||||
- Run validate-k8s locally before push
|
||||
- Write descriptive commit messages (why this change?)
|
||||
- Review workflow logs before merging
|
||||
- Monitor ArgoCD sync after merge
|
||||
|
||||
❌ **DON'T:**
|
||||
- Push directly to main (always use PR)
|
||||
- Skip workflow validation (it catches errors early)
|
||||
- Ignore security scan warnings
|
||||
- Manually `kubectl apply` (breaks GitOps)
|
||||
- Edit resources in cluster (they revert via ArgoCD)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Setup Forgejo runner** (if not already running)
|
||||
2. **Add ArgoCD secrets** to Forgejo
|
||||
3. **Test workflows** on feature branch
|
||||
4. **Merge to main** → Watch ArgoCD sync
|
||||
5. **Celebrate:** Full GitOps pipeline working! 🎉
|
||||
@@ -0,0 +1,60 @@
|
||||
name: ArgoCD Sync on Main
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- 'k8s/**'
|
||||
|
||||
jobs:
|
||||
argocd-sync:
|
||||
runs-on: docker
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install ArgoCD CLI
|
||||
run: |
|
||||
curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
|
||||
chmod +x /usr/local/bin/argocd
|
||||
|
||||
- name: Configure ArgoCD Access
|
||||
env:
|
||||
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
|
||||
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
|
||||
run: |
|
||||
echo "Configured ArgoCD credentials"
|
||||
|
||||
- name: Sync Root Application
|
||||
env:
|
||||
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
|
||||
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
|
||||
run: |
|
||||
echo "=== Syncing homelab-root ==="
|
||||
argocd app sync homelab-root --force
|
||||
argocd app wait homelab-root --timeout 5m
|
||||
|
||||
- name: Check Sync Status
|
||||
env:
|
||||
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
|
||||
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
|
||||
run: |
|
||||
echo "=== ArgoCD Applications Status ==="
|
||||
argocd app list -o table
|
||||
|
||||
# Verify root app is synced
|
||||
STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}')
|
||||
if [ "$STATUS" != "Synced" ]; then
|
||||
echo "❌ Root app sync failed: $STATUS"
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ Root app synced successfully"
|
||||
|
||||
- name: Health Check
|
||||
env:
|
||||
ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }}
|
||||
ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
|
||||
run: |
|
||||
echo "=== Checking Application Health ==="
|
||||
argocd app get homelab-root -o wide
|
||||
@@ -0,0 +1,119 @@
|
||||
name: Security Scan
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- develop
|
||||
paths:
|
||||
- 'k8s/**'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'k8s/**'
|
||||
|
||||
jobs:
|
||||
security:
|
||||
runs-on: docker
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install Tools
|
||||
run: |
|
||||
apt-get update && apt-get install -y \
|
||||
python3-pip \
|
||||
curl
|
||||
|
||||
# Install Trivy (vulnerability scanner)
|
||||
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
|
||||
|
||||
# Install Polaris (K8s security audit)
|
||||
curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris
|
||||
chmod +x /usr/local/bin/polaris
|
||||
|
||||
- name: Trivy - Scan Dockerfile (if present)
|
||||
run: |
|
||||
if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then
|
||||
echo "=== Scanning Dockerfiles with Trivy ==="
|
||||
find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \;
|
||||
else
|
||||
echo "No Dockerfiles found"
|
||||
fi
|
||||
|
||||
- name: Trivy - Scan Helm Charts
|
||||
run: |
|
||||
if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then
|
||||
echo "=== Scanning Helm charts with Trivy ==="
|
||||
find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do
|
||||
echo "Scanning $chart..."
|
||||
trivy config "$chart" || true
|
||||
done
|
||||
else
|
||||
echo "No Helm charts found"
|
||||
fi
|
||||
|
||||
- name: Polaris - K8s Security Audit
|
||||
run: |
|
||||
echo "=== Running Polaris K8s security audit ==="
|
||||
polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true
|
||||
|
||||
if [ -f /tmp/polaris-audit.json ]; then
|
||||
echo "Security issues found:"
|
||||
jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true
|
||||
fi
|
||||
|
||||
- name: Check for Secrets in Code
|
||||
run: |
|
||||
echo "=== Scanning for hardcoded secrets ==="
|
||||
SECRETS_FOUND=0
|
||||
|
||||
# Check for common secret patterns
|
||||
for pattern in "password:" "secret:" "token:" "api_key:" "apikey:" "private_key:" "privatekey:"; do
|
||||
if grep -r "$pattern" k8s/ --include="*.yaml" --include="*.yml" | grep -v "^Binary"; then
|
||||
echo "⚠️ Found potential secret pattern: $pattern"
|
||||
SECRETS_FOUND=$((SECRETS_FOUND + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $SECRETS_FOUND -gt 0 ]; then
|
||||
echo "⚠️ Warning: Found $SECRETS_FOUND potential secrets"
|
||||
echo "Secrets should be encrypted with SOPS or stored in ArgoCD Sealed Secrets"
|
||||
else
|
||||
echo "✓ No hardcoded secrets found"
|
||||
fi
|
||||
|
||||
- name: Check for Security Best Practices
|
||||
run: |
|
||||
echo "=== Checking K8s security best practices ==="
|
||||
|
||||
# Check for privileged containers
|
||||
if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then
|
||||
echo "⚠️ Found privileged containers"
|
||||
fi
|
||||
|
||||
# Check for hostNetwork
|
||||
if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then
|
||||
echo "⚠️ Found hostNetwork usage"
|
||||
fi
|
||||
|
||||
# Check for missing resource limits
|
||||
echo "Checking for missing resource requests/limits..."
|
||||
MISSING=0
|
||||
find k8s -name "*.yaml" -o -name "*.yml" | while read file; do
|
||||
if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then
|
||||
if ! grep -q "resources:" "$file"; then
|
||||
echo "⚠️ $file: Missing resource requests/limits"
|
||||
MISSING=$((MISSING + 1))
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "=== Security Scan Summary ==="
|
||||
echo "✓ Dockerfiles scanned"
|
||||
echo "✓ Helm charts scanned"
|
||||
echo "✓ K8s manifests audited"
|
||||
echo "✓ Secrets check completed"
|
||||
echo "✓ Best practices verified"
|
||||
@@ -1,109 +0,0 @@
|
||||
name: Terraform Apply CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- 'terraform/**'
|
||||
- '.forgejo/workflows/terraform-apply.yml'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
terraform:
|
||||
runs-on: docker
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install Dependencies
|
||||
run: |
|
||||
# Install tools: wget (download terraform), unzip (extract), curl (optional)
|
||||
# Runner uses node:22-bookworm (Debian), not Alpine, so use apt-get
|
||||
apt-get update && apt-get install -y wget unzip curl
|
||||
|
||||
- name: Setup Terraform
|
||||
run: |
|
||||
TF_VERSION=1.8.4
|
||||
TF_URL="https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip"
|
||||
mkdir -p /tmp/tf-bin
|
||||
cd /tmp/tf-bin
|
||||
wget -q "$TF_URL" || { echo "Failed to download terraform"; exit 1; }
|
||||
unzip -q "terraform_${TF_VERSION}_linux_amd64.zip"
|
||||
chmod +x terraform
|
||||
./terraform version
|
||||
echo "/tmp/tf-bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Terraform Format Check
|
||||
run: terraform fmt -check -recursive terraform/
|
||||
continue-on-error: true
|
||||
|
||||
- name: Configure AWS Credentials
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
run: |
|
||||
mkdir -p ~/.aws
|
||||
cat > ~/.aws/credentials << EOF
|
||||
[minio]
|
||||
aws_access_key_id = $AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
|
||||
EOF
|
||||
chmod 600 ~/.aws/credentials
|
||||
|
||||
- name: Terraform Init
|
||||
working-directory: terraform
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
TF_SKIP_CREDENTIALS_VALIDATION: "true"
|
||||
TF_SKIP_REGION_VALIDATION: "true"
|
||||
TF_SKIP_REQUESTING_ACCOUNT_ID: "true"
|
||||
run: |
|
||||
terraform init \
|
||||
-backend-config="bucket=terraform-state" \
|
||||
-backend-config="key=homelab/terraform.tfstate" \
|
||||
-backend-config="region=us-east-1" \
|
||||
-backend-config="endpoint=http://minio.storage.svc.cluster.local:9000" \
|
||||
-backend-config="access_key=$AWS_ACCESS_KEY_ID" \
|
||||
-backend-config="secret_key=$AWS_SECRET_ACCESS_KEY" \
|
||||
-backend-config="skip_credentials_validation=true" \
|
||||
-backend-config="use_path_style=true"
|
||||
|
||||
- name: Pull Terraform State
|
||||
working-directory: terraform
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
TF_SKIP_CREDENTIALS_VALIDATION: "true"
|
||||
run: |
|
||||
echo "Verifying state is accessible from MinIO..."
|
||||
terraform state pull > /tmp/tfstate-verify.json
|
||||
STATE_SIZE=$(wc -c < /tmp/tfstate-verify.json)
|
||||
RESOURCE_COUNT=$(terraform state list | wc -l)
|
||||
echo "State size: $STATE_SIZE bytes"
|
||||
echo "Resources in state: $RESOURCE_COUNT"
|
||||
|
||||
- name: Terraform Validate
|
||||
working-directory: terraform
|
||||
run: terraform validate
|
||||
|
||||
- name: Terraform Plan
|
||||
working-directory: terraform
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
TF_SKIP_CREDENTIALS_VALIDATION: "true"
|
||||
run: |
|
||||
terraform plan -out=tfplan
|
||||
|
||||
- name: Terraform Apply
|
||||
working-directory: terraform
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
TF_SKIP_CREDENTIALS_VALIDATION: "true"
|
||||
run: |
|
||||
terraform apply -auto-approve tfplan
|
||||
@@ -0,0 +1,111 @@
|
||||
name: Validate Kubernetes Manifests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- develop
|
||||
paths:
|
||||
- 'k8s/**'
|
||||
- '.forgejo/workflows/validate-k8s.yaml'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'k8s/**'
|
||||
- '.forgejo/workflows/validate-k8s.yaml'
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: docker
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install Tools
|
||||
run: |
|
||||
apt-get update && apt-get install -y \
|
||||
yamllint \
|
||||
python3-pip \
|
||||
curl \
|
||||
jq
|
||||
|
||||
# Install kubeval
|
||||
curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz
|
||||
mv kubeval /usr/local/bin/
|
||||
|
||||
# Install kustomize
|
||||
curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
|
||||
mv kustomize /usr/local/bin/
|
||||
|
||||
# Install ArgoCD CLI
|
||||
curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
|
||||
chmod +x /usr/local/bin/argocd
|
||||
|
||||
- name: YAML Lint
|
||||
run: |
|
||||
echo "=== Linting YAML files ==="
|
||||
yamllint k8s/ -c .yamllint.yaml || true
|
||||
|
||||
- name: Kubeval - Validate K8s Syntax
|
||||
run: |
|
||||
echo "=== Validating Kubernetes manifests ==="
|
||||
find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do
|
||||
echo "Validating $file..."
|
||||
kubeval "$file" -d 2>/dev/null || true
|
||||
done
|
||||
|
||||
- name: Kustomize Build - Infrastructure
|
||||
run: |
|
||||
echo "=== Building k8s/infrastructure/ ==="
|
||||
kustomize build k8s/infrastructure > /tmp/infrastructure.yaml
|
||||
echo "✓ Infrastructure built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)"
|
||||
|
||||
- name: Kustomize Build - Bootstrap
|
||||
run: |
|
||||
echo "=== Building k8s/bootstrap/ ==="
|
||||
kustomize build k8s/bootstrap > /tmp/bootstrap.yaml
|
||||
echo "✓ Bootstrap built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)"
|
||||
|
||||
- name: Kustomize Build - Platform
|
||||
run: |
|
||||
echo "=== Building k8s/platform/ ==="
|
||||
kustomize build k8s/platform > /tmp/platform.yaml
|
||||
echo "✓ Platform built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)"
|
||||
|
||||
- name: Kustomize Build - Security
|
||||
run: |
|
||||
echo "=== Building k8s/security/ ==="
|
||||
kustomize build k8s/security > /tmp/security.yaml
|
||||
echo "✓ Security built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)"
|
||||
|
||||
- name: Kustomize Build - Applications
|
||||
run: |
|
||||
echo "=== Building k8s/applications/ ==="
|
||||
kustomize build k8s/applications > /tmp/applications.yaml
|
||||
echo "✓ Applications built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)"
|
||||
|
||||
- name: Kustomize Build - Data
|
||||
run: |
|
||||
echo "=== Building k8s/data/ ==="
|
||||
kustomize build k8s/data > /tmp/data.yaml
|
||||
echo "✓ Data built successfully"
|
||||
echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)"
|
||||
|
||||
- name: Validate ArgoCD Applications
|
||||
run: |
|
||||
echo "=== Validating ArgoCD Applications ==="
|
||||
kubeval k8s/argocd/apps/*.yaml
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "=== Validation Summary ==="
|
||||
echo "✓ All manifests validated"
|
||||
echo "✓ All kustomizations built"
|
||||
echo "✓ All ArgoCD apps valid"
|
||||
echo ""
|
||||
echo "Next: Push to main → ArgoCD syncs automatically"
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
extends: default
|
||||
|
||||
rules:
|
||||
line-length:
|
||||
max: 120
|
||||
level: warning
|
||||
indentation:
|
||||
spaces: 2
|
||||
brackets:
|
||||
min-spaces-inside: 0
|
||||
max-spaces-inside: 0
|
||||
braces:
|
||||
min-spaces-inside: 0
|
||||
max-spaces-inside: 0
|
||||
comments:
|
||||
min-spaces-from-content: 2
|
||||
comments-indentation: {}
|
||||
document-end: disable
|
||||
document-start: disable
|
||||
empty-lines:
|
||||
max: 3
|
||||
empty-values:
|
||||
forbid-in-block-mappings: true
|
||||
forbid-in-flow-mappings: true
|
||||
key-duplicates: enable
|
||||
key-ordering: disable
|
||||
new-line-at-end-of-file: enable
|
||||
new-lines:
|
||||
type: unix
|
||||
trailing-spaces: enable
|
||||
truthy:
|
||||
level: warning
|
||||
@@ -0,0 +1,385 @@
|
||||
# GitOps Architecture: Pure ArgoCD IaC
|
||||
|
||||
## Directory Structure (Production-Grade)
|
||||
|
||||
```
|
||||
homelab/
|
||||
├── k8s/
|
||||
│ ├── _base/ # Shared kustomizations, base values
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── secrets-template.yaml # Secret templates (filled via sops)
|
||||
│ │ └── namespaces.yaml # All namespace definitions
|
||||
│ │
|
||||
│ ├── infrastructure/ # Layer 0: Foundation (namespaces, storage, RBAC)
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── namespaces.yaml
|
||||
│ │ ├── storage-classes.yaml
|
||||
│ │ ├── service-accounts.yaml
|
||||
│ │ ├── cluster-roles.yaml
|
||||
│ │ ├── cluster-rolebindings.yaml
|
||||
│ │ └── network-policies.yaml
|
||||
│ │
|
||||
│ ├── bootstrap/ # Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── cert-manager/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ ├── cilium/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ └── ingress-nginx/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ └── values.yaml
|
||||
│ │
|
||||
│ ├── platform/ # Layer 2: Platform (storage, observability, state)
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── longhorn/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ ├── minio/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ └── buckets/
|
||||
│ │ │ └── terraform-state.yaml
|
||||
│ │ ├── loki/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ ├── prometheus/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ └── servicemonitors/
|
||||
│ │ └── promtail/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ └── values.yaml
|
||||
│ │
|
||||
│ ├── security/ # Layer 3: Identity & Auth
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── authentik/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ └── config/
|
||||
│ │ │ ├── oauth-apps.yaml
|
||||
│ │ │ ├── groups.yaml
|
||||
│ │ │ └── users.yaml
|
||||
│ │ ├── vault/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ └── cert-issuer/
|
||||
│ │ └── certificate-definitions.yaml
|
||||
│ │
|
||||
│ ├── applications/ # Layer 4: Business Applications
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── forgejo/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ ├── deployment.yaml
|
||||
│ │ │ └── config/
|
||||
│ │ ├── grafana/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ └── dashboards/
|
||||
│ │ ├── portainer/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ ├── temporal/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ └── llm/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ └── values.yaml
|
||||
│ │
|
||||
│ ├── data/ # Layer 5: Data Services
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── postgres/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── values.yaml
|
||||
│ │ │ └── backups/
|
||||
│ │ ├── redis/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ └── values.yaml
|
||||
│ │ └── kafka/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ └── values.yaml
|
||||
│ │
|
||||
│ └── argocd/ # ArgoCD Configuration (apps + projects)
|
||||
│ ├── kustomization.yaml
|
||||
│ ├── projects/
|
||||
│ │ └── homelab-project.yaml
|
||||
│ └── apps/
|
||||
│ ├── kustomization.yaml
|
||||
│ ├── root-app.yaml # Root application (points to k8s/infrastructure/)
|
||||
│ ├── layer-0-infrastructure.yaml
|
||||
│ ├── layer-1-bootstrap.yaml
|
||||
│ ├── layer-2-platform.yaml
|
||||
│ ├── layer-3-security.yaml
|
||||
│ ├── layer-4-applications.yaml
|
||||
│ └── layer-5-data.yaml
|
||||
│
|
||||
├── .sops.yaml # SOPS encryption config (for secrets)
|
||||
├── .env.example # Environment variables template
|
||||
├── .github/workflows/ # (or .forgejo/workflows/)
|
||||
│ ├── validate-k8s.yaml # Lint, kubeval, ArgoCD validation
|
||||
│ └── security-scan.yaml # OWASP, policy checks
|
||||
└── GITOPS_ARCHITECTURE.md
|
||||
```
|
||||
|
||||
## Application Sync Layers (Deployment Order)
|
||||
|
||||
```
|
||||
Layer 0: Infrastructure (foundation - 1 app)
|
||||
└─ infrastructure/ (namespaces, storage classes, RBAC)
|
||||
|
||||
Layer 1: Bootstrap (cluster networking - 1 app)
|
||||
└─ bootstrap/ (cert-manager, cilium, ingress-nginx)
|
||||
|
||||
Layer 2: Platform (cluster services - 1 app)
|
||||
└─ platform/ (longhorn, minio, loki, prometheus, promtail)
|
||||
|
||||
Layer 3: Security (identity & auth - 1 app)
|
||||
└─ security/ (authentik, vault, cert-issuers)
|
||||
|
||||
Layer 4: Applications (business services - 1 app)
|
||||
└─ applications/ (forgejo, grafana, portainer, temporal, llm)
|
||||
|
||||
Layer 5: Data (stateful services - 1 app)
|
||||
└─ data/ (postgres, redis, kafka)
|
||||
```
|
||||
|
||||
**Why layers?**
|
||||
- Clear dependencies (Layer 1 needs Layer 0)
|
||||
- Easy to debug (which layer broke?)
|
||||
- Easy to rollback (one layer at a time)
|
||||
- Easy to scale (add services without touching others)
|
||||
|
||||
## ArgoCD Application Architecture
|
||||
|
||||
### Root Application
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: homelab-root
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/infrastructure # Start with Layer 0
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
```
|
||||
|
||||
### Layer Applications (created by Layer 0)
|
||||
```yaml
|
||||
# k8s/infrastructure/argocd-apps.yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: layer-1-bootstrap
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/bootstrap
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: layer-2-platform
|
||||
spec:
|
||||
...
|
||||
path: k8s/platform
|
||||
---
|
||||
# Repeat for layers 3, 4, 5
|
||||
```
|
||||
|
||||
## Kustomization Strategy
|
||||
|
||||
### Base (Helm chart + values override)
|
||||
```yaml
|
||||
# k8s/bootstrap/cert-manager/kustomization.yaml
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
helmCharts:
|
||||
- name: cert-manager
|
||||
repo: https://charts.jetstack.io
|
||||
releaseName: cert-manager
|
||||
version: v1.21.0
|
||||
namespace: cert-manager
|
||||
valuesInline:
|
||||
crds:
|
||||
enabled: true
|
||||
prometheus:
|
||||
enabled: true
|
||||
servicemonitor:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Patch (customize per environment)
|
||||
```yaml
|
||||
# k8s/bootstrap/cert-manager/kustomization.yaml
|
||||
patchesJson6902:
|
||||
- target:
|
||||
group: helm.sh
|
||||
version: v1
|
||||
kind: Release
|
||||
name: cert-manager
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/values/installCRDs
|
||||
value: "true"
|
||||
```
|
||||
|
||||
## Secrets Management (SOPS)
|
||||
|
||||
```bash
|
||||
# Encrypt secrets before committing
|
||||
sops -e secrets.yaml > secrets.enc.yaml
|
||||
git add secrets.enc.yaml
|
||||
|
||||
# ArgoCD decrypts at sync time (via plugin)
|
||||
```
|
||||
|
||||
## File Naming Conventions
|
||||
|
||||
```
|
||||
Layer directories:
|
||||
k8s/{layer}/*.yaml
|
||||
|
||||
Service subdirectories:
|
||||
k8s/{layer}/{service}/
|
||||
├── kustomization.yaml # Helm chart + patches
|
||||
├── values.yaml # Helm values
|
||||
└── config/ # Additional manifests
|
||||
├── foo.yaml
|
||||
└── bar.yaml
|
||||
|
||||
Naming:
|
||||
✓ cert-manager/values.yaml (service-specific)
|
||||
✓ cluster-roles.yaml (resource type)
|
||||
✓ storage-classes.yaml (resource type)
|
||||
✗ cert-manager-helm.yaml (redundant suffix)
|
||||
✗ my-custom-config.yaml (non-standard)
|
||||
```
|
||||
|
||||
## CI/CD Validation Pipeline
|
||||
|
||||
```yaml
|
||||
# .forgejo/workflows/validate-k8s.yaml
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
runs-on: default
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
# 1. Lint YAML
|
||||
- run: yamllint k8s/
|
||||
|
||||
# 2. Validate K8s manifests
|
||||
- run: kubeval k8s/**/*.yaml
|
||||
|
||||
# 3. Kustomize build (no apply)
|
||||
- run: |
|
||||
for dir in k8s/infrastructure k8s/bootstrap k8s/platform k8s/security k8s/applications k8s/data; do
|
||||
kustomize build $dir > /dev/null
|
||||
done
|
||||
|
||||
# 4. ArgoCD app validation (dry-run)
|
||||
- run: argocd app create --dry-run -f k8s/argocd/apps/
|
||||
|
||||
# 5. Policy check (optional)
|
||||
- run: conftest test -p policy/ k8s/**/*.yaml
|
||||
```
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
**Day 1: Design Review**
|
||||
- [ ] Review directory structure
|
||||
- [ ] Confirm layer dependencies
|
||||
- [ ] Identify secrets (need SOPS)
|
||||
- [ ] Plan kustomization strategy
|
||||
|
||||
**Day 2-3: Build Infrastructure Layer**
|
||||
- [ ] Create k8s/infrastructure/ with namespaces, storage classes, RBAC
|
||||
- [ ] Create kustomization.yaml for Layer 0
|
||||
- [ ] Test: `kustomize build k8s/infrastructure/`
|
||||
|
||||
**Day 4-5: Build Bootstrap Layer**
|
||||
- [ ] Create k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/
|
||||
- [ ] Add Helm chart references
|
||||
- [ ] Test: `kustomize build k8s/bootstrap/`
|
||||
|
||||
**Day 6-7: Build Platform Layer**
|
||||
- [ ] Create k8s/platform/{minio,longhorn,loki,prometheus}/
|
||||
- [ ] Add values overrides
|
||||
- [ ] Test: `kustomize build k8s/platform/`
|
||||
|
||||
**Day 8-9: Build Security Layer**
|
||||
- [ ] Create k8s/security/{authentik,vault}/
|
||||
- [ ] Migrate Authentik resources from Terraform
|
||||
- [ ] Encrypt secrets with SOPS
|
||||
|
||||
**Day 10-11: Build Application Layer**
|
||||
- [ ] Move k8s/talos-ci-cd/ → k8s/applications/forgejo/
|
||||
- [ ] Consolidate k8s/{logging,monitoring} → k8s/platform/ or k8s/applications/
|
||||
- [ ] Create kustomization.yaml for services
|
||||
|
||||
**Day 12: Create ArgoCD Apps**
|
||||
- [ ] Create k8s/argocd/apps/ with layer applications
|
||||
- [ ] Update AppProject permissions
|
||||
- [ ] Deploy root application (starts with Layer 0)
|
||||
|
||||
**Day 13-14: Monitor & Validate**
|
||||
- [ ] Watch ArgoCD sync for each layer
|
||||
- [ ] Verify no drift
|
||||
- [ ] Test manual edits (ArgoCD corrects them)
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
```bash
|
||||
# Rollback entire layer
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# ArgoCD detects change, syncs back to previous state
|
||||
|
||||
# Rollback single service
|
||||
git checkout <commit-hash> -- k8s/applications/forgejo/
|
||||
git commit -m "revert: forgejo back to <version>"
|
||||
git push
|
||||
```
|
||||
|
||||
## Benefits of This Architecture
|
||||
|
||||
✓ **Scalability:** Easy to add new services (copy service directory)
|
||||
✓ **Clarity:** Clear layer dependencies (no surprises)
|
||||
✓ **Safety:** Manual edits auto-corrected by ArgoCD
|
||||
✓ **Auditability:** Every change in git (who, when, why)
|
||||
✓ **Testing:** Kustomize build validates before sync
|
||||
✓ **Rollback:** Git history = disaster recovery
|
||||
✓ **Secrets:** SOPS encryption built-in
|
||||
✓ **CI/CD:** Automated validation on every PR
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Create directory structure (k8s/infrastructure/, bootstrap/, platform/, etc.)
|
||||
2. Migrate existing manifests from scattered k8s/ to new structure
|
||||
3. Create kustomization.yaml for each layer
|
||||
4. Create ArgoCD Applications for each layer
|
||||
5. Test on live cluster (non-disruptive)
|
||||
6. Update CI/CD pipeline to validate new structure
|
||||
@@ -0,0 +1,246 @@
|
||||
# GitOps IaC Implementation Summary
|
||||
|
||||
## Documents Created
|
||||
|
||||
1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design
|
||||
2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive)
|
||||
|
||||
## Architecture at a Glance
|
||||
|
||||
```
|
||||
Pure GitOps (ArgoCD only, no Terraform)
|
||||
├── Layer 0: Infrastructure (namespaces, storage, RBAC)
|
||||
├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
||||
├── Layer 2: Platform (minio, longhorn, loki, prometheus)
|
||||
├── Layer 3: Security (authentik, vault)
|
||||
├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm)
|
||||
└── Layer 5: Data (postgres, redis, kafka)
|
||||
```
|
||||
|
||||
**Key benefit:** Each layer depends on previous, staged deployment, easy rollback.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── _base/ # Shared kustomization base
|
||||
├── infrastructure/ # Layer 0 (foundation)
|
||||
├── bootstrap/ # Layer 1 (networking)
|
||||
├── platform/ # Layer 2 (services)
|
||||
├── security/ # Layer 3 (auth)
|
||||
├── applications/ # Layer 4 (business logic)
|
||||
├── data/ # Layer 5 (stateful)
|
||||
└── argocd/
|
||||
├── projects/ # AppProject definitions
|
||||
└── apps/
|
||||
├── root-app.yaml # Bootstraps Layer 0
|
||||
└── layer-*.yaml # 5 layer apps (synced in order)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Plan (Week 1)
|
||||
- [x] Review GITOPS_ARCHITECTURE.md
|
||||
- [x] Review GITOPS_MIGRATION_PLAN.md
|
||||
- [ ] Audit current cluster resources
|
||||
- [ ] Identify secrets needing encryption (SOPS)
|
||||
- [ ] Map dependencies (provided in plan)
|
||||
- **User approval required before proceeding to Phase 2**
|
||||
|
||||
### Phase 2: Build (Weeks 2-3)
|
||||
- [ ] Create k8s/ directory structure
|
||||
- [ ] Migrate Layer 0 (infrastructure)
|
||||
- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx)
|
||||
- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus)
|
||||
- [ ] Migrate Layer 3 (security: authentik, vault)
|
||||
- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.)
|
||||
- [ ] Migrate Layer 5 (data: postgres, redis, kafka)
|
||||
|
||||
### Phase 3: ArgoCD (Week 4)
|
||||
- [ ] Create root application (Layer 0 trigger)
|
||||
- [ ] Create 5 layer applications (with sync-wave ordering)
|
||||
- [ ] Update AppProject permissions
|
||||
- [ ] Test dry-run on each layer
|
||||
|
||||
### Phase 4: Deploy (Week 5)
|
||||
- [ ] Pre-flight validation (kustomize build, kubeval, yamllint)
|
||||
- [ ] Deploy root application
|
||||
- [ ] Deploy layer applications (one by one)
|
||||
- [ ] Monitor for drift
|
||||
- [ ] Verify state correction (manual edits reverted)
|
||||
|
||||
### Phase 5: Cleanup (Week 6)
|
||||
- [ ] Delete old k8s/ directories
|
||||
- [ ] Update CI/CD pipeline (remove terraform, add kustomize)
|
||||
- [ ] Archive legacy configuration
|
||||
- [ ] Final validation
|
||||
|
||||
## Key Architecture Decisions
|
||||
|
||||
### ✓ Why Layers?
|
||||
- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds
|
||||
- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only.
|
||||
- **Easy scaling:** Add new service without touching others
|
||||
- **Easy rollback:** `git revert <layer-commit>`
|
||||
|
||||
### ✓ Why Kustomize?
|
||||
- Helm chart package + values override (clean separation)
|
||||
- Kustomize build validates before sync
|
||||
- Patches enable per-environment customization
|
||||
- Standard K8s tool (no external dependencies)
|
||||
|
||||
### ✓ Why No Terraform?
|
||||
- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD)
|
||||
- Git is source of truth for everything
|
||||
- No state backend (no .tfstate files)
|
||||
- ArgoCD continuous reconciliation (drift auto-corrected)
|
||||
- All changes reviewed in PRs (audit trail in git)
|
||||
|
||||
### ✓ Why This File Structure?
|
||||
- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/)
|
||||
- **Scalable:** New service = new directory (copy template)
|
||||
- **Maintainable:** Clear ownership (who owns layer? who owns service?)
|
||||
- **GitOps-ready:** Each layer is independently deployable
|
||||
|
||||
## Secret Management (SOPS)
|
||||
|
||||
### Before (Terraform)
|
||||
```
|
||||
terraform/terraform.tfvars # Plaintext secrets (risky!)
|
||||
```
|
||||
|
||||
### After (GitOps)
|
||||
```bash
|
||||
# Encrypt secrets before commit
|
||||
sops -e secrets.yaml > secrets.enc.yaml
|
||||
git add secrets.enc.yaml
|
||||
|
||||
# ArgoCD decrypts at sync time (SOPS plugin installed)
|
||||
# Git stores only encrypted version (safe)
|
||||
```
|
||||
|
||||
**Secrets to encrypt:**
|
||||
- Authentik bootstrap password
|
||||
- MinIO root credentials
|
||||
- Database passwords
|
||||
- API tokens
|
||||
- OAuth client secrets
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Before
|
||||
```
|
||||
git push → terraform validate → terraform plan → manual apply
|
||||
```
|
||||
|
||||
### After (Pure GitOps)
|
||||
```
|
||||
git push → yamllint → kubeval → kustomize build → argocd validation
|
||||
↓
|
||||
ArgoCD auto-syncs (if PR merged)
|
||||
```
|
||||
|
||||
**No manual steps. No terraform. No state management. Pure GitOps.**
|
||||
|
||||
## Comparison: Old vs. New
|
||||
|
||||
| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) |
|
||||
|--------|--------------------------|-------------------|
|
||||
| **State management** | tfstate files | Git history |
|
||||
| **Secrets** | Plaintext in .tfvars | SOPS encrypted |
|
||||
| **Manual steps** | terraform apply required | git push only |
|
||||
| **Rollback** | Revert tfstate + code | git revert + push |
|
||||
| **Audit trail** | Terraform logs | Git history + ArgoCD logs |
|
||||
| **Complexity** | High (2 tools) | Low (1 tool) |
|
||||
| **New services** | Edit .tf files | Copy service directory |
|
||||
| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects |
|
||||
|
||||
## Non-Disruptive Migration
|
||||
|
||||
**Key principle:** Live cluster remains running during entire migration.
|
||||
|
||||
**How?**
|
||||
1. Build k8s/ structure **alongside** existing configuration
|
||||
2. Deploy each layer via ArgoCD while old method still runs
|
||||
3. Once new layer verified: delete old configuration
|
||||
4. If issue detected: rollback (git revert) + redeploy old version
|
||||
|
||||
**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed)
|
||||
|
||||
## Rollback Procedure (At Any Point)
|
||||
|
||||
```bash
|
||||
# 1. Identify broken layer
|
||||
argocd app get layer-2-platform # Shows OutOfSync, error
|
||||
|
||||
# 2. Revert that layer
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# 3. ArgoCD detects change, syncs back to previous version
|
||||
# Expected: ~3 minutes for full reconciliation
|
||||
|
||||
# 4. Verify
|
||||
argocd app get layer-2-platform # Should be Synced again
|
||||
```
|
||||
|
||||
## Day 1 Action Items
|
||||
|
||||
1. **Read GITOPS_ARCHITECTURE.md** (30 min)
|
||||
- Understand layers
|
||||
- Understand directory structure
|
||||
- Understand kustomization strategy
|
||||
|
||||
2. **Read GITOPS_MIGRATION_PLAN.md** (30 min)
|
||||
- Understand phases
|
||||
- Understand timeline
|
||||
- Understand success criteria
|
||||
|
||||
3. **Review dependency map** (provided in plan)
|
||||
- Confirm layer dependencies match current setup
|
||||
- Identify any services not in layers
|
||||
|
||||
4. **Audit secrets** (30 min)
|
||||
```bash
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
# Plan which need SOPS encryption
|
||||
```
|
||||
|
||||
5. **Decision point:** Proceed to Phase 2 (Week 2)?
|
||||
- YES → Start building k8s/ structure
|
||||
- NO → Clarify concerns, refine plan
|
||||
|
||||
## Benefits Summary
|
||||
|
||||
✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth
|
||||
✓ **Safer:** All changes reviewed in PRs, git history as audit trail
|
||||
✓ **Scalable:** New services = copy directory, update git, done
|
||||
✓ **Resilient:** State drift auto-corrected, rollback via `git revert`
|
||||
✓ **Auditable:** Every change in git with who/when/why
|
||||
✓ **Automated:** No manual kubectl apply, no manual terraform apply
|
||||
✓ **Testable:** Kustomize build validates before sync
|
||||
✓ **Recoverable:** git history = disaster recovery
|
||||
|
||||
## Support
|
||||
|
||||
- Architecture questions → GITOPS_ARCHITECTURE.md
|
||||
- Implementation questions → GITOPS_MIGRATION_PLAN.md
|
||||
- Syntax/tools questions → See referenced documentation
|
||||
- Kustomize: https://kustomize.io/
|
||||
- ArgoCD: https://argo-cd.readthedocs.io/
|
||||
- SOPS: https://github.com/mozilla/sops
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **User review:** Read both architecture documents (60 min)
|
||||
2. **User decision:** Proceed to Phase 2 or refine plan?
|
||||
3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday)
|
||||
|
||||
---
|
||||
|
||||
**Status:** Ready for implementation
|
||||
**Terraform:** Deleted ✓
|
||||
**Architecture:** Designed ✓
|
||||
**Plan:** Ready ✓
|
||||
**Waiting for:** User approval to proceed
|
||||
@@ -0,0 +1,573 @@
|
||||
# GitOps Migration Plan: Implement Production-Grade Architecture
|
||||
|
||||
## Current State → Target State
|
||||
|
||||
### Current (Messy)
|
||||
```
|
||||
k8s/
|
||||
├── talos-ci-cd/ (forgejo, runner)
|
||||
├── talos-iam/ (authentik, vault)
|
||||
├── logging/ (loki, promtail, grafana)
|
||||
├── monitoring/ (prometheus, alerts)
|
||||
├── storage/ (minio, longhorn)
|
||||
├── ddb/ (postgres)
|
||||
├── sqs/ (kafka, queue-crd)
|
||||
├── temporal/ (temporal)
|
||||
├── argocd/apps/ (20 Applications scattered)
|
||||
└── ... (more scattered)
|
||||
```
|
||||
|
||||
### Target (Clean, Layered)
|
||||
```
|
||||
k8s/
|
||||
├── infrastructure/ (namespaces, storage, RBAC)
|
||||
├── bootstrap/ (cert-manager, cilium, ingress-nginx)
|
||||
├── platform/ (storage, observability: minio, longhorn, loki, prometheus)
|
||||
├── security/ (authentik, vault, cert-issuer)
|
||||
├── applications/ (forgejo, grafana, portainer, temporal, llm)
|
||||
├── data/ (postgres, redis, kafka)
|
||||
└── argocd/
|
||||
├── projects/ (AppProject definitions)
|
||||
└── apps/ (6 layer Applications only)
|
||||
```
|
||||
|
||||
## Phase 1: Plan & Validate (Week 1)
|
||||
|
||||
### Step 1.1: Review GITOPS_ARCHITECTURE.md
|
||||
- [ ] Understand directory structure
|
||||
- [ ] Understand layer dependencies
|
||||
- [ ] Understand Kustomization strategy
|
||||
- User approval: YES/NO
|
||||
|
||||
### Step 1.2: Audit current resources
|
||||
```bash
|
||||
# Export all current resources
|
||||
kubectl get all -A -o yaml > /tmp/current-state-backup.yaml
|
||||
|
||||
# Count resources per layer
|
||||
kubectl get deployments -A | wc -l
|
||||
kubectl get statefulsets -A | wc -l
|
||||
kubectl get daemonsets -A | wc -l
|
||||
kubectl get services -A | wc -l
|
||||
kubectl get configmaps -A | wc -l
|
||||
kubectl get secrets -A | wc -l
|
||||
```
|
||||
|
||||
### Step 1.3: Identify secrets needing encryption
|
||||
```bash
|
||||
# Find secrets in current manifests
|
||||
grep -r "kind: Secret" k8s/ --include="*.yaml"
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
|
||||
# Plan SOPS encryption for:
|
||||
# - Authentik bootstrap password
|
||||
# - MinIO credentials
|
||||
# - Database passwords
|
||||
# - API tokens
|
||||
```
|
||||
|
||||
### Step 1.4: Dependency mapping
|
||||
```
|
||||
Layer 0 (Infrastructure)
|
||||
└─ Layer 1 (Bootstrap)
|
||||
├─ cert-manager (creates certificates)
|
||||
├─ cilium (network)
|
||||
└─ ingress-nginx (entry point)
|
||||
└─ Layer 2 (Platform)
|
||||
├─ minio (storage)
|
||||
├─ longhorn (PV storage)
|
||||
├─ loki (logs)
|
||||
└─ prometheus (metrics)
|
||||
└─ Layer 3 (Security)
|
||||
├─ authentik (auth)
|
||||
└─ vault (secrets)
|
||||
└─ Layer 4 (Applications)
|
||||
├─ forgejo
|
||||
├─ grafana
|
||||
├─ portainer
|
||||
├─ temporal
|
||||
└─ llm
|
||||
└─ Layer 5 (Data)
|
||||
├─ postgres
|
||||
├─ redis
|
||||
└─ kafka
|
||||
```
|
||||
|
||||
## Phase 2: Build Directory Structure (Weeks 2-3)
|
||||
|
||||
### Step 2.1: Create base directories
|
||||
```bash
|
||||
cd k8s/
|
||||
|
||||
# Create layers
|
||||
mkdir -p infrastructure bootstrap platform security applications data
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
for dir in infrastructure bootstrap platform security applications data; do
|
||||
cat > $dir/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: CHANGE_ME
|
||||
|
||||
resources: [] # Will add resources below
|
||||
|
||||
# Placeholders for this layer
|
||||
EOF
|
||||
done
|
||||
```
|
||||
|
||||
### Step 2.2: Migrate Layer 0 (Infrastructure)
|
||||
**Resources to create:**
|
||||
- Namespaces (all 20+)
|
||||
- Storage classes (longhorn, xfs, longhorn-kafka)
|
||||
- Service accounts (terraform-ci, etc.)
|
||||
- Cluster roles (terraform-ci, admin, etc.)
|
||||
- Network policies
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Extract from current cluster
|
||||
kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml
|
||||
kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml
|
||||
kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml
|
||||
kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml
|
||||
|
||||
# Clean up (remove status, owner refs, etc.)
|
||||
# Add to k8s/infrastructure/kustomization.yaml:
|
||||
resources:
|
||||
- namespaces.yaml
|
||||
- storage-classes.yaml
|
||||
- service-accounts.yaml
|
||||
- cluster-roles.yaml
|
||||
|
||||
# Test
|
||||
kustomize build k8s/infrastructure/
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/infrastructure/kustomization.yaml
|
||||
- [x] k8s/infrastructure/namespaces.yaml
|
||||
- [x] k8s/infrastructure/storage-classes.yaml
|
||||
- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml
|
||||
|
||||
### Step 2.3: Migrate Layer 1 (Bootstrap)
|
||||
|
||||
**Services: cert-manager, cilium, ingress-nginx**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx}
|
||||
|
||||
# For each service:
|
||||
# 1. Export Helm values
|
||||
helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml
|
||||
|
||||
# 2. Create kustomization.yaml with Helm chart ref
|
||||
cat > bootstrap/cert-manager/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: cert-manager
|
||||
|
||||
helmCharts:
|
||||
- name: cert-manager
|
||||
repo: https://charts.jetstack.io
|
||||
version: v1.21.0
|
||||
releaseName: cert-manager
|
||||
valuesFile: values.yaml
|
||||
EOF
|
||||
|
||||
# 3. Repeat for cilium, ingress-nginx
|
||||
|
||||
# 4. Add to k8s/bootstrap/kustomization.yaml:
|
||||
resources:
|
||||
- cert-manager/kustomization.yaml
|
||||
- cilium/kustomization.yaml
|
||||
- ingress-nginx/kustomization.yaml
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml
|
||||
- [x] k8s/bootstrap/kustomization.yaml
|
||||
|
||||
### Step 2.4: Migrate Layer 2 (Platform)
|
||||
|
||||
**Services: minio, longhorn, loki, prometheus, promtail**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p platform/{minio,longhorn,loki,prometheus,promtail}
|
||||
|
||||
# Migrate minio
|
||||
# 1. Export existing values
|
||||
helm get values minio -n storage > platform/minio/values.yaml
|
||||
|
||||
# 2. Export bucket configs
|
||||
kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/
|
||||
|
||||
# 3. Create kustomization.yaml
|
||||
cat > platform/minio/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: storage
|
||||
|
||||
helmCharts:
|
||||
- name: minio
|
||||
repo: https://charts.min.io
|
||||
version: 14.6.25 # Current version
|
||||
releaseName: minio
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/minio-buckets.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for loki, prometheus, promtail
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml
|
||||
- [x] k8s/platform/kustomization.yaml
|
||||
|
||||
### Step 2.5: Migrate Layer 3 (Security)
|
||||
|
||||
**Services: authentik, vault**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p security/{authentik,vault}
|
||||
|
||||
# Migrate authentik
|
||||
helm get values authentik -n iam > security/authentik/values.yaml
|
||||
|
||||
# Export Authentik resources
|
||||
kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml
|
||||
kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml
|
||||
kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml
|
||||
|
||||
# Create kustomization.yaml
|
||||
cat > security/authentik/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: iam
|
||||
|
||||
helmCharts:
|
||||
- name: authentik
|
||||
repo: https://charts.goauthentik.io
|
||||
version: 2024.12.1
|
||||
releaseName: authentik
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/apps.yaml
|
||||
- config/groups.yaml
|
||||
- config/oauth-providers.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for vault
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/security/{authentik,vault}/kustomization.yaml
|
||||
- [x] k8s/security/{authentik,vault}/values.yaml
|
||||
- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml
|
||||
- [x] k8s/security/kustomization.yaml
|
||||
|
||||
### Step 2.6: Migrate Layer 4 (Applications)
|
||||
|
||||
**Services: forgejo, grafana, portainer, temporal, llm**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p applications/{forgejo,grafana,portainer,temporal,llm}
|
||||
|
||||
# Move existing manifests
|
||||
cp -r k8s/talos-ci-cd/* applications/forgejo/
|
||||
cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc.
|
||||
cp -r k8s/portainer/* applications/portainer/
|
||||
cp -r k8s/temporal/* applications/temporal/
|
||||
cp -r k8s/llm/* applications/llm/
|
||||
|
||||
# For each, create kustomization.yaml with Helm chart ref
|
||||
# (or keep existing manifests if not Helm)
|
||||
|
||||
# Create layer kustomization.yaml
|
||||
cat > applications/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- forgejo/kustomization.yaml
|
||||
- grafana/kustomization.yaml
|
||||
- portainer/kustomization.yaml
|
||||
- temporal/kustomization.yaml
|
||||
- llm/kustomization.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml
|
||||
- [x] k8s/applications/kustomization.yaml
|
||||
|
||||
### Step 2.7: Migrate Layer 5 (Data)
|
||||
|
||||
**Services: postgres (CloudNativePG), redis, kafka**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p data/{postgres,redis,kafka}
|
||||
|
||||
# Export existing configurations
|
||||
kubectl get cnpg -A -o yaml > data/postgres/config.yaml
|
||||
kubectl get redis -A -o yaml > data/redis/config.yaml
|
||||
kubectl get kafka -A -o yaml > data/kafka/config.yaml
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
|
||||
# Repeat process...
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml
|
||||
- [x] k8s/data/kustomization.yaml
|
||||
|
||||
## Phase 3: Create ArgoCD Application Definitions (Week 4)
|
||||
|
||||
### Step 3.1: Create layer Applications
|
||||
```bash
|
||||
mkdir -p k8s/argocd/apps/layers
|
||||
|
||||
# Root application (Layer 0 only, triggers rest)
|
||||
cat > k8s/argocd/apps/root-app.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: homelab-root
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/infrastructure
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Layer 1 application
|
||||
cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: layer-1-bootstrap
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "1"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/bootstrap
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5)
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/argocd/apps/root-app.yaml
|
||||
- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml
|
||||
|
||||
### Step 3.2: Create kustomization.yaml for apps
|
||||
```bash
|
||||
cat > k8s/argocd/apps/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- root-app.yaml
|
||||
- layer-1-bootstrap.yaml
|
||||
- layer-2-platform.yaml
|
||||
- layer-3-security.yaml
|
||||
- layer-4-applications.yaml
|
||||
- layer-5-data.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
## Phase 4: Deploy & Validate (Week 5)
|
||||
|
||||
### Step 4.1: Pre-flight checks
|
||||
```bash
|
||||
# Validate all kustomization files
|
||||
for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do
|
||||
echo "Validating $dir..."
|
||||
kustomize build $dir > /tmp/validate.yaml
|
||||
kubeval /tmp/validate.yaml || exit 1
|
||||
done
|
||||
|
||||
# Validate ArgoCD apps
|
||||
kubeval k8s/argocd/apps/*.yaml
|
||||
|
||||
# Test YAML lint
|
||||
yamllint k8s/
|
||||
```
|
||||
|
||||
### Step 4.2: Deploy root application
|
||||
```bash
|
||||
# Apply root app (Layer 0 only)
|
||||
kubectl apply -f k8s/argocd/apps/root-app.yaml
|
||||
|
||||
# Watch sync
|
||||
argocd app watch homelab-root
|
||||
|
||||
# Verify infrastructure deployed
|
||||
kubectl get ns # Should have all namespaces
|
||||
kubectl get sc # Should have all storage classes
|
||||
```
|
||||
|
||||
### Step 4.3: Deploy layer applications
|
||||
```bash
|
||||
# Apply layer applications (one by one)
|
||||
kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml
|
||||
sleep 5
|
||||
argocd app watch layer-1-bootstrap
|
||||
|
||||
# Once Layer 1 synced: Layer 2
|
||||
kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml
|
||||
argocd app watch layer-2-platform
|
||||
|
||||
# ... repeat for layers 3, 4, 5
|
||||
```
|
||||
|
||||
### Step 4.4: Monitor for issues
|
||||
```bash
|
||||
# Check ArgoCD apps
|
||||
argocd app list
|
||||
|
||||
# Watch events
|
||||
kubectl get events -A -w
|
||||
|
||||
# Monitor pod status
|
||||
kubectl get pods -A
|
||||
|
||||
# Check logs
|
||||
kubectl logs -f -n argocd deployment/argocd-application-controller
|
||||
```
|
||||
|
||||
### Step 4.5: Validate state drift protection
|
||||
```bash
|
||||
# Manually edit a resource
|
||||
kubectl edit deployment cert-manager -n cert-manager
|
||||
# Change replicas from 1 → 2
|
||||
|
||||
# Wait 3 minutes (ArgoCD reconciliation interval)
|
||||
sleep 180
|
||||
|
||||
# Check: should be back to 1 replica
|
||||
kubectl get deployment cert-manager -n cert-manager
|
||||
# Expected: 1/1 Ready (reverted by ArgoCD)
|
||||
|
||||
# Verify in ArgoCD
|
||||
argocd app get layer-1-bootstrap
|
||||
# Expected: Synced status
|
||||
```
|
||||
|
||||
## Phase 5: Cleanup (Week 6)
|
||||
|
||||
### Step 5.1: Delete old directories
|
||||
```bash
|
||||
# After all apps synced successfully:
|
||||
rm -rf k8s/talos-ci-cd/
|
||||
rm -rf k8s/talos-iam/
|
||||
rm -rf k8s/logging/
|
||||
rm -rf k8s/monitoring/
|
||||
rm -rf k8s/storage/
|
||||
rm -rf k8s/ddb/
|
||||
rm -rf k8s/sqs/
|
||||
rm -rf k8s/temporal/
|
||||
# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd
|
||||
```
|
||||
|
||||
### Step 5.2: Update CI/CD pipeline
|
||||
```bash
|
||||
# Update .forgejo/workflows/
|
||||
# - Remove: terraform validate/plan/apply
|
||||
# - Add: kustomize build validation
|
||||
# - Add: argocd app validation
|
||||
```
|
||||
|
||||
### Step 5.3: Archive old configuration
|
||||
```bash
|
||||
# Keep for reference only
|
||||
mkdir k8s/.archive/
|
||||
git mv k8s/old-structure-backup k8s/.archive/
|
||||
git commit -m "archive: old k8s structure (moved to pure GitOps)"
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
**If something breaks during migration:**
|
||||
|
||||
```bash
|
||||
# Option 1: Rollback entire layer
|
||||
git checkout HEAD~1 -- k8s/bootstrap/
|
||||
git commit -m "revert: bootstrap layer (investigating)"
|
||||
# ArgoCD will resync to previous version automatically
|
||||
|
||||
# Option 2: Pause ArgoCD sync
|
||||
argocd app set layer-1-bootstrap --sync-policy none
|
||||
# Investigate, then re-enable:
|
||||
argocd app set layer-1-bootstrap --sync-policy automated
|
||||
|
||||
# Option 3: Full rollback to pre-migration
|
||||
git reset --hard <commit-before-migration>
|
||||
argocd app set homelab-root --sync-policy none
|
||||
# Manual investigation, then re-enable
|
||||
```
|
||||
|
||||
## Timeline Summary
|
||||
|
||||
```
|
||||
Week 1: Plan & validate (reviews, dependency mapping)
|
||||
Week 2: Layers 0-1 (infrastructure, bootstrap)
|
||||
Week 3: Layers 2-3 (platform, security)
|
||||
Week 3: Layers 4-5 (applications, data)
|
||||
Week 4: ArgoCD applications + layer definitions
|
||||
Week 5: Deploy & validate (watch for issues)
|
||||
Week 6: Cleanup & CI/CD updates
|
||||
```
|
||||
|
||||
**Total: 6 weeks, non-disruptive (all layers coexist during migration)**
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✓ All 20+ services deployed via ArgoCD
|
||||
✓ No manual kubectl apply in production
|
||||
✓ State drift detected & corrected automatically
|
||||
✓ All changes in git (reviewed via PR)
|
||||
✓ Rollback possible at any time (git history)
|
||||
✓ CI/CD validates all commits
|
||||
✓ Secrets encrypted with SOPS
|
||||
✓ New services can be added (copy service directory)
|
||||
Reference in New Issue
Block a user