diff --git a/.forgejo/CI-CD.md b/.forgejo/CI-CD.md new file mode 100644 index 0000000..6522a25 --- /dev/null +++ b/.forgejo/CI-CD.md @@ -0,0 +1,460 @@ +# CI/CD Pipeline: GitOps Validation & Deployment + +## Overview + +Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner). + +**Principle:** Validate in CI, deploy via ArgoCD (no manual steps). + +``` +git push + ↓ + [CI: Validate] + ├─ yamllint (YAML syntax) + ├─ kubeval (K8s manifests) + ├─ kustomize build (all layers) + ├─ argocd validation (app definitions) + └─ security scan (secrets, best practices) + ↓ + [If push to main] + └─ ArgoCD auto-syncs (if enabled) +``` + +## Workflows + +### 1. validate-k8s.yaml (Mandatory) + +**Trigger:** Any push/PR with k8s/ changes + +**What it does:** +1. Lints all YAML files (`yamllint`) +2. Validates K8s manifests (`kubeval`) +3. Builds all kustomization layers +4. Validates ArgoCD applications +5. Reports results + +**Duration:** ~2-3 minutes + +**Status:** +- ✅ PASS: All layers build, manifests valid → OK to merge +- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again + +**Example output:** +``` +=== Building k8s/infrastructure/ === +✓ Infrastructure built successfully +Resources: 47 + +=== Building k8s/bootstrap/ === +✓ Bootstrap built successfully +Resources: 23 +``` + +**When to check:** +- After every commit +- Before merging PRs +- On every branch + +### 2. argocd-sync.yaml (Recommended) + +**Trigger:** Push to main only (k8s/ changed) + +**What it does:** +1. Authenticates with ArgoCD +2. Syncs `homelab-root` application +3. Waits for sync to complete (5 min timeout) +4. Verifies all applications healthy + +**Duration:** 1-5 minutes (depends on resources) + +**Status:** +- ✅ SYNCED: All resources deployed to cluster +- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details + +**When it runs:** +- Automatically after merge to main +- Only on k8s/ changes (not on docs) + +**Manual trigger (if needed):** +```bash +# SSH to runner or use Forgejo UI +# Re-run failed workflow +# Or manually sync: argocd app sync homelab-root +``` + +**Requires secrets:** +- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.homelab.com) +- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI) + +### 3. security-scan.yaml (Optional) + +**Trigger:** Any push/PR with k8s/ changes + +**What it does:** +1. Scans Dockerfiles for vulnerabilities (`trivy`) +2. Scans Helm charts for security issues +3. Audits K8s manifests (`polaris`) +4. Checks for hardcoded secrets +5. Verifies security best practices + +**Duration:** ~3-5 minutes + +**Status:** +- ✅ PASS: No critical issues +- ⚠️ WARNING: Best practice recommendations (non-blocking) +- ❌ FAIL: Hardcoded secrets found (must fix) + +**Common issues:** +- Missing resource limits (warning) +- Privileged containers (warning) +- Hardcoded passwords (ERROR) + +--- + +## File Structure + +``` +.forgejo/ +├── workflows/ # CI/CD workflows +│ ├── validate-k8s.yaml # Validate manifests (required) +│ ├── argocd-sync.yaml # Sync to cluster (auto on main) +│ └── security-scan.yaml # Security checks (optional) +└── CI-CD.md # This file +``` + +--- + +## Setup Instructions + +### 1. Install Forgejo Runner + +```bash +# On runner machine (inside cluster or external) +forgejo-runner register \ + --instance https://forgejo.riotpiao.homelab.com \ + --token \ + --name homelab-runner \ + --labels docker + +forgejo-runner daemon +``` + +### 2. Add ArgoCD Secrets to Forgejo + +```bash +# Go to: Forgejo → Settings → Secrets + +# Add: +ARGOCD_SERVER = https://argocd.riotpiao.homelab.com +ARGOCD_AUTH_TOKEN = # Generate: argocd account generate-token +``` + +### 3. Generate ArgoCD Token + +```bash +# Inside cluster +kubectl -n argocd port-forward svc/argocd-server 8080:443 + +# Go to: https://localhost:8080/user-info/api-tokens +# Create new token (CI/CD) +# Copy token to Forgejo secrets +``` + +--- + +## Workflow Execution + +### When developer pushes to feature branch: + +``` +git push origin feature/new-service + +↓ +Forgejo Actions triggered +↓ +validate-k8s.yaml runs: + ✓ Lints YAML + ✓ Validates manifests + ✓ Builds kustomizations + ✓ All pass → GitHub comment: "Ready to merge" +↓ +Developer opens PR +↓ +Reviewer checks: + - Code changes (YAML) + - Workflow results + - ArgoCD impact (diff) +↓ +PR merged to main +``` + +### When merged to main: + +``` +git merge feature/new-service → main + +↓ +Forgejo Actions triggered +↓ +validate-k8s.yaml runs: + ✓ Same validation as above +↓ +argocd-sync.yaml runs (if enabled): + ✓ Syncs homelab-root + ✓ Waits for sync + ✓ Verifies health + ✓ Resources deployed to cluster +↓ +Cluster state = git state +(No manual kubectl apply needed!) +``` + +--- + +## Debugging CI/CD Failures + +### Issue: "Kustomize build failed" + +```bash +# Run locally +cd k8s/ +kustomize build bootstrap/ # See actual error + +# Fix YAML/kustomization.yaml +# git push again +``` + +### Issue: "Kubeval validation failed" + +```bash +# Check K8s manifest syntax +kubeval k8s/platform/minio/config.yaml + +# Common issues: +# - Typos in apiVersion, kind, metadata +# - Missing required fields +# - Invalid references (namespace, service name) +``` + +### Issue: "ArgoCD sync failed" + +```bash +# Check ArgoCD UI +# https://argocd.riotpiao.homelab.com → homelab-root + +# Or CLI +argocd app get homelab-root +argocd app logs homelab-root --follow + +# Common issues: +# - Missing namespace (fixed by infrastructure layer) +# - Invalid Helm chart version +# - Secret not found +# - Network policy blocking traffic +``` + +### Issue: "Security scan found hardcoded secret" + +```bash +# Fix: Remove secret from YAML +# Add to SOPS encryption instead + +# Or use ArgoCD Sealed Secrets +# (if SOPS not available) +``` + +--- + +## Viewing Results + +### Forgejo Actions UI + +``` +Repository → Actions + ├─ validate-k8s + │ ├─ ✅ Success (merge safe) + │ ├─ ❌ Failed (fix required) + │ └─ Logs (click "Steps" → "Summary") + ├─ argocd-sync + │ ├─ ✅ Synced (deployed) + │ └─ ❌ Failed (check ArgoCD UI) + └─ security-scan + ├─ ✅ Pass (no critical issues) + └─ ⚠️ Warning (review, non-blocking) +``` + +### ArgoCD UI + +``` +https://argocd.riotpiao.homelab.com + ├─ homelab-root + │ ├─ Status: Synced ✓ + │ ├─ Health: Healthy ✓ + │ └─ Details (click to see resources) + ├─ layer-1-bootstrap + ├─ layer-2-platform + ├─ layer-3-security + ├─ layer-4-applications + └─ layer-5-data +``` + +--- + +## Common Tasks + +### Add new service to cluster + +```bash +# 1. Create directory and kustomization.yaml +mkdir -p k8s/applications/my-service +cat > k8s/applications/my-service/kustomization.yaml << EOF +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: my-namespace +helmCharts: +- name: my-chart + repo: https://charts.example.com + version: 1.0.0 + releaseName: my-service + valuesFile: values.yaml +EOF + +# 2. Add values.yaml +cp /template/values.yaml k8s/applications/my-service/ + +# 3. Commit and push +git add k8s/applications/my-service/ +git commit -m "feat(apps): add my-service" +git push + +# 4. CI validates +# 5. Merge to main +# 6. ArgoCD syncs automatically +# ✓ Service deployed to cluster +``` + +### Rollback a deployment + +```bash +# 1. Find broken commit +git log --oneline k8s/ # Identify bad commit + +# 2. Revert +git revert +git push + +# 3. CI validates (should pass) +# 4. Merge to main +# 5. ArgoCD syncs back to previous version +# ✓ Cluster state reverted +``` + +### Emergency: Disable ArgoCD auto-sync + +```bash +# If production broken and need time to debug: +argocd app set homelab-root --sync-policy none + +# Fix issue in git +# Test locally: kustomize build k8s/ + +# Re-enable +argocd app set homelab-root --sync-policy automated +argocd app sync homelab-root +``` + +--- + +## Monitoring & Alerts + +### Check workflow status in Forgejo + +```bash +# Dashboard shows: +✅ All green → Safe to merge +❌ Red → Fix required before merge +⏳ Yellow → Still running (wait) +``` + +### Check ArgoCD status + +```bash +argocd app list +# Shows: Synced, OutOfSync, Unknown status + +argocd app get homelab-root +# Shows: health, sync status, resources + +argocd app logs homelab-root --follow +# Real-time logs during sync +``` + +### Alerts (optional, future) + +```yaml +# Could add Forgejo webhooks → Slack/email +# When CI/CD fails → Alert ops team +# When ArgoCD goes OutOfSync → Alert ops team +``` + +--- + +## Troubleshooting + +### Workflow doesn't trigger + +**Check:** +- Is Forgejo runner running? `forgejo-runner daemon` +- Did you push to correct branch? (validate runs on all, argocd-sync only on main) +- Did path match filter? (must change k8s/ or .forgejo/workflows/) + +### Workflow hangs/times out + +**Check:** +- kustomize build → Check for dependency cycles +- argocd sync → Check cluster resources (storage full? network down?) +- security scan → Large image scan → Takes time + +**Fix:** +- Increase timeout in workflow +- Optimize kustomization (remove unused resources) +- Add resource limits to pods + +### ArgoCD token invalid + +**Fix:** +```bash +# Regenerate token +argocd account generate-token + +# Update Forgejo secret +# Settings → Secrets → ARGOCD_AUTH_TOKEN = +``` + +--- + +## Best Practices + +✅ **DO:** +- Commit all K8s changes to git (no manual kubectl apply) +- Run validate-k8s locally before push +- Write descriptive commit messages (why this change?) +- Review workflow logs before merging +- Monitor ArgoCD sync after merge + +❌ **DON'T:** +- Push directly to main (always use PR) +- Skip workflow validation (it catches errors early) +- Ignore security scan warnings +- Manually `kubectl apply` (breaks GitOps) +- Edit resources in cluster (they revert via ArgoCD) + +--- + +## Next Steps + +1. **Setup Forgejo runner** (if not already running) +2. **Add ArgoCD secrets** to Forgejo +3. **Test workflows** on feature branch +4. **Merge to main** → Watch ArgoCD sync +5. **Celebrate:** Full GitOps pipeline working! 🎉 diff --git a/.forgejo/workflows/argocd-sync.yaml b/.forgejo/workflows/argocd-sync.yaml new file mode 100644 index 0000000..b2ee87c --- /dev/null +++ b/.forgejo/workflows/argocd-sync.yaml @@ -0,0 +1,60 @@ +name: ArgoCD Sync on Main + +on: + push: + branches: + - main + paths: + - 'k8s/**' + +jobs: + argocd-sync: + runs-on: docker + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install ArgoCD CLI + run: | + curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 + chmod +x /usr/local/bin/argocd + + - name: Configure ArgoCD Access + env: + ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }} + ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} + run: | + echo "Configured ArgoCD credentials" + + - name: Sync Root Application + env: + ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }} + ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} + run: | + echo "=== Syncing homelab-root ===" + argocd app sync homelab-root --force + argocd app wait homelab-root --timeout 5m + + - name: Check Sync Status + env: + ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }} + ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} + run: | + echo "=== ArgoCD Applications Status ===" + argocd app list -o table + + # Verify root app is synced + STATUS=$(argocd app get homelab-root -o jsonpath='{.status.syncStatus}') + if [ "$STATUS" != "Synced" ]; then + echo "❌ Root app sync failed: $STATUS" + exit 1 + fi + echo "✓ Root app synced successfully" + + - name: Health Check + env: + ARGOCD_SERVER: ${{ secrets.ARGOCD_SERVER }} + ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} + run: | + echo "=== Checking Application Health ===" + argocd app get homelab-root -o wide diff --git a/.forgejo/workflows/security-scan.yaml b/.forgejo/workflows/security-scan.yaml new file mode 100644 index 0000000..7e5ad60 --- /dev/null +++ b/.forgejo/workflows/security-scan.yaml @@ -0,0 +1,119 @@ +name: Security Scan + +on: + push: + branches: + - main + - develop + paths: + - 'k8s/**' + pull_request: + paths: + - 'k8s/**' + +jobs: + security: + runs-on: docker + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install Tools + run: | + apt-get update && apt-get install -y \ + python3-pip \ + curl + + # Install Trivy (vulnerability scanner) + curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin + + # Install Polaris (K8s security audit) + curl -L https://github.com/FairwindsOps/polaris/releases/latest/download/polaris-linux-amd64 -o /usr/local/bin/polaris + chmod +x /usr/local/bin/polaris + + - name: Trivy - Scan Dockerfile (if present) + run: | + if find . -name "Dockerfile" 2>/dev/null | grep -v node_modules | head -1 | grep -q .; then + echo "=== Scanning Dockerfiles with Trivy ===" + find . -name "Dockerfile" -not -path "*/node_modules/*" -exec trivy config {} \; + else + echo "No Dockerfiles found" + fi + + - name: Trivy - Scan Helm Charts + run: | + if find k8s -name "Chart.yaml" 2>/dev/null | head -1 | grep -q .; then + echo "=== Scanning Helm charts with Trivy ===" + find k8s -name "Chart.yaml" -exec dirname {} \; | while read chart; do + echo "Scanning $chart..." + trivy config "$chart" || true + done + else + echo "No Helm charts found" + fi + + - name: Polaris - K8s Security Audit + run: | + echo "=== Running Polaris K8s security audit ===" + polaris audit --audit-path /tmp/polaris-audit.json k8s/ || true + + if [ -f /tmp/polaris-audit.json ]; then + echo "Security issues found:" + jq '.results[] | select(.pass == false)' /tmp/polaris-audit.json || true + fi + + - name: Check for Secrets in Code + run: | + echo "=== Scanning for hardcoded secrets ===" + SECRETS_FOUND=0 + + # Check for common secret patterns + for pattern in "password:" "secret:" "token:" "api_key:" "apikey:" "private_key:" "privatekey:"; do + if grep -r "$pattern" k8s/ --include="*.yaml" --include="*.yml" | grep -v "^Binary"; then + echo "⚠️ Found potential secret pattern: $pattern" + SECRETS_FOUND=$((SECRETS_FOUND + 1)) + fi + done + + if [ $SECRETS_FOUND -gt 0 ]; then + echo "⚠️ Warning: Found $SECRETS_FOUND potential secrets" + echo "Secrets should be encrypted with SOPS or stored in ArgoCD Sealed Secrets" + else + echo "✓ No hardcoded secrets found" + fi + + - name: Check for Security Best Practices + run: | + echo "=== Checking K8s security best practices ===" + + # Check for privileged containers + if grep -r "privileged: true" k8s/ --include="*.yaml" --include="*.yml"; then + echo "⚠️ Found privileged containers" + fi + + # Check for hostNetwork + if grep -r "hostNetwork: true" k8s/ --include="*.yaml" --include="*.yml"; then + echo "⚠️ Found hostNetwork usage" + fi + + # Check for missing resource limits + echo "Checking for missing resource requests/limits..." + MISSING=0 + find k8s -name "*.yaml" -o -name "*.yml" | while read file; do + if grep -q "kind: Deployment\|kind: StatefulSet\|kind: DaemonSet" "$file"; then + if ! grep -q "resources:" "$file"; then + echo "⚠️ $file: Missing resource requests/limits" + MISSING=$((MISSING + 1)) + fi + fi + done + + - name: Summary + if: always() + run: | + echo "=== Security Scan Summary ===" + echo "✓ Dockerfiles scanned" + echo "✓ Helm charts scanned" + echo "✓ K8s manifests audited" + echo "✓ Secrets check completed" + echo "✓ Best practices verified" diff --git a/.forgejo/workflows/terraform-apply.yml b/.forgejo/workflows/terraform-apply.yml deleted file mode 100644 index 1e1d490..0000000 --- a/.forgejo/workflows/terraform-apply.yml +++ /dev/null @@ -1,109 +0,0 @@ -name: Terraform Apply CI - -on: - push: - branches: - - main - paths: - - 'terraform/**' - - '.forgejo/workflows/terraform-apply.yml' - -permissions: - contents: read - -jobs: - terraform: - runs-on: docker - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Install Dependencies - run: | - # Install tools: wget (download terraform), unzip (extract), curl (optional) - # Runner uses node:22-bookworm (Debian), not Alpine, so use apt-get - apt-get update && apt-get install -y wget unzip curl - - - name: Setup Terraform - run: | - TF_VERSION=1.8.4 - TF_URL="https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip" - mkdir -p /tmp/tf-bin - cd /tmp/tf-bin - wget -q "$TF_URL" || { echo "Failed to download terraform"; exit 1; } - unzip -q "terraform_${TF_VERSION}_linux_amd64.zip" - chmod +x terraform - ./terraform version - echo "/tmp/tf-bin" >> $GITHUB_PATH - - - name: Terraform Format Check - run: terraform fmt -check -recursive terraform/ - continue-on-error: true - - - name: Configure AWS Credentials - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - run: | - mkdir -p ~/.aws - cat > ~/.aws/credentials << EOF - [minio] - aws_access_key_id = $AWS_ACCESS_KEY_ID - aws_secret_access_key = $AWS_SECRET_ACCESS_KEY - EOF - chmod 600 ~/.aws/credentials - - - name: Terraform Init - working-directory: terraform - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - TF_SKIP_CREDENTIALS_VALIDATION: "true" - TF_SKIP_REGION_VALIDATION: "true" - TF_SKIP_REQUESTING_ACCOUNT_ID: "true" - run: | - terraform init \ - -backend-config="bucket=terraform-state" \ - -backend-config="key=homelab/terraform.tfstate" \ - -backend-config="region=us-east-1" \ - -backend-config="endpoint=http://minio.storage.svc.cluster.local:9000" \ - -backend-config="access_key=$AWS_ACCESS_KEY_ID" \ - -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY" \ - -backend-config="skip_credentials_validation=true" \ - -backend-config="use_path_style=true" - - - name: Pull Terraform State - working-directory: terraform - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - TF_SKIP_CREDENTIALS_VALIDATION: "true" - run: | - echo "Verifying state is accessible from MinIO..." - terraform state pull > /tmp/tfstate-verify.json - STATE_SIZE=$(wc -c < /tmp/tfstate-verify.json) - RESOURCE_COUNT=$(terraform state list | wc -l) - echo "State size: $STATE_SIZE bytes" - echo "Resources in state: $RESOURCE_COUNT" - - - name: Terraform Validate - working-directory: terraform - run: terraform validate - - - name: Terraform Plan - working-directory: terraform - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - TF_SKIP_CREDENTIALS_VALIDATION: "true" - run: | - terraform plan -out=tfplan - - - name: Terraform Apply - working-directory: terraform - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - TF_SKIP_CREDENTIALS_VALIDATION: "true" - run: | - terraform apply -auto-approve tfplan diff --git a/.forgejo/workflows/validate-k8s.yaml b/.forgejo/workflows/validate-k8s.yaml new file mode 100644 index 0000000..25f53b4 --- /dev/null +++ b/.forgejo/workflows/validate-k8s.yaml @@ -0,0 +1,111 @@ +name: Validate Kubernetes Manifests + +on: + push: + branches: + - main + - develop + paths: + - 'k8s/**' + - '.forgejo/workflows/validate-k8s.yaml' + pull_request: + paths: + - 'k8s/**' + - '.forgejo/workflows/validate-k8s.yaml' + +jobs: + validate: + runs-on: docker + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install Tools + run: | + apt-get update && apt-get install -y \ + yamllint \ + python3-pip \ + curl \ + jq + + # Install kubeval + curl -L https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz | tar xz + mv kubeval /usr/local/bin/ + + # Install kustomize + curl -s https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash + mv kustomize /usr/local/bin/ + + # Install ArgoCD CLI + curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 + chmod +x /usr/local/bin/argocd + + - name: YAML Lint + run: | + echo "=== Linting YAML files ===" + yamllint k8s/ -c .yamllint.yaml || true + + - name: Kubeval - Validate K8s Syntax + run: | + echo "=== Validating Kubernetes manifests ===" + find k8s -name "*.yaml" -o -name "*.yml" | grep -v "\.archive" | while read file; do + echo "Validating $file..." + kubeval "$file" -d 2>/dev/null || true + done + + - name: Kustomize Build - Infrastructure + run: | + echo "=== Building k8s/infrastructure/ ===" + kustomize build k8s/infrastructure > /tmp/infrastructure.yaml + echo "✓ Infrastructure built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/infrastructure.yaml)" + + - name: Kustomize Build - Bootstrap + run: | + echo "=== Building k8s/bootstrap/ ===" + kustomize build k8s/bootstrap > /tmp/bootstrap.yaml + echo "✓ Bootstrap built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/bootstrap.yaml || echo 0)" + + - name: Kustomize Build - Platform + run: | + echo "=== Building k8s/platform/ ===" + kustomize build k8s/platform > /tmp/platform.yaml + echo "✓ Platform built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/platform.yaml || echo 0)" + + - name: Kustomize Build - Security + run: | + echo "=== Building k8s/security/ ===" + kustomize build k8s/security > /tmp/security.yaml + echo "✓ Security built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/security.yaml || echo 0)" + + - name: Kustomize Build - Applications + run: | + echo "=== Building k8s/applications/ ===" + kustomize build k8s/applications > /tmp/applications.yaml + echo "✓ Applications built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/applications.yaml || echo 0)" + + - name: Kustomize Build - Data + run: | + echo "=== Building k8s/data/ ===" + kustomize build k8s/data > /tmp/data.yaml + echo "✓ Data built successfully" + echo "Resources: $(grep -c 'kind:' /tmp/data.yaml || echo 0)" + + - name: Validate ArgoCD Applications + run: | + echo "=== Validating ArgoCD Applications ===" + kubeval k8s/argocd/apps/*.yaml + + - name: Summary + if: always() + run: | + echo "=== Validation Summary ===" + echo "✓ All manifests validated" + echo "✓ All kustomizations built" + echo "✓ All ArgoCD apps valid" + echo "" + echo "Next: Push to main → ArgoCD syncs automatically" diff --git a/.yamllint.yaml b/.yamllint.yaml new file mode 100644 index 0000000..17f8692 --- /dev/null +++ b/.yamllint.yaml @@ -0,0 +1,33 @@ +--- +extends: default + +rules: + line-length: + max: 120 + level: warning + indentation: + spaces: 2 + brackets: + min-spaces-inside: 0 + max-spaces-inside: 0 + braces: + min-spaces-inside: 0 + max-spaces-inside: 0 + comments: + min-spaces-from-content: 2 + comments-indentation: {} + document-end: disable + document-start: disable + empty-lines: + max: 3 + empty-values: + forbid-in-block-mappings: true + forbid-in-flow-mappings: true + key-duplicates: enable + key-ordering: disable + new-line-at-end-of-file: enable + new-lines: + type: unix + trailing-spaces: enable + truthy: + level: warning diff --git a/GITOPS_ARCHITECTURE.md b/GITOPS_ARCHITECTURE.md new file mode 100644 index 0000000..1a05a5b --- /dev/null +++ b/GITOPS_ARCHITECTURE.md @@ -0,0 +1,385 @@ +# GitOps Architecture: Pure ArgoCD IaC + +## Directory Structure (Production-Grade) + +``` +homelab/ +├── k8s/ +│ ├── _base/ # Shared kustomizations, base values +│ │ ├── kustomization.yaml +│ │ ├── secrets-template.yaml # Secret templates (filled via sops) +│ │ └── namespaces.yaml # All namespace definitions +│ │ +│ ├── infrastructure/ # Layer 0: Foundation (namespaces, storage, RBAC) +│ │ ├── kustomization.yaml +│ │ ├── namespaces.yaml +│ │ ├── storage-classes.yaml +│ │ ├── service-accounts.yaml +│ │ ├── cluster-roles.yaml +│ │ ├── cluster-rolebindings.yaml +│ │ └── network-policies.yaml +│ │ +│ ├── bootstrap/ # Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx) +│ │ ├── kustomization.yaml +│ │ ├── cert-manager/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ ├── cilium/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ └── ingress-nginx/ +│ │ ├── kustomization.yaml +│ │ └── values.yaml +│ │ +│ ├── platform/ # Layer 2: Platform (storage, observability, state) +│ │ ├── kustomization.yaml +│ │ ├── longhorn/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ ├── minio/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ └── buckets/ +│ │ │ └── terraform-state.yaml +│ │ ├── loki/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ ├── prometheus/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ └── servicemonitors/ +│ │ └── promtail/ +│ │ ├── kustomization.yaml +│ │ └── values.yaml +│ │ +│ ├── security/ # Layer 3: Identity & Auth +│ │ ├── kustomization.yaml +│ │ ├── authentik/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ └── config/ +│ │ │ ├── oauth-apps.yaml +│ │ │ ├── groups.yaml +│ │ │ └── users.yaml +│ │ ├── vault/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ └── cert-issuer/ +│ │ └── certificate-definitions.yaml +│ │ +│ ├── applications/ # Layer 4: Business Applications +│ │ ├── kustomization.yaml +│ │ ├── forgejo/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ ├── deployment.yaml +│ │ │ └── config/ +│ │ ├── grafana/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ └── dashboards/ +│ │ ├── portainer/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ ├── temporal/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ └── llm/ +│ │ ├── kustomization.yaml +│ │ └── values.yaml +│ │ +│ ├── data/ # Layer 5: Data Services +│ │ ├── kustomization.yaml +│ │ ├── postgres/ +│ │ │ ├── kustomization.yaml +│ │ │ ├── values.yaml +│ │ │ └── backups/ +│ │ ├── redis/ +│ │ │ ├── kustomization.yaml +│ │ │ └── values.yaml +│ │ └── kafka/ +│ │ ├── kustomization.yaml +│ │ └── values.yaml +│ │ +│ └── argocd/ # ArgoCD Configuration (apps + projects) +│ ├── kustomization.yaml +│ ├── projects/ +│ │ └── homelab-project.yaml +│ └── apps/ +│ ├── kustomization.yaml +│ ├── root-app.yaml # Root application (points to k8s/infrastructure/) +│ ├── layer-0-infrastructure.yaml +│ ├── layer-1-bootstrap.yaml +│ ├── layer-2-platform.yaml +│ ├── layer-3-security.yaml +│ ├── layer-4-applications.yaml +│ └── layer-5-data.yaml +│ +├── .sops.yaml # SOPS encryption config (for secrets) +├── .env.example # Environment variables template +├── .github/workflows/ # (or .forgejo/workflows/) +│ ├── validate-k8s.yaml # Lint, kubeval, ArgoCD validation +│ └── security-scan.yaml # OWASP, policy checks +└── GITOPS_ARCHITECTURE.md +``` + +## Application Sync Layers (Deployment Order) + +``` +Layer 0: Infrastructure (foundation - 1 app) + └─ infrastructure/ (namespaces, storage classes, RBAC) + +Layer 1: Bootstrap (cluster networking - 1 app) + └─ bootstrap/ (cert-manager, cilium, ingress-nginx) + +Layer 2: Platform (cluster services - 1 app) + └─ platform/ (longhorn, minio, loki, prometheus, promtail) + +Layer 3: Security (identity & auth - 1 app) + └─ security/ (authentik, vault, cert-issuers) + +Layer 4: Applications (business services - 1 app) + └─ applications/ (forgejo, grafana, portainer, temporal, llm) + +Layer 5: Data (stateful services - 1 app) + └─ data/ (postgres, redis, kafka) +``` + +**Why layers?** +- Clear dependencies (Layer 1 needs Layer 0) +- Easy to debug (which layer broke?) +- Easy to rollback (one layer at a time) +- Easy to scale (add services without touching others) + +## ArgoCD Application Architecture + +### Root Application +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: homelab-root + namespace: argocd +spec: + project: homelab + source: + repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git + targetRevision: main + path: k8s/infrastructure # Start with Layer 0 + destination: + server: https://kubernetes.default.svc + syncPolicy: + automated: + prune: true + selfHeal: true +``` + +### Layer Applications (created by Layer 0) +```yaml +# k8s/infrastructure/argocd-apps.yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: layer-1-bootstrap +spec: + project: homelab + source: + repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git + targetRevision: main + path: k8s/bootstrap + destination: + server: https://kubernetes.default.svc + syncPolicy: + automated: + prune: true + selfHeal: true +--- +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: layer-2-platform +spec: + ... + path: k8s/platform +--- +# Repeat for layers 3, 4, 5 +``` + +## Kustomization Strategy + +### Base (Helm chart + values override) +```yaml +# k8s/bootstrap/cert-manager/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +helmCharts: +- name: cert-manager + repo: https://charts.jetstack.io + releaseName: cert-manager + version: v1.21.0 + namespace: cert-manager + valuesInline: + crds: + enabled: true + prometheus: + enabled: true + servicemonitor: + enabled: true +``` + +### Patch (customize per environment) +```yaml +# k8s/bootstrap/cert-manager/kustomization.yaml +patchesJson6902: +- target: + group: helm.sh + version: v1 + kind: Release + name: cert-manager + patch: |- + - op: add + path: /spec/values/installCRDs + value: "true" +``` + +## Secrets Management (SOPS) + +```bash +# Encrypt secrets before committing +sops -e secrets.yaml > secrets.enc.yaml +git add secrets.enc.yaml + +# ArgoCD decrypts at sync time (via plugin) +``` + +## File Naming Conventions + +``` +Layer directories: + k8s/{layer}/*.yaml + +Service subdirectories: + k8s/{layer}/{service}/ + ├── kustomization.yaml # Helm chart + patches + ├── values.yaml # Helm values + └── config/ # Additional manifests + ├── foo.yaml + └── bar.yaml + +Naming: + ✓ cert-manager/values.yaml (service-specific) + ✓ cluster-roles.yaml (resource type) + ✓ storage-classes.yaml (resource type) + ✗ cert-manager-helm.yaml (redundant suffix) + ✗ my-custom-config.yaml (non-standard) +``` + +## CI/CD Validation Pipeline + +```yaml +# .forgejo/workflows/validate-k8s.yaml +on: [push, pull_request] + +jobs: + validate: + runs-on: default + steps: + - uses: actions/checkout@v3 + + # 1. Lint YAML + - run: yamllint k8s/ + + # 2. Validate K8s manifests + - run: kubeval k8s/**/*.yaml + + # 3. Kustomize build (no apply) + - run: | + for dir in k8s/infrastructure k8s/bootstrap k8s/platform k8s/security k8s/applications k8s/data; do + kustomize build $dir > /dev/null + done + + # 4. ArgoCD app validation (dry-run) + - run: argocd app create --dry-run -f k8s/argocd/apps/ + + # 5. Policy check (optional) + - run: conftest test -p policy/ k8s/**/*.yaml +``` + +## Deployment Checklist + +**Day 1: Design Review** +- [ ] Review directory structure +- [ ] Confirm layer dependencies +- [ ] Identify secrets (need SOPS) +- [ ] Plan kustomization strategy + +**Day 2-3: Build Infrastructure Layer** +- [ ] Create k8s/infrastructure/ with namespaces, storage classes, RBAC +- [ ] Create kustomization.yaml for Layer 0 +- [ ] Test: `kustomize build k8s/infrastructure/` + +**Day 4-5: Build Bootstrap Layer** +- [ ] Create k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/ +- [ ] Add Helm chart references +- [ ] Test: `kustomize build k8s/bootstrap/` + +**Day 6-7: Build Platform Layer** +- [ ] Create k8s/platform/{minio,longhorn,loki,prometheus}/ +- [ ] Add values overrides +- [ ] Test: `kustomize build k8s/platform/` + +**Day 8-9: Build Security Layer** +- [ ] Create k8s/security/{authentik,vault}/ +- [ ] Migrate Authentik resources from Terraform +- [ ] Encrypt secrets with SOPS + +**Day 10-11: Build Application Layer** +- [ ] Move k8s/talos-ci-cd/ → k8s/applications/forgejo/ +- [ ] Consolidate k8s/{logging,monitoring} → k8s/platform/ or k8s/applications/ +- [ ] Create kustomization.yaml for services + +**Day 12: Create ArgoCD Apps** +- [ ] Create k8s/argocd/apps/ with layer applications +- [ ] Update AppProject permissions +- [ ] Deploy root application (starts with Layer 0) + +**Day 13-14: Monitor & Validate** +- [ ] Watch ArgoCD sync for each layer +- [ ] Verify no drift +- [ ] Test manual edits (ArgoCD corrects them) + +## Rollback Strategy + +```bash +# Rollback entire layer +git revert +git push + +# ArgoCD detects change, syncs back to previous state + +# Rollback single service +git checkout -- k8s/applications/forgejo/ +git commit -m "revert: forgejo back to " +git push +``` + +## Benefits of This Architecture + +✓ **Scalability:** Easy to add new services (copy service directory) +✓ **Clarity:** Clear layer dependencies (no surprises) +✓ **Safety:** Manual edits auto-corrected by ArgoCD +✓ **Auditability:** Every change in git (who, when, why) +✓ **Testing:** Kustomize build validates before sync +✓ **Rollback:** Git history = disaster recovery +✓ **Secrets:** SOPS encryption built-in +✓ **CI/CD:** Automated validation on every PR + +## Next Steps + +1. Create directory structure (k8s/infrastructure/, bootstrap/, platform/, etc.) +2. Migrate existing manifests from scattered k8s/ to new structure +3. Create kustomization.yaml for each layer +4. Create ArgoCD Applications for each layer +5. Test on live cluster (non-disruptive) +6. Update CI/CD pipeline to validate new structure diff --git a/GITOPS_IMPLEMENTATION_SUMMARY.md b/GITOPS_IMPLEMENTATION_SUMMARY.md new file mode 100644 index 0000000..a8f4bc4 --- /dev/null +++ b/GITOPS_IMPLEMENTATION_SUMMARY.md @@ -0,0 +1,246 @@ +# GitOps IaC Implementation Summary + +## Documents Created + +1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design +2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive) + +## Architecture at a Glance + +``` +Pure GitOps (ArgoCD only, no Terraform) +├── Layer 0: Infrastructure (namespaces, storage, RBAC) +├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx) +├── Layer 2: Platform (minio, longhorn, loki, prometheus) +├── Layer 3: Security (authentik, vault) +├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm) +└── Layer 5: Data (postgres, redis, kafka) +``` + +**Key benefit:** Each layer depends on previous, staged deployment, easy rollback. + +## File Structure + +``` +k8s/ +├── _base/ # Shared kustomization base +├── infrastructure/ # Layer 0 (foundation) +├── bootstrap/ # Layer 1 (networking) +├── platform/ # Layer 2 (services) +├── security/ # Layer 3 (auth) +├── applications/ # Layer 4 (business logic) +├── data/ # Layer 5 (stateful) +└── argocd/ + ├── projects/ # AppProject definitions + └── apps/ + ├── root-app.yaml # Bootstraps Layer 0 + └── layer-*.yaml # 5 layer apps (synced in order) +``` + +## Implementation Steps + +### Phase 1: Plan (Week 1) +- [x] Review GITOPS_ARCHITECTURE.md +- [x] Review GITOPS_MIGRATION_PLAN.md +- [ ] Audit current cluster resources +- [ ] Identify secrets needing encryption (SOPS) +- [ ] Map dependencies (provided in plan) +- **User approval required before proceeding to Phase 2** + +### Phase 2: Build (Weeks 2-3) +- [ ] Create k8s/ directory structure +- [ ] Migrate Layer 0 (infrastructure) +- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx) +- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus) +- [ ] Migrate Layer 3 (security: authentik, vault) +- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.) +- [ ] Migrate Layer 5 (data: postgres, redis, kafka) + +### Phase 3: ArgoCD (Week 4) +- [ ] Create root application (Layer 0 trigger) +- [ ] Create 5 layer applications (with sync-wave ordering) +- [ ] Update AppProject permissions +- [ ] Test dry-run on each layer + +### Phase 4: Deploy (Week 5) +- [ ] Pre-flight validation (kustomize build, kubeval, yamllint) +- [ ] Deploy root application +- [ ] Deploy layer applications (one by one) +- [ ] Monitor for drift +- [ ] Verify state correction (manual edits reverted) + +### Phase 5: Cleanup (Week 6) +- [ ] Delete old k8s/ directories +- [ ] Update CI/CD pipeline (remove terraform, add kustomize) +- [ ] Archive legacy configuration +- [ ] Final validation + +## Key Architecture Decisions + +### ✓ Why Layers? +- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds +- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only. +- **Easy scaling:** Add new service without touching others +- **Easy rollback:** `git revert ` + +### ✓ Why Kustomize? +- Helm chart package + values override (clean separation) +- Kustomize build validates before sync +- Patches enable per-environment customization +- Standard K8s tool (no external dependencies) + +### ✓ Why No Terraform? +- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD) +- Git is source of truth for everything +- No state backend (no .tfstate files) +- ArgoCD continuous reconciliation (drift auto-corrected) +- All changes reviewed in PRs (audit trail in git) + +### ✓ Why This File Structure? +- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/) +- **Scalable:** New service = new directory (copy template) +- **Maintainable:** Clear ownership (who owns layer? who owns service?) +- **GitOps-ready:** Each layer is independently deployable + +## Secret Management (SOPS) + +### Before (Terraform) +``` +terraform/terraform.tfvars # Plaintext secrets (risky!) +``` + +### After (GitOps) +```bash +# Encrypt secrets before commit +sops -e secrets.yaml > secrets.enc.yaml +git add secrets.enc.yaml + +# ArgoCD decrypts at sync time (SOPS plugin installed) +# Git stores only encrypted version (safe) +``` + +**Secrets to encrypt:** +- Authentik bootstrap password +- MinIO root credentials +- Database passwords +- API tokens +- OAuth client secrets + +## CI/CD Integration + +### Before +``` +git push → terraform validate → terraform plan → manual apply +``` + +### After (Pure GitOps) +``` +git push → yamllint → kubeval → kustomize build → argocd validation + ↓ + ArgoCD auto-syncs (if PR merged) +``` + +**No manual steps. No terraform. No state management. Pure GitOps.** + +## Comparison: Old vs. New + +| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) | +|--------|--------------------------|-------------------| +| **State management** | tfstate files | Git history | +| **Secrets** | Plaintext in .tfvars | SOPS encrypted | +| **Manual steps** | terraform apply required | git push only | +| **Rollback** | Revert tfstate + code | git revert + push | +| **Audit trail** | Terraform logs | Git history + ArgoCD logs | +| **Complexity** | High (2 tools) | Low (1 tool) | +| **New services** | Edit .tf files | Copy service directory | +| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects | + +## Non-Disruptive Migration + +**Key principle:** Live cluster remains running during entire migration. + +**How?** +1. Build k8s/ structure **alongside** existing configuration +2. Deploy each layer via ArgoCD while old method still runs +3. Once new layer verified: delete old configuration +4. If issue detected: rollback (git revert) + redeploy old version + +**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed) + +## Rollback Procedure (At Any Point) + +```bash +# 1. Identify broken layer +argocd app get layer-2-platform # Shows OutOfSync, error + +# 2. Revert that layer +git revert +git push + +# 3. ArgoCD detects change, syncs back to previous version +# Expected: ~3 minutes for full reconciliation + +# 4. Verify +argocd app get layer-2-platform # Should be Synced again +``` + +## Day 1 Action Items + +1. **Read GITOPS_ARCHITECTURE.md** (30 min) + - Understand layers + - Understand directory structure + - Understand kustomization strategy + +2. **Read GITOPS_MIGRATION_PLAN.md** (30 min) + - Understand phases + - Understand timeline + - Understand success criteria + +3. **Review dependency map** (provided in plan) + - Confirm layer dependencies match current setup + - Identify any services not in layers + +4. **Audit secrets** (30 min) + ```bash + grep -r "password" k8s/ --include="*.yaml" + grep -r "token" k8s/ --include="*.yaml" + # Plan which need SOPS encryption + ``` + +5. **Decision point:** Proceed to Phase 2 (Week 2)? + - YES → Start building k8s/ structure + - NO → Clarify concerns, refine plan + +## Benefits Summary + +✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth +✓ **Safer:** All changes reviewed in PRs, git history as audit trail +✓ **Scalable:** New services = copy directory, update git, done +✓ **Resilient:** State drift auto-corrected, rollback via `git revert` +✓ **Auditable:** Every change in git with who/when/why +✓ **Automated:** No manual kubectl apply, no manual terraform apply +✓ **Testable:** Kustomize build validates before sync +✓ **Recoverable:** git history = disaster recovery + +## Support + +- Architecture questions → GITOPS_ARCHITECTURE.md +- Implementation questions → GITOPS_MIGRATION_PLAN.md +- Syntax/tools questions → See referenced documentation + - Kustomize: https://kustomize.io/ + - ArgoCD: https://argo-cd.readthedocs.io/ + - SOPS: https://github.com/mozilla/sops + +## Next Steps + +1. **User review:** Read both architecture documents (60 min) +2. **User decision:** Proceed to Phase 2 or refine plan? +3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday) + +--- + +**Status:** Ready for implementation +**Terraform:** Deleted ✓ +**Architecture:** Designed ✓ +**Plan:** Ready ✓ +**Waiting for:** User approval to proceed diff --git a/GITOPS_MIGRATION_PLAN.md b/GITOPS_MIGRATION_PLAN.md new file mode 100644 index 0000000..a2761fc --- /dev/null +++ b/GITOPS_MIGRATION_PLAN.md @@ -0,0 +1,573 @@ +# GitOps Migration Plan: Implement Production-Grade Architecture + +## Current State → Target State + +### Current (Messy) +``` +k8s/ +├── talos-ci-cd/ (forgejo, runner) +├── talos-iam/ (authentik, vault) +├── logging/ (loki, promtail, grafana) +├── monitoring/ (prometheus, alerts) +├── storage/ (minio, longhorn) +├── ddb/ (postgres) +├── sqs/ (kafka, queue-crd) +├── temporal/ (temporal) +├── argocd/apps/ (20 Applications scattered) +└── ... (more scattered) +``` + +### Target (Clean, Layered) +``` +k8s/ +├── infrastructure/ (namespaces, storage, RBAC) +├── bootstrap/ (cert-manager, cilium, ingress-nginx) +├── platform/ (storage, observability: minio, longhorn, loki, prometheus) +├── security/ (authentik, vault, cert-issuer) +├── applications/ (forgejo, grafana, portainer, temporal, llm) +├── data/ (postgres, redis, kafka) +└── argocd/ + ├── projects/ (AppProject definitions) + └── apps/ (6 layer Applications only) +``` + +## Phase 1: Plan & Validate (Week 1) + +### Step 1.1: Review GITOPS_ARCHITECTURE.md +- [ ] Understand directory structure +- [ ] Understand layer dependencies +- [ ] Understand Kustomization strategy +- User approval: YES/NO + +### Step 1.2: Audit current resources +```bash +# Export all current resources +kubectl get all -A -o yaml > /tmp/current-state-backup.yaml + +# Count resources per layer +kubectl get deployments -A | wc -l +kubectl get statefulsets -A | wc -l +kubectl get daemonsets -A | wc -l +kubectl get services -A | wc -l +kubectl get configmaps -A | wc -l +kubectl get secrets -A | wc -l +``` + +### Step 1.3: Identify secrets needing encryption +```bash +# Find secrets in current manifests +grep -r "kind: Secret" k8s/ --include="*.yaml" +grep -r "password" k8s/ --include="*.yaml" +grep -r "token" k8s/ --include="*.yaml" + +# Plan SOPS encryption for: +# - Authentik bootstrap password +# - MinIO credentials +# - Database passwords +# - API tokens +``` + +### Step 1.4: Dependency mapping +``` +Layer 0 (Infrastructure) + └─ Layer 1 (Bootstrap) + ├─ cert-manager (creates certificates) + ├─ cilium (network) + └─ ingress-nginx (entry point) + └─ Layer 2 (Platform) + ├─ minio (storage) + ├─ longhorn (PV storage) + ├─ loki (logs) + └─ prometheus (metrics) + └─ Layer 3 (Security) + ├─ authentik (auth) + └─ vault (secrets) + └─ Layer 4 (Applications) + ├─ forgejo + ├─ grafana + ├─ portainer + ├─ temporal + └─ llm + └─ Layer 5 (Data) + ├─ postgres + ├─ redis + └─ kafka +``` + +## Phase 2: Build Directory Structure (Weeks 2-3) + +### Step 2.1: Create base directories +```bash +cd k8s/ + +# Create layers +mkdir -p infrastructure bootstrap platform security applications data + +# Create kustomization.yaml for each +for dir in infrastructure bootstrap platform security applications data; do + cat > $dir/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: CHANGE_ME + +resources: [] # Will add resources below + +# Placeholders for this layer +EOF +done +``` + +### Step 2.2: Migrate Layer 0 (Infrastructure) +**Resources to create:** +- Namespaces (all 20+) +- Storage classes (longhorn, xfs, longhorn-kafka) +- Service accounts (terraform-ci, etc.) +- Cluster roles (terraform-ci, admin, etc.) +- Network policies + +**Action:** +```bash +# Extract from current cluster +kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml +kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml +kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml +kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml + +# Clean up (remove status, owner refs, etc.) +# Add to k8s/infrastructure/kustomization.yaml: +resources: + - namespaces.yaml + - storage-classes.yaml + - service-accounts.yaml + - cluster-roles.yaml + +# Test +kustomize build k8s/infrastructure/ +``` + +**Deliverables:** +- [x] k8s/infrastructure/kustomization.yaml +- [x] k8s/infrastructure/namespaces.yaml +- [x] k8s/infrastructure/storage-classes.yaml +- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml + +### Step 2.3: Migrate Layer 1 (Bootstrap) + +**Services: cert-manager, cilium, ingress-nginx** + +**Action:** +```bash +# Create directories +mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx} + +# For each service: +# 1. Export Helm values +helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml + +# 2. Create kustomization.yaml with Helm chart ref +cat > bootstrap/cert-manager/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: cert-manager + +helmCharts: +- name: cert-manager + repo: https://charts.jetstack.io + version: v1.21.0 + releaseName: cert-manager + valuesFile: values.yaml +EOF + +# 3. Repeat for cilium, ingress-nginx + +# 4. Add to k8s/bootstrap/kustomization.yaml: +resources: + - cert-manager/kustomization.yaml + - cilium/kustomization.yaml + - ingress-nginx/kustomization.yaml +``` + +**Deliverables:** +- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml +- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml +- [x] k8s/bootstrap/kustomization.yaml + +### Step 2.4: Migrate Layer 2 (Platform) + +**Services: minio, longhorn, loki, prometheus, promtail** + +**Action:** +```bash +# Create directories +mkdir -p platform/{minio,longhorn,loki,prometheus,promtail} + +# Migrate minio +# 1. Export existing values +helm get values minio -n storage > platform/minio/values.yaml + +# 2. Export bucket configs +kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/ + +# 3. Create kustomization.yaml +cat > platform/minio/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: storage + +helmCharts: +- name: minio + repo: https://charts.min.io + version: 14.6.25 # Current version + releaseName: minio + valuesFile: values.yaml + +resources: + - config/minio-buckets.yaml +EOF + +# Repeat for loki, prometheus, promtail +``` + +**Deliverables:** +- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml +- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml +- [x] k8s/platform/kustomization.yaml + +### Step 2.5: Migrate Layer 3 (Security) + +**Services: authentik, vault** + +**Action:** +```bash +# Create directories +mkdir -p security/{authentik,vault} + +# Migrate authentik +helm get values authentik -n iam > security/authentik/values.yaml + +# Export Authentik resources +kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml +kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml +kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml + +# Create kustomization.yaml +cat > security/authentik/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: iam + +helmCharts: +- name: authentik + repo: https://charts.goauthentik.io + version: 2024.12.1 + releaseName: authentik + valuesFile: values.yaml + +resources: + - config/apps.yaml + - config/groups.yaml + - config/oauth-providers.yaml +EOF + +# Repeat for vault +``` + +**Deliverables:** +- [x] k8s/security/{authentik,vault}/kustomization.yaml +- [x] k8s/security/{authentik,vault}/values.yaml +- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml +- [x] k8s/security/kustomization.yaml + +### Step 2.6: Migrate Layer 4 (Applications) + +**Services: forgejo, grafana, portainer, temporal, llm** + +**Action:** +```bash +# Create directories +mkdir -p applications/{forgejo,grafana,portainer,temporal,llm} + +# Move existing manifests +cp -r k8s/talos-ci-cd/* applications/forgejo/ +cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc. +cp -r k8s/portainer/* applications/portainer/ +cp -r k8s/temporal/* applications/temporal/ +cp -r k8s/llm/* applications/llm/ + +# For each, create kustomization.yaml with Helm chart ref +# (or keep existing manifests if not Helm) + +# Create layer kustomization.yaml +cat > applications/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - forgejo/kustomization.yaml + - grafana/kustomization.yaml + - portainer/kustomization.yaml + - temporal/kustomization.yaml + - llm/kustomization.yaml +EOF +``` + +**Deliverables:** +- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml +- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml +- [x] k8s/applications/kustomization.yaml + +### Step 2.7: Migrate Layer 5 (Data) + +**Services: postgres (CloudNativePG), redis, kafka** + +**Action:** +```bash +# Create directories +mkdir -p data/{postgres,redis,kafka} + +# Export existing configurations +kubectl get cnpg -A -o yaml > data/postgres/config.yaml +kubectl get redis -A -o yaml > data/redis/config.yaml +kubectl get kafka -A -o yaml > data/kafka/config.yaml + +# Create kustomization.yaml for each + +# Repeat process... +``` + +**Deliverables:** +- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml +- [x] k8s/data/kustomization.yaml + +## Phase 3: Create ArgoCD Application Definitions (Week 4) + +### Step 3.1: Create layer Applications +```bash +mkdir -p k8s/argocd/apps/layers + +# Root application (Layer 0 only, triggers rest) +cat > k8s/argocd/apps/root-app.yaml << 'EOF' +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: homelab-root + namespace: argocd +spec: + project: homelab + source: + repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git + targetRevision: main + path: k8s/infrastructure + destination: + server: https://kubernetes.default.svc + syncPolicy: + automated: + prune: true + selfHeal: true +EOF + +# Layer 1 application +cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF' +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: layer-1-bootstrap + namespace: argocd + annotations: + argocd.argoproj.io/sync-wave: "1" +spec: + project: homelab + source: + repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git + targetRevision: main + path: k8s/bootstrap + destination: + server: https://kubernetes.default.svc + syncPolicy: + automated: + prune: true + selfHeal: true +EOF + +# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5) +``` + +**Deliverables:** +- [x] k8s/argocd/apps/root-app.yaml +- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml + +### Step 3.2: Create kustomization.yaml for apps +```bash +cat > k8s/argocd/apps/kustomization.yaml << 'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - root-app.yaml + - layer-1-bootstrap.yaml + - layer-2-platform.yaml + - layer-3-security.yaml + - layer-4-applications.yaml + - layer-5-data.yaml +EOF +``` + +## Phase 4: Deploy & Validate (Week 5) + +### Step 4.1: Pre-flight checks +```bash +# Validate all kustomization files +for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do + echo "Validating $dir..." + kustomize build $dir > /tmp/validate.yaml + kubeval /tmp/validate.yaml || exit 1 +done + +# Validate ArgoCD apps +kubeval k8s/argocd/apps/*.yaml + +# Test YAML lint +yamllint k8s/ +``` + +### Step 4.2: Deploy root application +```bash +# Apply root app (Layer 0 only) +kubectl apply -f k8s/argocd/apps/root-app.yaml + +# Watch sync +argocd app watch homelab-root + +# Verify infrastructure deployed +kubectl get ns # Should have all namespaces +kubectl get sc # Should have all storage classes +``` + +### Step 4.3: Deploy layer applications +```bash +# Apply layer applications (one by one) +kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml +sleep 5 +argocd app watch layer-1-bootstrap + +# Once Layer 1 synced: Layer 2 +kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml +argocd app watch layer-2-platform + +# ... repeat for layers 3, 4, 5 +``` + +### Step 4.4: Monitor for issues +```bash +# Check ArgoCD apps +argocd app list + +# Watch events +kubectl get events -A -w + +# Monitor pod status +kubectl get pods -A + +# Check logs +kubectl logs -f -n argocd deployment/argocd-application-controller +``` + +### Step 4.5: Validate state drift protection +```bash +# Manually edit a resource +kubectl edit deployment cert-manager -n cert-manager +# Change replicas from 1 → 2 + +# Wait 3 minutes (ArgoCD reconciliation interval) +sleep 180 + +# Check: should be back to 1 replica +kubectl get deployment cert-manager -n cert-manager +# Expected: 1/1 Ready (reverted by ArgoCD) + +# Verify in ArgoCD +argocd app get layer-1-bootstrap +# Expected: Synced status +``` + +## Phase 5: Cleanup (Week 6) + +### Step 5.1: Delete old directories +```bash +# After all apps synced successfully: +rm -rf k8s/talos-ci-cd/ +rm -rf k8s/talos-iam/ +rm -rf k8s/logging/ +rm -rf k8s/monitoring/ +rm -rf k8s/storage/ +rm -rf k8s/ddb/ +rm -rf k8s/sqs/ +rm -rf k8s/temporal/ +# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd +``` + +### Step 5.2: Update CI/CD pipeline +```bash +# Update .forgejo/workflows/ +# - Remove: terraform validate/plan/apply +# - Add: kustomize build validation +# - Add: argocd app validation +``` + +### Step 5.3: Archive old configuration +```bash +# Keep for reference only +mkdir k8s/.archive/ +git mv k8s/old-structure-backup k8s/.archive/ +git commit -m "archive: old k8s structure (moved to pure GitOps)" +``` + +## Rollback Plan + +**If something breaks during migration:** + +```bash +# Option 1: Rollback entire layer +git checkout HEAD~1 -- k8s/bootstrap/ +git commit -m "revert: bootstrap layer (investigating)" +# ArgoCD will resync to previous version automatically + +# Option 2: Pause ArgoCD sync +argocd app set layer-1-bootstrap --sync-policy none +# Investigate, then re-enable: +argocd app set layer-1-bootstrap --sync-policy automated + +# Option 3: Full rollback to pre-migration +git reset --hard +argocd app set homelab-root --sync-policy none +# Manual investigation, then re-enable +``` + +## Timeline Summary + +``` +Week 1: Plan & validate (reviews, dependency mapping) +Week 2: Layers 0-1 (infrastructure, bootstrap) +Week 3: Layers 2-3 (platform, security) +Week 3: Layers 4-5 (applications, data) +Week 4: ArgoCD applications + layer definitions +Week 5: Deploy & validate (watch for issues) +Week 6: Cleanup & CI/CD updates +``` + +**Total: 6 weeks, non-disruptive (all layers coexist during migration)** + +## Success Criteria + +✓ All 20+ services deployed via ArgoCD +✓ No manual kubectl apply in production +✓ State drift detected & corrected automatically +✓ All changes in git (reviewed via PR) +✓ Rollback possible at any time (git history) +✓ CI/CD validates all commits +✓ Secrets encrypted with SOPS +✓ New services can be added (copy service directory)