homelab/GITOPS_ARCHITECTURE.md

GitOps Architecture: Pure ArgoCD IaC

Directory Structure (Production-Grade)

homelab/
├── k8s/
│   ├── _base/                          # Shared kustomizations, base values
│   │   ├── kustomization.yaml
│   │   ├── secrets-template.yaml       # Secret templates (filled via sops)
│   │   └── namespaces.yaml             # All namespace definitions
│   │
│   ├── infrastructure/                 # Layer 0: Foundation (namespaces, storage, RBAC)
│   │   ├── kustomization.yaml
│   │   ├── namespaces.yaml
│   │   ├── storage-classes.yaml
│   │   ├── service-accounts.yaml
│   │   ├── cluster-roles.yaml
│   │   ├── cluster-rolebindings.yaml
│   │   └── network-policies.yaml
│   │
│   ├── bootstrap/                      # Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
│   │   ├── kustomization.yaml
│   │   ├── cert-manager/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   ├── cilium/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   └── ingress-nginx/
│   │       ├── kustomization.yaml
│   │       └── values.yaml
│   │
│   ├── platform/                       # Layer 2: Platform (storage, observability, state)
│   │   ├── kustomization.yaml
│   │   ├── longhorn/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   ├── minio/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   └── buckets/
│   │   │       └── terraform-state.yaml
│   │   ├── loki/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   ├── prometheus/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   └── servicemonitors/
│   │   └── promtail/
│   │       ├── kustomization.yaml
│   │       └── values.yaml
│   │
│   ├── security/                       # Layer 3: Identity & Auth
│   │   ├── kustomization.yaml
│   │   ├── authentik/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   └── config/
│   │   │       ├── oauth-apps.yaml
│   │   │       ├── groups.yaml
│   │   │       └── users.yaml
│   │   ├── vault/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   └── cert-issuer/
│   │       └── certificate-definitions.yaml
│   │
│   ├── applications/                   # Layer 4: Business Applications
│   │   ├── kustomization.yaml
│   │   ├── forgejo/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   ├── deployment.yaml
│   │   │   └── config/
│   │   ├── grafana/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   └── dashboards/
│   │   ├── portainer/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   ├── temporal/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   └── llm/
│   │       ├── kustomization.yaml
│   │       └── values.yaml
│   │
│   ├── data/                           # Layer 5: Data Services
│   │   ├── kustomization.yaml
│   │   ├── postgres/
│   │   │   ├── kustomization.yaml
│   │   │   ├── values.yaml
│   │   │   └── backups/
│   │   ├── redis/
│   │   │   ├── kustomization.yaml
│   │   │   └── values.yaml
│   │   └── kafka/
│   │       ├── kustomization.yaml
│   │       └── values.yaml
│   │
│   └── argocd/                         # ArgoCD Configuration (apps + projects)
│       ├── kustomization.yaml
│       ├── projects/
│       │   └── homelab-project.yaml
│       └── apps/
│           ├── kustomization.yaml
│           ├── root-app.yaml           # Root application (points to k8s/infrastructure/)
│           ├── layer-0-infrastructure.yaml
│           ├── layer-1-bootstrap.yaml
│           ├── layer-2-platform.yaml
│           ├── layer-3-security.yaml
│           ├── layer-4-applications.yaml
│           └── layer-5-data.yaml
│
├── .sops.yaml                          # SOPS encryption config (for secrets)
├── .env.example                        # Environment variables template
├── .github/workflows/                  # (or .forgejo/workflows/)
│   ├── validate-k8s.yaml              # Lint, kubeval, ArgoCD validation
│   └── security-scan.yaml             # OWASP, policy checks
└── GITOPS_ARCHITECTURE.md

Application Sync Layers (Deployment Order)

Layer 0: Infrastructure (foundation - 1 app)
  └─ infrastructure/ (namespaces, storage classes, RBAC)

Layer 1: Bootstrap (cluster networking - 1 app)
  └─ bootstrap/ (cert-manager, cilium, ingress-nginx)

Layer 2: Platform (cluster services - 1 app)
  └─ platform/ (longhorn, minio, loki, prometheus, promtail)

Layer 3: Security (identity & auth - 1 app)
  └─ security/ (authentik, vault, cert-issuers)

Layer 4: Applications (business services - 1 app)
  └─ applications/ (forgejo, grafana, portainer, temporal, llm)

Layer 5: Data (stateful services - 1 app)
  └─ data/ (postgres, redis, kafka)

Why layers?

• Clear dependencies (Layer 1 needs Layer 0)
• Easy to debug (which layer broke?)
• Easy to rollback (one layer at a time)
• Easy to scale (add services without touching others)

ArgoCD Application Architecture

Root Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: homelab-root
  namespace: argocd
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/infrastructure  # Start with Layer 0
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Layer Applications (created by Layer 0)

# k8s/infrastructure/argocd-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: layer-1-bootstrap
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/bootstrap
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: layer-2-platform
spec:
  ...
  path: k8s/platform
---
# Repeat for layers 3, 4, 5

Kustomization Strategy

Base (Helm chart + values override)

# k8s/bootstrap/cert-manager/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: cert-manager
  repo: https://charts.jetstack.io
  releaseName: cert-manager
  version: v1.21.0
  namespace: cert-manager
  valuesInline:
    crds:
      enabled: true
    prometheus:
      enabled: true
      servicemonitor:
        enabled: true

Patch (customize per environment)

# k8s/bootstrap/cert-manager/kustomization.yaml
patchesJson6902:
- target:
    group: helm.sh
    version: v1
    kind: Release
    name: cert-manager
  patch: |-
    - op: add
      path: /spec/values/installCRDs
      value: "true"

Secrets Management (SOPS)

# Encrypt secrets before committing
sops -e secrets.yaml > secrets.enc.yaml
git add secrets.enc.yaml

# ArgoCD decrypts at sync time (via plugin)

File Naming Conventions

Layer directories:
  k8s/{layer}/*.yaml
  
Service subdirectories:
  k8s/{layer}/{service}/
    ├── kustomization.yaml        # Helm chart + patches
    ├── values.yaml               # Helm values
    └── config/                   # Additional manifests
        ├── foo.yaml
        └── bar.yaml

Naming:
  ✓ cert-manager/values.yaml     (service-specific)
  ✓ cluster-roles.yaml            (resource type)
  ✓ storage-classes.yaml          (resource type)
  ✗ cert-manager-helm.yaml        (redundant suffix)
  ✗ my-custom-config.yaml         (non-standard)

CI/CD Validation Pipeline

# .forgejo/workflows/validate-k8s.yaml
on: [push, pull_request]

jobs:
  validate:
    runs-on: default
    steps:
      - uses: actions/checkout@v3
      
      # 1. Lint YAML
      - run: yamllint k8s/
      
      # 2. Validate K8s manifests
      - run: kubeval k8s/**/*.yaml
      
      # 3. Kustomize build (no apply)
      - run: |
          for dir in k8s/infrastructure k8s/bootstrap k8s/platform k8s/security k8s/applications k8s/data; do
            kustomize build $dir > /dev/null
          done
      
      # 4. ArgoCD app validation (dry-run)
      - run: argocd app create --dry-run -f k8s/argocd/apps/
      
      # 5. Policy check (optional)
      - run: conftest test -p policy/ k8s/**/*.yaml

Deployment Checklist

Day 1: Design Review

• Review directory structure
• Confirm layer dependencies
• Identify secrets (need SOPS)
• Plan kustomization strategy

Day 2-3: Build Infrastructure Layer

• Create k8s/infrastructure/ with namespaces, storage classes, RBAC
• Create kustomization.yaml for Layer 0
• Test: kustomize build k8s/infrastructure/

Day 4-5: Build Bootstrap Layer

• Create k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/
• Add Helm chart references
• Test: kustomize build k8s/bootstrap/

Day 6-7: Build Platform Layer

• Create k8s/platform/{minio,longhorn,loki,prometheus}/
• Add values overrides
• Test: kustomize build k8s/platform/

Day 8-9: Build Security Layer

• Create k8s/security/{authentik,vault}/
• Migrate Authentik resources from Terraform
• Encrypt secrets with SOPS

Day 10-11: Build Application Layer

• Move k8s/talos-ci-cd/ → k8s/applications/forgejo/
• Consolidate k8s/{logging,monitoring} → k8s/platform/ or k8s/applications/
• Create kustomization.yaml for services

Day 12: Create ArgoCD Apps

• Create k8s/argocd/apps/ with layer applications
• Update AppProject permissions
• Deploy root application (starts with Layer 0)

Day 13-14: Monitor & Validate

• Watch ArgoCD sync for each layer
• Verify no drift
• Test manual edits (ArgoCD corrects them)

Rollback Strategy

# Rollback entire layer
git revert <commit-hash>
git push

# ArgoCD detects change, syncs back to previous state

# Rollback single service
git checkout <commit-hash> -- k8s/applications/forgejo/
git commit -m "revert: forgejo back to <version>"
git push

Benefits of This Architecture

✓ Scalability: Easy to add new services (copy service directory) ✓ Clarity: Clear layer dependencies (no surprises) ✓ Safety: Manual edits auto-corrected by ArgoCD ✓ Auditability: Every change in git (who, when, why) ✓ Testing: Kustomize build validates before sync ✓ Rollback: Git history = disaster recovery ✓ Secrets: SOPS encryption built-in ✓ CI/CD: Automated validation on every PR

Next Steps

1. Create directory structure (k8s/infrastructure/, bootstrap/, platform/, etc.)
2. Migrate existing manifests from scattered k8s/ to new structure
3. Create kustomization.yaml for each layer
4. Create ArgoCD Applications for each layer
5. Test on live cluster (non-disruptive)
6. Update CI/CD pipeline to validate new structure