rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
32281ee9235e75a17621aa12503f581df9e0eb68
homelab/GITOPS_ARCHITECTURE.md
T

386 lines
12 KiB
Markdown
Raw Blame History

# GitOps Architecture: Pure ArgoCD IaC
## Directory Structure (Production-Grade)
```
homelab/
├── k8s/
│ ├── _base/ # Shared kustomizations, base values
│ │ ├── kustomization.yaml
│ │ ├── secrets-template.yaml # Secret templates (filled via sops)
│ │ └── namespaces.yaml # All namespace definitions
│ │
│ ├── infrastructure/ # Layer 0: Foundation (namespaces, storage, RBAC)
│ │ ├── kustomization.yaml
│ │ ├── namespaces.yaml
│ │ ├── storage-classes.yaml
│ │ ├── service-accounts.yaml
│ │ ├── cluster-roles.yaml
│ │ ├── cluster-rolebindings.yaml
│ │ └── network-policies.yaml
│ │
│ ├── bootstrap/ # Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
│ │ ├── kustomization.yaml
│ │ ├── cert-manager/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ ├── cilium/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ └── ingress-nginx/
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ │
│ ├── platform/ # Layer 2: Platform (storage, observability, state)
│ │ ├── kustomization.yaml
│ │ ├── longhorn/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ ├── minio/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ └── buckets/
│ │ │ └── terraform-state.yaml
│ │ ├── loki/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ ├── prometheus/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ └── servicemonitors/
│ │ └── promtail/
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ │
│ ├── security/ # Layer 3: Identity & Auth
│ │ ├── kustomization.yaml
│ │ ├── authentik/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ └── config/
│ │ │ ├── oauth-apps.yaml
│ │ │ ├── groups.yaml
│ │ │ └── users.yaml
│ │ ├── vault/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ └── cert-issuer/
│ │ └── certificate-definitions.yaml
│ │
│ ├── applications/ # Layer 4: Business Applications
│ │ ├── kustomization.yaml
│ │ ├── forgejo/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ ├── deployment.yaml
│ │ │ └── config/
│ │ ├── grafana/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ └── dashboards/
│ │ ├── portainer/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ ├── temporal/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ └── llm/
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ │
│ ├── data/ # Layer 5: Data Services
│ │ ├── kustomization.yaml
│ │ ├── postgres/
│ │ │ ├── kustomization.yaml
│ │ │ ├── values.yaml
│ │ │ └── backups/
│ │ ├── redis/
│ │ │ ├── kustomization.yaml
│ │ │ └── values.yaml
│ │ └── kafka/
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ │
│ └── argocd/ # ArgoCD Configuration (apps + projects)
│ ├── kustomization.yaml
│ ├── projects/
│ │ └── homelab-project.yaml
│ └── apps/
│ ├── kustomization.yaml
│ ├── root-app.yaml # Root application (points to k8s/infrastructure/)
│ ├── layer-0-infrastructure.yaml
│ ├── layer-1-bootstrap.yaml
│ ├── layer-2-platform.yaml
│ ├── layer-3-security.yaml
│ ├── layer-4-applications.yaml
│ └── layer-5-data.yaml
│
├── .sops.yaml # SOPS encryption config (for secrets)
├── .env.example # Environment variables template
├── .github/workflows/ # (or .forgejo/workflows/)
│ ├── validate-k8s.yaml # Lint, kubeval, ArgoCD validation
│ └── security-scan.yaml # OWASP, policy checks
└── GITOPS_ARCHITECTURE.md
```
## Application Sync Layers (Deployment Order)
```
Layer 0: Infrastructure (foundation - 1 app)
└─ infrastructure/ (namespaces, storage classes, RBAC)
Layer 1: Bootstrap (cluster networking - 1 app)
└─ bootstrap/ (cert-manager, cilium, ingress-nginx)
Layer 2: Platform (cluster services - 1 app)
└─ platform/ (longhorn, minio, loki, prometheus, promtail)
Layer 3: Security (identity & auth - 1 app)
└─ security/ (authentik, vault, cert-issuers)
Layer 4: Applications (business services - 1 app)
└─ applications/ (forgejo, grafana, portainer, temporal, llm)
Layer 5: Data (stateful services - 1 app)
└─ data/ (postgres, redis, kafka)
```
**Why layers?**
- Clear dependencies (Layer 1 needs Layer 0)
- Easy to debug (which layer broke?)
- Easy to rollback (one layer at a time)
- Easy to scale (add services without touching others)
## ArgoCD Application Architecture
### Root Application
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: homelab-root
namespace: argocd
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
path: k8s/infrastructure # Start with Layer 0
destination:
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
selfHeal: true
```
### Layer Applications (created by Layer 0)
```yaml
# k8s/infrastructure/argocd-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: layer-1-bootstrap
spec:
project: homelab
source:
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
path: k8s/bootstrap
destination:
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: layer-2-platform
spec:
...
path: k8s/platform
---
# Repeat for layers 3, 4, 5
```
## Kustomization Strategy
### Base (Helm chart + values override)
```yaml
# k8s/bootstrap/cert-manager/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: cert-manager
repo: https://charts.jetstack.io
releaseName: cert-manager
version: v1.21.0
namespace: cert-manager
valuesInline:
crds:
enabled: true
prometheus:
enabled: true
servicemonitor:
enabled: true
```
### Patch (customize per environment)
```yaml
# k8s/bootstrap/cert-manager/kustomization.yaml
patchesJson6902:
- target:
group: helm.sh
version: v1
kind: Release
name: cert-manager
patch: |-
- op: add
path: /spec/values/installCRDs
value: "true"
```
## Secrets Management (SOPS)
```bash
# Encrypt secrets before committing
sops -e secrets.yaml > secrets.enc.yaml
git add secrets.enc.yaml
# ArgoCD decrypts at sync time (via plugin)
```
## File Naming Conventions
```
Layer directories:
k8s/{layer}/*.yaml
Service subdirectories:
k8s/{layer}/{service}/
├── kustomization.yaml # Helm chart + patches
├── values.yaml # Helm values
└── config/ # Additional manifests
├── foo.yaml
└── bar.yaml
Naming:
✓ cert-manager/values.yaml (service-specific)
✓ cluster-roles.yaml (resource type)
✓ storage-classes.yaml (resource type)
✗ cert-manager-helm.yaml (redundant suffix)
✗ my-custom-config.yaml (non-standard)
```
## CI/CD Validation Pipeline
```yaml
# .forgejo/workflows/validate-k8s.yaml
on: [push, pull_request]
jobs:
validate:
runs-on: default
steps:
- uses: actions/checkout@v3
# 1. Lint YAML
- run: yamllint k8s/
# 2. Validate K8s manifests
- run: kubeval k8s/**/*.yaml
# 3. Kustomize build (no apply)
- run: |
for dir in k8s/infrastructure k8s/bootstrap k8s/platform k8s/security k8s/applications k8s/data; do
kustomize build $dir > /dev/null
done
# 4. ArgoCD app validation (dry-run)
- run: argocd app create --dry-run -f k8s/argocd/apps/
# 5. Policy check (optional)
- run: conftest test -p policy/ k8s/**/*.yaml
```
## Deployment Checklist
**Day 1: Design Review**
- [ ] Review directory structure
- [ ] Confirm layer dependencies
- [ ] Identify secrets (need SOPS)
- [ ] Plan kustomization strategy
**Day 2-3: Build Infrastructure Layer**
- [ ] Create k8s/infrastructure/ with namespaces, storage classes, RBAC
- [ ] Create kustomization.yaml for Layer 0
- [ ] Test: `kustomize build k8s/infrastructure/`
**Day 4-5: Build Bootstrap Layer**
- [ ] Create k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/
- [ ] Add Helm chart references
- [ ] Test: `kustomize build k8s/bootstrap/`
**Day 6-7: Build Platform Layer**
- [ ] Create k8s/platform/{minio,longhorn,loki,prometheus}/
- [ ] Add values overrides
- [ ] Test: `kustomize build k8s/platform/`
**Day 8-9: Build Security Layer**
- [ ] Create k8s/security/{authentik,vault}/
- [ ] Migrate Authentik resources from Terraform
- [ ] Encrypt secrets with SOPS
**Day 10-11: Build Application Layer**
- [ ] Move k8s/talos-ci-cd/ → k8s/applications/forgejo/
- [ ] Consolidate k8s/{logging,monitoring} → k8s/platform/ or k8s/applications/
- [ ] Create kustomization.yaml for services
**Day 12: Create ArgoCD Apps**
- [ ] Create k8s/argocd/apps/ with layer applications
- [ ] Update AppProject permissions
- [ ] Deploy root application (starts with Layer 0)
**Day 13-14: Monitor & Validate**
- [ ] Watch ArgoCD sync for each layer
- [ ] Verify no drift
- [ ] Test manual edits (ArgoCD corrects them)
## Rollback Strategy
```bash
# Rollback entire layer
git revert <commit-hash>
git push
# ArgoCD detects change, syncs back to previous state
# Rollback single service
git checkout <commit-hash> -- k8s/applications/forgejo/
git commit -m "revert: forgejo back to <version>"
git push
```
## Benefits of This Architecture
✓ **Scalability:** Easy to add new services (copy service directory)
✓ **Clarity:** Clear layer dependencies (no surprises)
✓ **Safety:** Manual edits auto-corrected by ArgoCD
✓ **Auditability:** Every change in git (who, when, why)
✓ **Testing:** Kustomize build validates before sync
✓ **Rollback:** Git history = disaster recovery
✓ **Secrets:** SOPS encryption built-in
✓ **CI/CD:** Automated validation on every PR
## Next Steps
1. Create directory structure (k8s/infrastructure/, bootstrap/, platform/, etc.)
2. Migrate existing manifests from scattered k8s/ to new structure
3. Create kustomization.yaml for each layer
4. Create ArgoCD Applications for each layer
5. Test on live cluster (non-disruptive)
6. Update CI/CD pipeline to validate new structure
Reference in New Issue View Git Blame Copy Permalink