chore: remove scratch planning docs — not meant for the repo
This commit is contained in:
Story Crater Bot
@@ -1,385 +0,0 @@
|
|||||||
# GitOps Architecture: Pure ArgoCD IaC
|
|
||||||
|
|
||||||
## Directory Structure (Production-Grade)
|
|
||||||
|
|
||||||
```
|
|
||||||
homelab/
|
|
||||||
├── k8s/
|
|
||||||
│ ├── _base/ # Shared kustomizations, base values
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── secrets-template.yaml # Secret templates (filled via sops)
|
|
||||||
│ │ └── namespaces.yaml # All namespace definitions
|
|
||||||
│ │
|
|
||||||
│ ├── infrastructure/ # Layer 0: Foundation (namespaces, storage, RBAC)
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── namespaces.yaml
|
|
||||||
│ │ ├── storage-classes.yaml
|
|
||||||
│ │ ├── service-accounts.yaml
|
|
||||||
│ │ ├── cluster-roles.yaml
|
|
||||||
│ │ ├── cluster-rolebindings.yaml
|
|
||||||
│ │ └── network-policies.yaml
|
|
||||||
│ │
|
|
||||||
│ ├── bootstrap/ # Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── cert-manager/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ ├── cilium/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ └── ingress-nginx/
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ └── values.yaml
|
|
||||||
│ │
|
|
||||||
│ ├── platform/ # Layer 2: Platform (storage, observability, state)
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── longhorn/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ ├── minio/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ └── buckets/
|
|
||||||
│ │ │ └── terraform-state.yaml
|
|
||||||
│ │ ├── loki/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ ├── prometheus/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ └── servicemonitors/
|
|
||||||
│ │ └── promtail/
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ └── values.yaml
|
|
||||||
│ │
|
|
||||||
│ ├── security/ # Layer 3: Identity & Auth
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── authentik/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ └── config/
|
|
||||||
│ │ │ ├── oauth-apps.yaml
|
|
||||||
│ │ │ ├── groups.yaml
|
|
||||||
│ │ │ └── users.yaml
|
|
||||||
│ │ ├── vault/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ └── cert-issuer/
|
|
||||||
│ │ └── certificate-definitions.yaml
|
|
||||||
│ │
|
|
||||||
│ ├── applications/ # Layer 4: Business Applications
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── forgejo/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ ├── deployment.yaml
|
|
||||||
│ │ │ └── config/
|
|
||||||
│ │ ├── grafana/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ └── dashboards/
|
|
||||||
│ │ ├── portainer/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ ├── temporal/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ └── llm/
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ └── values.yaml
|
|
||||||
│ │
|
|
||||||
│ ├── data/ # Layer 5: Data Services
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ ├── postgres/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ ├── values.yaml
|
|
||||||
│ │ │ └── backups/
|
|
||||||
│ │ ├── redis/
|
|
||||||
│ │ │ ├── kustomization.yaml
|
|
||||||
│ │ │ └── values.yaml
|
|
||||||
│ │ └── kafka/
|
|
||||||
│ │ ├── kustomization.yaml
|
|
||||||
│ │ └── values.yaml
|
|
||||||
│ │
|
|
||||||
│ └── argocd/ # ArgoCD Configuration (apps + projects)
|
|
||||||
│ ├── kustomization.yaml
|
|
||||||
│ ├── projects/
|
|
||||||
│ │ └── homelab-project.yaml
|
|
||||||
│ └── apps/
|
|
||||||
│ ├── kustomization.yaml
|
|
||||||
│ ├── root-app.yaml # Root application (points to k8s/infrastructure/)
|
|
||||||
│ ├── layer-0-infrastructure.yaml
|
|
||||||
│ ├── layer-1-bootstrap.yaml
|
|
||||||
│ ├── layer-2-platform.yaml
|
|
||||||
│ ├── layer-3-security.yaml
|
|
||||||
│ ├── layer-4-applications.yaml
|
|
||||||
│ └── layer-5-data.yaml
|
|
||||||
│
|
|
||||||
├── .sops.yaml # SOPS encryption config (for secrets)
|
|
||||||
├── .env.example # Environment variables template
|
|
||||||
├── .github/workflows/ # (or .forgejo/workflows/)
|
|
||||||
│ ├── validate-k8s.yaml # Lint, kubeval, ArgoCD validation
|
|
||||||
│ └── security-scan.yaml # OWASP, policy checks
|
|
||||||
└── GITOPS_ARCHITECTURE.md
|
|
||||||
```
|
|
||||||
|
|
||||||
## Application Sync Layers (Deployment Order)
|
|
||||||
|
|
||||||
```
|
|
||||||
Layer 0: Infrastructure (foundation - 1 app)
|
|
||||||
└─ infrastructure/ (namespaces, storage classes, RBAC)
|
|
||||||
|
|
||||||
Layer 1: Bootstrap (cluster networking - 1 app)
|
|
||||||
└─ bootstrap/ (cert-manager, cilium, ingress-nginx)
|
|
||||||
|
|
||||||
Layer 2: Platform (cluster services - 1 app)
|
|
||||||
└─ platform/ (longhorn, minio, loki, prometheus, promtail)
|
|
||||||
|
|
||||||
Layer 3: Security (identity & auth - 1 app)
|
|
||||||
└─ security/ (authentik, vault, cert-issuers)
|
|
||||||
|
|
||||||
Layer 4: Applications (business services - 1 app)
|
|
||||||
└─ applications/ (forgejo, grafana, portainer, temporal, llm)
|
|
||||||
|
|
||||||
Layer 5: Data (stateful services - 1 app)
|
|
||||||
└─ data/ (postgres, redis, kafka)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why layers?**
|
|
||||||
- Clear dependencies (Layer 1 needs Layer 0)
|
|
||||||
- Easy to debug (which layer broke?)
|
|
||||||
- Easy to rollback (one layer at a time)
|
|
||||||
- Easy to scale (add services without touching others)
|
|
||||||
|
|
||||||
## ArgoCD Application Architecture
|
|
||||||
|
|
||||||
### Root Application
|
|
||||||
```yaml
|
|
||||||
apiVersion: argoproj.io/v1alpha1
|
|
||||||
kind: Application
|
|
||||||
metadata:
|
|
||||||
name: homelab-root
|
|
||||||
namespace: argocd
|
|
||||||
spec:
|
|
||||||
project: homelab
|
|
||||||
source:
|
|
||||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
||||||
targetRevision: main
|
|
||||||
path: k8s/infrastructure # Start with Layer 0
|
|
||||||
destination:
|
|
||||||
server: https://kubernetes.default.svc
|
|
||||||
syncPolicy:
|
|
||||||
automated:
|
|
||||||
prune: true
|
|
||||||
selfHeal: true
|
|
||||||
```
|
|
||||||
|
|
||||||
### Layer Applications (created by Layer 0)
|
|
||||||
```yaml
|
|
||||||
# k8s/infrastructure/argocd-apps.yaml
|
|
||||||
apiVersion: argoproj.io/v1alpha1
|
|
||||||
kind: Application
|
|
||||||
metadata:
|
|
||||||
name: layer-1-bootstrap
|
|
||||||
spec:
|
|
||||||
project: homelab
|
|
||||||
source:
|
|
||||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
||||||
targetRevision: main
|
|
||||||
path: k8s/bootstrap
|
|
||||||
destination:
|
|
||||||
server: https://kubernetes.default.svc
|
|
||||||
syncPolicy:
|
|
||||||
automated:
|
|
||||||
prune: true
|
|
||||||
selfHeal: true
|
|
||||||
---
|
|
||||||
apiVersion: argoproj.io/v1alpha1
|
|
||||||
kind: Application
|
|
||||||
metadata:
|
|
||||||
name: layer-2-platform
|
|
||||||
spec:
|
|
||||||
...
|
|
||||||
path: k8s/platform
|
|
||||||
---
|
|
||||||
# Repeat for layers 3, 4, 5
|
|
||||||
```
|
|
||||||
|
|
||||||
## Kustomization Strategy
|
|
||||||
|
|
||||||
### Base (Helm chart + values override)
|
|
||||||
```yaml
|
|
||||||
# k8s/bootstrap/cert-manager/kustomization.yaml
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
helmCharts:
|
|
||||||
- name: cert-manager
|
|
||||||
repo: https://charts.jetstack.io
|
|
||||||
releaseName: cert-manager
|
|
||||||
version: v1.21.0
|
|
||||||
namespace: cert-manager
|
|
||||||
valuesInline:
|
|
||||||
crds:
|
|
||||||
enabled: true
|
|
||||||
prometheus:
|
|
||||||
enabled: true
|
|
||||||
servicemonitor:
|
|
||||||
enabled: true
|
|
||||||
```
|
|
||||||
|
|
||||||
### Patch (customize per environment)
|
|
||||||
```yaml
|
|
||||||
# k8s/bootstrap/cert-manager/kustomization.yaml
|
|
||||||
patchesJson6902:
|
|
||||||
- target:
|
|
||||||
group: helm.sh
|
|
||||||
version: v1
|
|
||||||
kind: Release
|
|
||||||
name: cert-manager
|
|
||||||
patch: |-
|
|
||||||
- op: add
|
|
||||||
path: /spec/values/installCRDs
|
|
||||||
value: "true"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Secrets Management (SOPS)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Encrypt secrets before committing
|
|
||||||
sops -e secrets.yaml > secrets.enc.yaml
|
|
||||||
git add secrets.enc.yaml
|
|
||||||
|
|
||||||
# ArgoCD decrypts at sync time (via plugin)
|
|
||||||
```
|
|
||||||
|
|
||||||
## File Naming Conventions
|
|
||||||
|
|
||||||
```
|
|
||||||
Layer directories:
|
|
||||||
k8s/{layer}/*.yaml
|
|
||||||
|
|
||||||
Service subdirectories:
|
|
||||||
k8s/{layer}/{service}/
|
|
||||||
├── kustomization.yaml # Helm chart + patches
|
|
||||||
├── values.yaml # Helm values
|
|
||||||
└── config/ # Additional manifests
|
|
||||||
├── foo.yaml
|
|
||||||
└── bar.yaml
|
|
||||||
|
|
||||||
Naming:
|
|
||||||
✓ cert-manager/values.yaml (service-specific)
|
|
||||||
✓ cluster-roles.yaml (resource type)
|
|
||||||
✓ storage-classes.yaml (resource type)
|
|
||||||
✗ cert-manager-helm.yaml (redundant suffix)
|
|
||||||
✗ my-custom-config.yaml (non-standard)
|
|
||||||
```
|
|
||||||
|
|
||||||
## CI/CD Validation Pipeline
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# .forgejo/workflows/validate-k8s.yaml
|
|
||||||
on: [push, pull_request]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
validate:
|
|
||||||
runs-on: default
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v3
|
|
||||||
|
|
||||||
# 1. Lint YAML
|
|
||||||
- run: yamllint k8s/
|
|
||||||
|
|
||||||
# 2. Validate K8s manifests
|
|
||||||
- run: kubeval k8s/**/*.yaml
|
|
||||||
|
|
||||||
# 3. Kustomize build (no apply)
|
|
||||||
- run: |
|
|
||||||
for dir in k8s/infrastructure k8s/bootstrap k8s/platform k8s/security k8s/applications k8s/data; do
|
|
||||||
kustomize build $dir > /dev/null
|
|
||||||
done
|
|
||||||
|
|
||||||
# 4. ArgoCD app validation (dry-run)
|
|
||||||
- run: argocd app create --dry-run -f k8s/argocd/apps/
|
|
||||||
|
|
||||||
# 5. Policy check (optional)
|
|
||||||
- run: conftest test -p policy/ k8s/**/*.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Deployment Checklist
|
|
||||||
|
|
||||||
**Day 1: Design Review**
|
|
||||||
- [ ] Review directory structure
|
|
||||||
- [ ] Confirm layer dependencies
|
|
||||||
- [ ] Identify secrets (need SOPS)
|
|
||||||
- [ ] Plan kustomization strategy
|
|
||||||
|
|
||||||
**Day 2-3: Build Infrastructure Layer**
|
|
||||||
- [ ] Create k8s/infrastructure/ with namespaces, storage classes, RBAC
|
|
||||||
- [ ] Create kustomization.yaml for Layer 0
|
|
||||||
- [ ] Test: `kustomize build k8s/infrastructure/`
|
|
||||||
|
|
||||||
**Day 4-5: Build Bootstrap Layer**
|
|
||||||
- [ ] Create k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/
|
|
||||||
- [ ] Add Helm chart references
|
|
||||||
- [ ] Test: `kustomize build k8s/bootstrap/`
|
|
||||||
|
|
||||||
**Day 6-7: Build Platform Layer**
|
|
||||||
- [ ] Create k8s/platform/{minio,longhorn,loki,prometheus}/
|
|
||||||
- [ ] Add values overrides
|
|
||||||
- [ ] Test: `kustomize build k8s/platform/`
|
|
||||||
|
|
||||||
**Day 8-9: Build Security Layer**
|
|
||||||
- [ ] Create k8s/security/{authentik,vault}/
|
|
||||||
- [ ] Migrate Authentik resources from Terraform
|
|
||||||
- [ ] Encrypt secrets with SOPS
|
|
||||||
|
|
||||||
**Day 10-11: Build Application Layer**
|
|
||||||
- [ ] Move k8s/talos-ci-cd/ → k8s/applications/forgejo/
|
|
||||||
- [ ] Consolidate k8s/{logging,monitoring} → k8s/platform/ or k8s/applications/
|
|
||||||
- [ ] Create kustomization.yaml for services
|
|
||||||
|
|
||||||
**Day 12: Create ArgoCD Apps**
|
|
||||||
- [ ] Create k8s/argocd/apps/ with layer applications
|
|
||||||
- [ ] Update AppProject permissions
|
|
||||||
- [ ] Deploy root application (starts with Layer 0)
|
|
||||||
|
|
||||||
**Day 13-14: Monitor & Validate**
|
|
||||||
- [ ] Watch ArgoCD sync for each layer
|
|
||||||
- [ ] Verify no drift
|
|
||||||
- [ ] Test manual edits (ArgoCD corrects them)
|
|
||||||
|
|
||||||
## Rollback Strategy
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Rollback entire layer
|
|
||||||
git revert <commit-hash>
|
|
||||||
git push
|
|
||||||
|
|
||||||
# ArgoCD detects change, syncs back to previous state
|
|
||||||
|
|
||||||
# Rollback single service
|
|
||||||
git checkout <commit-hash> -- k8s/applications/forgejo/
|
|
||||||
git commit -m "revert: forgejo back to <version>"
|
|
||||||
git push
|
|
||||||
```
|
|
||||||
|
|
||||||
## Benefits of This Architecture
|
|
||||||
|
|
||||||
✓ **Scalability:** Easy to add new services (copy service directory)
|
|
||||||
✓ **Clarity:** Clear layer dependencies (no surprises)
|
|
||||||
✓ **Safety:** Manual edits auto-corrected by ArgoCD
|
|
||||||
✓ **Auditability:** Every change in git (who, when, why)
|
|
||||||
✓ **Testing:** Kustomize build validates before sync
|
|
||||||
✓ **Rollback:** Git history = disaster recovery
|
|
||||||
✓ **Secrets:** SOPS encryption built-in
|
|
||||||
✓ **CI/CD:** Automated validation on every PR
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. Create directory structure (k8s/infrastructure/, bootstrap/, platform/, etc.)
|
|
||||||
2. Migrate existing manifests from scattered k8s/ to new structure
|
|
||||||
3. Create kustomization.yaml for each layer
|
|
||||||
4. Create ArgoCD Applications for each layer
|
|
||||||
5. Test on live cluster (non-disruptive)
|
|
||||||
6. Update CI/CD pipeline to validate new structure
|
|
||||||
@@ -1,246 +0,0 @@
|
|||||||
# GitOps IaC Implementation Summary
|
|
||||||
|
|
||||||
## Documents Created
|
|
||||||
|
|
||||||
1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design
|
|
||||||
2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive)
|
|
||||||
|
|
||||||
## Architecture at a Glance
|
|
||||||
|
|
||||||
```
|
|
||||||
Pure GitOps (ArgoCD only, no Terraform)
|
|
||||||
├── Layer 0: Infrastructure (namespaces, storage, RBAC)
|
|
||||||
├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
|
||||||
├── Layer 2: Platform (minio, longhorn, loki, prometheus)
|
|
||||||
├── Layer 3: Security (authentik, vault)
|
|
||||||
├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm)
|
|
||||||
└── Layer 5: Data (postgres, redis, kafka)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key benefit:** Each layer depends on previous, staged deployment, easy rollback.
|
|
||||||
|
|
||||||
## File Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
k8s/
|
|
||||||
├── _base/ # Shared kustomization base
|
|
||||||
├── infrastructure/ # Layer 0 (foundation)
|
|
||||||
├── bootstrap/ # Layer 1 (networking)
|
|
||||||
├── platform/ # Layer 2 (services)
|
|
||||||
├── security/ # Layer 3 (auth)
|
|
||||||
├── applications/ # Layer 4 (business logic)
|
|
||||||
├── data/ # Layer 5 (stateful)
|
|
||||||
└── argocd/
|
|
||||||
├── projects/ # AppProject definitions
|
|
||||||
└── apps/
|
|
||||||
├── root-app.yaml # Bootstraps Layer 0
|
|
||||||
└── layer-*.yaml # 5 layer apps (synced in order)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Implementation Steps
|
|
||||||
|
|
||||||
### Phase 1: Plan (Week 1)
|
|
||||||
- [x] Review GITOPS_ARCHITECTURE.md
|
|
||||||
- [x] Review GITOPS_MIGRATION_PLAN.md
|
|
||||||
- [ ] Audit current cluster resources
|
|
||||||
- [ ] Identify secrets needing encryption (SOPS)
|
|
||||||
- [ ] Map dependencies (provided in plan)
|
|
||||||
- **User approval required before proceeding to Phase 2**
|
|
||||||
|
|
||||||
### Phase 2: Build (Weeks 2-3)
|
|
||||||
- [ ] Create k8s/ directory structure
|
|
||||||
- [ ] Migrate Layer 0 (infrastructure)
|
|
||||||
- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx)
|
|
||||||
- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus)
|
|
||||||
- [ ] Migrate Layer 3 (security: authentik, vault)
|
|
||||||
- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.)
|
|
||||||
- [ ] Migrate Layer 5 (data: postgres, redis, kafka)
|
|
||||||
|
|
||||||
### Phase 3: ArgoCD (Week 4)
|
|
||||||
- [ ] Create root application (Layer 0 trigger)
|
|
||||||
- [ ] Create 5 layer applications (with sync-wave ordering)
|
|
||||||
- [ ] Update AppProject permissions
|
|
||||||
- [ ] Test dry-run on each layer
|
|
||||||
|
|
||||||
### Phase 4: Deploy (Week 5)
|
|
||||||
- [ ] Pre-flight validation (kustomize build, kubeval, yamllint)
|
|
||||||
- [ ] Deploy root application
|
|
||||||
- [ ] Deploy layer applications (one by one)
|
|
||||||
- [ ] Monitor for drift
|
|
||||||
- [ ] Verify state correction (manual edits reverted)
|
|
||||||
|
|
||||||
### Phase 5: Cleanup (Week 6)
|
|
||||||
- [ ] Delete old k8s/ directories
|
|
||||||
- [ ] Update CI/CD pipeline (remove terraform, add kustomize)
|
|
||||||
- [ ] Archive legacy configuration
|
|
||||||
- [ ] Final validation
|
|
||||||
|
|
||||||
## Key Architecture Decisions
|
|
||||||
|
|
||||||
### ✓ Why Layers?
|
|
||||||
- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds
|
|
||||||
- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only.
|
|
||||||
- **Easy scaling:** Add new service without touching others
|
|
||||||
- **Easy rollback:** `git revert <layer-commit>`
|
|
||||||
|
|
||||||
### ✓ Why Kustomize?
|
|
||||||
- Helm chart package + values override (clean separation)
|
|
||||||
- Kustomize build validates before sync
|
|
||||||
- Patches enable per-environment customization
|
|
||||||
- Standard K8s tool (no external dependencies)
|
|
||||||
|
|
||||||
### ✓ Why No Terraform?
|
|
||||||
- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD)
|
|
||||||
- Git is source of truth for everything
|
|
||||||
- No state backend (no .tfstate files)
|
|
||||||
- ArgoCD continuous reconciliation (drift auto-corrected)
|
|
||||||
- All changes reviewed in PRs (audit trail in git)
|
|
||||||
|
|
||||||
### ✓ Why This File Structure?
|
|
||||||
- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/)
|
|
||||||
- **Scalable:** New service = new directory (copy template)
|
|
||||||
- **Maintainable:** Clear ownership (who owns layer? who owns service?)
|
|
||||||
- **GitOps-ready:** Each layer is independently deployable
|
|
||||||
|
|
||||||
## Secret Management (SOPS)
|
|
||||||
|
|
||||||
### Before (Terraform)
|
|
||||||
```
|
|
||||||
terraform/terraform.tfvars # Plaintext secrets (risky!)
|
|
||||||
```
|
|
||||||
|
|
||||||
### After (GitOps)
|
|
||||||
```bash
|
|
||||||
# Encrypt secrets before commit
|
|
||||||
sops -e secrets.yaml > secrets.enc.yaml
|
|
||||||
git add secrets.enc.yaml
|
|
||||||
|
|
||||||
# ArgoCD decrypts at sync time (SOPS plugin installed)
|
|
||||||
# Git stores only encrypted version (safe)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Secrets to encrypt:**
|
|
||||||
- Authentik bootstrap password
|
|
||||||
- MinIO root credentials
|
|
||||||
- Database passwords
|
|
||||||
- API tokens
|
|
||||||
- OAuth client secrets
|
|
||||||
|
|
||||||
## CI/CD Integration
|
|
||||||
|
|
||||||
### Before
|
|
||||||
```
|
|
||||||
git push → terraform validate → terraform plan → manual apply
|
|
||||||
```
|
|
||||||
|
|
||||||
### After (Pure GitOps)
|
|
||||||
```
|
|
||||||
git push → yamllint → kubeval → kustomize build → argocd validation
|
|
||||||
↓
|
|
||||||
ArgoCD auto-syncs (if PR merged)
|
|
||||||
```
|
|
||||||
|
|
||||||
**No manual steps. No terraform. No state management. Pure GitOps.**
|
|
||||||
|
|
||||||
## Comparison: Old vs. New
|
|
||||||
|
|
||||||
| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) |
|
|
||||||
|--------|--------------------------|-------------------|
|
|
||||||
| **State management** | tfstate files | Git history |
|
|
||||||
| **Secrets** | Plaintext in .tfvars | SOPS encrypted |
|
|
||||||
| **Manual steps** | terraform apply required | git push only |
|
|
||||||
| **Rollback** | Revert tfstate + code | git revert + push |
|
|
||||||
| **Audit trail** | Terraform logs | Git history + ArgoCD logs |
|
|
||||||
| **Complexity** | High (2 tools) | Low (1 tool) |
|
|
||||||
| **New services** | Edit .tf files | Copy service directory |
|
|
||||||
| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects |
|
|
||||||
|
|
||||||
## Non-Disruptive Migration
|
|
||||||
|
|
||||||
**Key principle:** Live cluster remains running during entire migration.
|
|
||||||
|
|
||||||
**How?**
|
|
||||||
1. Build k8s/ structure **alongside** existing configuration
|
|
||||||
2. Deploy each layer via ArgoCD while old method still runs
|
|
||||||
3. Once new layer verified: delete old configuration
|
|
||||||
4. If issue detected: rollback (git revert) + redeploy old version
|
|
||||||
|
|
||||||
**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed)
|
|
||||||
|
|
||||||
## Rollback Procedure (At Any Point)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Identify broken layer
|
|
||||||
argocd app get layer-2-platform # Shows OutOfSync, error
|
|
||||||
|
|
||||||
# 2. Revert that layer
|
|
||||||
git revert <commit-hash>
|
|
||||||
git push
|
|
||||||
|
|
||||||
# 3. ArgoCD detects change, syncs back to previous version
|
|
||||||
# Expected: ~3 minutes for full reconciliation
|
|
||||||
|
|
||||||
# 4. Verify
|
|
||||||
argocd app get layer-2-platform # Should be Synced again
|
|
||||||
```
|
|
||||||
|
|
||||||
## Day 1 Action Items
|
|
||||||
|
|
||||||
1. **Read GITOPS_ARCHITECTURE.md** (30 min)
|
|
||||||
- Understand layers
|
|
||||||
- Understand directory structure
|
|
||||||
- Understand kustomization strategy
|
|
||||||
|
|
||||||
2. **Read GITOPS_MIGRATION_PLAN.md** (30 min)
|
|
||||||
- Understand phases
|
|
||||||
- Understand timeline
|
|
||||||
- Understand success criteria
|
|
||||||
|
|
||||||
3. **Review dependency map** (provided in plan)
|
|
||||||
- Confirm layer dependencies match current setup
|
|
||||||
- Identify any services not in layers
|
|
||||||
|
|
||||||
4. **Audit secrets** (30 min)
|
|
||||||
```bash
|
|
||||||
grep -r "password" k8s/ --include="*.yaml"
|
|
||||||
grep -r "token" k8s/ --include="*.yaml"
|
|
||||||
# Plan which need SOPS encryption
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Decision point:** Proceed to Phase 2 (Week 2)?
|
|
||||||
- YES → Start building k8s/ structure
|
|
||||||
- NO → Clarify concerns, refine plan
|
|
||||||
|
|
||||||
## Benefits Summary
|
|
||||||
|
|
||||||
✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth
|
|
||||||
✓ **Safer:** All changes reviewed in PRs, git history as audit trail
|
|
||||||
✓ **Scalable:** New services = copy directory, update git, done
|
|
||||||
✓ **Resilient:** State drift auto-corrected, rollback via `git revert`
|
|
||||||
✓ **Auditable:** Every change in git with who/when/why
|
|
||||||
✓ **Automated:** No manual kubectl apply, no manual terraform apply
|
|
||||||
✓ **Testable:** Kustomize build validates before sync
|
|
||||||
✓ **Recoverable:** git history = disaster recovery
|
|
||||||
|
|
||||||
## Support
|
|
||||||
|
|
||||||
- Architecture questions → GITOPS_ARCHITECTURE.md
|
|
||||||
- Implementation questions → GITOPS_MIGRATION_PLAN.md
|
|
||||||
- Syntax/tools questions → See referenced documentation
|
|
||||||
- Kustomize: https://kustomize.io/
|
|
||||||
- ArgoCD: https://argo-cd.readthedocs.io/
|
|
||||||
- SOPS: https://github.com/mozilla/sops
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. **User review:** Read both architecture documents (60 min)
|
|
||||||
2. **User decision:** Proceed to Phase 2 or refine plan?
|
|
||||||
3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Status:** Ready for implementation
|
|
||||||
**Terraform:** Deleted ✓
|
|
||||||
**Architecture:** Designed ✓
|
|
||||||
**Plan:** Ready ✓
|
|
||||||
**Waiting for:** User approval to proceed
|
|
||||||
@@ -1,573 +0,0 @@
|
|||||||
# GitOps Migration Plan: Implement Production-Grade Architecture
|
|
||||||
|
|
||||||
## Current State → Target State
|
|
||||||
|
|
||||||
### Current (Messy)
|
|
||||||
```
|
|
||||||
k8s/
|
|
||||||
├── talos-ci-cd/ (forgejo, runner)
|
|
||||||
├── talos-iam/ (authentik, vault)
|
|
||||||
├── logging/ (loki, promtail, grafana)
|
|
||||||
├── monitoring/ (prometheus, alerts)
|
|
||||||
├── storage/ (minio, longhorn)
|
|
||||||
├── ddb/ (postgres)
|
|
||||||
├── sqs/ (kafka, queue-crd)
|
|
||||||
├── temporal/ (temporal)
|
|
||||||
├── argocd/apps/ (20 Applications scattered)
|
|
||||||
└── ... (more scattered)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Target (Clean, Layered)
|
|
||||||
```
|
|
||||||
k8s/
|
|
||||||
├── infrastructure/ (namespaces, storage, RBAC)
|
|
||||||
├── bootstrap/ (cert-manager, cilium, ingress-nginx)
|
|
||||||
├── platform/ (storage, observability: minio, longhorn, loki, prometheus)
|
|
||||||
├── security/ (authentik, vault, cert-issuer)
|
|
||||||
├── applications/ (forgejo, grafana, portainer, temporal, llm)
|
|
||||||
├── data/ (postgres, redis, kafka)
|
|
||||||
└── argocd/
|
|
||||||
├── projects/ (AppProject definitions)
|
|
||||||
└── apps/ (6 layer Applications only)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Phase 1: Plan & Validate (Week 1)
|
|
||||||
|
|
||||||
### Step 1.1: Review GITOPS_ARCHITECTURE.md
|
|
||||||
- [ ] Understand directory structure
|
|
||||||
- [ ] Understand layer dependencies
|
|
||||||
- [ ] Understand Kustomization strategy
|
|
||||||
- User approval: YES/NO
|
|
||||||
|
|
||||||
### Step 1.2: Audit current resources
|
|
||||||
```bash
|
|
||||||
# Export all current resources
|
|
||||||
kubectl get all -A -o yaml > /tmp/current-state-backup.yaml
|
|
||||||
|
|
||||||
# Count resources per layer
|
|
||||||
kubectl get deployments -A | wc -l
|
|
||||||
kubectl get statefulsets -A | wc -l
|
|
||||||
kubectl get daemonsets -A | wc -l
|
|
||||||
kubectl get services -A | wc -l
|
|
||||||
kubectl get configmaps -A | wc -l
|
|
||||||
kubectl get secrets -A | wc -l
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 1.3: Identify secrets needing encryption
|
|
||||||
```bash
|
|
||||||
# Find secrets in current manifests
|
|
||||||
grep -r "kind: Secret" k8s/ --include="*.yaml"
|
|
||||||
grep -r "password" k8s/ --include="*.yaml"
|
|
||||||
grep -r "token" k8s/ --include="*.yaml"
|
|
||||||
|
|
||||||
# Plan SOPS encryption for:
|
|
||||||
# - Authentik bootstrap password
|
|
||||||
# - MinIO credentials
|
|
||||||
# - Database passwords
|
|
||||||
# - API tokens
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 1.4: Dependency mapping
|
|
||||||
```
|
|
||||||
Layer 0 (Infrastructure)
|
|
||||||
└─ Layer 1 (Bootstrap)
|
|
||||||
├─ cert-manager (creates certificates)
|
|
||||||
├─ cilium (network)
|
|
||||||
└─ ingress-nginx (entry point)
|
|
||||||
└─ Layer 2 (Platform)
|
|
||||||
├─ minio (storage)
|
|
||||||
├─ longhorn (PV storage)
|
|
||||||
├─ loki (logs)
|
|
||||||
└─ prometheus (metrics)
|
|
||||||
└─ Layer 3 (Security)
|
|
||||||
├─ authentik (auth)
|
|
||||||
└─ vault (secrets)
|
|
||||||
└─ Layer 4 (Applications)
|
|
||||||
├─ forgejo
|
|
||||||
├─ grafana
|
|
||||||
├─ portainer
|
|
||||||
├─ temporal
|
|
||||||
└─ llm
|
|
||||||
└─ Layer 5 (Data)
|
|
||||||
├─ postgres
|
|
||||||
├─ redis
|
|
||||||
└─ kafka
|
|
||||||
```
|
|
||||||
|
|
||||||
## Phase 2: Build Directory Structure (Weeks 2-3)
|
|
||||||
|
|
||||||
### Step 2.1: Create base directories
|
|
||||||
```bash
|
|
||||||
cd k8s/
|
|
||||||
|
|
||||||
# Create layers
|
|
||||||
mkdir -p infrastructure bootstrap platform security applications data
|
|
||||||
|
|
||||||
# Create kustomization.yaml for each
|
|
||||||
for dir in infrastructure bootstrap platform security applications data; do
|
|
||||||
cat > $dir/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
namespace: CHANGE_ME
|
|
||||||
|
|
||||||
resources: [] # Will add resources below
|
|
||||||
|
|
||||||
# Placeholders for this layer
|
|
||||||
EOF
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2.2: Migrate Layer 0 (Infrastructure)
|
|
||||||
**Resources to create:**
|
|
||||||
- Namespaces (all 20+)
|
|
||||||
- Storage classes (longhorn, xfs, longhorn-kafka)
|
|
||||||
- Service accounts (terraform-ci, etc.)
|
|
||||||
- Cluster roles (terraform-ci, admin, etc.)
|
|
||||||
- Network policies
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Extract from current cluster
|
|
||||||
kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml
|
|
||||||
kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml
|
|
||||||
kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml
|
|
||||||
kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml
|
|
||||||
|
|
||||||
# Clean up (remove status, owner refs, etc.)
|
|
||||||
# Add to k8s/infrastructure/kustomization.yaml:
|
|
||||||
resources:
|
|
||||||
- namespaces.yaml
|
|
||||||
- storage-classes.yaml
|
|
||||||
- service-accounts.yaml
|
|
||||||
- cluster-roles.yaml
|
|
||||||
|
|
||||||
# Test
|
|
||||||
kustomize build k8s/infrastructure/
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/infrastructure/kustomization.yaml
|
|
||||||
- [x] k8s/infrastructure/namespaces.yaml
|
|
||||||
- [x] k8s/infrastructure/storage-classes.yaml
|
|
||||||
- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml
|
|
||||||
|
|
||||||
### Step 2.3: Migrate Layer 1 (Bootstrap)
|
|
||||||
|
|
||||||
**Services: cert-manager, cilium, ingress-nginx**
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Create directories
|
|
||||||
mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx}
|
|
||||||
|
|
||||||
# For each service:
|
|
||||||
# 1. Export Helm values
|
|
||||||
helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml
|
|
||||||
|
|
||||||
# 2. Create kustomization.yaml with Helm chart ref
|
|
||||||
cat > bootstrap/cert-manager/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
namespace: cert-manager
|
|
||||||
|
|
||||||
helmCharts:
|
|
||||||
- name: cert-manager
|
|
||||||
repo: https://charts.jetstack.io
|
|
||||||
version: v1.21.0
|
|
||||||
releaseName: cert-manager
|
|
||||||
valuesFile: values.yaml
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# 3. Repeat for cilium, ingress-nginx
|
|
||||||
|
|
||||||
# 4. Add to k8s/bootstrap/kustomization.yaml:
|
|
||||||
resources:
|
|
||||||
- cert-manager/kustomization.yaml
|
|
||||||
- cilium/kustomization.yaml
|
|
||||||
- ingress-nginx/kustomization.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml
|
|
||||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml
|
|
||||||
- [x] k8s/bootstrap/kustomization.yaml
|
|
||||||
|
|
||||||
### Step 2.4: Migrate Layer 2 (Platform)
|
|
||||||
|
|
||||||
**Services: minio, longhorn, loki, prometheus, promtail**
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Create directories
|
|
||||||
mkdir -p platform/{minio,longhorn,loki,prometheus,promtail}
|
|
||||||
|
|
||||||
# Migrate minio
|
|
||||||
# 1. Export existing values
|
|
||||||
helm get values minio -n storage > platform/minio/values.yaml
|
|
||||||
|
|
||||||
# 2. Export bucket configs
|
|
||||||
kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/
|
|
||||||
|
|
||||||
# 3. Create kustomization.yaml
|
|
||||||
cat > platform/minio/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
namespace: storage
|
|
||||||
|
|
||||||
helmCharts:
|
|
||||||
- name: minio
|
|
||||||
repo: https://charts.min.io
|
|
||||||
version: 14.6.25 # Current version
|
|
||||||
releaseName: minio
|
|
||||||
valuesFile: values.yaml
|
|
||||||
|
|
||||||
resources:
|
|
||||||
- config/minio-buckets.yaml
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# Repeat for loki, prometheus, promtail
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml
|
|
||||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml
|
|
||||||
- [x] k8s/platform/kustomization.yaml
|
|
||||||
|
|
||||||
### Step 2.5: Migrate Layer 3 (Security)
|
|
||||||
|
|
||||||
**Services: authentik, vault**
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Create directories
|
|
||||||
mkdir -p security/{authentik,vault}
|
|
||||||
|
|
||||||
# Migrate authentik
|
|
||||||
helm get values authentik -n iam > security/authentik/values.yaml
|
|
||||||
|
|
||||||
# Export Authentik resources
|
|
||||||
kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml
|
|
||||||
kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml
|
|
||||||
kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml
|
|
||||||
|
|
||||||
# Create kustomization.yaml
|
|
||||||
cat > security/authentik/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
namespace: iam
|
|
||||||
|
|
||||||
helmCharts:
|
|
||||||
- name: authentik
|
|
||||||
repo: https://charts.goauthentik.io
|
|
||||||
version: 2024.12.1
|
|
||||||
releaseName: authentik
|
|
||||||
valuesFile: values.yaml
|
|
||||||
|
|
||||||
resources:
|
|
||||||
- config/apps.yaml
|
|
||||||
- config/groups.yaml
|
|
||||||
- config/oauth-providers.yaml
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# Repeat for vault
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/security/{authentik,vault}/kustomization.yaml
|
|
||||||
- [x] k8s/security/{authentik,vault}/values.yaml
|
|
||||||
- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml
|
|
||||||
- [x] k8s/security/kustomization.yaml
|
|
||||||
|
|
||||||
### Step 2.6: Migrate Layer 4 (Applications)
|
|
||||||
|
|
||||||
**Services: forgejo, grafana, portainer, temporal, llm**
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Create directories
|
|
||||||
mkdir -p applications/{forgejo,grafana,portainer,temporal,llm}
|
|
||||||
|
|
||||||
# Move existing manifests
|
|
||||||
cp -r k8s/talos-ci-cd/* applications/forgejo/
|
|
||||||
cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc.
|
|
||||||
cp -r k8s/portainer/* applications/portainer/
|
|
||||||
cp -r k8s/temporal/* applications/temporal/
|
|
||||||
cp -r k8s/llm/* applications/llm/
|
|
||||||
|
|
||||||
# For each, create kustomization.yaml with Helm chart ref
|
|
||||||
# (or keep existing manifests if not Helm)
|
|
||||||
|
|
||||||
# Create layer kustomization.yaml
|
|
||||||
cat > applications/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
resources:
|
|
||||||
- forgejo/kustomization.yaml
|
|
||||||
- grafana/kustomization.yaml
|
|
||||||
- portainer/kustomization.yaml
|
|
||||||
- temporal/kustomization.yaml
|
|
||||||
- llm/kustomization.yaml
|
|
||||||
EOF
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml
|
|
||||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml
|
|
||||||
- [x] k8s/applications/kustomization.yaml
|
|
||||||
|
|
||||||
### Step 2.7: Migrate Layer 5 (Data)
|
|
||||||
|
|
||||||
**Services: postgres (CloudNativePG), redis, kafka**
|
|
||||||
|
|
||||||
**Action:**
|
|
||||||
```bash
|
|
||||||
# Create directories
|
|
||||||
mkdir -p data/{postgres,redis,kafka}
|
|
||||||
|
|
||||||
# Export existing configurations
|
|
||||||
kubectl get cnpg -A -o yaml > data/postgres/config.yaml
|
|
||||||
kubectl get redis -A -o yaml > data/redis/config.yaml
|
|
||||||
kubectl get kafka -A -o yaml > data/kafka/config.yaml
|
|
||||||
|
|
||||||
# Create kustomization.yaml for each
|
|
||||||
|
|
||||||
# Repeat process...
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml
|
|
||||||
- [x] k8s/data/kustomization.yaml
|
|
||||||
|
|
||||||
## Phase 3: Create ArgoCD Application Definitions (Week 4)
|
|
||||||
|
|
||||||
### Step 3.1: Create layer Applications
|
|
||||||
```bash
|
|
||||||
mkdir -p k8s/argocd/apps/layers
|
|
||||||
|
|
||||||
# Root application (Layer 0 only, triggers rest)
|
|
||||||
cat > k8s/argocd/apps/root-app.yaml << 'EOF'
|
|
||||||
apiVersion: argoproj.io/v1alpha1
|
|
||||||
kind: Application
|
|
||||||
metadata:
|
|
||||||
name: homelab-root
|
|
||||||
namespace: argocd
|
|
||||||
spec:
|
|
||||||
project: homelab
|
|
||||||
source:
|
|
||||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
||||||
targetRevision: main
|
|
||||||
path: k8s/infrastructure
|
|
||||||
destination:
|
|
||||||
server: https://kubernetes.default.svc
|
|
||||||
syncPolicy:
|
|
||||||
automated:
|
|
||||||
prune: true
|
|
||||||
selfHeal: true
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# Layer 1 application
|
|
||||||
cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF'
|
|
||||||
apiVersion: argoproj.io/v1alpha1
|
|
||||||
kind: Application
|
|
||||||
metadata:
|
|
||||||
name: layer-1-bootstrap
|
|
||||||
namespace: argocd
|
|
||||||
annotations:
|
|
||||||
argocd.argoproj.io/sync-wave: "1"
|
|
||||||
spec:
|
|
||||||
project: homelab
|
|
||||||
source:
|
|
||||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
||||||
targetRevision: main
|
|
||||||
path: k8s/bootstrap
|
|
||||||
destination:
|
|
||||||
server: https://kubernetes.default.svc
|
|
||||||
syncPolicy:
|
|
||||||
automated:
|
|
||||||
prune: true
|
|
||||||
selfHeal: true
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Deliverables:**
|
|
||||||
- [x] k8s/argocd/apps/root-app.yaml
|
|
||||||
- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml
|
|
||||||
|
|
||||||
### Step 3.2: Create kustomization.yaml for apps
|
|
||||||
```bash
|
|
||||||
cat > k8s/argocd/apps/kustomization.yaml << 'EOF'
|
|
||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
||||||
kind: Kustomization
|
|
||||||
|
|
||||||
resources:
|
|
||||||
- root-app.yaml
|
|
||||||
- layer-1-bootstrap.yaml
|
|
||||||
- layer-2-platform.yaml
|
|
||||||
- layer-3-security.yaml
|
|
||||||
- layer-4-applications.yaml
|
|
||||||
- layer-5-data.yaml
|
|
||||||
EOF
|
|
||||||
```
|
|
||||||
|
|
||||||
## Phase 4: Deploy & Validate (Week 5)
|
|
||||||
|
|
||||||
### Step 4.1: Pre-flight checks
|
|
||||||
```bash
|
|
||||||
# Validate all kustomization files
|
|
||||||
for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do
|
|
||||||
echo "Validating $dir..."
|
|
||||||
kustomize build $dir > /tmp/validate.yaml
|
|
||||||
kubeval /tmp/validate.yaml || exit 1
|
|
||||||
done
|
|
||||||
|
|
||||||
# Validate ArgoCD apps
|
|
||||||
kubeval k8s/argocd/apps/*.yaml
|
|
||||||
|
|
||||||
# Test YAML lint
|
|
||||||
yamllint k8s/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4.2: Deploy root application
|
|
||||||
```bash
|
|
||||||
# Apply root app (Layer 0 only)
|
|
||||||
kubectl apply -f k8s/argocd/apps/root-app.yaml
|
|
||||||
|
|
||||||
# Watch sync
|
|
||||||
argocd app watch homelab-root
|
|
||||||
|
|
||||||
# Verify infrastructure deployed
|
|
||||||
kubectl get ns # Should have all namespaces
|
|
||||||
kubectl get sc # Should have all storage classes
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4.3: Deploy layer applications
|
|
||||||
```bash
|
|
||||||
# Apply layer applications (one by one)
|
|
||||||
kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml
|
|
||||||
sleep 5
|
|
||||||
argocd app watch layer-1-bootstrap
|
|
||||||
|
|
||||||
# Once Layer 1 synced: Layer 2
|
|
||||||
kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml
|
|
||||||
argocd app watch layer-2-platform
|
|
||||||
|
|
||||||
# ... repeat for layers 3, 4, 5
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4.4: Monitor for issues
|
|
||||||
```bash
|
|
||||||
# Check ArgoCD apps
|
|
||||||
argocd app list
|
|
||||||
|
|
||||||
# Watch events
|
|
||||||
kubectl get events -A -w
|
|
||||||
|
|
||||||
# Monitor pod status
|
|
||||||
kubectl get pods -A
|
|
||||||
|
|
||||||
# Check logs
|
|
||||||
kubectl logs -f -n argocd deployment/argocd-application-controller
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4.5: Validate state drift protection
|
|
||||||
```bash
|
|
||||||
# Manually edit a resource
|
|
||||||
kubectl edit deployment cert-manager -n cert-manager
|
|
||||||
# Change replicas from 1 → 2
|
|
||||||
|
|
||||||
# Wait 3 minutes (ArgoCD reconciliation interval)
|
|
||||||
sleep 180
|
|
||||||
|
|
||||||
# Check: should be back to 1 replica
|
|
||||||
kubectl get deployment cert-manager -n cert-manager
|
|
||||||
# Expected: 1/1 Ready (reverted by ArgoCD)
|
|
||||||
|
|
||||||
# Verify in ArgoCD
|
|
||||||
argocd app get layer-1-bootstrap
|
|
||||||
# Expected: Synced status
|
|
||||||
```
|
|
||||||
|
|
||||||
## Phase 5: Cleanup (Week 6)
|
|
||||||
|
|
||||||
### Step 5.1: Delete old directories
|
|
||||||
```bash
|
|
||||||
# After all apps synced successfully:
|
|
||||||
rm -rf k8s/talos-ci-cd/
|
|
||||||
rm -rf k8s/talos-iam/
|
|
||||||
rm -rf k8s/logging/
|
|
||||||
rm -rf k8s/monitoring/
|
|
||||||
rm -rf k8s/storage/
|
|
||||||
rm -rf k8s/ddb/
|
|
||||||
rm -rf k8s/sqs/
|
|
||||||
rm -rf k8s/temporal/
|
|
||||||
# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 5.2: Update CI/CD pipeline
|
|
||||||
```bash
|
|
||||||
# Update .forgejo/workflows/
|
|
||||||
# - Remove: terraform validate/plan/apply
|
|
||||||
# - Add: kustomize build validation
|
|
||||||
# - Add: argocd app validation
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 5.3: Archive old configuration
|
|
||||||
```bash
|
|
||||||
# Keep for reference only
|
|
||||||
mkdir k8s/.archive/
|
|
||||||
git mv k8s/old-structure-backup k8s/.archive/
|
|
||||||
git commit -m "archive: old k8s structure (moved to pure GitOps)"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Rollback Plan
|
|
||||||
|
|
||||||
**If something breaks during migration:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Option 1: Rollback entire layer
|
|
||||||
git checkout HEAD~1 -- k8s/bootstrap/
|
|
||||||
git commit -m "revert: bootstrap layer (investigating)"
|
|
||||||
# ArgoCD will resync to previous version automatically
|
|
||||||
|
|
||||||
# Option 2: Pause ArgoCD sync
|
|
||||||
argocd app set layer-1-bootstrap --sync-policy none
|
|
||||||
# Investigate, then re-enable:
|
|
||||||
argocd app set layer-1-bootstrap --sync-policy automated
|
|
||||||
|
|
||||||
# Option 3: Full rollback to pre-migration
|
|
||||||
git reset --hard <commit-before-migration>
|
|
||||||
argocd app set homelab-root --sync-policy none
|
|
||||||
# Manual investigation, then re-enable
|
|
||||||
```
|
|
||||||
|
|
||||||
## Timeline Summary
|
|
||||||
|
|
||||||
```
|
|
||||||
Week 1: Plan & validate (reviews, dependency mapping)
|
|
||||||
Week 2: Layers 0-1 (infrastructure, bootstrap)
|
|
||||||
Week 3: Layers 2-3 (platform, security)
|
|
||||||
Week 3: Layers 4-5 (applications, data)
|
|
||||||
Week 4: ArgoCD applications + layer definitions
|
|
||||||
Week 5: Deploy & validate (watch for issues)
|
|
||||||
Week 6: Cleanup & CI/CD updates
|
|
||||||
```
|
|
||||||
|
|
||||||
**Total: 6 weeks, non-disruptive (all layers coexist during migration)**
|
|
||||||
|
|
||||||
## Success Criteria
|
|
||||||
|
|
||||||
✓ All 20+ services deployed via ArgoCD
|
|
||||||
✓ No manual kubectl apply in production
|
|
||||||
✓ State drift detected & corrected automatically
|
|
||||||
✓ All changes in git (reviewed via PR)
|
|
||||||
✓ Rollback possible at any time (git history)
|
|
||||||
✓ CI/CD validates all commits
|
|
||||||
✓ Secrets encrypted with SOPS
|
|
||||||
✓ New services can be added (copy service directory)
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
# Cluster Recovery Plan
|
|
||||||
|
|
||||||
**Context:** Control-plane node (`talos-cp-1`, 192.168.1.213) had a corrupted machine CA in `terraform.tfvars`, causing a "broken key size" TLS error. During troubleshooting, an `apply-config` run with `install.wipe: true` (before the fix) wiped the OS/etcd disk (`sda`). Longhorn storage disks (`sdb`/`sdc`/`sdd`, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is `NotReady` (no CNI yet).
|
|
||||||
|
|
||||||
## Phase 1 — Get CNI up (blocks everything else)
|
|
||||||
|
|
||||||
Cluster config has `cni: name: none` (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
helm repo add cilium https://helm.cilium.io/
|
|
||||||
helm repo update
|
|
||||||
helm install cilium cilium/cilium --namespace kube-system \
|
|
||||||
--set ipam.mode=kubernetes \
|
|
||||||
--set kubeProxyReplacement=true \
|
|
||||||
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
|
|
||||||
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
|
|
||||||
--set cgroup.autoMount.enabled=false \
|
|
||||||
--set cgroup.hostRoot=/sys/fs/cgroup \
|
|
||||||
--set k8sServiceHost=localhost \
|
|
||||||
--set k8sServicePort=7445
|
|
||||||
```
|
|
||||||
|
|
||||||
Verify: node flips to `Ready`, `cilium status` reports OK.
|
|
||||||
|
|
||||||
## Phase 2 — Bootstrap ArgoCD (app-of-apps)
|
|
||||||
|
|
||||||
If ArgoCD's own manifests + root Application live in git (`k8s/argocd/`), recovery is a 3-step process:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl create namespace argocd
|
|
||||||
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
|
|
||||||
# wait for argocd-server to be Ready
|
|
||||||
kubectl apply -f k8s/argocd/root-app.yaml # app-of-apps entrypoint
|
|
||||||
```
|
|
||||||
|
|
||||||
Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs.
|
|
||||||
|
|
||||||
## Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall)
|
|
||||||
|
|
||||||
Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica."
|
|
||||||
|
|
||||||
1. Confirm disk path via `longhorn-disk.cfg` (already confirmed: `/var/lib/longhorn-disk{1,2,3}`)
|
|
||||||
2. Locate `replicas/<volume-name>/` directories, read `volume.meta` for size
|
|
||||||
3. Launch a temporary `longhornio/longhorn-engine` container against the replica directory to expose it as a block device
|
|
||||||
4. Mount and copy data out manually, per volume
|
|
||||||
|
|
||||||
Not automatic — budget real time, one volume at a time.
|
|
||||||
|
|
||||||
## Phase 4 — WireGuard / router path
|
|
||||||
|
|
||||||
DuckDNS (`riotpiao.duckdns.org`) resolves correctly to current public IP (`24.114.42.68`, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from **off-LAN** — needs router-side verification:
|
|
||||||
- Confirm port-forward rule is **UDP 51820 → 192.168.1.213:51820** exactly
|
|
||||||
- If double-NAT, forward must be on the outermost internet-facing router
|
|
||||||
- Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding)
|
|
||||||
|
|
||||||
**Separately — LAN-side symptom:** tunnel also fails to connect **from the same LAN** as the server, via the public DDNS hostname. Root cause: **NAT hairpin/loopback not supported by the router** — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug.
|
|
||||||
|
|
||||||
Fixes, in order of robustness:
|
|
||||||
1. Enable NAT hairpinning/loopback in router settings, if supported
|
|
||||||
2. Split-DNS: resolve the DDNS hostname to the LAN IP (`192.168.1.213`) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS)
|
|
||||||
3. Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at `192.168.1.213:51820` while on LAN; switch back to the DDNS hostname when away
|
|
||||||
|
|
||||||
## Phase 5 — Redeploy DuckDNS updater
|
|
||||||
|
|
||||||
The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a `CronJob` running `curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip="` every few minutes.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Recommended order:** Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency).
|
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
- [How to Install Cilium on Talos Linux Step by Step](https://oneuptime.com/blog/post/2026-03-03-install-cilium-on-talos-linux-step-by-step/view)
|
|
||||||
- [Deploy Cilium CNI - Sidero Documentation](https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium)
|
|
||||||
- [Cluster Bootstrapping - Argo CD](https://argo-cd.readthedocs.io/en/latest/operator-manual/cluster-bootstrapping/)
|
|
||||||
- [How to Bootstrap an Entire Cluster with ArgoCD App-of-Apps](https://oneuptime.com/blog/post/2026-02-26-argocd-bootstrap-cluster-app-of-apps/view)
|
|
||||||
- [Restoring Data from an Orphaned Replica Directory - Longhorn KB](https://longhorn.io/kb/restoring-data-from-an-orphaned-replica-directory/)
|
|
||||||
- [Port Forwarding for WireGuard](https://portforward.com/wireguard/)
|
|
||||||
- [Guide Wireguard-portforwarding - SNBForums](https://www.snbforums.com/threads/guide-wireguard-portforwarding.89737/)
|
|
||||||
- [Help with hairpin NAT for wireguard on ubuntu server - Ubiquiti Community](https://community.ui.com/questions/Help-with-hairpin-NAT-for-wireguard-on-ubuntu-server/12d4cbe7-f1f5-448d-9d77-ea8226c18f16)
|
|
||||||
- [Wireguard VPN on Pi4 - cannot connect client - Raspberry Pi Forums](https://forums.raspberrypi.com/viewtopic.php?t=339692)
|
|
||||||
Reference in New Issue
Block a user