chore: remove scratch planning docs — not meant for the repo
This commit is contained in:
Story Crater Bot
@@ -1,246 +0,0 @@
|
||||
# GitOps IaC Implementation Summary
|
||||
|
||||
## Documents Created
|
||||
|
||||
1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design
|
||||
2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive)
|
||||
|
||||
## Architecture at a Glance
|
||||
|
||||
```
|
||||
Pure GitOps (ArgoCD only, no Terraform)
|
||||
├── Layer 0: Infrastructure (namespaces, storage, RBAC)
|
||||
├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx)
|
||||
├── Layer 2: Platform (minio, longhorn, loki, prometheus)
|
||||
├── Layer 3: Security (authentik, vault)
|
||||
├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm)
|
||||
└── Layer 5: Data (postgres, redis, kafka)
|
||||
```
|
||||
|
||||
**Key benefit:** Each layer depends on previous, staged deployment, easy rollback.
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── _base/ # Shared kustomization base
|
||||
├── infrastructure/ # Layer 0 (foundation)
|
||||
├── bootstrap/ # Layer 1 (networking)
|
||||
├── platform/ # Layer 2 (services)
|
||||
├── security/ # Layer 3 (auth)
|
||||
├── applications/ # Layer 4 (business logic)
|
||||
├── data/ # Layer 5 (stateful)
|
||||
└── argocd/
|
||||
├── projects/ # AppProject definitions
|
||||
└── apps/
|
||||
├── root-app.yaml # Bootstraps Layer 0
|
||||
└── layer-*.yaml # 5 layer apps (synced in order)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Plan (Week 1)
|
||||
- [x] Review GITOPS_ARCHITECTURE.md
|
||||
- [x] Review GITOPS_MIGRATION_PLAN.md
|
||||
- [ ] Audit current cluster resources
|
||||
- [ ] Identify secrets needing encryption (SOPS)
|
||||
- [ ] Map dependencies (provided in plan)
|
||||
- **User approval required before proceeding to Phase 2**
|
||||
|
||||
### Phase 2: Build (Weeks 2-3)
|
||||
- [ ] Create k8s/ directory structure
|
||||
- [ ] Migrate Layer 0 (infrastructure)
|
||||
- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx)
|
||||
- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus)
|
||||
- [ ] Migrate Layer 3 (security: authentik, vault)
|
||||
- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.)
|
||||
- [ ] Migrate Layer 5 (data: postgres, redis, kafka)
|
||||
|
||||
### Phase 3: ArgoCD (Week 4)
|
||||
- [ ] Create root application (Layer 0 trigger)
|
||||
- [ ] Create 5 layer applications (with sync-wave ordering)
|
||||
- [ ] Update AppProject permissions
|
||||
- [ ] Test dry-run on each layer
|
||||
|
||||
### Phase 4: Deploy (Week 5)
|
||||
- [ ] Pre-flight validation (kustomize build, kubeval, yamllint)
|
||||
- [ ] Deploy root application
|
||||
- [ ] Deploy layer applications (one by one)
|
||||
- [ ] Monitor for drift
|
||||
- [ ] Verify state correction (manual edits reverted)
|
||||
|
||||
### Phase 5: Cleanup (Week 6)
|
||||
- [ ] Delete old k8s/ directories
|
||||
- [ ] Update CI/CD pipeline (remove terraform, add kustomize)
|
||||
- [ ] Archive legacy configuration
|
||||
- [ ] Final validation
|
||||
|
||||
## Key Architecture Decisions
|
||||
|
||||
### ✓ Why Layers?
|
||||
- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds
|
||||
- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only.
|
||||
- **Easy scaling:** Add new service without touching others
|
||||
- **Easy rollback:** `git revert <layer-commit>`
|
||||
|
||||
### ✓ Why Kustomize?
|
||||
- Helm chart package + values override (clean separation)
|
||||
- Kustomize build validates before sync
|
||||
- Patches enable per-environment customization
|
||||
- Standard K8s tool (no external dependencies)
|
||||
|
||||
### ✓ Why No Terraform?
|
||||
- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD)
|
||||
- Git is source of truth for everything
|
||||
- No state backend (no .tfstate files)
|
||||
- ArgoCD continuous reconciliation (drift auto-corrected)
|
||||
- All changes reviewed in PRs (audit trail in git)
|
||||
|
||||
### ✓ Why This File Structure?
|
||||
- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/)
|
||||
- **Scalable:** New service = new directory (copy template)
|
||||
- **Maintainable:** Clear ownership (who owns layer? who owns service?)
|
||||
- **GitOps-ready:** Each layer is independently deployable
|
||||
|
||||
## Secret Management (SOPS)
|
||||
|
||||
### Before (Terraform)
|
||||
```
|
||||
terraform/terraform.tfvars # Plaintext secrets (risky!)
|
||||
```
|
||||
|
||||
### After (GitOps)
|
||||
```bash
|
||||
# Encrypt secrets before commit
|
||||
sops -e secrets.yaml > secrets.enc.yaml
|
||||
git add secrets.enc.yaml
|
||||
|
||||
# ArgoCD decrypts at sync time (SOPS plugin installed)
|
||||
# Git stores only encrypted version (safe)
|
||||
```
|
||||
|
||||
**Secrets to encrypt:**
|
||||
- Authentik bootstrap password
|
||||
- MinIO root credentials
|
||||
- Database passwords
|
||||
- API tokens
|
||||
- OAuth client secrets
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Before
|
||||
```
|
||||
git push → terraform validate → terraform plan → manual apply
|
||||
```
|
||||
|
||||
### After (Pure GitOps)
|
||||
```
|
||||
git push → yamllint → kubeval → kustomize build → argocd validation
|
||||
↓
|
||||
ArgoCD auto-syncs (if PR merged)
|
||||
```
|
||||
|
||||
**No manual steps. No terraform. No state management. Pure GitOps.**
|
||||
|
||||
## Comparison: Old vs. New
|
||||
|
||||
| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) |
|
||||
|--------|--------------------------|-------------------|
|
||||
| **State management** | tfstate files | Git history |
|
||||
| **Secrets** | Plaintext in .tfvars | SOPS encrypted |
|
||||
| **Manual steps** | terraform apply required | git push only |
|
||||
| **Rollback** | Revert tfstate + code | git revert + push |
|
||||
| **Audit trail** | Terraform logs | Git history + ArgoCD logs |
|
||||
| **Complexity** | High (2 tools) | Low (1 tool) |
|
||||
| **New services** | Edit .tf files | Copy service directory |
|
||||
| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects |
|
||||
|
||||
## Non-Disruptive Migration
|
||||
|
||||
**Key principle:** Live cluster remains running during entire migration.
|
||||
|
||||
**How?**
|
||||
1. Build k8s/ structure **alongside** existing configuration
|
||||
2. Deploy each layer via ArgoCD while old method still runs
|
||||
3. Once new layer verified: delete old configuration
|
||||
4. If issue detected: rollback (git revert) + redeploy old version
|
||||
|
||||
**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed)
|
||||
|
||||
## Rollback Procedure (At Any Point)
|
||||
|
||||
```bash
|
||||
# 1. Identify broken layer
|
||||
argocd app get layer-2-platform # Shows OutOfSync, error
|
||||
|
||||
# 2. Revert that layer
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# 3. ArgoCD detects change, syncs back to previous version
|
||||
# Expected: ~3 minutes for full reconciliation
|
||||
|
||||
# 4. Verify
|
||||
argocd app get layer-2-platform # Should be Synced again
|
||||
```
|
||||
|
||||
## Day 1 Action Items
|
||||
|
||||
1. **Read GITOPS_ARCHITECTURE.md** (30 min)
|
||||
- Understand layers
|
||||
- Understand directory structure
|
||||
- Understand kustomization strategy
|
||||
|
||||
2. **Read GITOPS_MIGRATION_PLAN.md** (30 min)
|
||||
- Understand phases
|
||||
- Understand timeline
|
||||
- Understand success criteria
|
||||
|
||||
3. **Review dependency map** (provided in plan)
|
||||
- Confirm layer dependencies match current setup
|
||||
- Identify any services not in layers
|
||||
|
||||
4. **Audit secrets** (30 min)
|
||||
```bash
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
# Plan which need SOPS encryption
|
||||
```
|
||||
|
||||
5. **Decision point:** Proceed to Phase 2 (Week 2)?
|
||||
- YES → Start building k8s/ structure
|
||||
- NO → Clarify concerns, refine plan
|
||||
|
||||
## Benefits Summary
|
||||
|
||||
✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth
|
||||
✓ **Safer:** All changes reviewed in PRs, git history as audit trail
|
||||
✓ **Scalable:** New services = copy directory, update git, done
|
||||
✓ **Resilient:** State drift auto-corrected, rollback via `git revert`
|
||||
✓ **Auditable:** Every change in git with who/when/why
|
||||
✓ **Automated:** No manual kubectl apply, no manual terraform apply
|
||||
✓ **Testable:** Kustomize build validates before sync
|
||||
✓ **Recoverable:** git history = disaster recovery
|
||||
|
||||
## Support
|
||||
|
||||
- Architecture questions → GITOPS_ARCHITECTURE.md
|
||||
- Implementation questions → GITOPS_MIGRATION_PLAN.md
|
||||
- Syntax/tools questions → See referenced documentation
|
||||
- Kustomize: https://kustomize.io/
|
||||
- ArgoCD: https://argo-cd.readthedocs.io/
|
||||
- SOPS: https://github.com/mozilla/sops
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **User review:** Read both architecture documents (60 min)
|
||||
2. **User decision:** Proceed to Phase 2 or refine plan?
|
||||
3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday)
|
||||
|
||||
---
|
||||
|
||||
**Status:** Ready for implementation
|
||||
**Terraform:** Deleted ✓
|
||||
**Architecture:** Designed ✓
|
||||
**Plan:** Ready ✓
|
||||
**Waiting for:** User approval to proceed
|
||||
@@ -1,573 +0,0 @@
|
||||
# GitOps Migration Plan: Implement Production-Grade Architecture
|
||||
|
||||
## Current State → Target State
|
||||
|
||||
### Current (Messy)
|
||||
```
|
||||
k8s/
|
||||
├── talos-ci-cd/ (forgejo, runner)
|
||||
├── talos-iam/ (authentik, vault)
|
||||
├── logging/ (loki, promtail, grafana)
|
||||
├── monitoring/ (prometheus, alerts)
|
||||
├── storage/ (minio, longhorn)
|
||||
├── ddb/ (postgres)
|
||||
├── sqs/ (kafka, queue-crd)
|
||||
├── temporal/ (temporal)
|
||||
├── argocd/apps/ (20 Applications scattered)
|
||||
└── ... (more scattered)
|
||||
```
|
||||
|
||||
### Target (Clean, Layered)
|
||||
```
|
||||
k8s/
|
||||
├── infrastructure/ (namespaces, storage, RBAC)
|
||||
├── bootstrap/ (cert-manager, cilium, ingress-nginx)
|
||||
├── platform/ (storage, observability: minio, longhorn, loki, prometheus)
|
||||
├── security/ (authentik, vault, cert-issuer)
|
||||
├── applications/ (forgejo, grafana, portainer, temporal, llm)
|
||||
├── data/ (postgres, redis, kafka)
|
||||
└── argocd/
|
||||
├── projects/ (AppProject definitions)
|
||||
└── apps/ (6 layer Applications only)
|
||||
```
|
||||
|
||||
## Phase 1: Plan & Validate (Week 1)
|
||||
|
||||
### Step 1.1: Review GITOPS_ARCHITECTURE.md
|
||||
- [ ] Understand directory structure
|
||||
- [ ] Understand layer dependencies
|
||||
- [ ] Understand Kustomization strategy
|
||||
- User approval: YES/NO
|
||||
|
||||
### Step 1.2: Audit current resources
|
||||
```bash
|
||||
# Export all current resources
|
||||
kubectl get all -A -o yaml > /tmp/current-state-backup.yaml
|
||||
|
||||
# Count resources per layer
|
||||
kubectl get deployments -A | wc -l
|
||||
kubectl get statefulsets -A | wc -l
|
||||
kubectl get daemonsets -A | wc -l
|
||||
kubectl get services -A | wc -l
|
||||
kubectl get configmaps -A | wc -l
|
||||
kubectl get secrets -A | wc -l
|
||||
```
|
||||
|
||||
### Step 1.3: Identify secrets needing encryption
|
||||
```bash
|
||||
# Find secrets in current manifests
|
||||
grep -r "kind: Secret" k8s/ --include="*.yaml"
|
||||
grep -r "password" k8s/ --include="*.yaml"
|
||||
grep -r "token" k8s/ --include="*.yaml"
|
||||
|
||||
# Plan SOPS encryption for:
|
||||
# - Authentik bootstrap password
|
||||
# - MinIO credentials
|
||||
# - Database passwords
|
||||
# - API tokens
|
||||
```
|
||||
|
||||
### Step 1.4: Dependency mapping
|
||||
```
|
||||
Layer 0 (Infrastructure)
|
||||
└─ Layer 1 (Bootstrap)
|
||||
├─ cert-manager (creates certificates)
|
||||
├─ cilium (network)
|
||||
└─ ingress-nginx (entry point)
|
||||
└─ Layer 2 (Platform)
|
||||
├─ minio (storage)
|
||||
├─ longhorn (PV storage)
|
||||
├─ loki (logs)
|
||||
└─ prometheus (metrics)
|
||||
└─ Layer 3 (Security)
|
||||
├─ authentik (auth)
|
||||
└─ vault (secrets)
|
||||
└─ Layer 4 (Applications)
|
||||
├─ forgejo
|
||||
├─ grafana
|
||||
├─ portainer
|
||||
├─ temporal
|
||||
└─ llm
|
||||
└─ Layer 5 (Data)
|
||||
├─ postgres
|
||||
├─ redis
|
||||
└─ kafka
|
||||
```
|
||||
|
||||
## Phase 2: Build Directory Structure (Weeks 2-3)
|
||||
|
||||
### Step 2.1: Create base directories
|
||||
```bash
|
||||
cd k8s/
|
||||
|
||||
# Create layers
|
||||
mkdir -p infrastructure bootstrap platform security applications data
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
for dir in infrastructure bootstrap platform security applications data; do
|
||||
cat > $dir/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: CHANGE_ME
|
||||
|
||||
resources: [] # Will add resources below
|
||||
|
||||
# Placeholders for this layer
|
||||
EOF
|
||||
done
|
||||
```
|
||||
|
||||
### Step 2.2: Migrate Layer 0 (Infrastructure)
|
||||
**Resources to create:**
|
||||
- Namespaces (all 20+)
|
||||
- Storage classes (longhorn, xfs, longhorn-kafka)
|
||||
- Service accounts (terraform-ci, etc.)
|
||||
- Cluster roles (terraform-ci, admin, etc.)
|
||||
- Network policies
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Extract from current cluster
|
||||
kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml
|
||||
kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml
|
||||
kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml
|
||||
kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml
|
||||
|
||||
# Clean up (remove status, owner refs, etc.)
|
||||
# Add to k8s/infrastructure/kustomization.yaml:
|
||||
resources:
|
||||
- namespaces.yaml
|
||||
- storage-classes.yaml
|
||||
- service-accounts.yaml
|
||||
- cluster-roles.yaml
|
||||
|
||||
# Test
|
||||
kustomize build k8s/infrastructure/
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/infrastructure/kustomization.yaml
|
||||
- [x] k8s/infrastructure/namespaces.yaml
|
||||
- [x] k8s/infrastructure/storage-classes.yaml
|
||||
- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml
|
||||
|
||||
### Step 2.3: Migrate Layer 1 (Bootstrap)
|
||||
|
||||
**Services: cert-manager, cilium, ingress-nginx**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx}
|
||||
|
||||
# For each service:
|
||||
# 1. Export Helm values
|
||||
helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml
|
||||
|
||||
# 2. Create kustomization.yaml with Helm chart ref
|
||||
cat > bootstrap/cert-manager/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: cert-manager
|
||||
|
||||
helmCharts:
|
||||
- name: cert-manager
|
||||
repo: https://charts.jetstack.io
|
||||
version: v1.21.0
|
||||
releaseName: cert-manager
|
||||
valuesFile: values.yaml
|
||||
EOF
|
||||
|
||||
# 3. Repeat for cilium, ingress-nginx
|
||||
|
||||
# 4. Add to k8s/bootstrap/kustomization.yaml:
|
||||
resources:
|
||||
- cert-manager/kustomization.yaml
|
||||
- cilium/kustomization.yaml
|
||||
- ingress-nginx/kustomization.yaml
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml
|
||||
- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml
|
||||
- [x] k8s/bootstrap/kustomization.yaml
|
||||
|
||||
### Step 2.4: Migrate Layer 2 (Platform)
|
||||
|
||||
**Services: minio, longhorn, loki, prometheus, promtail**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p platform/{minio,longhorn,loki,prometheus,promtail}
|
||||
|
||||
# Migrate minio
|
||||
# 1. Export existing values
|
||||
helm get values minio -n storage > platform/minio/values.yaml
|
||||
|
||||
# 2. Export bucket configs
|
||||
kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/
|
||||
|
||||
# 3. Create kustomization.yaml
|
||||
cat > platform/minio/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: storage
|
||||
|
||||
helmCharts:
|
||||
- name: minio
|
||||
repo: https://charts.min.io
|
||||
version: 14.6.25 # Current version
|
||||
releaseName: minio
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/minio-buckets.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for loki, prometheus, promtail
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml
|
||||
- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml
|
||||
- [x] k8s/platform/kustomization.yaml
|
||||
|
||||
### Step 2.5: Migrate Layer 3 (Security)
|
||||
|
||||
**Services: authentik, vault**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p security/{authentik,vault}
|
||||
|
||||
# Migrate authentik
|
||||
helm get values authentik -n iam > security/authentik/values.yaml
|
||||
|
||||
# Export Authentik resources
|
||||
kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml
|
||||
kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml
|
||||
kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml
|
||||
|
||||
# Create kustomization.yaml
|
||||
cat > security/authentik/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: iam
|
||||
|
||||
helmCharts:
|
||||
- name: authentik
|
||||
repo: https://charts.goauthentik.io
|
||||
version: 2024.12.1
|
||||
releaseName: authentik
|
||||
valuesFile: values.yaml
|
||||
|
||||
resources:
|
||||
- config/apps.yaml
|
||||
- config/groups.yaml
|
||||
- config/oauth-providers.yaml
|
||||
EOF
|
||||
|
||||
# Repeat for vault
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/security/{authentik,vault}/kustomization.yaml
|
||||
- [x] k8s/security/{authentik,vault}/values.yaml
|
||||
- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml
|
||||
- [x] k8s/security/kustomization.yaml
|
||||
|
||||
### Step 2.6: Migrate Layer 4 (Applications)
|
||||
|
||||
**Services: forgejo, grafana, portainer, temporal, llm**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p applications/{forgejo,grafana,portainer,temporal,llm}
|
||||
|
||||
# Move existing manifests
|
||||
cp -r k8s/talos-ci-cd/* applications/forgejo/
|
||||
cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc.
|
||||
cp -r k8s/portainer/* applications/portainer/
|
||||
cp -r k8s/temporal/* applications/temporal/
|
||||
cp -r k8s/llm/* applications/llm/
|
||||
|
||||
# For each, create kustomization.yaml with Helm chart ref
|
||||
# (or keep existing manifests if not Helm)
|
||||
|
||||
# Create layer kustomization.yaml
|
||||
cat > applications/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- forgejo/kustomization.yaml
|
||||
- grafana/kustomization.yaml
|
||||
- portainer/kustomization.yaml
|
||||
- temporal/kustomization.yaml
|
||||
- llm/kustomization.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml
|
||||
- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml
|
||||
- [x] k8s/applications/kustomization.yaml
|
||||
|
||||
### Step 2.7: Migrate Layer 5 (Data)
|
||||
|
||||
**Services: postgres (CloudNativePG), redis, kafka**
|
||||
|
||||
**Action:**
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p data/{postgres,redis,kafka}
|
||||
|
||||
# Export existing configurations
|
||||
kubectl get cnpg -A -o yaml > data/postgres/config.yaml
|
||||
kubectl get redis -A -o yaml > data/redis/config.yaml
|
||||
kubectl get kafka -A -o yaml > data/kafka/config.yaml
|
||||
|
||||
# Create kustomization.yaml for each
|
||||
|
||||
# Repeat process...
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml
|
||||
- [x] k8s/data/kustomization.yaml
|
||||
|
||||
## Phase 3: Create ArgoCD Application Definitions (Week 4)
|
||||
|
||||
### Step 3.1: Create layer Applications
|
||||
```bash
|
||||
mkdir -p k8s/argocd/apps/layers
|
||||
|
||||
# Root application (Layer 0 only, triggers rest)
|
||||
cat > k8s/argocd/apps/root-app.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: homelab-root
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/infrastructure
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Layer 1 application
|
||||
cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF'
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: layer-1-bootstrap
|
||||
namespace: argocd
|
||||
annotations:
|
||||
argocd.argoproj.io/sync-wave: "1"
|
||||
spec:
|
||||
project: homelab
|
||||
source:
|
||||
repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
path: k8s/bootstrap
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
EOF
|
||||
|
||||
# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5)
|
||||
```
|
||||
|
||||
**Deliverables:**
|
||||
- [x] k8s/argocd/apps/root-app.yaml
|
||||
- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml
|
||||
|
||||
### Step 3.2: Create kustomization.yaml for apps
|
||||
```bash
|
||||
cat > k8s/argocd/apps/kustomization.yaml << 'EOF'
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- root-app.yaml
|
||||
- layer-1-bootstrap.yaml
|
||||
- layer-2-platform.yaml
|
||||
- layer-3-security.yaml
|
||||
- layer-4-applications.yaml
|
||||
- layer-5-data.yaml
|
||||
EOF
|
||||
```
|
||||
|
||||
## Phase 4: Deploy & Validate (Week 5)
|
||||
|
||||
### Step 4.1: Pre-flight checks
|
||||
```bash
|
||||
# Validate all kustomization files
|
||||
for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do
|
||||
echo "Validating $dir..."
|
||||
kustomize build $dir > /tmp/validate.yaml
|
||||
kubeval /tmp/validate.yaml || exit 1
|
||||
done
|
||||
|
||||
# Validate ArgoCD apps
|
||||
kubeval k8s/argocd/apps/*.yaml
|
||||
|
||||
# Test YAML lint
|
||||
yamllint k8s/
|
||||
```
|
||||
|
||||
### Step 4.2: Deploy root application
|
||||
```bash
|
||||
# Apply root app (Layer 0 only)
|
||||
kubectl apply -f k8s/argocd/apps/root-app.yaml
|
||||
|
||||
# Watch sync
|
||||
argocd app watch homelab-root
|
||||
|
||||
# Verify infrastructure deployed
|
||||
kubectl get ns # Should have all namespaces
|
||||
kubectl get sc # Should have all storage classes
|
||||
```
|
||||
|
||||
### Step 4.3: Deploy layer applications
|
||||
```bash
|
||||
# Apply layer applications (one by one)
|
||||
kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml
|
||||
sleep 5
|
||||
argocd app watch layer-1-bootstrap
|
||||
|
||||
# Once Layer 1 synced: Layer 2
|
||||
kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml
|
||||
argocd app watch layer-2-platform
|
||||
|
||||
# ... repeat for layers 3, 4, 5
|
||||
```
|
||||
|
||||
### Step 4.4: Monitor for issues
|
||||
```bash
|
||||
# Check ArgoCD apps
|
||||
argocd app list
|
||||
|
||||
# Watch events
|
||||
kubectl get events -A -w
|
||||
|
||||
# Monitor pod status
|
||||
kubectl get pods -A
|
||||
|
||||
# Check logs
|
||||
kubectl logs -f -n argocd deployment/argocd-application-controller
|
||||
```
|
||||
|
||||
### Step 4.5: Validate state drift protection
|
||||
```bash
|
||||
# Manually edit a resource
|
||||
kubectl edit deployment cert-manager -n cert-manager
|
||||
# Change replicas from 1 → 2
|
||||
|
||||
# Wait 3 minutes (ArgoCD reconciliation interval)
|
||||
sleep 180
|
||||
|
||||
# Check: should be back to 1 replica
|
||||
kubectl get deployment cert-manager -n cert-manager
|
||||
# Expected: 1/1 Ready (reverted by ArgoCD)
|
||||
|
||||
# Verify in ArgoCD
|
||||
argocd app get layer-1-bootstrap
|
||||
# Expected: Synced status
|
||||
```
|
||||
|
||||
## Phase 5: Cleanup (Week 6)
|
||||
|
||||
### Step 5.1: Delete old directories
|
||||
```bash
|
||||
# After all apps synced successfully:
|
||||
rm -rf k8s/talos-ci-cd/
|
||||
rm -rf k8s/talos-iam/
|
||||
rm -rf k8s/logging/
|
||||
rm -rf k8s/monitoring/
|
||||
rm -rf k8s/storage/
|
||||
rm -rf k8s/ddb/
|
||||
rm -rf k8s/sqs/
|
||||
rm -rf k8s/temporal/
|
||||
# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd
|
||||
```
|
||||
|
||||
### Step 5.2: Update CI/CD pipeline
|
||||
```bash
|
||||
# Update .forgejo/workflows/
|
||||
# - Remove: terraform validate/plan/apply
|
||||
# - Add: kustomize build validation
|
||||
# - Add: argocd app validation
|
||||
```
|
||||
|
||||
### Step 5.3: Archive old configuration
|
||||
```bash
|
||||
# Keep for reference only
|
||||
mkdir k8s/.archive/
|
||||
git mv k8s/old-structure-backup k8s/.archive/
|
||||
git commit -m "archive: old k8s structure (moved to pure GitOps)"
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
**If something breaks during migration:**
|
||||
|
||||
```bash
|
||||
# Option 1: Rollback entire layer
|
||||
git checkout HEAD~1 -- k8s/bootstrap/
|
||||
git commit -m "revert: bootstrap layer (investigating)"
|
||||
# ArgoCD will resync to previous version automatically
|
||||
|
||||
# Option 2: Pause ArgoCD sync
|
||||
argocd app set layer-1-bootstrap --sync-policy none
|
||||
# Investigate, then re-enable:
|
||||
argocd app set layer-1-bootstrap --sync-policy automated
|
||||
|
||||
# Option 3: Full rollback to pre-migration
|
||||
git reset --hard <commit-before-migration>
|
||||
argocd app set homelab-root --sync-policy none
|
||||
# Manual investigation, then re-enable
|
||||
```
|
||||
|
||||
## Timeline Summary
|
||||
|
||||
```
|
||||
Week 1: Plan & validate (reviews, dependency mapping)
|
||||
Week 2: Layers 0-1 (infrastructure, bootstrap)
|
||||
Week 3: Layers 2-3 (platform, security)
|
||||
Week 3: Layers 4-5 (applications, data)
|
||||
Week 4: ArgoCD applications + layer definitions
|
||||
Week 5: Deploy & validate (watch for issues)
|
||||
Week 6: Cleanup & CI/CD updates
|
||||
```
|
||||
|
||||
**Total: 6 weeks, non-disruptive (all layers coexist during migration)**
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✓ All 20+ services deployed via ArgoCD
|
||||
✓ No manual kubectl apply in production
|
||||
✓ State drift detected & corrected automatically
|
||||
✓ All changes in git (reviewed via PR)
|
||||
✓ Rollback possible at any time (git history)
|
||||
✓ CI/CD validates all commits
|
||||
✓ Secrets encrypted with SOPS
|
||||
✓ New services can be added (copy service directory)
|
||||
@@ -1,81 +0,0 @@
|
||||
# Cluster Recovery Plan
|
||||
|
||||
**Context:** Control-plane node (`talos-cp-1`, 192.168.1.213) had a corrupted machine CA in `terraform.tfvars`, causing a "broken key size" TLS error. During troubleshooting, an `apply-config` run with `install.wipe: true` (before the fix) wiped the OS/etcd disk (`sda`). Longhorn storage disks (`sdb`/`sdc`/`sdd`, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is `NotReady` (no CNI yet).
|
||||
|
||||
## Phase 1 — Get CNI up (blocks everything else)
|
||||
|
||||
Cluster config has `cni: name: none` (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint:
|
||||
|
||||
```bash
|
||||
helm repo add cilium https://helm.cilium.io/
|
||||
helm repo update
|
||||
helm install cilium cilium/cilium --namespace kube-system \
|
||||
--set ipam.mode=kubernetes \
|
||||
--set kubeProxyReplacement=true \
|
||||
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
|
||||
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
|
||||
--set cgroup.autoMount.enabled=false \
|
||||
--set cgroup.hostRoot=/sys/fs/cgroup \
|
||||
--set k8sServiceHost=localhost \
|
||||
--set k8sServicePort=7445
|
||||
```
|
||||
|
||||
Verify: node flips to `Ready`, `cilium status` reports OK.
|
||||
|
||||
## Phase 2 — Bootstrap ArgoCD (app-of-apps)
|
||||
|
||||
If ArgoCD's own manifests + root Application live in git (`k8s/argocd/`), recovery is a 3-step process:
|
||||
|
||||
```bash
|
||||
kubectl create namespace argocd
|
||||
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
|
||||
# wait for argocd-server to be Ready
|
||||
kubectl apply -f k8s/argocd/root-app.yaml # app-of-apps entrypoint
|
||||
```
|
||||
|
||||
Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs.
|
||||
|
||||
## Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall)
|
||||
|
||||
Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica."
|
||||
|
||||
1. Confirm disk path via `longhorn-disk.cfg` (already confirmed: `/var/lib/longhorn-disk{1,2,3}`)
|
||||
2. Locate `replicas/<volume-name>/` directories, read `volume.meta` for size
|
||||
3. Launch a temporary `longhornio/longhorn-engine` container against the replica directory to expose it as a block device
|
||||
4. Mount and copy data out manually, per volume
|
||||
|
||||
Not automatic — budget real time, one volume at a time.
|
||||
|
||||
## Phase 4 — WireGuard / router path
|
||||
|
||||
DuckDNS (`riotpiao.duckdns.org`) resolves correctly to current public IP (`24.114.42.68`, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from **off-LAN** — needs router-side verification:
|
||||
- Confirm port-forward rule is **UDP 51820 → 192.168.1.213:51820** exactly
|
||||
- If double-NAT, forward must be on the outermost internet-facing router
|
||||
- Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding)
|
||||
|
||||
**Separately — LAN-side symptom:** tunnel also fails to connect **from the same LAN** as the server, via the public DDNS hostname. Root cause: **NAT hairpin/loopback not supported by the router** — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug.
|
||||
|
||||
Fixes, in order of robustness:
|
||||
1. Enable NAT hairpinning/loopback in router settings, if supported
|
||||
2. Split-DNS: resolve the DDNS hostname to the LAN IP (`192.168.1.213`) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS)
|
||||
3. Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at `192.168.1.213:51820` while on LAN; switch back to the DDNS hostname when away
|
||||
|
||||
## Phase 5 — Redeploy DuckDNS updater
|
||||
|
||||
The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a `CronJob` running `curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip="` every few minutes.
|
||||
|
||||
---
|
||||
|
||||
**Recommended order:** Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency).
|
||||
|
||||
## Sources
|
||||
|
||||
- [How to Install Cilium on Talos Linux Step by Step](https://oneuptime.com/blog/post/2026-03-03-install-cilium-on-talos-linux-step-by-step/view)
|
||||
- [Deploy Cilium CNI - Sidero Documentation](https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium)
|
||||
- [Cluster Bootstrapping - Argo CD](https://argo-cd.readthedocs.io/en/latest/operator-manual/cluster-bootstrapping/)
|
||||
- [How to Bootstrap an Entire Cluster with ArgoCD App-of-Apps](https://oneuptime.com/blog/post/2026-02-26-argocd-bootstrap-cluster-app-of-apps/view)
|
||||
- [Restoring Data from an Orphaned Replica Directory - Longhorn KB](https://longhorn.io/kb/restoring-data-from-an-orphaned-replica-directory/)
|
||||
- [Port Forwarding for WireGuard](https://portforward.com/wireguard/)
|
||||
- [Guide Wireguard-portforwarding - SNBForums](https://www.snbforums.com/threads/guide-wireguard-portforwarding.89737/)
|
||||
- [Help with hairpin NAT for wireguard on ubuntu server - Ubiquiti Community](https://community.ui.com/questions/Help-with-hairpin-NAT-for-wireguard-on-ubuntu-server/12d4cbe7-f1f5-448d-9d77-ea8226c18f16)
|
||||
- [Wireguard VPN on Pi4 - cannot connect client - Raspberry Pi Forums](https://forums.raspberrypi.com/viewtopic.php?t=339692)
|
||||
Reference in New Issue
Block a user