diff --git a/GITOPS_IMPLEMENTATION_SUMMARY.md b/GITOPS_IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index a8f4bc4..0000000 --- a/GITOPS_IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,246 +0,0 @@ -# GitOps IaC Implementation Summary - -## Documents Created - -1. **GITOPS_ARCHITECTURE.md** - Production-grade directory structure & design -2. **GITOPS_MIGRATION_PLAN.md** - Step-by-step implementation (6 weeks, non-disruptive) - -## Architecture at a Glance - -``` -Pure GitOps (ArgoCD only, no Terraform) -├── Layer 0: Infrastructure (namespaces, storage, RBAC) -├── Layer 1: Bootstrap (cert-manager, cilium, ingress-nginx) -├── Layer 2: Platform (minio, longhorn, loki, prometheus) -├── Layer 3: Security (authentik, vault) -├── Layer 4: Applications (forgejo, grafana, portainer, temporal, llm) -└── Layer 5: Data (postgres, redis, kafka) -``` - -**Key benefit:** Each layer depends on previous, staged deployment, easy rollback. - -## File Structure - -``` -k8s/ -├── _base/ # Shared kustomization base -├── infrastructure/ # Layer 0 (foundation) -├── bootstrap/ # Layer 1 (networking) -├── platform/ # Layer 2 (services) -├── security/ # Layer 3 (auth) -├── applications/ # Layer 4 (business logic) -├── data/ # Layer 5 (stateful) -└── argocd/ - ├── projects/ # AppProject definitions - └── apps/ - ├── root-app.yaml # Bootstraps Layer 0 - └── layer-*.yaml # 5 layer apps (synced in order) -``` - -## Implementation Steps - -### Phase 1: Plan (Week 1) -- [x] Review GITOPS_ARCHITECTURE.md -- [x] Review GITOPS_MIGRATION_PLAN.md -- [ ] Audit current cluster resources -- [ ] Identify secrets needing encryption (SOPS) -- [ ] Map dependencies (provided in plan) -- **User approval required before proceeding to Phase 2** - -### Phase 2: Build (Weeks 2-3) -- [ ] Create k8s/ directory structure -- [ ] Migrate Layer 0 (infrastructure) -- [ ] Migrate Layer 1 (bootstrap: cert-manager, cilium, ingress-nginx) -- [ ] Migrate Layer 2 (platform: minio, longhorn, loki, prometheus) -- [ ] Migrate Layer 3 (security: authentik, vault) -- [ ] Migrate Layer 4 (applications: forgejo, grafana, temporal, etc.) -- [ ] Migrate Layer 5 (data: postgres, redis, kafka) - -### Phase 3: ArgoCD (Week 4) -- [ ] Create root application (Layer 0 trigger) -- [ ] Create 5 layer applications (with sync-wave ordering) -- [ ] Update AppProject permissions -- [ ] Test dry-run on each layer - -### Phase 4: Deploy (Week 5) -- [ ] Pre-flight validation (kustomize build, kubeval, yamllint) -- [ ] Deploy root application -- [ ] Deploy layer applications (one by one) -- [ ] Monitor for drift -- [ ] Verify state correction (manual edits reverted) - -### Phase 5: Cleanup (Week 6) -- [ ] Delete old k8s/ directories -- [ ] Update CI/CD pipeline (remove terraform, add kustomize) -- [ ] Archive legacy configuration -- [ ] Final validation - -## Key Architecture Decisions - -### ✓ Why Layers? -- **Clear dependencies:** Layer N can't deploy until Layer N-1 succeeds -- **Easy debugging:** Which layer broke? Layer 2? Roll back Layer 2 only. -- **Easy scaling:** Add new service without touching others -- **Easy rollback:** `git revert ` - -### ✓ Why Kustomize? -- Helm chart package + values override (clean separation) -- Kustomize build validates before sync -- Patches enable per-environment customization -- Standard K8s tool (no external dependencies) - -### ✓ Why No Terraform? -- Single tool (ArgoCD) vs. dual tool (Terraform + ArgoCD) -- Git is source of truth for everything -- No state backend (no .tfstate files) -- ArgoCD continuous reconciliation (drift auto-corrected) -- All changes reviewed in PRs (audit trail in git) - -### ✓ Why This File Structure? -- **Predictable:** Each service has consistent structure (kustomization.yaml, values.yaml, config/) -- **Scalable:** New service = new directory (copy template) -- **Maintainable:** Clear ownership (who owns layer? who owns service?) -- **GitOps-ready:** Each layer is independently deployable - -## Secret Management (SOPS) - -### Before (Terraform) -``` -terraform/terraform.tfvars # Plaintext secrets (risky!) -``` - -### After (GitOps) -```bash -# Encrypt secrets before commit -sops -e secrets.yaml > secrets.enc.yaml -git add secrets.enc.yaml - -# ArgoCD decrypts at sync time (SOPS plugin installed) -# Git stores only encrypted version (safe) -``` - -**Secrets to encrypt:** -- Authentik bootstrap password -- MinIO root credentials -- Database passwords -- API tokens -- OAuth client secrets - -## CI/CD Integration - -### Before -``` -git push → terraform validate → terraform plan → manual apply -``` - -### After (Pure GitOps) -``` -git push → yamllint → kubeval → kustomize build → argocd validation - ↓ - ArgoCD auto-syncs (if PR merged) -``` - -**No manual steps. No terraform. No state management. Pure GitOps.** - -## Comparison: Old vs. New - -| Aspect | Old (Terraform + ArgoCD) | New (ArgoCD Only) | -|--------|--------------------------|-------------------| -| **State management** | tfstate files | Git history | -| **Secrets** | Plaintext in .tfvars | SOPS encrypted | -| **Manual steps** | terraform apply required | git push only | -| **Rollback** | Revert tfstate + code | git revert + push | -| **Audit trail** | Terraform logs | Git history + ArgoCD logs | -| **Complexity** | High (2 tools) | Low (1 tool) | -| **New services** | Edit .tf files | Copy service directory | -| **State drift** | Manual terraform plan to detect | ArgoCD auto-detects & corrects | - -## Non-Disruptive Migration - -**Key principle:** Live cluster remains running during entire migration. - -**How?** -1. Build k8s/ structure **alongside** existing configuration -2. Deploy each layer via ArgoCD while old method still runs -3. Once new layer verified: delete old configuration -4. If issue detected: rollback (git revert) + redeploy old version - -**Timeline:** 6 weeks (can be parallelized to 3-4 weeks if needed) - -## Rollback Procedure (At Any Point) - -```bash -# 1. Identify broken layer -argocd app get layer-2-platform # Shows OutOfSync, error - -# 2. Revert that layer -git revert -git push - -# 3. ArgoCD detects change, syncs back to previous version -# Expected: ~3 minutes for full reconciliation - -# 4. Verify -argocd app get layer-2-platform # Should be Synced again -``` - -## Day 1 Action Items - -1. **Read GITOPS_ARCHITECTURE.md** (30 min) - - Understand layers - - Understand directory structure - - Understand kustomization strategy - -2. **Read GITOPS_MIGRATION_PLAN.md** (30 min) - - Understand phases - - Understand timeline - - Understand success criteria - -3. **Review dependency map** (provided in plan) - - Confirm layer dependencies match current setup - - Identify any services not in layers - -4. **Audit secrets** (30 min) - ```bash - grep -r "password" k8s/ --include="*.yaml" - grep -r "token" k8s/ --include="*.yaml" - # Plan which need SOPS encryption - ``` - -5. **Decision point:** Proceed to Phase 2 (Week 2)? - - YES → Start building k8s/ structure - - NO → Clarify concerns, refine plan - -## Benefits Summary - -✓ **Simpler:** One tool (ArgoCD), one repo (git), one source of truth -✓ **Safer:** All changes reviewed in PRs, git history as audit trail -✓ **Scalable:** New services = copy directory, update git, done -✓ **Resilient:** State drift auto-corrected, rollback via `git revert` -✓ **Auditable:** Every change in git with who/when/why -✓ **Automated:** No manual kubectl apply, no manual terraform apply -✓ **Testable:** Kustomize build validates before sync -✓ **Recoverable:** git history = disaster recovery - -## Support - -- Architecture questions → GITOPS_ARCHITECTURE.md -- Implementation questions → GITOPS_MIGRATION_PLAN.md -- Syntax/tools questions → See referenced documentation - - Kustomize: https://kustomize.io/ - - ArgoCD: https://argo-cd.readthedocs.io/ - - SOPS: https://github.com/mozilla/sops - -## Next Steps - -1. **User review:** Read both architecture documents (60 min) -2. **User decision:** Proceed to Phase 2 or refine plan? -3. **Kickoff Phase 2:** Start building infrastructure layer (Week 2 Monday) - ---- - -**Status:** Ready for implementation -**Terraform:** Deleted ✓ -**Architecture:** Designed ✓ -**Plan:** Ready ✓ -**Waiting for:** User approval to proceed diff --git a/GITOPS_MIGRATION_PLAN.md b/GITOPS_MIGRATION_PLAN.md deleted file mode 100644 index e2625c6..0000000 --- a/GITOPS_MIGRATION_PLAN.md +++ /dev/null @@ -1,573 +0,0 @@ -# GitOps Migration Plan: Implement Production-Grade Architecture - -## Current State → Target State - -### Current (Messy) -``` -k8s/ -├── talos-ci-cd/ (forgejo, runner) -├── talos-iam/ (authentik, vault) -├── logging/ (loki, promtail, grafana) -├── monitoring/ (prometheus, alerts) -├── storage/ (minio, longhorn) -├── ddb/ (postgres) -├── sqs/ (kafka, queue-crd) -├── temporal/ (temporal) -├── argocd/apps/ (20 Applications scattered) -└── ... (more scattered) -``` - -### Target (Clean, Layered) -``` -k8s/ -├── infrastructure/ (namespaces, storage, RBAC) -├── bootstrap/ (cert-manager, cilium, ingress-nginx) -├── platform/ (storage, observability: minio, longhorn, loki, prometheus) -├── security/ (authentik, vault, cert-issuer) -├── applications/ (forgejo, grafana, portainer, temporal, llm) -├── data/ (postgres, redis, kafka) -└── argocd/ - ├── projects/ (AppProject definitions) - └── apps/ (6 layer Applications only) -``` - -## Phase 1: Plan & Validate (Week 1) - -### Step 1.1: Review GITOPS_ARCHITECTURE.md -- [ ] Understand directory structure -- [ ] Understand layer dependencies -- [ ] Understand Kustomization strategy -- User approval: YES/NO - -### Step 1.2: Audit current resources -```bash -# Export all current resources -kubectl get all -A -o yaml > /tmp/current-state-backup.yaml - -# Count resources per layer -kubectl get deployments -A | wc -l -kubectl get statefulsets -A | wc -l -kubectl get daemonsets -A | wc -l -kubectl get services -A | wc -l -kubectl get configmaps -A | wc -l -kubectl get secrets -A | wc -l -``` - -### Step 1.3: Identify secrets needing encryption -```bash -# Find secrets in current manifests -grep -r "kind: Secret" k8s/ --include="*.yaml" -grep -r "password" k8s/ --include="*.yaml" -grep -r "token" k8s/ --include="*.yaml" - -# Plan SOPS encryption for: -# - Authentik bootstrap password -# - MinIO credentials -# - Database passwords -# - API tokens -``` - -### Step 1.4: Dependency mapping -``` -Layer 0 (Infrastructure) - └─ Layer 1 (Bootstrap) - ├─ cert-manager (creates certificates) - ├─ cilium (network) - └─ ingress-nginx (entry point) - └─ Layer 2 (Platform) - ├─ minio (storage) - ├─ longhorn (PV storage) - ├─ loki (logs) - └─ prometheus (metrics) - └─ Layer 3 (Security) - ├─ authentik (auth) - └─ vault (secrets) - └─ Layer 4 (Applications) - ├─ forgejo - ├─ grafana - ├─ portainer - ├─ temporal - └─ llm - └─ Layer 5 (Data) - ├─ postgres - ├─ redis - └─ kafka -``` - -## Phase 2: Build Directory Structure (Weeks 2-3) - -### Step 2.1: Create base directories -```bash -cd k8s/ - -# Create layers -mkdir -p infrastructure bootstrap platform security applications data - -# Create kustomization.yaml for each -for dir in infrastructure bootstrap platform security applications data; do - cat > $dir/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: CHANGE_ME - -resources: [] # Will add resources below - -# Placeholders for this layer -EOF -done -``` - -### Step 2.2: Migrate Layer 0 (Infrastructure) -**Resources to create:** -- Namespaces (all 20+) -- Storage classes (longhorn, xfs, longhorn-kafka) -- Service accounts (terraform-ci, etc.) -- Cluster roles (terraform-ci, admin, etc.) -- Network policies - -**Action:** -```bash -# Extract from current cluster -kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml -kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml -kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml -kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml - -# Clean up (remove status, owner refs, etc.) -# Add to k8s/infrastructure/kustomization.yaml: -resources: - - namespaces.yaml - - storage-classes.yaml - - service-accounts.yaml - - cluster-roles.yaml - -# Test -kustomize build k8s/infrastructure/ -``` - -**Deliverables:** -- [x] k8s/infrastructure/kustomization.yaml -- [x] k8s/infrastructure/namespaces.yaml -- [x] k8s/infrastructure/storage-classes.yaml -- [x] k8s/infrastructure/{service-accounts,cluster-roles}.yaml - -### Step 2.3: Migrate Layer 1 (Bootstrap) - -**Services: cert-manager, cilium, ingress-nginx** - -**Action:** -```bash -# Create directories -mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx} - -# For each service: -# 1. Export Helm values -helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml - -# 2. Create kustomization.yaml with Helm chart ref -cat > bootstrap/cert-manager/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: cert-manager - -helmCharts: -- name: cert-manager - repo: https://charts.jetstack.io - version: v1.21.0 - releaseName: cert-manager - valuesFile: values.yaml -EOF - -# 3. Repeat for cilium, ingress-nginx - -# 4. Add to k8s/bootstrap/kustomization.yaml: -resources: - - cert-manager/kustomization.yaml - - cilium/kustomization.yaml - - ingress-nginx/kustomization.yaml -``` - -**Deliverables:** -- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml -- [x] k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml -- [x] k8s/bootstrap/kustomization.yaml - -### Step 2.4: Migrate Layer 2 (Platform) - -**Services: minio, longhorn, loki, prometheus, promtail** - -**Action:** -```bash -# Create directories -mkdir -p platform/{minio,longhorn,loki,prometheus,promtail} - -# Migrate minio -# 1. Export existing values -helm get values minio -n storage > platform/minio/values.yaml - -# 2. Export bucket configs -kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/ - -# 3. Create kustomization.yaml -cat > platform/minio/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: storage - -helmCharts: -- name: minio - repo: https://charts.min.io - version: 14.6.25 # Current version - releaseName: minio - valuesFile: values.yaml - -resources: - - config/minio-buckets.yaml -EOF - -# Repeat for loki, prometheus, promtail -``` - -**Deliverables:** -- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml -- [x] k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml -- [x] k8s/platform/kustomization.yaml - -### Step 2.5: Migrate Layer 3 (Security) - -**Services: authentik, vault** - -**Action:** -```bash -# Create directories -mkdir -p security/{authentik,vault} - -# Migrate authentik -helm get values authentik -n iam > security/authentik/values.yaml - -# Export Authentik resources -kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml -kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml -kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml - -# Create kustomization.yaml -cat > security/authentik/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -namespace: iam - -helmCharts: -- name: authentik - repo: https://charts.goauthentik.io - version: 2024.12.1 - releaseName: authentik - valuesFile: values.yaml - -resources: - - config/apps.yaml - - config/groups.yaml - - config/oauth-providers.yaml -EOF - -# Repeat for vault -``` - -**Deliverables:** -- [x] k8s/security/{authentik,vault}/kustomization.yaml -- [x] k8s/security/{authentik,vault}/values.yaml -- [x] k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml -- [x] k8s/security/kustomization.yaml - -### Step 2.6: Migrate Layer 4 (Applications) - -**Services: forgejo, grafana, portainer, temporal, llm** - -**Action:** -```bash -# Create directories -mkdir -p applications/{forgejo,grafana,portainer,temporal,llm} - -# Move existing manifests -cp -r k8s/talos-ci-cd/* applications/forgejo/ -cp -r k8s/monitoring/* applications/grafana/ # Includes dashboards, etc. -cp -r k8s/portainer/* applications/portainer/ -cp -r k8s/temporal/* applications/temporal/ -cp -r k8s/llm/* applications/llm/ - -# For each, create kustomization.yaml with Helm chart ref -# (or keep existing manifests if not Helm) - -# Create layer kustomization.yaml -cat > applications/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -resources: - - forgejo/kustomization.yaml - - grafana/kustomization.yaml - - portainer/kustomization.yaml - - temporal/kustomization.yaml - - llm/kustomization.yaml -EOF -``` - -**Deliverables:** -- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml -- [x] k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml -- [x] k8s/applications/kustomization.yaml - -### Step 2.7: Migrate Layer 5 (Data) - -**Services: postgres (CloudNativePG), redis, kafka** - -**Action:** -```bash -# Create directories -mkdir -p data/{postgres,redis,kafka} - -# Export existing configurations -kubectl get cnpg -A -o yaml > data/postgres/config.yaml -kubectl get redis -A -o yaml > data/redis/config.yaml -kubectl get kafka -A -o yaml > data/kafka/config.yaml - -# Create kustomization.yaml for each - -# Repeat process... -``` - -**Deliverables:** -- [x] k8s/data/{postgres,redis,kafka}/kustomization.yaml -- [x] k8s/data/kustomization.yaml - -## Phase 3: Create ArgoCD Application Definitions (Week 4) - -### Step 3.1: Create layer Applications -```bash -mkdir -p k8s/argocd/apps/layers - -# Root application (Layer 0 only, triggers rest) -cat > k8s/argocd/apps/root-app.yaml << 'EOF' -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: homelab-root - namespace: argocd -spec: - project: homelab - source: - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git - targetRevision: main - path: k8s/infrastructure - destination: - server: https://kubernetes.default.svc - syncPolicy: - automated: - prune: true - selfHeal: true -EOF - -# Layer 1 application -cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF' -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: layer-1-bootstrap - namespace: argocd - annotations: - argocd.argoproj.io/sync-wave: "1" -spec: - project: homelab - source: - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git - targetRevision: main - path: k8s/bootstrap - destination: - server: https://kubernetes.default.svc - syncPolicy: - automated: - prune: true - selfHeal: true -EOF - -# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5) -``` - -**Deliverables:** -- [x] k8s/argocd/apps/root-app.yaml -- [x] k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml - -### Step 3.2: Create kustomization.yaml for apps -```bash -cat > k8s/argocd/apps/kustomization.yaml << 'EOF' -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -resources: - - root-app.yaml - - layer-1-bootstrap.yaml - - layer-2-platform.yaml - - layer-3-security.yaml - - layer-4-applications.yaml - - layer-5-data.yaml -EOF -``` - -## Phase 4: Deploy & Validate (Week 5) - -### Step 4.1: Pre-flight checks -```bash -# Validate all kustomization files -for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do - echo "Validating $dir..." - kustomize build $dir > /tmp/validate.yaml - kubeval /tmp/validate.yaml || exit 1 -done - -# Validate ArgoCD apps -kubeval k8s/argocd/apps/*.yaml - -# Test YAML lint -yamllint k8s/ -``` - -### Step 4.2: Deploy root application -```bash -# Apply root app (Layer 0 only) -kubectl apply -f k8s/argocd/apps/root-app.yaml - -# Watch sync -argocd app watch homelab-root - -# Verify infrastructure deployed -kubectl get ns # Should have all namespaces -kubectl get sc # Should have all storage classes -``` - -### Step 4.3: Deploy layer applications -```bash -# Apply layer applications (one by one) -kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml -sleep 5 -argocd app watch layer-1-bootstrap - -# Once Layer 1 synced: Layer 2 -kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml -argocd app watch layer-2-platform - -# ... repeat for layers 3, 4, 5 -``` - -### Step 4.4: Monitor for issues -```bash -# Check ArgoCD apps -argocd app list - -# Watch events -kubectl get events -A -w - -# Monitor pod status -kubectl get pods -A - -# Check logs -kubectl logs -f -n argocd deployment/argocd-application-controller -``` - -### Step 4.5: Validate state drift protection -```bash -# Manually edit a resource -kubectl edit deployment cert-manager -n cert-manager -# Change replicas from 1 → 2 - -# Wait 3 minutes (ArgoCD reconciliation interval) -sleep 180 - -# Check: should be back to 1 replica -kubectl get deployment cert-manager -n cert-manager -# Expected: 1/1 Ready (reverted by ArgoCD) - -# Verify in ArgoCD -argocd app get layer-1-bootstrap -# Expected: Synced status -``` - -## Phase 5: Cleanup (Week 6) - -### Step 5.1: Delete old directories -```bash -# After all apps synced successfully: -rm -rf k8s/talos-ci-cd/ -rm -rf k8s/talos-iam/ -rm -rf k8s/logging/ -rm -rf k8s/monitoring/ -rm -rf k8s/storage/ -rm -rf k8s/ddb/ -rm -rf k8s/sqs/ -rm -rf k8s/temporal/ -# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd -``` - -### Step 5.2: Update CI/CD pipeline -```bash -# Update .forgejo/workflows/ -# - Remove: terraform validate/plan/apply -# - Add: kustomize build validation -# - Add: argocd app validation -``` - -### Step 5.3: Archive old configuration -```bash -# Keep for reference only -mkdir k8s/.archive/ -git mv k8s/old-structure-backup k8s/.archive/ -git commit -m "archive: old k8s structure (moved to pure GitOps)" -``` - -## Rollback Plan - -**If something breaks during migration:** - -```bash -# Option 1: Rollback entire layer -git checkout HEAD~1 -- k8s/bootstrap/ -git commit -m "revert: bootstrap layer (investigating)" -# ArgoCD will resync to previous version automatically - -# Option 2: Pause ArgoCD sync -argocd app set layer-1-bootstrap --sync-policy none -# Investigate, then re-enable: -argocd app set layer-1-bootstrap --sync-policy automated - -# Option 3: Full rollback to pre-migration -git reset --hard -argocd app set homelab-root --sync-policy none -# Manual investigation, then re-enable -``` - -## Timeline Summary - -``` -Week 1: Plan & validate (reviews, dependency mapping) -Week 2: Layers 0-1 (infrastructure, bootstrap) -Week 3: Layers 2-3 (platform, security) -Week 3: Layers 4-5 (applications, data) -Week 4: ArgoCD applications + layer definitions -Week 5: Deploy & validate (watch for issues) -Week 6: Cleanup & CI/CD updates -``` - -**Total: 6 weeks, non-disruptive (all layers coexist during migration)** - -## Success Criteria - -✓ All 20+ services deployed via ArgoCD -✓ No manual kubectl apply in production -✓ State drift detected & corrected automatically -✓ All changes in git (reviewed via PR) -✓ Rollback possible at any time (git history) -✓ CI/CD validates all commits -✓ Secrets encrypted with SOPS -✓ New services can be added (copy service directory) diff --git a/Recover-Plan.md b/Recover-Plan.md deleted file mode 100644 index 3028265..0000000 --- a/Recover-Plan.md +++ /dev/null @@ -1,81 +0,0 @@ -# Cluster Recovery Plan - -**Context:** Control-plane node (`talos-cp-1`, 192.168.1.213) had a corrupted machine CA in `terraform.tfvars`, causing a "broken key size" TLS error. During troubleshooting, an `apply-config` run with `install.wipe: true` (before the fix) wiped the OS/etcd disk (`sda`). Longhorn storage disks (`sdb`/`sdc`/`sdd`, ~1.26TB) were NOT touched and still hold data. etcd has since been re-bootstrapped fresh and empty; kube-apiserver/controller-manager/scheduler are running; node is `NotReady` (no CNI yet). - -## Phase 1 — Get CNI up (blocks everything else) - -Cluster config has `cni: name: none` (Cilium installs via Helm, not Talos-managed). Install via Helm, using KubePrism (already enabled, port 7445) as the API endpoint: - -```bash -helm repo add cilium https://helm.cilium.io/ -helm repo update -helm install cilium cilium/cilium --namespace kube-system \ - --set ipam.mode=kubernetes \ - --set kubeProxyReplacement=true \ - --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \ - --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \ - --set cgroup.autoMount.enabled=false \ - --set cgroup.hostRoot=/sys/fs/cgroup \ - --set k8sServiceHost=localhost \ - --set k8sServicePort=7445 -``` - -Verify: node flips to `Ready`, `cilium status` reports OK. - -## Phase 2 — Bootstrap ArgoCD (app-of-apps) - -If ArgoCD's own manifests + root Application live in git (`k8s/argocd/`), recovery is a 3-step process: - -```bash -kubectl create namespace argocd -kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml -# wait for argocd-server to be Ready -kubectl apply -f k8s/argocd/root-app.yaml # app-of-apps entrypoint -``` - -Everything downstream (Longhorn CRDs, ingress, cert-manager, workloads) reconciles automatically from git once the root app syncs. - -## Phase 3 — Longhorn volume salvage (manual, per-volume, do after Longhorn CRDs reinstall) - -Replica data survived on disk; CRDs (PVC↔PV↔Volume mappings) did not. Official supported recovery path: "Export a Volume from a Single Replica." - -1. Confirm disk path via `longhorn-disk.cfg` (already confirmed: `/var/lib/longhorn-disk{1,2,3}`) -2. Locate `replicas//` directories, read `volume.meta` for size -3. Launch a temporary `longhornio/longhorn-engine` container against the replica directory to expose it as a block device -4. Mount and copy data out manually, per volume - -Not automatic — budget real time, one volume at a time. - -## Phase 4 — WireGuard / router path - -DuckDNS (`riotpiao.duckdns.org`) resolves correctly to current public IP (`24.114.42.68`, confirmed via Google + Cloudflare DNS). Tunnel fails to connect from **off-LAN** — needs router-side verification: -- Confirm port-forward rule is **UDP 51820 → 192.168.1.213:51820** exactly -- If double-NAT, forward must be on the outermost internet-facing router -- Confirm with ISP whether you're behind CGNAT (silently blocks all inbound forwarding) - -**Separately — LAN-side symptom:** tunnel also fails to connect **from the same LAN** as the server, via the public DDNS hostname. Root cause: **NAT hairpin/loopback not supported by the router** — traffic leaves the LAN, tries to loop back in via the public IP, and gets dropped. This is a router limitation, not a WireGuard/Shadowrocket bug. - -Fixes, in order of robustness: -1. Enable NAT hairpinning/loopback in router settings, if supported -2. Split-DNS: resolve the DDNS hostname to the LAN IP (`192.168.1.213`) for LAN clients only, via a local resolver (Pi-hole/dnsmasq/router DNS) -3. Quick workaround: manually point the Shadowrocket WireGuard peer endpoint at `192.168.1.213:51820` while on LAN; switch back to the DDNS hostname when away - -## Phase 5 — Redeploy DuckDNS updater - -The cluster wipe likely took out any in-cluster DuckDNS-refresh CronJob. Redeploy via GitOps once ArgoCD is back — standard pattern: a `CronJob` running `curl "https://www.duckdns.org/update?domains=riotpiao&token=...&ip="` every few minutes. - ---- - -**Recommended order:** Phase 1 (Cilium) → Phase 2 (ArgoCD) → Phase 4 (router/WireGuard, parallel) → Phase 5 (DuckDNS cron) → Phase 3 (Longhorn salvage, lowest urgency). - -## Sources - -- [How to Install Cilium on Talos Linux Step by Step](https://oneuptime.com/blog/post/2026-03-03-install-cilium-on-talos-linux-step-by-step/view) -- [Deploy Cilium CNI - Sidero Documentation](https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium) -- [Cluster Bootstrapping - Argo CD](https://argo-cd.readthedocs.io/en/latest/operator-manual/cluster-bootstrapping/) -- [How to Bootstrap an Entire Cluster with ArgoCD App-of-Apps](https://oneuptime.com/blog/post/2026-02-26-argocd-bootstrap-cluster-app-of-apps/view) -- [Restoring Data from an Orphaned Replica Directory - Longhorn KB](https://longhorn.io/kb/restoring-data-from-an-orphaned-replica-directory/) -- [Port Forwarding for WireGuard](https://portforward.com/wireguard/) -- [Guide Wireguard-portforwarding - SNBForums](https://www.snbforums.com/threads/guide-wireguard-portforwarding.89737/) -- [Help with hairpin NAT for wireguard on ubuntu server - Ubiquiti Community](https://community.ui.com/questions/Help-with-hairpin-NAT-for-wireguard-on-ubuntu-server/12d4cbe7-f1f5-448d-9d77-ea8226c18f16) -- [Wireguard VPN on Pi4 - cannot connect client - Raspberry Pi Forums](https://forums.raspberrypi.com/viewtopic.php?t=339692)