homelab/GITOPS_MIGRATION_PLAN.md

GitOps Migration Plan: Implement Production-Grade Architecture

Current State → Target State

Current (Messy)

k8s/
├── talos-ci-cd/        (forgejo, runner)
├── talos-iam/          (authentik, vault)
├── logging/            (loki, promtail, grafana)
├── monitoring/         (prometheus, alerts)
├── storage/            (minio, longhorn)
├── ddb/                (postgres)
├── sqs/                (kafka, queue-crd)
├── temporal/           (temporal)
├── argocd/apps/        (20 Applications scattered)
└── ... (more scattered)

Target (Clean, Layered)

k8s/
├── infrastructure/     (namespaces, storage, RBAC)
├── bootstrap/          (cert-manager, cilium, ingress-nginx)
├── platform/           (storage, observability: minio, longhorn, loki, prometheus)
├── security/           (authentik, vault, cert-issuer)
├── applications/       (forgejo, grafana, portainer, temporal, llm)
├── data/               (postgres, redis, kafka)
└── argocd/
    ├── projects/       (AppProject definitions)
    └── apps/           (6 layer Applications only)

Phase 1: Plan & Validate (Week 1)

Step 1.1: Review GITOPS_ARCHITECTURE.md

• Understand directory structure
• Understand layer dependencies
• Understand Kustomization strategy
• User approval: YES/NO

Step 1.2: Audit current resources

# Export all current resources
kubectl get all -A -o yaml > /tmp/current-state-backup.yaml

# Count resources per layer
kubectl get deployments -A | wc -l
kubectl get statefulsets -A | wc -l
kubectl get daemonsets -A | wc -l
kubectl get services -A | wc -l
kubectl get configmaps -A | wc -l
kubectl get secrets -A | wc -l

Step 1.3: Identify secrets needing encryption

# Find secrets in current manifests
grep -r "kind: Secret" k8s/ --include="*.yaml"
grep -r "password" k8s/ --include="*.yaml"
grep -r "token" k8s/ --include="*.yaml"

# Plan SOPS encryption for:
# - Authentik bootstrap password
# - MinIO credentials
# - Database passwords
# - API tokens

Step 1.4: Dependency mapping

Layer 0 (Infrastructure)
  └─ Layer 1 (Bootstrap)
      ├─ cert-manager (creates certificates)
      ├─ cilium (network)
      └─ ingress-nginx (entry point)
          └─ Layer 2 (Platform)
              ├─ minio (storage)
              ├─ longhorn (PV storage)
              ├─ loki (logs)
              └─ prometheus (metrics)
                  └─ Layer 3 (Security)
                      ├─ authentik (auth)
                      └─ vault (secrets)
                          └─ Layer 4 (Applications)
                              ├─ forgejo
                              ├─ grafana
                              ├─ portainer
                              ├─ temporal
                              └─ llm
                          └─ Layer 5 (Data)
                              ├─ postgres
                              ├─ redis
                              └─ kafka

Phase 2: Build Directory Structure (Weeks 2-3)

Step 2.1: Create base directories

cd k8s/

# Create layers
mkdir -p infrastructure bootstrap platform security applications data

# Create kustomization.yaml for each
for dir in infrastructure bootstrap platform security applications data; do
  cat > $dir/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: CHANGE_ME

resources: []  # Will add resources below

# Placeholders for this layer
EOF
done

Step 2.2: Migrate Layer 0 (Infrastructure)

Resources to create:

• Namespaces (all 20+)
• Storage classes (longhorn, xfs, longhorn-kafka)
• Service accounts (terraform-ci, etc.)
• Cluster roles (terraform-ci, admin, etc.)
• Network policies

Action:

# Extract from current cluster
kubectl get namespace -o yaml > k8s/infrastructure/namespaces.yaml
kubectl get storageclass -o yaml > k8s/infrastructure/storage-classes.yaml
kubectl get serviceaccount -A -o yaml > k8s/infrastructure/service-accounts.yaml
kubectl get clusterrole -o yaml > k8s/infrastructure/cluster-roles.yaml

# Clean up (remove status, owner refs, etc.)
# Add to k8s/infrastructure/kustomization.yaml:
resources:
  - namespaces.yaml
  - storage-classes.yaml
  - service-accounts.yaml
  - cluster-roles.yaml

# Test
kustomize build k8s/infrastructure/

Deliverables:

• k8s/infrastructure/kustomization.yaml
• k8s/infrastructure/namespaces.yaml
• k8s/infrastructure/storage-classes.yaml
• k8s/infrastructure/{service-accounts,cluster-roles}.yaml

Step 2.3: Migrate Layer 1 (Bootstrap)

Services: cert-manager, cilium, ingress-nginx

Action:

# Create directories
mkdir -p bootstrap/{cert-manager,cilium,ingress-nginx}

# For each service:
# 1. Export Helm values
helm get values cert-manager -n cert-manager > bootstrap/cert-manager/values.yaml

# 2. Create kustomization.yaml with Helm chart ref
cat > bootstrap/cert-manager/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: cert-manager

helmCharts:
- name: cert-manager
  repo: https://charts.jetstack.io
  version: v1.21.0
  releaseName: cert-manager
  valuesFile: values.yaml
EOF

# 3. Repeat for cilium, ingress-nginx

# 4. Add to k8s/bootstrap/kustomization.yaml:
resources:
  - cert-manager/kustomization.yaml
  - cilium/kustomization.yaml
  - ingress-nginx/kustomization.yaml

Deliverables:

• k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/kustomization.yaml
• k8s/bootstrap/{cert-manager,cilium,ingress-nginx}/values.yaml
• k8s/bootstrap/kustomization.yaml

Step 2.4: Migrate Layer 2 (Platform)

Services: minio, longhorn, loki, prometheus, promtail

Action:

# Create directories
mkdir -p platform/{minio,longhorn,loki,prometheus,promtail}

# Migrate minio
# 1. Export existing values
helm get values minio -n storage > platform/minio/values.yaml

# 2. Export bucket configs
kubectl get jobs,configmaps,secrets -n storage -o yaml > platform/minio/config/

# 3. Create kustomization.yaml
cat > platform/minio/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: storage

helmCharts:
- name: minio
  repo: https://charts.min.io
  version: 14.6.25  # Current version
  releaseName: minio
  valuesFile: values.yaml

resources:
  - config/minio-buckets.yaml
EOF

# Repeat for loki, prometheus, promtail

Deliverables:

• k8s/platform/{minio,longhorn,loki,prometheus,promtail}/kustomization.yaml
• k8s/platform/{minio,longhorn,loki,prometheus,promtail}/values.yaml
• k8s/platform/kustomization.yaml

Step 2.5: Migrate Layer 3 (Security)

Services: authentik, vault

Action:

# Create directories
mkdir -p security/{authentik,vault}

# Migrate authentik
helm get values authentik -n iam > security/authentik/values.yaml

# Export Authentik resources
kubectl get -n iam authentik_application -o yaml > security/authentik/config/apps.yaml
kubectl get -n iam authentik_group -o yaml > security/authentik/config/groups.yaml
kubectl get -n iam authentik_provider_oauth2 -o yaml > security/authentik/config/oauth-providers.yaml

# Create kustomization.yaml
cat > security/authentik/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: iam

helmCharts:
- name: authentik
  repo: https://charts.goauthentik.io
  version: 2024.12.1
  releaseName: authentik
  valuesFile: values.yaml

resources:
  - config/apps.yaml
  - config/groups.yaml
  - config/oauth-providers.yaml
EOF

# Repeat for vault

Deliverables:

• k8s/security/{authentik,vault}/kustomization.yaml
• k8s/security/{authentik,vault}/values.yaml
• k8s/security/authentik/config/{apps,groups,oauth-providers}.yaml
• k8s/security/kustomization.yaml

Step 2.6: Migrate Layer 4 (Applications)

Services: forgejo, grafana, portainer, temporal, llm

Action:

# Create directories
mkdir -p applications/{forgejo,grafana,portainer,temporal,llm}

# Move existing manifests
cp -r k8s/talos-ci-cd/* applications/forgejo/
cp -r k8s/monitoring/* applications/grafana/  # Includes dashboards, etc.
cp -r k8s/portainer/* applications/portainer/
cp -r k8s/temporal/* applications/temporal/
cp -r k8s/llm/* applications/llm/

# For each, create kustomization.yaml with Helm chart ref
# (or keep existing manifests if not Helm)

# Create layer kustomization.yaml
cat > applications/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - forgejo/kustomization.yaml
  - grafana/kustomization.yaml
  - portainer/kustomization.yaml
  - temporal/kustomization.yaml
  - llm/kustomization.yaml
EOF

Deliverables:

• k8s/applications/{forgejo,grafana,portainer,temporal,llm}/kustomization.yaml
• k8s/applications/{forgejo,grafana,portainer,temporal,llm}/values.yaml
• k8s/applications/kustomization.yaml

Step 2.7: Migrate Layer 5 (Data)

Services: postgres (CloudNativePG), redis, kafka

Action:

# Create directories
mkdir -p data/{postgres,redis,kafka}

# Export existing configurations
kubectl get cnpg -A -o yaml > data/postgres/config.yaml
kubectl get redis -A -o yaml > data/redis/config.yaml
kubectl get kafka -A -o yaml > data/kafka/config.yaml

# Create kustomization.yaml for each

# Repeat process...

Deliverables:

• k8s/data/{postgres,redis,kafka}/kustomization.yaml
• k8s/data/kustomization.yaml

Phase 3: Create ArgoCD Application Definitions (Week 4)

Step 3.1: Create layer Applications

mkdir -p k8s/argocd/apps/layers

# Root application (Layer 0 only, triggers rest)
cat > k8s/argocd/apps/root-app.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: homelab-root
  namespace: argocd
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/infrastructure
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

# Layer 1 application
cat > k8s/argocd/apps/layer-1-bootstrap.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: layer-1-bootstrap
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git
    targetRevision: main
    path: k8s/bootstrap
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

# Repeat for layers 2, 3, 4, 5 (sync-wave: 2, 3, 4, 5)

Deliverables:

• k8s/argocd/apps/root-app.yaml
• k8s/argocd/apps/layer-{1,2,3,4,5}-*.yaml

Step 3.2: Create kustomization.yaml for apps

cat > k8s/argocd/apps/kustomization.yaml << 'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - root-app.yaml
  - layer-1-bootstrap.yaml
  - layer-2-platform.yaml
  - layer-3-security.yaml
  - layer-4-applications.yaml
  - layer-5-data.yaml
EOF

Phase 4: Deploy & Validate (Week 5)

Step 4.1: Pre-flight checks

# Validate all kustomization files
for dir in k8s/{infrastructure,bootstrap,platform,security,applications,data}; do
  echo "Validating $dir..."
  kustomize build $dir > /tmp/validate.yaml
  kubeval /tmp/validate.yaml || exit 1
done

# Validate ArgoCD apps
kubeval k8s/argocd/apps/*.yaml

# Test YAML lint
yamllint k8s/

Step 4.2: Deploy root application

# Apply root app (Layer 0 only)
kubectl apply -f k8s/argocd/apps/root-app.yaml

# Watch sync
argocd app watch homelab-root

# Verify infrastructure deployed
kubectl get ns  # Should have all namespaces
kubectl get sc  # Should have all storage classes

Step 4.3: Deploy layer applications

# Apply layer applications (one by one)
kubectl apply -f k8s/argocd/apps/layer-1-bootstrap.yaml
sleep 5
argocd app watch layer-1-bootstrap

# Once Layer 1 synced: Layer 2
kubectl apply -f k8s/argocd/apps/layer-2-platform.yaml
argocd app watch layer-2-platform

# ... repeat for layers 3, 4, 5

Step 4.4: Monitor for issues

# Check ArgoCD apps
argocd app list

# Watch events
kubectl get events -A -w

# Monitor pod status
kubectl get pods -A

# Check logs
kubectl logs -f -n argocd deployment/argocd-application-controller

Step 4.5: Validate state drift protection

# Manually edit a resource
kubectl edit deployment cert-manager -n cert-manager
# Change replicas from 1 → 2

# Wait 3 minutes (ArgoCD reconciliation interval)
sleep 180

# Check: should be back to 1 replica
kubectl get deployment cert-manager -n cert-manager
# Expected: 1/1 Ready (reverted by ArgoCD)

# Verify in ArgoCD
argocd app get layer-1-bootstrap
# Expected: Synced status

Phase 5: Cleanup (Week 6)

Step 5.1: Delete old directories

# After all apps synced successfully:
rm -rf k8s/talos-ci-cd/
rm -rf k8s/talos-iam/
rm -rf k8s/logging/
rm -rf k8s/monitoring/
rm -rf k8s/storage/
rm -rf k8s/ddb/
rm -rf k8s/sqs/
rm -rf k8s/temporal/
# Keep only: infrastructure, bootstrap, platform, security, applications, data, argocd

Step 5.2: Update CI/CD pipeline

# Update .forgejo/workflows/
# - Remove: terraform validate/plan/apply
# - Add: kustomize build validation
# - Add: argocd app validation

Step 5.3: Archive old configuration

# Keep for reference only
mkdir k8s/.archive/
git mv k8s/old-structure-backup k8s/.archive/
git commit -m "archive: old k8s structure (moved to pure GitOps)"

Rollback Plan

If something breaks during migration:

# Option 1: Rollback entire layer
git checkout HEAD~1 -- k8s/bootstrap/
git commit -m "revert: bootstrap layer (investigating)"
# ArgoCD will resync to previous version automatically

# Option 2: Pause ArgoCD sync
argocd app set layer-1-bootstrap --sync-policy none
# Investigate, then re-enable:
argocd app set layer-1-bootstrap --sync-policy automated

# Option 3: Full rollback to pre-migration
git reset --hard <commit-before-migration>
argocd app set homelab-root --sync-policy none
# Manual investigation, then re-enable

Timeline Summary

Week 1: Plan & validate (reviews, dependency mapping)
Week 2: Layers 0-1 (infrastructure, bootstrap)
Week 3: Layers 2-3 (platform, security)
Week 3: Layers 4-5 (applications, data)
Week 4: ArgoCD applications + layer definitions
Week 5: Deploy & validate (watch for issues)
Week 6: Cleanup & CI/CD updates

Total: 6 weeks, non-disruptive (all layers coexist during migration)

Success Criteria

✓ All 20+ services deployed via ArgoCD ✓ No manual kubectl apply in production ✓ State drift detected & corrected automatically ✓ All changes in git (reviewed via PR) ✓ Rollback possible at any time (git history) ✓ CI/CD validates all commits ✓ Secrets encrypted with SOPS ✓ New services can be added (copy service directory)