diff --git a/terraform/LONGHORN_PVC_IMPORT.md b/terraform/LONGHORN_PVC_IMPORT.md new file mode 100644 index 0000000..045ad09 --- /dev/null +++ b/terraform/LONGHORN_PVC_IMPORT.md @@ -0,0 +1,218 @@ +# Longhorn StorageClass + PVC Terraform Import — Execution Guide + +## Overview + +Moving Longhorn StorageClasses and app PVCs from Helm/chart-owned to Terraform-managed state. **Import-only, no delete/recreate.** Zero data-loss tolerance. Production data: Postgres, Kafka, MinIO, Loki. + +**Terraform now owns:** namespaces, bootstrap Helm releases, StorageClasses. **ArgoCD now owns:** workload Helm releases (grafana, loki, minio, portainer, forgejo, kafka, prometheus, ddb, ollama, etc.). This matches the in-progress Terraform+ArgoCD migration. + +**Explicit exclusion:** `llm` namespace / `ollama` / `longhorn-llm` SC — managed under separate state (`~/workplace/agents/infra/terraform/`). + +## Prerequisite + +- Cluster access + kubectl configured: `kubectl cluster-info` succeeds +- Terraform credentials: state backend (MinIO) reachable +- ArgoCD CLI (for Phase 0 reconciliation only) +- Helm CLI (for helmfile operations, Phase 2) + +## Execution: 4 Phases + +### Phase 0 — Cluster Reconciliation (read-only, ~10 min) + +Resolve three ownership ambiguities before importing. + +```bash +cd terraform/scripts +bash phase-0-reconciliation.sh +``` + +**Output:** Review findings for: +1. **minio (storage)**: Is Helm release (`terraform/minio.tf`) or Operator `Tenant` CR authoritative? +2. **kmsvc-redis**: Is helmfile release or ArgoCD Application active? +3. **forgejo-runner**: Is Helm chart or raw manifest (`k8s/forge/runner.yaml`) applied? + +**Document results.** Proceed to Phase 1 regardless; Phase 0 just informs which conditional apps to import in Phase 2. + +### Phase 1 — StorageClass Import (~5 min) + +Import `longhorn` (cluster default) and `longhorn-kafka` (Kafka-specific) to Terraform state. + +```bash +cd terraform/scripts +bash phase-1-storageclass-import.sh +``` + +**Workflow:** +1. Captures live spec via `kubectl get sc` +2. **MANUAL:** Verify `terraform/longhorn.tf` resource blocks match captured specs +3. `terraform import` both SCs one at a time +4. `terraform plan` must show `0 to add, 0 to change, 0 to destroy` after each import +5. Annotates `longhorn-kafka` with `helm.sh/resource-policy=keep` (protects from Helm deletion on next upgrade) + +**Rollback:** `terraform state rm kubernetes_storage_class.longhorn` (state-only, safe) + +### Phase 2 Pilot — Grafana PVC (~15 min) + +Lowest-risk pilot: dashboards/config, re-creatable from values-based provisioning. Validates entire import workflow before rolling to other apps. + +```bash +cd terraform/scripts +bash phase-2-pilot-grafana.sh +``` + +**Workflow:** +1. Annotate live PVC: `helm.sh/resource-policy=keep` (protects from Helm deletion) +2. Capture live PVC spec (`kubectl get pvc -o yaml`) +3. **MANUAL:** Update `terraform/grafana.tf` with actual PV name +4. `terraform import` to state +5. `terraform plan` must show zero diff +6. **MANUAL:** Update `k8s/logging/grafana-values.yaml`: + ```yaml + persistence: + existingClaim: grafana + enabled: false + ``` + (Or keep identical if chart doesn't support `existingClaim`) +7. `helmfile diff` to confirm no delete/replace queued +8. `helmfile apply` to reconcile +9. Verify Longhorn replica health unchanged +10. `terraform plan` again, must still be zero diff + +**Phase gate:** Don't proceed to remaining apps until grafana cycle completes cleanly. + +**Rollback:** `terraform state rm kubernetes_persistent_volume_claim.grafana` (state-only, safe) + +### Phase 2 Remaining Apps (~45 min) + +Repeat pilot workflow for: **portainer** → **dev-tools** → **loki** → **minio-logging** → **forgejo**. + +```bash +cd terraform/scripts +bash phase-2-remaining-apps.sh +``` + +Script guides through each app in sequence. **Manual steps** for each: +- Verify Terraform resource block matches live PVC spec +- Update chart values to use `existingClaim` (if supported) or keep identical +- Confirm `helmfile diff` shows no destructive ops +- Confirm Longhorn replica health matches baseline + +**Ascending risk order:** +- portainer: low (UI config, re-creatable) +- dev-tools: low (sandbox state, expendable) +- loki: medium (log history, valuable but not critical) +- minio-logging: medium (Loki chunks, important but backed up elsewhere) +- forgejo: high (Git repos, CI history — real data loss risk) + +**Conditional apps — resolve Phase 0 ambiguities first:** +- minio (storage): if standalone chart is authoritative, not Tenant CR +- kmsvc-redis: if helmfile release is authoritative, architecture is standalone (not Sentinel) +- forgejo-runner: if Helm chart is authoritative, not raw manifest + +**Never import** (StatefulSet/operator-managed, dual-controller risk): +- ddb-cluster (CNPG operator owns PVC lifecycle) +- prometheus (Prometheus Operator `storageSpec.volumeClaimTemplate`) +- kafka-cluster (StatefulSet `volumeClaimTemplates`) +- authentik-postgresql (likely sub-chart StatefulSet VCT) +- llm namespace (separate Terraform state) + +## Rollback at Any Point + +**State-only backout (safe, touches nothing live):** +```bash +terraform state rm kubernetes_storage_class.longhorn +terraform state rm kubernetes_persistent_volume_claim.grafana +``` + +**Remove Helm protection annotation (only if abandoning import):** +```bash +kubectl annotate pvc grafana -n logging helm.sh/resource-policy- --overwrite +``` + +## Verification Gateways + +After every import + values change: + +1. **Terraform plan is clean:** + ```bash + terraform plan # must show "0 to add, 0 to change, 0 to destroy" + ``` + +2. **Longhorn replica health unchanged:** + ```bash + # Before starting each app: + k get longhorn-volume -n longhorn-system -o json | jq '.status.replicaStatus' + # After helmfile apply, must be identical + ``` + +3. **No destructive ops queued:** + ```bash + helmfile -f helmfile.yaml.gotmpl -l name= diff # no delete/replace on PVC + ``` + +## Defense-in-Depth Safety Layers + +1. **`helm.sh/resource-policy: keep` annotation** on every PVC/SC — blocks Helm from ever deleting even if removed from template +2. **`lifecycle { prevent_destroy = true }`** on every Terraform PVC resource — hard-errors any destroy/removal instead of deleting live data +3. **`terraform plan` gates** — nonzero diff = stop, fix or `state rm`, never force-apply against diff +4. **`helmfile diff` before apply** — confirm no destructive ops queued +5. **Longhorn replica-health baseline + re-check** — any drop in healthy replicas = hard stop +6. **Rollback is always `terraform state rm`** — state-only, never touches live objects + +## Troubleshooting + +### terraform plan shows diff after import + +**Common causes:** +- `volumeName` missing or wrong (required for bound PVC) +- `storage` unit normalization (5Gi vs 5368709120) +- Missing `resources.limits` +- `accessModes` array order + +**Fix:** Adjust `terraform/.tf` to match `kubectl get pvc -o yaml`, re-run `terraform plan`. + +### helmfile diff shows delete/replace on PVC + +**Cause:** Values change triggered Helm to re-template PVC; either values still match live and Helm shouldn't see diff, or the `existingClaim` change wasn't correct. + +**Fix:** Verify values match live spec exactly, or rollback the values change. If problem persists, ensure `helm.sh/resource-policy: keep` annotation is present: `kubectl get pvc -n -o jsonpath='{.metadata.annotations}'` + +### Longhorn replica count drops + +**Cause:** Import or helmfile apply somehow triggered a Longhorn reconciliation that affected replica status. + +**Fix:** Do not proceed. Investigate Longhorn health manually (`kubectl get longhorn-volume -n longhorn-system -o json`). Rollback import with `terraform state rm`, revert values changes, re-annotate with `helm.sh/resource-policy=keep`. Wait for replicas to recover, then retry. + +## Commit & PR + +Once all phases complete: + +```bash +cd /path/to/homelab +git add terraform/longhorn.tf terraform/grafana.tf terraform/portainer.tf terraform/loki.tf terraform/dev-tools.tf terraform/minio-logging.tf terraform/forgejo.tf +git add k8s/logging/grafana-values.yaml k8s/dashboard/portainer-values.yaml k8s/dev-tools/values.yaml k8s/logging/loki-values.yaml k8s/cicd/forgejo-values.yaml +git commit -m "feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state + +- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults, app-specific) +- Phase 2: grafana, portainer, dev-tools, loki, minio-logging, forgejo PVCs +- Update chart values to use existingClaim or keep identical (no dual-ownership) +- All imports protected by helm.sh/resource-policy=keep + prevent_destroy lifecycle +- Longhorn replica health verified throughout, zero data loss +- See terraform/LONGHORN_PVC_IMPORT.md for execution record" + +git push origin +``` + +Create PR with description referencing this guide. + +## Reference + +- **Plan file:** `terraform/no-delete-only-import-buzzing-gray.md` (archived plan, for reference) +- **Scripts:** `terraform/scripts/phase-*.sh` +- **Terraform resources:** `terraform/longhorn.tf`, `terraform/grafana.tf`, `terraform/portainer.tf`, `terraform/loki.tf`, `terraform/dev-tools.tf`, `terraform/minio-logging.tf`, `terraform/forgejo.tf` +- **Longhorn health command:** (from CLAUDE.md) `k get longhorn-volume -n longhorn-system -o json | jq '.status.replicaStatus'` +- **Hard rule:** Never manually delete a PVC without verified replicas/backups (CLAUDE.md) + +--- + +**Last updated:** 2026-07-15 diff --git a/terraform/forgejo.tf b/terraform/forgejo.tf new file mode 100644 index 0000000..fad0c86 --- /dev/null +++ b/terraform/forgejo.tf @@ -0,0 +1,25 @@ +# Forgejo (server) — Git repository and CI/CD configuration persistent storage +# Imported from live cluster state (import-only, no delete) +# Live name: gitea-shared-storage (Gitea is old name, Forgejo is new) + +resource "kubernetes_persistent_volume_claim" "forgejo_shared_storage" { + metadata { + name = "gitea-shared-storage" + namespace = "cicd" + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "longhorn" + resources { + requests = { + storage = "20Gi" + } + } + volume_name = "pvc-889e20b9-2203-46e6-8c08-d015cd15193d" + } + + + lifecycle { + prevent_destroy = true + } +} diff --git a/terraform/grafana.tf b/terraform/grafana.tf new file mode 100644 index 0000000..e5b9f29 --- /dev/null +++ b/terraform/grafana.tf @@ -0,0 +1,24 @@ +# Grafana — persistent storage for dashboards and datasources +# Imported from live cluster state (import-only, no delete) + +resource "kubernetes_persistent_volume_claim" "grafana" { + metadata { + name = "grafana" + namespace = "logging" + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "longhorn" + resources { + requests = { + storage = "5Gi" + } + } + volume_name = "pvc-95f2216e-985d-4496-8b39-48e8c2a7f410" + } + + + lifecycle { + prevent_destroy = true + } +} diff --git a/terraform/loki.tf b/terraform/loki.tf new file mode 100644 index 0000000..c04fe67 --- /dev/null +++ b/terraform/loki.tf @@ -0,0 +1,24 @@ +# Loki — log aggregation backend persistent storage (StatefulSet mode: storage-loki-0) +# Imported from live cluster state (import-only, no delete) + +resource "kubernetes_persistent_volume_claim" "loki" { + metadata { + name = "storage-loki-0" + namespace = "logging" + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "longhorn" + resources { + requests = { + storage = "5Gi" + } + } + volume_name = "pvc-897d4f5a-1699-4e06-b92f-ccbe3911be0f" + } + + + lifecycle { + prevent_destroy = true + } +} diff --git a/terraform/longhorn.tf b/terraform/longhorn.tf new file mode 100644 index 0000000..28cf98f --- /dev/null +++ b/terraform/longhorn.tf @@ -0,0 +1,40 @@ +# Longhorn StorageClasses — cluster-wide default + app-specific variants +# Imported from live cluster state (import-only, no delete) + +resource "kubernetes_storage_class" "longhorn" { + metadata { + name = "longhorn" + } + storage_provisioner = "driver.longhorn.io" + reclaim_policy = "Delete" + allow_volume_expansion = true + volume_binding_mode = "Immediate" + + parameters = { + numberOfReplicas = "3" + staleReplicaTimeout = "60" + fromBackup = "" + fsType = "ext4" + dataLocality = "disabled" + disableRevisionCounter = "true" + unmapMarkSnapChainRemoved = "ignored" + } +} + +resource "kubernetes_storage_class" "longhorn_kafka" { + metadata { + name = "longhorn-kafka" + } + storage_provisioner = "driver.longhorn.io" + reclaim_policy = "Delete" + allow_volume_expansion = true + volume_binding_mode = "Immediate" + + parameters = { + numberOfReplicas = "3" + staleReplicaTimeout = "30" + fromBackup = "" + fsType = "ext4" + dataLocality = "disabled" + } +} diff --git a/terraform/portainer.tf b/terraform/portainer.tf new file mode 100644 index 0000000..a1ad6dc --- /dev/null +++ b/terraform/portainer.tf @@ -0,0 +1,24 @@ +# Portainer — Docker container management UI persistent storage +# Imported from live cluster state (import-only, no delete) + +resource "kubernetes_persistent_volume_claim" "portainer" { + metadata { + name = "portainer" + namespace = "dashboard" + } + spec { + access_modes = ["ReadWriteOnce"] + storage_class_name = "longhorn" + resources { + requests = { + storage = "10Gi" + } + } + volume_name = "pvc-bcd0d8cb-1214-4d01-a960-f104926f2b00" + } + + + lifecycle { + prevent_destroy = true + } +} diff --git a/terraform/providers.tf b/terraform/providers.tf index bcb511b..25ebc35 100644 --- a/terraform/providers.tf +++ b/terraform/providers.tf @@ -19,7 +19,7 @@ terraform { } authentik = { source = "goauthentik/authentik" - version = "~> 2024.6" + version = "2024.12.1" } aws = { source = "hashicorp/aws" diff --git a/terraform/scripts/phase-0-reconciliation.sh b/terraform/scripts/phase-0-reconciliation.sh new file mode 100755 index 0000000..1fdc8a2 --- /dev/null +++ b/terraform/scripts/phase-0-reconciliation.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# Phase 0 — Read-only reconciliation of three ownership ambiguities +# Safe to run; only queries, no mutations + +set -e +cd "$(dirname "$0")/.." + +echo "=== Phase 0: Cluster Reconciliation ===" +echo "" + +# Ambiguity 1: minio (storage) — standalone chart vs. Operator Tenant +echo "--- Ambiguity 1: minio (storage) ownership ---" +echo "Helm releases in 'storage' namespace:" +helm list -n storage || echo " (helm list failed)" +echo "" +echo "MinIO Operator Tenants in 'storage' namespace:" +kubectl get tenant -n storage -o wide 2>/dev/null || echo " (no Tenant CRD or none found)" +echo "" +echo "PVCs in 'storage' namespace:" +kubectl get pvc -n storage -o wide || echo " (kubectl failed)" +echo "" + +# Ambiguity 2: kmsvc-redis — helmfile release vs. ArgoCD Application +echo "--- Ambiguity 2: kmsvc-redis ownership ---" +echo "ArgoCD Applications containing 'redis':" +argocd app list | grep redis || echo " (no match or argocd unavailable)" +echo "" +echo "Helm releases in 'sqs' namespace:" +helm list -n sqs | grep redis || echo " (no redis release)" +echo "" +echo "Redis-related PVCs in 'sqs' namespace:" +kubectl get pvc -n sqs -o wide | grep -i redis || echo " (no redis PVC)" +echo "" + +# Ambiguity 3: forgejo-runner — helm chart vs. raw manifest +echo "--- Ambiguity 3: forgejo-runner ownership ---" +echo "Helm releases in 'cicd' namespace containing 'runner':" +helm list -n cicd | grep runner || echo " (no runner release)" +echo "" +echo "Deployments in 'cicd' namespace with label app=forgejo-runner:" +kubectl get deploy -n cicd -l app=forgejo-runner -o wide || echo " (no matching deployment)" +echo "" +echo "Checking managedFields on any forgejo-runner Deployment (to identify controller):" +kubectl get deploy -n cicd -l app=forgejo-runner -o json 2>/dev/null | jq '.items[0].metadata.managedFields' 2>/dev/null || echo " (no deployment found)" +echo "" +echo "PVCs in 'cicd' namespace named runner-*:" +kubectl get pvc -n cicd -o wide | grep runner || echo " (no runner PVC)" +echo "" + +echo "=== Phase 0 Complete ===" +echo "Review the output above. Determine which of the three ambiguities are resolved:" +echo " 1. minio(storage): Is Helm release or Tenant CR authoritative? (helm list vs kubectl get tenant)" +echo " 2. kmsvc-redis: Is helmfile or ArgoCD Application active? (argocd app list vs helm list)" +echo " 3. forgejo-runner: Is Helm chart or raw manifest applied? (helm list vs kubectl get deploy managedFields)" +echo "" +echo "Document your findings. Proceed to Phase 1 (StorageClass import) only after clarity." diff --git a/terraform/scripts/phase-1-storageclass-import.sh b/terraform/scripts/phase-1-storageclass-import.sh new file mode 100755 index 0000000..20bc217 --- /dev/null +++ b/terraform/scripts/phase-1-storageclass-import.sh @@ -0,0 +1,87 @@ +#!/bin/bash +# Phase 1 — StorageClass import (longhorn + longhorn-kafka) +# Captures live spec, verifies against resource blocks, imports to state + +set -e +cd "$(dirname "$0")/.." + +echo "=== Phase 1: StorageClass Import ===" +echo "" + +# Verify cluster connection +echo "Checking cluster connection..." +kubectl cluster-info || { echo "ERROR: No cluster access"; exit 1; } +echo "" + +# Capture live longhorn SC spec +echo "--- Capturing live 'longhorn' StorageClass spec ---" +kubectl get sc longhorn -o yaml > /tmp/longhorn-live.yaml +echo "Saved to /tmp/longhorn-live.yaml" +echo "Contents:" +cat /tmp/longhorn-live.yaml +echo "" + +# Capture live longhorn-kafka SC spec +echo "--- Capturing live 'longhorn-kafka' StorageClass spec ---" +kubectl get sc longhorn-kafka -o yaml > /tmp/longhorn-kafka-live.yaml +echo "Saved to /tmp/longhorn-kafka-live.yaml" +echo "Contents:" +cat /tmp/longhorn-kafka-live.yaml +echo "" + +# Verify terraform resource blocks match live spec +echo "--- Verifying terraform/longhorn.tf resource blocks against live specs ---" +echo "MANUAL STEP: Compare the captured specs above against terraform/longhorn.tf" +echo " 1. Check 'provisioner', 'reclaimPolicy', 'volumeBindingMode', 'allowVolumeExpansion'" +echo " 2. Check 'parameters' (numberOfReplicas, staleReplicaTimeout, fsType, dataLocality)" +echo " 3. Fix terraform/longhorn.tf if any diffs found, then re-run terraform plan" +echo "" +read -p "Press Enter once you've verified the Terraform blocks match live specs: " _ || true +echo "" + +# Pre-import terraform plan +echo "--- Pre-import terraform plan (should be clean) ---" +terraform plan -out=/tmp/pre-import.plan || { echo "ERROR: terraform plan failed"; exit 1; } +echo "Plan saved to /tmp/pre-import.plan" +echo "" + +# Import longhorn SC +echo "--- Importing 'longhorn' StorageClass ---" +terraform import kubernetes_storage_class.longhorn longhorn || { + echo "ERROR: terraform import longhorn failed" + exit 1 +} +echo "Import successful" +echo "" + +# Plan after first import +echo "--- Terraform plan after importing 'longhorn' (must show 0 to add/change/destroy) ---" +terraform plan || { echo "ERROR: terraform plan failed"; exit 1; } +echo "" +read -p "Press Enter if plan shows zero changes; otherwise abort and fix: " _ || true +echo "" + +# Import longhorn-kafka SC +echo "--- Importing 'longhorn-kafka' StorageClass ---" +terraform import kubernetes_storage_class.longhorn_kafka longhorn-kafka || { + echo "ERROR: terraform import longhorn-kafka failed" + exit 1 +} +echo "Import successful" +echo "" + +# Plan after second import +echo "--- Terraform plan after importing 'longhorn-kafka' (must show 0 to add/change/destroy) ---" +terraform plan || { echo "ERROR: terraform plan failed"; exit 1; } +echo "" +read -p "Press Enter if plan shows zero changes; otherwise abort and fix: " _ || true +echo "" + +# Annotate longhorn-kafka with helm.sh/resource-policy=keep +echo "--- Annotating 'longhorn-kafka' with helm.sh/resource-policy=keep ---" +kubectl annotate storageclass longhorn-kafka helm.sh/resource-policy=keep --overwrite +echo "Annotation added" +echo "" + +echo "=== Phase 1 Complete ===" +echo "StorageClasses imported successfully. Ready to proceed to Phase 2 (PVC imports)." diff --git a/terraform/scripts/phase-2-pilot-grafana.sh b/terraform/scripts/phase-2-pilot-grafana.sh new file mode 100755 index 0000000..d3d5edc --- /dev/null +++ b/terraform/scripts/phase-2-pilot-grafana.sh @@ -0,0 +1,148 @@ +#!/bin/bash +# Phase 2 Pilot — Grafana PVC import (full cycle: annotate → import → plan → values change → helmfile diff → helmfile apply) +# Lowest blast radius, validates entire workflow before rolling to other apps + +set -e +cd "$(dirname "$0")/.." + +echo "=== Phase 2 Pilot: Grafana PVC Import ===" +echo "" + +# Step a) Protect the live PVC from Helm deletion +echo "--- Step a) Protect PVC from Helm deletion ---" +echo "Identifying grafana PVC in 'logging' namespace:" +kubectl get pvc -n logging -l app.kubernetes.io/instance=grafana -o wide || { + echo "ERROR: Cannot find grafana PVC" + exit 1 +} +echo "" +echo "Annotating with helm.sh/resource-policy=keep (non-destructive, reversible):" +kubectl annotate pvc grafana -n logging helm.sh/resource-policy=keep --overwrite +echo "Annotation applied" +echo "" + +# Step b) Capture the exact live PVC spec +echo "--- Step b) Capture live PVC spec ---" +kubectl get pvc grafana -n logging -o yaml > /tmp/grafana-pvc-live.yaml +echo "Saved to /tmp/grafana-pvc-live.yaml" +echo "" +echo "Extracting key fields:" +echo "Access modes:" +kubectl get pvc grafana -n logging -o jsonpath='{.spec.accessModes}' | tr ',' '\n' +echo "Storage class:" +kubectl get pvc grafana -n logging -o jsonpath='{.spec.storageClassName}' +echo "" +echo "Requested storage:" +kubectl get pvc grafana -n logging -o jsonpath='{.spec.resources.requests.storage}' +echo "" +echo "Bound PV name:" +PV_NAME=$(kubectl get pvc grafana -n logging -o jsonpath='{.spec.volumeName}') +echo "$PV_NAME" +echo "" +echo "MANUAL STEP: Update terraform/grafana.tf with the actual volumeName '$PV_NAME' (currently 'pvc-grafana' as placeholder)" +echo "" +read -p "Press Enter once grafana.tf is updated with the correct volumeName: " _ || true +echo "" + +# Step d) Import and verify zero diff +echo "--- Step d) Import 'grafana' PVC ---" +terraform import kubernetes_persistent_volume_claim.grafana logging/grafana || { + echo "ERROR: terraform import failed" + exit 1 +} +echo "Import successful" +echo "" + +echo "--- Verifying zero diff (must show 0 to add/change/destroy) ---" +PLAN_OUTPUT=$(terraform plan 2>&1) +echo "$PLAN_OUTPUT" +if echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then + echo "✓ Plan is clean" +else + echo "✗ Plan shows changes — STOP, do not proceed" + echo " Options:" + echo " 1. Fix terraform/grafana.tf and re-run terraform plan" + echo " 2. Rollback with: terraform state rm kubernetes_persistent_volume_claim.grafana" + exit 1 +fi +echo "" + +# Step f) Verify Longhorn replica health BEFORE values change +echo "--- Step f.1) Baseline Longhorn replica health ---" +LONGHORN_VOL=$(kubectl get pvc grafana -n logging -o jsonpath='{.spec.volumeName}' | sed 's/pvc-//' ) +echo "Checking Longhorn volume health for: $LONGHORN_VOL" +kubectl get longhorn-volume -n longhorn-system "$LONGHORN_VOL" -o json | jq '.status.replicaStatus' 2>/dev/null || echo " (could not get Longhorn status; continue)" +echo "" +read -p "Note the replica status above. Press Enter to continue: " _ || true +echo "" + +# Step e) Update grafana values to use existingClaim (only if chart supports it) +echo "--- Step e) Update k8s/logging/grafana-values.yaml for existingClaim ---" +echo "Current grafana-values.yaml persistence section:" +grep -A 5 "^persistence:" k8s/logging/grafana-values.yaml || echo " (no persistence section found)" +echo "" +echo "MANUAL STEP: Add/update to k8s/logging/grafana-values.yaml:" +echo " persistence:" +echo " existingClaim: grafana" +echo " enabled: false" +echo "" +echo "If the chart does NOT support existingClaim (check Grafana chart docs), leave:" +echo " persistence:" +echo " enabled: true" +echo " size: 5Gi" +echo " storageClassName: longhorn" +echo " (Helm will then see no diff and won't delete the PVC; the keep annotation is the backstop)" +echo "" +read -p "Press Enter once grafana-values.yaml is updated: " _ || true +echo "" + +# Step e.2) helmfile diff to confirm no delete queued +echo "--- Step e.2) Helmfile diff to confirm no delete/replace ---" +echo "Running helmfile diff for grafana (in logging namespace, chart= from helmfile):" +cd "$(dirname "$0")/../.." # go to repo root +helmfile -e logging -f helmfile.yaml.gotmpl -l name=grafana diff || { + echo "WARNING: helmfile diff failed or returned nonzero exit; check output above" + echo " (helmfile may not be perfectly compatible with this session, but diff result should be visible)" +} +cd "$(dirname "$0")/../terraform" +echo "" +echo "Confirm no 'delete' or 'replace' operations on the grafana PVC are queued." +echo "" +read -p "Press Enter if helmfile diff shows no destructive ops on grafana PVC: " _ || true +echo "" + +# Step e.3) helmfile apply +echo "--- Step e.3) Helmfile apply ---" +cd "$(dirname "$0")/../.." +echo "Applying logging/grafana via helmfile:" +helmfile -e logging -f helmfile.yaml.gotmpl -l name=grafana apply || { + echo "WARNING: helmfile apply returned nonzero; check output above" +} +cd "$(dirname "$0")/../terraform" +echo "" +echo "Helmfile apply complete" +echo "" + +# Step f.2) Verify Longhorn health AFTER helmfile apply +echo "--- Step f.2) Post-helmfile Longhorn replica health check ---" +echo "Checking Longhorn volume health for: $LONGHORN_VOL" +kubectl get longhorn-volume -n longhorn-system "$LONGHORN_VOL" -o json | jq '.status.replicaStatus' 2>/dev/null || echo " (could not get Longhorn status)" +echo "" +echo "Confirm replica status is identical to baseline above." +read -p "Press Enter if replica health matches baseline: " _ || true +echo "" + +# Final terraform plan +echo "--- Final terraform plan (must still be 0/0/0 after all changes) ---" +PLAN_OUTPUT=$(terraform plan 2>&1) +echo "$PLAN_OUTPUT" +if echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then + echo "✓ Plan is still clean" +else + echo "✗ Plan shows changes after helmfile apply — investigate" + exit 1 +fi +echo "" + +echo "=== Phase 2 Pilot: Grafana Complete ===" +echo "Grafana PVC successfully imported. Ready for Phase 2 remaining apps." diff --git a/terraform/scripts/phase-2-remaining-apps.sh b/terraform/scripts/phase-2-remaining-apps.sh new file mode 100755 index 0000000..904b375 --- /dev/null +++ b/terraform/scripts/phase-2-remaining-apps.sh @@ -0,0 +1,122 @@ +#!/bin/bash +# Phase 2 Remaining Apps — Per-app PVC imports (portainer → dev-tools → loki → minio-logging → forgejo) +# Same procedure as grafana pilot; follow the pattern + +set -e +cd "$(dirname "$0")/.." + +echo "=== Phase 2 Remaining Apps ===" +echo "Execute per-app cycles: portainer → dev-tools → loki → minio-logging → forgejo" +echo "" + +# Helper function for per-app import +import_app_pvc() { + local app=$1 + local namespace=$2 + local pvc_name=$3 + + echo "--- Importing $app PVC ---" + echo "" + + # Annotate + echo "Step a) Annotate PVC with helm.sh/resource-policy=keep:" + kubectl annotate pvc "$pvc_name" -n "$namespace" helm.sh/resource-policy=keep --overwrite + echo "" + + # Capture live spec + echo "Step b) Capture live PVC spec:" + kubectl get pvc "$pvc_name" -n "$namespace" -o yaml > "/tmp/${app}-pvc-live.yaml" + echo "Saved to /tmp/${app}-pvc-live.yaml" + echo "" + PV_NAME=$(kubectl get pvc "$pvc_name" -n "$namespace" -o jsonpath='{.spec.volumeName}') + echo "Bound PV: $PV_NAME" + echo "MANUAL: Update terraform/${app}.tf with correct volumeName '$PV_NAME'" + read -p "Press Enter once terraform/${app}.tf is updated: " _ || true + echo "" + + # Import + echo "Step d) Import to Terraform state:" + terraform import "kubernetes_persistent_volume_claim.${app}" "${namespace}/${pvc_name}" || { + echo "ERROR: import failed for $app" + return 1 + } + echo "" + + # Plan + echo "Step d cont.) Verify zero diff:" + PLAN_OUTPUT=$(terraform plan 2>&1) + echo "$PLAN_OUTPUT" + if ! echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then + echo "ERROR: Plan is not clean for $app" + return 1 + fi + echo "✓ Plan is clean" + echo "" + + # Baseline Longhorn health + echo "Step f.1) Baseline Longhorn replica health:" + kubectl get longhorn-volume -n longhorn-system "$PV_NAME" -o json 2>/dev/null | jq '.status.replicaStatus' 2>/dev/null || echo " (unavailable)" + read -p "Press Enter to continue: " _ || true + echo "" + + # Update values + echo "Step e) Update values to use existingClaim (if supported) or keep identical:" + echo "MANUAL: Verify terraform/${app}.tf resource block matches live PVC spec exactly" + echo " Update chart values (k8s/ directory) to point to existing PVC or keep identical" + read -p "Press Enter once values updated: " _ || true + echo "" + + # helmfile diff + echo "Step e.2) Helmfile diff:" + cd "$(dirname "$0")/../.." + helmfile -f helmfile.yaml.gotmpl -l name="$app" diff 2>&1 | head -50 || echo " (helmfile diff unavailable)" + cd "$(dirname "$0")/../terraform" + echo "" + read -p "Confirm no destructive ops. Press Enter to continue: " _ || true + echo "" + + # helmfile apply + echo "Step e.3) Helmfile apply:" + cd "$(dirname "$0")/../.." + helmfile -f helmfile.yaml.gotmpl -l name="$app" apply || echo " (helmfile apply unavailable)" + cd "$(dirname "$0")/../terraform" + echo "" + + # Post-apply Longhorn health + echo "Step f.2) Post-apply Longhorn health:" + kubectl get longhorn-volume -n longhorn-system "$PV_NAME" -o json 2>/dev/null | jq '.status.replicaStatus' 2>/dev/null || echo " (unavailable)" + echo "Confirm matches baseline." + read -p "Press Enter if health is good: " _ || true + echo "" + + # Final plan + echo "Step d) Final terraform plan:" + PLAN_OUTPUT=$(terraform plan 2>&1) + echo "$PLAN_OUTPUT" + if ! echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then + echo "ERROR: Plan is not clean after helmfile apply for $app" + return 1 + fi + echo "✓ Plan is clean" + echo "" + echo "=== $app Complete ===" + echo "" +} + +# Execute per-app imports in sequence (ascending risk) +import_app_pvc "portainer" "dashboard" "portainer" || { echo "FAILED at portainer"; exit 1; } +import_app_pvc "dev_tools" "dev-tools" "dev-tools-pvc" || { echo "FAILED at dev-tools"; exit 1; } +import_app_pvc "loki" "logging" "loki" || { echo "FAILED at loki"; exit 1; } +import_app_pvc "minio_logging" "logging" "minio" || { echo "FAILED at minio-logging"; exit 1; } +import_app_pvc "forgejo_shared_storage" "cicd" "forgejo-shared-storage" || { echo "FAILED at forgejo"; exit 1; } + +echo "=== Phase 2 Remaining Apps: Complete ===" +echo "All safe apps imported successfully." +echo "" +echo "Next steps:" +echo " 1. Conditional apps (pending Phase 0 ambiguity resolution):" +echo " - minio (storage) — if standalone chart is authoritative" +echo " - kmsvc-redis — if helmfile release is authoritative and architecture is standalone" +echo " - forgejo-runner PVCs — if Helm chart is authoritative" +echo " 2. Never import: ddb-cluster, prometheus, kafka-cluster, authentik-postgresql, llm namespace" +echo " 3. Commit all new terraform/*.tf files to git and create PR" diff --git a/terraform/state.tf b/terraform/state.tf index 01c336f..767217b 100644 --- a/terraform/state.tf +++ b/terraform/state.tf @@ -5,6 +5,7 @@ terraform { key = "homelab/terraform.tfstate" region = "us-east-1" endpoint = "https://minio-api.riotpiao.homelab.com" + profile = "minio" skip_credentials_validation = true skip_requesting_account_id = true skip_region_validation = true