feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state

- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults) - Phase 2 pilot: grafana, loki, portainer, forgejo PVCs - All imports protected by lifecycle.prevent_destroy - Removes Helm annotations (meta.helm.sh/*) to prevent dual-ownership conflicts - Remote state backend (MinIO S3) syncs automatically on plan/apply - Import-only approach: zero data loss, existing volumes untouched - See terraform/LONGHORN_PVC_IMPORT.md for execution record Co-Authored-By: Claude Haiku 4.5 <[email protected]>
2026-08-18 15:08:01 -07:00
parent 3a7471fc9b
commit f2b6ad1c60
12 changed files with 770 additions and 1 deletions
@@ -0,0 +1,218 @@
+# Longhorn StorageClass + PVC Terraform Import — Execution Guide
+
+## Overview
+
+Moving Longhorn StorageClasses and app PVCs from Helm/chart-owned to Terraform-managed state. **Import-only, no delete/recreate.** Zero data-loss tolerance. Production data: Postgres, Kafka, MinIO, Loki.
+
+**Terraform now owns:** namespaces, bootstrap Helm releases, StorageClasses. **ArgoCD now owns:** workload Helm releases (grafana, loki, minio, portainer, forgejo, kafka, prometheus, ddb, ollama, etc.). This matches the in-progress Terraform+ArgoCD migration.
+
+**Explicit exclusion:** `llm` namespace / `ollama` / `longhorn-llm` SC — managed under separate state (`~/workplace/agents/infra/terraform/`).
+
+## Prerequisite
+
+- Cluster access + kubectl configured: `kubectl cluster-info` succeeds
+- Terraform credentials: state backend (MinIO) reachable
+- ArgoCD CLI (for Phase 0 reconciliation only)
+- Helm CLI (for helmfile operations, Phase 2)
+
+## Execution: 4 Phases
+
+### Phase 0 — Cluster Reconciliation (read-only, ~10 min)
+
+Resolve three ownership ambiguities before importing.
+
+```bash
+cd terraform/scripts
+bash phase-0-reconciliation.sh
+```
+
+**Output:** Review findings for:
+1. **minio (storage)**: Is Helm release (`terraform/minio.tf`) or Operator `Tenant` CR authoritative?
+2. **kmsvc-redis**: Is helmfile release or ArgoCD Application active?
+3. **forgejo-runner**: Is Helm chart or raw manifest (`k8s/forge/runner.yaml`) applied?
+
+**Document results.** Proceed to Phase 1 regardless; Phase 0 just informs which conditional apps to import in Phase 2.
+
+### Phase 1 — StorageClass Import (~5 min)
+
+Import `longhorn` (cluster default) and `longhorn-kafka` (Kafka-specific) to Terraform state.
+
+```bash
+cd terraform/scripts
+bash phase-1-storageclass-import.sh
+```
+
+**Workflow:**
+1. Captures live spec via `kubectl get sc`
+2. **MANUAL:** Verify `terraform/longhorn.tf` resource blocks match captured specs
+3. `terraform import` both SCs one at a time
+4. `terraform plan` must show `0 to add, 0 to change, 0 to destroy` after each import
+5. Annotates `longhorn-kafka` with `helm.sh/resource-policy=keep` (protects from Helm deletion on next upgrade)
+
+**Rollback:** `terraform state rm kubernetes_storage_class.longhorn` (state-only, safe)
+
+### Phase 2 Pilot — Grafana PVC (~15 min)
+
+Lowest-risk pilot: dashboards/config, re-creatable from values-based provisioning. Validates entire import workflow before rolling to other apps.
+
+```bash
+cd terraform/scripts
+bash phase-2-pilot-grafana.sh
+```
+
+**Workflow:**
+1. Annotate live PVC: `helm.sh/resource-policy=keep` (protects from Helm deletion)
+2. Capture live PVC spec (`kubectl get pvc -o yaml`)
+3. **MANUAL:** Update `terraform/grafana.tf` with actual PV name
+4. `terraform import` to state
+5. `terraform plan` must show zero diff
+6. **MANUAL:** Update `k8s/logging/grafana-values.yaml`:
+   ```yaml
+   persistence:
+     existingClaim: grafana
+     enabled: false
+   ```
+   (Or keep identical if chart doesn't support `existingClaim`)
+7. `helmfile diff` to confirm no delete/replace queued
+8. `helmfile apply` to reconcile
+9. Verify Longhorn replica health unchanged
+10. `terraform plan` again, must still be zero diff
+
+**Phase gate:** Don't proceed to remaining apps until grafana cycle completes cleanly.
+
+**Rollback:** `terraform state rm kubernetes_persistent_volume_claim.grafana` (state-only, safe)
+
+### Phase 2 Remaining Apps (~45 min)
+
+Repeat pilot workflow for: **portainer** → **dev-tools** → **loki** → **minio-logging** → **forgejo**.
+
+```bash
+cd terraform/scripts
+bash phase-2-remaining-apps.sh
+```
+
+Script guides through each app in sequence. **Manual steps** for each:
+- Verify Terraform resource block matches live PVC spec
+- Update chart values to use `existingClaim` (if supported) or keep identical
+- Confirm `helmfile diff` shows no destructive ops
+- Confirm Longhorn replica health matches baseline
+
+**Ascending risk order:**
+- portainer: low (UI config, re-creatable)
+- dev-tools: low (sandbox state, expendable)
+- loki: medium (log history, valuable but not critical)
+- minio-logging: medium (Loki chunks, important but backed up elsewhere)
+- forgejo: high (Git repos, CI history — real data loss risk)
+
+**Conditional apps — resolve Phase 0 ambiguities first:**
+- minio (storage): if standalone chart is authoritative, not Tenant CR
+- kmsvc-redis: if helmfile release is authoritative, architecture is standalone (not Sentinel)
+- forgejo-runner: if Helm chart is authoritative, not raw manifest
+
+**Never import** (StatefulSet/operator-managed, dual-controller risk):
+- ddb-cluster (CNPG operator owns PVC lifecycle)
+- prometheus (Prometheus Operator `storageSpec.volumeClaimTemplate`)
+- kafka-cluster (StatefulSet `volumeClaimTemplates`)
+- authentik-postgresql (likely sub-chart StatefulSet VCT)
+- llm namespace (separate Terraform state)
+
+## Rollback at Any Point
+
+**State-only backout (safe, touches nothing live):**
+```bash
+terraform state rm kubernetes_storage_class.longhorn
+terraform state rm kubernetes_persistent_volume_claim.grafana
+```
+
+**Remove Helm protection annotation (only if abandoning import):**
+```bash
+kubectl annotate pvc grafana -n logging helm.sh/resource-policy- --overwrite
+```
+
+## Verification Gateways
+
+After every import + values change:
+
+1. **Terraform plan is clean:**
+   ```bash
+   terraform plan   # must show "0 to add, 0 to change, 0 to destroy"
+   ```
+
+2. **Longhorn replica health unchanged:**
+   ```bash
+   # Before starting each app:
+   k get longhorn-volume -n longhorn-system <pv-name> -o json | jq '.status.replicaStatus'
+   # After helmfile apply, must be identical
+   ```
+
+3. **No destructive ops queued:**
+   ```bash
+   helmfile -f helmfile.yaml.gotmpl -l name=<app> diff   # no delete/replace on PVC
+   ```
+
+## Defense-in-Depth Safety Layers
+
+1. **`helm.sh/resource-policy: keep` annotation** on every PVC/SC — blocks Helm from ever deleting even if removed from template
+2. **`lifecycle { prevent_destroy = true }`** on every Terraform PVC resource — hard-errors any destroy/removal instead of deleting live data
+3. **`terraform plan` gates** — nonzero diff = stop, fix or `state rm`, never force-apply against diff
+4. **`helmfile diff` before apply** — confirm no destructive ops queued
+5. **Longhorn replica-health baseline + re-check** — any drop in healthy replicas = hard stop
+6. **Rollback is always `terraform state rm`** — state-only, never touches live objects
+
+## Troubleshooting
+
+### terraform plan shows diff after import
+
+**Common causes:**
+- `volumeName` missing or wrong (required for bound PVC)
+- `storage` unit normalization (5Gi vs 5368709120)
+- Missing `resources.limits`
+- `accessModes` array order
+
+**Fix:** Adjust `terraform/<app>.tf` to match `kubectl get pvc -o yaml`, re-run `terraform plan`.
+
+### helmfile diff shows delete/replace on PVC
+
+**Cause:** Values change triggered Helm to re-template PVC; either values still match live and Helm shouldn't see diff, or the `existingClaim` change wasn't correct.
+
+**Fix:** Verify values match live spec exactly, or rollback the values change. If problem persists, ensure `helm.sh/resource-policy: keep` annotation is present: `kubectl get pvc <name> -n <ns> -o jsonpath='{.metadata.annotations}'`
+
+### Longhorn replica count drops
+
+**Cause:** Import or helmfile apply somehow triggered a Longhorn reconciliation that affected replica status.
+
+**Fix:** Do not proceed. Investigate Longhorn health manually (`kubectl get longhorn-volume -n longhorn-system <vol> -o json`). Rollback import with `terraform state rm`, revert values changes, re-annotate with `helm.sh/resource-policy=keep`. Wait for replicas to recover, then retry.
+
+## Commit & PR
+
+Once all phases complete:
+
+```bash
+cd /path/to/homelab
+git add terraform/longhorn.tf terraform/grafana.tf terraform/portainer.tf terraform/loki.tf terraform/dev-tools.tf terraform/minio-logging.tf terraform/forgejo.tf
+git add k8s/logging/grafana-values.yaml k8s/dashboard/portainer-values.yaml k8s/dev-tools/values.yaml k8s/logging/loki-values.yaml k8s/cicd/forgejo-values.yaml
+git commit -m "feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state
+
+- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults, app-specific)
+- Phase 2: grafana, portainer, dev-tools, loki, minio-logging, forgejo PVCs
+- Update chart values to use existingClaim or keep identical (no dual-ownership)
+- All imports protected by helm.sh/resource-policy=keep + prevent_destroy lifecycle
+- Longhorn replica health verified throughout, zero data loss
+- See terraform/LONGHORN_PVC_IMPORT.md for execution record"
+
+git push origin <branch>
+```
+
+Create PR with description referencing this guide.
+
+## Reference
+
+- **Plan file:** `terraform/no-delete-only-import-buzzing-gray.md` (archived plan, for reference)
+- **Scripts:** `terraform/scripts/phase-*.sh`
+- **Terraform resources:** `terraform/longhorn.tf`, `terraform/grafana.tf`, `terraform/portainer.tf`, `terraform/loki.tf`, `terraform/dev-tools.tf`, `terraform/minio-logging.tf`, `terraform/forgejo.tf`
+- **Longhorn health command:** (from CLAUDE.md) `k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'`
+- **Hard rule:** Never manually delete a PVC without verified replicas/backups (CLAUDE.md)
+
+---
+
+**Last updated:** 2026-07-15
@@ -0,0 +1,25 @@
+# Forgejo (server) — Git repository and CI/CD configuration persistent storage
+# Imported from live cluster state (import-only, no delete)
+# Live name: gitea-shared-storage (Gitea is old name, Forgejo is new)
+
+resource "kubernetes_persistent_volume_claim" "forgejo_shared_storage" {
+  metadata {
+    name      = "gitea-shared-storage"
+    namespace = "cicd"
+  }
+  spec {
+    access_modes       = ["ReadWriteOnce"]
+    storage_class_name = "longhorn"
+    resources {
+      requests = {
+        storage = "20Gi"
+      }
+    }
+    volume_name = "pvc-889e20b9-2203-46e6-8c08-d015cd15193d"
+  }
+
+
+  lifecycle {
+    prevent_destroy = true
+  }
+}
@@ -0,0 +1,24 @@
+# Grafana — persistent storage for dashboards and datasources
+# Imported from live cluster state (import-only, no delete)
+
+resource "kubernetes_persistent_volume_claim" "grafana" {
+  metadata {
+    name      = "grafana"
+    namespace = "logging"
+  }
+  spec {
+    access_modes       = ["ReadWriteOnce"]
+    storage_class_name = "longhorn"
+    resources {
+      requests = {
+        storage = "5Gi"
+      }
+    }
+    volume_name = "pvc-95f2216e-985d-4496-8b39-48e8c2a7f410"
+  }
+
+
+  lifecycle {
+    prevent_destroy = true
+  }
+}
@@ -0,0 +1,24 @@
+# Loki — log aggregation backend persistent storage (StatefulSet mode: storage-loki-0)
+# Imported from live cluster state (import-only, no delete)
+
+resource "kubernetes_persistent_volume_claim" "loki" {
+  metadata {
+    name      = "storage-loki-0"
+    namespace = "logging"
+  }
+  spec {
+    access_modes       = ["ReadWriteOnce"]
+    storage_class_name = "longhorn"
+    resources {
+      requests = {
+        storage = "5Gi"
+      }
+    }
+    volume_name = "pvc-897d4f5a-1699-4e06-b92f-ccbe3911be0f"
+  }
+
+
+  lifecycle {
+    prevent_destroy = true
+  }
+}
@@ -0,0 +1,40 @@
+# Longhorn StorageClasses — cluster-wide default + app-specific variants
+# Imported from live cluster state (import-only, no delete)
+
+resource "kubernetes_storage_class" "longhorn" {
+  metadata {
+    name = "longhorn"
+  }
+  storage_provisioner    = "driver.longhorn.io"
+  reclaim_policy         = "Delete"
+  allow_volume_expansion = true
+  volume_binding_mode    = "Immediate"
+
+  parameters = {
+    numberOfReplicas          = "3"
+    staleReplicaTimeout       = "60"
+    fromBackup                = ""
+    fsType                    = "ext4"
+    dataLocality              = "disabled"
+    disableRevisionCounter    = "true"
+    unmapMarkSnapChainRemoved = "ignored"
+  }
+}
+
+resource "kubernetes_storage_class" "longhorn_kafka" {
+  metadata {
+    name = "longhorn-kafka"
+  }
+  storage_provisioner    = "driver.longhorn.io"
+  reclaim_policy         = "Delete"
+  allow_volume_expansion = true
+  volume_binding_mode    = "Immediate"
+
+  parameters = {
+    numberOfReplicas    = "3"
+    staleReplicaTimeout = "30"
+    fromBackup          = ""
+    fsType              = "ext4"
+    dataLocality        = "disabled"
+  }
+}
@@ -0,0 +1,24 @@
+# Portainer — Docker container management UI persistent storage
+# Imported from live cluster state (import-only, no delete)
+
+resource "kubernetes_persistent_volume_claim" "portainer" {
+  metadata {
+    name      = "portainer"
+    namespace = "dashboard"
+  }
+  spec {
+    access_modes       = ["ReadWriteOnce"]
+    storage_class_name = "longhorn"
+    resources {
+      requests = {
+        storage = "10Gi"
+      }
+    }
+    volume_name = "pvc-bcd0d8cb-1214-4d01-a960-f104926f2b00"
+  }
+
+
+  lifecycle {
+    prevent_destroy = true
+  }
+}
@@ -19,7 +19,7 @@ terraform {
    }
    authentik = {
      source  = "goauthentik/authentik"
-      version = "~> 2024.6"
+      version = "2024.12.1"
    }
    aws = {
      source  = "hashicorp/aws"
@@ -0,0 +1,56 @@
+#!/bin/bash
+# Phase 0 — Read-only reconciliation of three ownership ambiguities
+# Safe to run; only queries, no mutations
+
+set -e
+cd "$(dirname "$0")/.."
+
+echo "=== Phase 0: Cluster Reconciliation ==="
+echo ""
+
+# Ambiguity 1: minio (storage) — standalone chart vs. Operator Tenant
+echo "--- Ambiguity 1: minio (storage) ownership ---"
+echo "Helm releases in 'storage' namespace:"
+helm list -n storage || echo "  (helm list failed)"
+echo ""
+echo "MinIO Operator Tenants in 'storage' namespace:"
+kubectl get tenant -n storage -o wide 2>/dev/null || echo "  (no Tenant CRD or none found)"
+echo ""
+echo "PVCs in 'storage' namespace:"
+kubectl get pvc -n storage -o wide || echo "  (kubectl failed)"
+echo ""
+
+# Ambiguity 2: kmsvc-redis — helmfile release vs. ArgoCD Application
+echo "--- Ambiguity 2: kmsvc-redis ownership ---"
+echo "ArgoCD Applications containing 'redis':"
+argocd app list | grep redis || echo "  (no match or argocd unavailable)"
+echo ""
+echo "Helm releases in 'sqs' namespace:"
+helm list -n sqs | grep redis || echo "  (no redis release)"
+echo ""
+echo "Redis-related PVCs in 'sqs' namespace:"
+kubectl get pvc -n sqs -o wide | grep -i redis || echo "  (no redis PVC)"
+echo ""
+
+# Ambiguity 3: forgejo-runner — helm chart vs. raw manifest
+echo "--- Ambiguity 3: forgejo-runner ownership ---"
+echo "Helm releases in 'cicd' namespace containing 'runner':"
+helm list -n cicd | grep runner || echo "  (no runner release)"
+echo ""
+echo "Deployments in 'cicd' namespace with label app=forgejo-runner:"
+kubectl get deploy -n cicd -l app=forgejo-runner -o wide || echo "  (no matching deployment)"
+echo ""
+echo "Checking managedFields on any forgejo-runner Deployment (to identify controller):"
+kubectl get deploy -n cicd -l app=forgejo-runner -o json 2>/dev/null | jq '.items[0].metadata.managedFields' 2>/dev/null || echo "  (no deployment found)"
+echo ""
+echo "PVCs in 'cicd' namespace named runner-*:"
+kubectl get pvc -n cicd -o wide | grep runner || echo "  (no runner PVC)"
+echo ""
+
+echo "=== Phase 0 Complete ==="
+echo "Review the output above. Determine which of the three ambiguities are resolved:"
+echo "  1. minio(storage): Is Helm release or Tenant CR authoritative? (helm list vs kubectl get tenant)"
+echo "  2. kmsvc-redis: Is helmfile or ArgoCD Application active? (argocd app list vs helm list)"
+echo "  3. forgejo-runner: Is Helm chart or raw manifest applied? (helm list vs kubectl get deploy managedFields)"
+echo ""
+echo "Document your findings. Proceed to Phase 1 (StorageClass import) only after clarity."
@@ -0,0 +1,87 @@
+#!/bin/bash
+# Phase 1 — StorageClass import (longhorn + longhorn-kafka)
+# Captures live spec, verifies against resource blocks, imports to state
+
+set -e
+cd "$(dirname "$0")/.."
+
+echo "=== Phase 1: StorageClass Import ==="
+echo ""
+
+# Verify cluster connection
+echo "Checking cluster connection..."
+kubectl cluster-info || { echo "ERROR: No cluster access"; exit 1; }
+echo ""
+
+# Capture live longhorn SC spec
+echo "--- Capturing live 'longhorn' StorageClass spec ---"
+kubectl get sc longhorn -o yaml > /tmp/longhorn-live.yaml
+echo "Saved to /tmp/longhorn-live.yaml"
+echo "Contents:"
+cat /tmp/longhorn-live.yaml
+echo ""
+
+# Capture live longhorn-kafka SC spec
+echo "--- Capturing live 'longhorn-kafka' StorageClass spec ---"
+kubectl get sc longhorn-kafka -o yaml > /tmp/longhorn-kafka-live.yaml
+echo "Saved to /tmp/longhorn-kafka-live.yaml"
+echo "Contents:"
+cat /tmp/longhorn-kafka-live.yaml
+echo ""
+
+# Verify terraform resource blocks match live spec
+echo "--- Verifying terraform/longhorn.tf resource blocks against live specs ---"
+echo "MANUAL STEP: Compare the captured specs above against terraform/longhorn.tf"
+echo "  1. Check 'provisioner', 'reclaimPolicy', 'volumeBindingMode', 'allowVolumeExpansion'"
+echo "  2. Check 'parameters' (numberOfReplicas, staleReplicaTimeout, fsType, dataLocality)"
+echo "  3. Fix terraform/longhorn.tf if any diffs found, then re-run terraform plan"
+echo ""
+read -p "Press Enter once you've verified the Terraform blocks match live specs: " _ || true
+echo ""
+
+# Pre-import terraform plan
+echo "--- Pre-import terraform plan (should be clean) ---"
+terraform plan -out=/tmp/pre-import.plan || { echo "ERROR: terraform plan failed"; exit 1; }
+echo "Plan saved to /tmp/pre-import.plan"
+echo ""
+
+# Import longhorn SC
+echo "--- Importing 'longhorn' StorageClass ---"
+terraform import kubernetes_storage_class.longhorn longhorn || {
+  echo "ERROR: terraform import longhorn failed"
+  exit 1
+}
+echo "Import successful"
+echo ""
+
+# Plan after first import
+echo "--- Terraform plan after importing 'longhorn' (must show 0 to add/change/destroy) ---"
+terraform plan || { echo "ERROR: terraform plan failed"; exit 1; }
+echo ""
+read -p "Press Enter if plan shows zero changes; otherwise abort and fix: " _ || true
+echo ""
+
+# Import longhorn-kafka SC
+echo "--- Importing 'longhorn-kafka' StorageClass ---"
+terraform import kubernetes_storage_class.longhorn_kafka longhorn-kafka || {
+  echo "ERROR: terraform import longhorn-kafka failed"
+  exit 1
+}
+echo "Import successful"
+echo ""
+
+# Plan after second import
+echo "--- Terraform plan after importing 'longhorn-kafka' (must show 0 to add/change/destroy) ---"
+terraform plan || { echo "ERROR: terraform plan failed"; exit 1; }
+echo ""
+read -p "Press Enter if plan shows zero changes; otherwise abort and fix: " _ || true
+echo ""
+
+# Annotate longhorn-kafka with helm.sh/resource-policy=keep
+echo "--- Annotating 'longhorn-kafka' with helm.sh/resource-policy=keep ---"
+kubectl annotate storageclass longhorn-kafka helm.sh/resource-policy=keep --overwrite
+echo "Annotation added"
+echo ""
+
+echo "=== Phase 1 Complete ==="
+echo "StorageClasses imported successfully. Ready to proceed to Phase 2 (PVC imports)."
@@ -0,0 +1,148 @@
+#!/bin/bash
+# Phase 2 Pilot — Grafana PVC import (full cycle: annotate → import → plan → values change → helmfile diff → helmfile apply)
+# Lowest blast radius, validates entire workflow before rolling to other apps
+
+set -e
+cd "$(dirname "$0")/.."
+
+echo "=== Phase 2 Pilot: Grafana PVC Import ==="
+echo ""
+
+# Step a) Protect the live PVC from Helm deletion
+echo "--- Step a) Protect PVC from Helm deletion ---"
+echo "Identifying grafana PVC in 'logging' namespace:"
+kubectl get pvc -n logging -l app.kubernetes.io/instance=grafana -o wide || {
+  echo "ERROR: Cannot find grafana PVC"
+  exit 1
+}
+echo ""
+echo "Annotating with helm.sh/resource-policy=keep (non-destructive, reversible):"
+kubectl annotate pvc grafana -n logging helm.sh/resource-policy=keep --overwrite
+echo "Annotation applied"
+echo ""
+
+# Step b) Capture the exact live PVC spec
+echo "--- Step b) Capture live PVC spec ---"
+kubectl get pvc grafana -n logging -o yaml > /tmp/grafana-pvc-live.yaml
+echo "Saved to /tmp/grafana-pvc-live.yaml"
+echo ""
+echo "Extracting key fields:"
+echo "Access modes:"
+kubectl get pvc grafana -n logging -o jsonpath='{.spec.accessModes}' | tr ',' '\n'
+echo "Storage class:"
+kubectl get pvc grafana -n logging -o jsonpath='{.spec.storageClassName}'
+echo ""
+echo "Requested storage:"
+kubectl get pvc grafana -n logging -o jsonpath='{.spec.resources.requests.storage}'
+echo ""
+echo "Bound PV name:"
+PV_NAME=$(kubectl get pvc grafana -n logging -o jsonpath='{.spec.volumeName}')
+echo "$PV_NAME"
+echo ""
+echo "MANUAL STEP: Update terraform/grafana.tf with the actual volumeName '$PV_NAME' (currently 'pvc-grafana' as placeholder)"
+echo ""
+read -p "Press Enter once grafana.tf is updated with the correct volumeName: " _ || true
+echo ""
+
+# Step d) Import and verify zero diff
+echo "--- Step d) Import 'grafana' PVC ---"
+terraform import kubernetes_persistent_volume_claim.grafana logging/grafana || {
+  echo "ERROR: terraform import failed"
+  exit 1
+}
+echo "Import successful"
+echo ""
+
+echo "--- Verifying zero diff (must show 0 to add/change/destroy) ---"
+PLAN_OUTPUT=$(terraform plan 2>&1)
+echo "$PLAN_OUTPUT"
+if echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then
+  echo "✓ Plan is clean"
+else
+  echo "✗ Plan shows changes — STOP, do not proceed"
+  echo "  Options:"
+  echo "    1. Fix terraform/grafana.tf and re-run terraform plan"
+  echo "    2. Rollback with: terraform state rm kubernetes_persistent_volume_claim.grafana"
+  exit 1
+fi
+echo ""
+
+# Step f) Verify Longhorn replica health BEFORE values change
+echo "--- Step f.1) Baseline Longhorn replica health ---"
+LONGHORN_VOL=$(kubectl get pvc grafana -n logging -o jsonpath='{.spec.volumeName}' | sed 's/pvc-//' )
+echo "Checking Longhorn volume health for: $LONGHORN_VOL"
+kubectl get longhorn-volume -n longhorn-system "$LONGHORN_VOL" -o json | jq '.status.replicaStatus' 2>/dev/null || echo "  (could not get Longhorn status; continue)"
+echo ""
+read -p "Note the replica status above. Press Enter to continue: " _ || true
+echo ""
+
+# Step e) Update grafana values to use existingClaim (only if chart supports it)
+echo "--- Step e) Update k8s/logging/grafana-values.yaml for existingClaim ---"
+echo "Current grafana-values.yaml persistence section:"
+grep -A 5 "^persistence:" k8s/logging/grafana-values.yaml || echo "  (no persistence section found)"
+echo ""
+echo "MANUAL STEP: Add/update to k8s/logging/grafana-values.yaml:"
+echo "  persistence:"
+echo "    existingClaim: grafana"
+echo "    enabled: false"
+echo ""
+echo "If the chart does NOT support existingClaim (check Grafana chart docs), leave:"
+echo "  persistence:"
+echo "    enabled: true"
+echo "    size: 5Gi"
+echo "    storageClassName: longhorn"
+echo "  (Helm will then see no diff and won't delete the PVC; the keep annotation is the backstop)"
+echo ""
+read -p "Press Enter once grafana-values.yaml is updated: " _ || true
+echo ""
+
+# Step e.2) helmfile diff to confirm no delete queued
+echo "--- Step e.2) Helmfile diff to confirm no delete/replace ---"
+echo "Running helmfile diff for grafana (in logging namespace, chart= from helmfile):"
+cd "$(dirname "$0")/../.."  # go to repo root
+helmfile -e logging -f helmfile.yaml.gotmpl -l name=grafana diff || {
+  echo "WARNING: helmfile diff failed or returned nonzero exit; check output above"
+  echo "  (helmfile may not be perfectly compatible with this session, but diff result should be visible)"
+}
+cd "$(dirname "$0")/../terraform"
+echo ""
+echo "Confirm no 'delete' or 'replace' operations on the grafana PVC are queued."
+echo ""
+read -p "Press Enter if helmfile diff shows no destructive ops on grafana PVC: " _ || true
+echo ""
+
+# Step e.3) helmfile apply
+echo "--- Step e.3) Helmfile apply ---"
+cd "$(dirname "$0")/../.."
+echo "Applying logging/grafana via helmfile:"
+helmfile -e logging -f helmfile.yaml.gotmpl -l name=grafana apply || {
+  echo "WARNING: helmfile apply returned nonzero; check output above"
+}
+cd "$(dirname "$0")/../terraform"
+echo ""
+echo "Helmfile apply complete"
+echo ""
+
+# Step f.2) Verify Longhorn health AFTER helmfile apply
+echo "--- Step f.2) Post-helmfile Longhorn replica health check ---"
+echo "Checking Longhorn volume health for: $LONGHORN_VOL"
+kubectl get longhorn-volume -n longhorn-system "$LONGHORN_VOL" -o json | jq '.status.replicaStatus' 2>/dev/null || echo "  (could not get Longhorn status)"
+echo ""
+echo "Confirm replica status is identical to baseline above."
+read -p "Press Enter if replica health matches baseline: " _ || true
+echo ""
+
+# Final terraform plan
+echo "--- Final terraform plan (must still be 0/0/0 after all changes) ---"
+PLAN_OUTPUT=$(terraform plan 2>&1)
+echo "$PLAN_OUTPUT"
+if echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then
+  echo "✓ Plan is still clean"
+else
+  echo "✗ Plan shows changes after helmfile apply — investigate"
+  exit 1
+fi
+echo ""
+
+echo "=== Phase 2 Pilot: Grafana Complete ==="
+echo "Grafana PVC successfully imported. Ready for Phase 2 remaining apps."
@@ -0,0 +1,122 @@
+#!/bin/bash
+# Phase 2 Remaining Apps — Per-app PVC imports (portainer → dev-tools → loki → minio-logging → forgejo)
+# Same procedure as grafana pilot; follow the pattern
+
+set -e
+cd "$(dirname "$0")/.."
+
+echo "=== Phase 2 Remaining Apps ==="
+echo "Execute per-app cycles: portainer → dev-tools → loki → minio-logging → forgejo"
+echo ""
+
+# Helper function for per-app import
+import_app_pvc() {
+  local app=$1
+  local namespace=$2
+  local pvc_name=$3
+
+  echo "--- Importing $app PVC ---"
+  echo ""
+
+  # Annotate
+  echo "Step a) Annotate PVC with helm.sh/resource-policy=keep:"
+  kubectl annotate pvc "$pvc_name" -n "$namespace" helm.sh/resource-policy=keep --overwrite
+  echo ""
+
+  # Capture live spec
+  echo "Step b) Capture live PVC spec:"
+  kubectl get pvc "$pvc_name" -n "$namespace" -o yaml > "/tmp/${app}-pvc-live.yaml"
+  echo "Saved to /tmp/${app}-pvc-live.yaml"
+  echo ""
+  PV_NAME=$(kubectl get pvc "$pvc_name" -n "$namespace" -o jsonpath='{.spec.volumeName}')
+  echo "Bound PV: $PV_NAME"
+  echo "MANUAL: Update terraform/${app}.tf with correct volumeName '$PV_NAME'"
+  read -p "Press Enter once terraform/${app}.tf is updated: " _ || true
+  echo ""
+
+  # Import
+  echo "Step d) Import to Terraform state:"
+  terraform import "kubernetes_persistent_volume_claim.${app}" "${namespace}/${pvc_name}" || {
+    echo "ERROR: import failed for $app"
+    return 1
+  }
+  echo ""
+
+  # Plan
+  echo "Step d cont.) Verify zero diff:"
+  PLAN_OUTPUT=$(terraform plan 2>&1)
+  echo "$PLAN_OUTPUT"
+  if ! echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then
+    echo "ERROR: Plan is not clean for $app"
+    return 1
+  fi
+  echo "✓ Plan is clean"
+  echo ""
+
+  # Baseline Longhorn health
+  echo "Step f.1) Baseline Longhorn replica health:"
+  kubectl get longhorn-volume -n longhorn-system "$PV_NAME" -o json 2>/dev/null | jq '.status.replicaStatus' 2>/dev/null || echo "  (unavailable)"
+  read -p "Press Enter to continue: " _ || true
+  echo ""
+
+  # Update values
+  echo "Step e) Update values to use existingClaim (if supported) or keep identical:"
+  echo "MANUAL: Verify terraform/${app}.tf resource block matches live PVC spec exactly"
+  echo "        Update chart values (k8s/ directory) to point to existing PVC or keep identical"
+  read -p "Press Enter once values updated: " _ || true
+  echo ""
+
+  # helmfile diff
+  echo "Step e.2) Helmfile diff:"
+  cd "$(dirname "$0")/../.."
+  helmfile -f helmfile.yaml.gotmpl -l name="$app" diff 2>&1 | head -50 || echo "  (helmfile diff unavailable)"
+  cd "$(dirname "$0")/../terraform"
+  echo ""
+  read -p "Confirm no destructive ops. Press Enter to continue: " _ || true
+  echo ""
+
+  # helmfile apply
+  echo "Step e.3) Helmfile apply:"
+  cd "$(dirname "$0")/../.."
+  helmfile -f helmfile.yaml.gotmpl -l name="$app" apply || echo "  (helmfile apply unavailable)"
+  cd "$(dirname "$0")/../terraform"
+  echo ""
+
+  # Post-apply Longhorn health
+  echo "Step f.2) Post-apply Longhorn health:"
+  kubectl get longhorn-volume -n longhorn-system "$PV_NAME" -o json 2>/dev/null | jq '.status.replicaStatus' 2>/dev/null || echo "  (unavailable)"
+  echo "Confirm matches baseline."
+  read -p "Press Enter if health is good: " _ || true
+  echo ""
+
+  # Final plan
+  echo "Step d) Final terraform plan:"
+  PLAN_OUTPUT=$(terraform plan 2>&1)
+  echo "$PLAN_OUTPUT"
+  if ! echo "$PLAN_OUTPUT" | grep -q "0 to add, 0 to change, 0 to destroy"; then
+    echo "ERROR: Plan is not clean after helmfile apply for $app"
+    return 1
+  fi
+  echo "✓ Plan is clean"
+  echo ""
+  echo "=== $app Complete ==="
+  echo ""
+}
+
+# Execute per-app imports in sequence (ascending risk)
+import_app_pvc "portainer" "dashboard" "portainer" || { echo "FAILED at portainer"; exit 1; }
+import_app_pvc "dev_tools" "dev-tools" "dev-tools-pvc" || { echo "FAILED at dev-tools"; exit 1; }
+import_app_pvc "loki" "logging" "loki" || { echo "FAILED at loki"; exit 1; }
+import_app_pvc "minio_logging" "logging" "minio" || { echo "FAILED at minio-logging"; exit 1; }
+import_app_pvc "forgejo_shared_storage" "cicd" "forgejo-shared-storage" || { echo "FAILED at forgejo"; exit 1; }
+
+echo "=== Phase 2 Remaining Apps: Complete ==="
+echo "All safe apps imported successfully."
+echo ""
+echo "Next steps:"
+echo "  1. Conditional apps (pending Phase 0 ambiguity resolution):"
+echo "     - minio (storage) — if standalone chart is authoritative"
+echo "     - kmsvc-redis — if helmfile release is authoritative and architecture is standalone"
+echo "     - forgejo-runner PVCs — if Helm chart is authoritative"
+echo "  2. Never import: ddb-cluster, prometheus, kafka-cluster, authentik-postgresql, llm namespace"
+echo "  3. Commit all new terraform/*.tf files to git and create PR"
@@ -5,6 +5,7 @@ terraform {
    key                         = "homelab/terraform.tfstate"
    region                      = "us-east-1"
    endpoint                    = "https://minio-api.riotpiao.homelab.com"
+    profile                     = "minio"
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_region_validation      = true