homelab/terraform/LONGHORN_PVC_IMPORT.md

Longhorn StorageClass + PVC Terraform Import — Execution Guide

Overview

Moving Longhorn StorageClasses and app PVCs from Helm/chart-owned to Terraform-managed state. Import-only, no delete/recreate. Zero data-loss tolerance. Production data: Postgres, Kafka, MinIO, Loki.

Terraform now owns: namespaces, bootstrap Helm releases, StorageClasses. ArgoCD now owns: workload Helm releases (grafana, loki, minio, portainer, forgejo, kafka, prometheus, ddb, ollama, etc.). This matches the in-progress Terraform+ArgoCD migration.

Explicit exclusion: llm namespace / ollama / longhorn-llm SC — managed under separate state (~/workplace/agents/infra/terraform/).

Prerequisite

• Cluster access + kubectl configured: kubectl cluster-info succeeds
• Terraform credentials: state backend (MinIO) reachable
• ArgoCD CLI (for Phase 0 reconciliation only)
• Helm CLI (for helmfile operations, Phase 2)

Execution: 4 Phases

Phase 0 — Cluster Reconciliation (read-only, ~10 min)

Resolve three ownership ambiguities before importing.

cd terraform/scripts
bash phase-0-reconciliation.sh

Output: Review findings for:

1. minio (storage): Is Helm release (terraform/minio.tf) or Operator Tenant CR authoritative?
2. kmsvc-redis: Is helmfile release or ArgoCD Application active?
3. forgejo-runner: Is Helm chart or raw manifest (k8s/forge/runner.yaml) applied?

Document results. Proceed to Phase 1 regardless; Phase 0 just informs which conditional apps to import in Phase 2.

Phase 1 — StorageClass Import (~5 min)

Import longhorn (cluster default) and longhorn-kafka (Kafka-specific) to Terraform state.

cd terraform/scripts
bash phase-1-storageclass-import.sh

Workflow:

1. Captures live spec via kubectl get sc
2. MANUAL: Verify terraform/longhorn.tf resource blocks match captured specs
3. terraform import both SCs one at a time
4. terraform plan must show 0 to add, 0 to change, 0 to destroy after each import
5. Annotates longhorn-kafka with helm.sh/resource-policy=keep (protects from Helm deletion on next upgrade)

Rollback: terraform state rm kubernetes_storage_class.longhorn (state-only, safe)

Phase 2 Pilot — Grafana PVC (~15 min)

Lowest-risk pilot: dashboards/config, re-creatable from values-based provisioning. Validates entire import workflow before rolling to other apps.

cd terraform/scripts
bash phase-2-pilot-grafana.sh

Workflow:

1. Annotate live PVC: helm.sh/resource-policy=keep (protects from Helm deletion)
2. Capture live PVC spec (kubectl get pvc -o yaml)
3. MANUAL: Update terraform/grafana.tf with actual PV name
4. terraform import to state
5. terraform plan must show zero diff
6. MANUAL: Update k8s/logging/grafana-values.yaml:

   persistence:
     existingClaim: grafana
     enabled: false
   

   (Or keep identical if chart doesn't support existingClaim)
7. helmfile diff to confirm no delete/replace queued
8. helmfile apply to reconcile
9. Verify Longhorn replica health unchanged
10. terraform plan again, must still be zero diff

Phase gate: Don't proceed to remaining apps until grafana cycle completes cleanly.

Rollback: terraform state rm kubernetes_persistent_volume_claim.grafana (state-only, safe)

Phase 2 Remaining Apps (~45 min)

Repeat pilot workflow for: portainer → dev-tools → loki → minio-logging → forgejo.

cd terraform/scripts
bash phase-2-remaining-apps.sh

Script guides through each app in sequence. Manual steps for each:

• Verify Terraform resource block matches live PVC spec
• Update chart values to use existingClaim (if supported) or keep identical
• Confirm helmfile diff shows no destructive ops
• Confirm Longhorn replica health matches baseline

Ascending risk order:

• portainer: low (UI config, re-creatable)
• dev-tools: low (sandbox state, expendable)
• loki: medium (log history, valuable but not critical)
• minio-logging: medium (Loki chunks, important but backed up elsewhere)
• forgejo: high (Git repos, CI history — real data loss risk)

Conditional apps — resolve Phase 0 ambiguities first:

• minio (storage): if standalone chart is authoritative, not Tenant CR
• kmsvc-redis: if helmfile release is authoritative, architecture is standalone (not Sentinel)
• forgejo-runner: if Helm chart is authoritative, not raw manifest

Never import (StatefulSet/operator-managed, dual-controller risk):

• ddb-cluster (CNPG operator owns PVC lifecycle)
• prometheus (Prometheus Operator storageSpec.volumeClaimTemplate)
• kafka-cluster (StatefulSet volumeClaimTemplates)
• authentik-postgresql (likely sub-chart StatefulSet VCT)
• llm namespace (separate Terraform state)

Rollback at Any Point

State-only backout (safe, touches nothing live):

terraform state rm kubernetes_storage_class.longhorn
terraform state rm kubernetes_persistent_volume_claim.grafana

Remove Helm protection annotation (only if abandoning import):

kubectl annotate pvc grafana -n logging helm.sh/resource-policy- --overwrite

Verification Gateways

After every import + values change:

1. Terraform plan is clean:

   terraform plan   # must show "0 to add, 0 to change, 0 to destroy"
   

2. Longhorn replica health unchanged:

   # Before starting each app:
   k get longhorn-volume -n longhorn-system <pv-name> -o json | jq '.status.replicaStatus'
   # After helmfile apply, must be identical
   

3. No destructive ops queued:

   helmfile -f helmfile.yaml.gotmpl -l name=<app> diff   # no delete/replace on PVC
   

Defense-in-Depth Safety Layers

1. helm.sh/resource-policy: keep annotation on every PVC/SC — blocks Helm from ever deleting even if removed from template
2. lifecycle { prevent_destroy = true } on every Terraform PVC resource — hard-errors any destroy/removal instead of deleting live data
3. terraform plan gates — nonzero diff = stop, fix or state rm, never force-apply against diff
4. helmfile diff before apply — confirm no destructive ops queued
5. Longhorn replica-health baseline + re-check — any drop in healthy replicas = hard stop
6. Rollback is always terraform state rm — state-only, never touches live objects

Troubleshooting

terraform plan shows diff after import

Common causes:

• volumeName missing or wrong (required for bound PVC)
• storage unit normalization (5Gi vs 5368709120)
• Missing resources.limits
• accessModes array order

Fix: Adjust terraform/<app>.tf to match kubectl get pvc -o yaml, re-run terraform plan.

helmfile diff shows delete/replace on PVC

Cause: Values change triggered Helm to re-template PVC; either values still match live and Helm shouldn't see diff, or the existingClaim change wasn't correct.

Fix: Verify values match live spec exactly, or rollback the values change. If problem persists, ensure helm.sh/resource-policy: keep annotation is present: kubectl get pvc <name> -n <ns> -o jsonpath='{.metadata.annotations}'

Longhorn replica count drops

Cause: Import or helmfile apply somehow triggered a Longhorn reconciliation that affected replica status.

Fix: Do not proceed. Investigate Longhorn health manually (kubectl get longhorn-volume -n longhorn-system <vol> -o json). Rollback import with terraform state rm, revert values changes, re-annotate with helm.sh/resource-policy=keep. Wait for replicas to recover, then retry.

Commit & PR

Once all phases complete:

cd /path/to/homelab
git add terraform/longhorn.tf terraform/grafana.tf terraform/portainer.tf terraform/loki.tf terraform/dev-tools.tf terraform/minio-logging.tf terraform/forgejo.tf
git add k8s/logging/grafana-values.yaml k8s/dashboard/portainer-values.yaml k8s/dev-tools/values.yaml k8s/logging/loki-values.yaml k8s/cicd/forgejo-values.yaml
git commit -m "feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state

- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults, app-specific)
- Phase 2: grafana, portainer, dev-tools, loki, minio-logging, forgejo PVCs
- Update chart values to use existingClaim or keep identical (no dual-ownership)
- All imports protected by helm.sh/resource-policy=keep + prevent_destroy lifecycle
- Longhorn replica health verified throughout, zero data loss
- See terraform/LONGHORN_PVC_IMPORT.md for execution record"

git push origin <branch>

Create PR with description referencing this guide.

Reference

• Plan file: terraform/no-delete-only-import-buzzing-gray.md (archived plan, for reference)
• Scripts: terraform/scripts/phase-*.sh
• Terraform resources: terraform/longhorn.tf, terraform/grafana.tf, terraform/portainer.tf, terraform/loki.tf, terraform/dev-tools.tf, terraform/minio-logging.tf, terraform/forgejo.tf
• Longhorn health command: (from CLAUDE.md) k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'
• Hard rule: Never manually delete a PVC without verified replicas/backups (CLAUDE.md)

---

Last updated: 2026-07-15