homelab/terraform/LONGHORN_PVC_IMPORT.md

# Longhorn StorageClass + PVC Terraform Import — Execution Guide

## Overview

Moving Longhorn StorageClasses and app PVCs from Helm/chart-owned to Terraform-managed state. **Import-only, no delete/recreate.** Zero data-loss tolerance. Production data: Postgres, Kafka, MinIO, Loki.

**Terraform now owns:** namespaces, bootstrap Helm releases, StorageClasses. **ArgoCD now owns:** workload Helm releases (grafana, loki, minio, portainer, forgejo, kafka, prometheus, ddb, ollama, etc.). This matches the in-progress Terraform+ArgoCD migration.

**Explicit exclusion:** `llm` namespace / `ollama` / `longhorn-llm` SC — managed under separate state (`~/workplace/agents/infra/terraform/`).

## Prerequisite

- Cluster access + kubectl configured: `kubectl cluster-info` succeeds
- Terraform credentials: state backend (MinIO) reachable
- ArgoCD CLI (for Phase 0 reconciliation only)
- Helm CLI (for helmfile operations, Phase 2)

## Execution: 4 Phases

### Phase 0 — Cluster Reconciliation (read-only, ~10 min)

Resolve three ownership ambiguities before importing.

```bash
cd terraform/scripts
bash phase-0-reconciliation.sh
```

**Output:** Review findings for:
1. **minio (storage)**: Is Helm release (`terraform/minio.tf`) or Operator `Tenant` CR authoritative?
2. **kmsvc-redis**: Is helmfile release or ArgoCD Application active?
3. **forgejo-runner**: Is Helm chart or raw manifest (`k8s/forge/runner.yaml`) applied?

**Document results.** Proceed to Phase 1 regardless; Phase 0 just informs which conditional apps to import in Phase 2.

### Phase 1 — StorageClass Import (~5 min)

Import `longhorn` (cluster default) and `longhorn-kafka` (Kafka-specific) to Terraform state.

```bash
cd terraform/scripts
bash phase-1-storageclass-import.sh
```

**Workflow:**
1. Captures live spec via `kubectl get sc`
2. **MANUAL:** Verify `terraform/longhorn.tf` resource blocks match captured specs
3. `terraform import` both SCs one at a time
4. `terraform plan` must show `0 to add, 0 to change, 0 to destroy` after each import
5. Annotates `longhorn-kafka` with `helm.sh/resource-policy=keep` (protects from Helm deletion on next upgrade)

**Rollback:** `terraform state rm kubernetes_storage_class.longhorn` (state-only, safe)

### Phase 2 Pilot — Grafana PVC (~15 min)

Lowest-risk pilot: dashboards/config, re-creatable from values-based provisioning. Validates entire import workflow before rolling to other apps.

```bash
cd terraform/scripts
bash phase-2-pilot-grafana.sh
```

**Workflow:**
1. Annotate live PVC: `helm.sh/resource-policy=keep` (protects from Helm deletion)
2. Capture live PVC spec (`kubectl get pvc -o yaml`)
3. **MANUAL:** Update `terraform/grafana.tf` with actual PV name
4. `terraform import` to state
5. `terraform plan` must show zero diff
6. **MANUAL:** Update `k8s/logging/grafana-values.yaml`:
   ```yaml
   persistence:
     existingClaim: grafana
     enabled: false
   ```
   (Or keep identical if chart doesn't support `existingClaim`)
7. `helmfile diff` to confirm no delete/replace queued
8. `helmfile apply` to reconcile
9. Verify Longhorn replica health unchanged
10. `terraform plan` again, must still be zero diff

**Phase gate:** Don't proceed to remaining apps until grafana cycle completes cleanly.

**Rollback:** `terraform state rm kubernetes_persistent_volume_claim.grafana` (state-only, safe)

### Phase 2 Remaining Apps (~45 min)

Repeat pilot workflow for: **portainer** → **dev-tools** → **loki** → **minio-logging** → **forgejo**.

```bash
cd terraform/scripts
bash phase-2-remaining-apps.sh
```

Script guides through each app in sequence. **Manual steps** for each:
- Verify Terraform resource block matches live PVC spec
- Update chart values to use `existingClaim` (if supported) or keep identical
- Confirm `helmfile diff` shows no destructive ops
- Confirm Longhorn replica health matches baseline

**Ascending risk order:**
- portainer: low (UI config, re-creatable)
- dev-tools: low (sandbox state, expendable)
- loki: medium (log history, valuable but not critical)
- minio-logging: medium (Loki chunks, important but backed up elsewhere)
- forgejo: high (Git repos, CI history — real data loss risk)

**Conditional apps — resolve Phase 0 ambiguities first:**
- minio (storage): if standalone chart is authoritative, not Tenant CR
- kmsvc-redis: if helmfile release is authoritative, architecture is standalone (not Sentinel)
- forgejo-runner: if Helm chart is authoritative, not raw manifest

**Never import** (StatefulSet/operator-managed, dual-controller risk):
- ddb-cluster (CNPG operator owns PVC lifecycle)
- prometheus (Prometheus Operator `storageSpec.volumeClaimTemplate`)
- kafka-cluster (StatefulSet `volumeClaimTemplates`)
- authentik-postgresql (likely sub-chart StatefulSet VCT)
- llm namespace (separate Terraform state)

## Rollback at Any Point

**State-only backout (safe, touches nothing live):**
```bash
terraform state rm kubernetes_storage_class.longhorn
terraform state rm kubernetes_persistent_volume_claim.grafana
```

**Remove Helm protection annotation (only if abandoning import):**
```bash
kubectl annotate pvc grafana -n logging helm.sh/resource-policy- --overwrite
```

## Verification Gateways

After every import + values change:

1. **Terraform plan is clean:**
   ```bash
   terraform plan   # must show "0 to add, 0 to change, 0 to destroy"
   ```

2. **Longhorn replica health unchanged:**
   ```bash
   # Before starting each app:
   k get longhorn-volume -n longhorn-system <pv-name> -o json | jq '.status.replicaStatus'
   # After helmfile apply, must be identical
   ```

3. **No destructive ops queued:**
   ```bash
   helmfile -f helmfile.yaml.gotmpl -l name=<app> diff   # no delete/replace on PVC
   ```

## Defense-in-Depth Safety Layers

1. **`helm.sh/resource-policy: keep` annotation** on every PVC/SC — blocks Helm from ever deleting even if removed from template
2. **`lifecycle { prevent_destroy = true }`** on every Terraform PVC resource — hard-errors any destroy/removal instead of deleting live data
3. **`terraform plan` gates** — nonzero diff = stop, fix or `state rm`, never force-apply against diff
4. **`helmfile diff` before apply** — confirm no destructive ops queued
5. **Longhorn replica-health baseline + re-check** — any drop in healthy replicas = hard stop
6. **Rollback is always `terraform state rm`** — state-only, never touches live objects

## Troubleshooting

### terraform plan shows diff after import

**Common causes:**
- `volumeName` missing or wrong (required for bound PVC)
- `storage` unit normalization (5Gi vs 5368709120)
- Missing `resources.limits`
- `accessModes` array order

**Fix:** Adjust `terraform/<app>.tf` to match `kubectl get pvc -o yaml`, re-run `terraform plan`.

### helmfile diff shows delete/replace on PVC

**Cause:** Values change triggered Helm to re-template PVC; either values still match live and Helm shouldn't see diff, or the `existingClaim` change wasn't correct.

**Fix:** Verify values match live spec exactly, or rollback the values change. If problem persists, ensure `helm.sh/resource-policy: keep` annotation is present: `kubectl get pvc <name> -n <ns> -o jsonpath='{.metadata.annotations}'`

### Longhorn replica count drops

**Cause:** Import or helmfile apply somehow triggered a Longhorn reconciliation that affected replica status.

**Fix:** Do not proceed. Investigate Longhorn health manually (`kubectl get longhorn-volume -n longhorn-system <vol> -o json`). Rollback import with `terraform state rm`, revert values changes, re-annotate with `helm.sh/resource-policy=keep`. Wait for replicas to recover, then retry.

## Commit & PR

Once all phases complete:

```bash
cd /path/to/homelab
git add terraform/longhorn.tf terraform/grafana.tf terraform/portainer.tf terraform/loki.tf terraform/dev-tools.tf terraform/minio-logging.tf terraform/forgejo.tf
git add k8s/logging/grafana-values.yaml k8s/dashboard/portainer-values.yaml k8s/dev-tools/values.yaml k8s/logging/loki-values.yaml k8s/cicd/forgejo-values.yaml
git commit -m "feat(terraform): import Longhorn StorageClasses and app PVCs to Terraform state

- Phase 1: longhorn, longhorn-kafka StorageClasses (cluster-wide defaults, app-specific)
- Phase 2: grafana, portainer, dev-tools, loki, minio-logging, forgejo PVCs
- Update chart values to use existingClaim or keep identical (no dual-ownership)
- All imports protected by helm.sh/resource-policy=keep + prevent_destroy lifecycle
- Longhorn replica health verified throughout, zero data loss
- See terraform/LONGHORN_PVC_IMPORT.md for execution record"

git push origin <branch>
```

Create PR with description referencing this guide.

## Reference

- **Plan file:** `terraform/no-delete-only-import-buzzing-gray.md` (archived plan, for reference)
- **Scripts:** `terraform/scripts/phase-*.sh`
- **Terraform resources:** `terraform/longhorn.tf`, `terraform/grafana.tf`, `terraform/portainer.tf`, `terraform/loki.tf`, `terraform/dev-tools.tf`, `terraform/minio-logging.tf`, `terraform/forgejo.tf`
- **Longhorn health command:** (from CLAUDE.md) `k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'`
- **Hard rule:** Never manually delete a PVC without verified replicas/backups (CLAUDE.md)

---

**Last updated:** 2026-07-15