docs(terraform): add state management script and best practices guide

docs/TERRAFORM-STATE.md

@@ -0,0 +1,113 @@
+# Terraform State Management
+
+## Overview
+
+Terraform state for the homelab cluster is managed using a hybrid approach:
+- **Remote backend:** S3 (MinIO) for centralized, shared state
+- **Local backup:** Git-ignored backups for disaster recovery
+
+## Backend Configuration
+
+State is stored in MinIO S3:
+
+```
+Bucket: terraform-state
+Key: homelab/terraform.tfstate
+Endpoint: https://minio-api.riotpiao.homelab.com
+Profile: minio
+```
+
+Configuration: `terraform/state.tf`
+
+## Accessing State
+
+### Pull state from S3
+```bash
+cd terraform
+terraform state pull > terraform.tfstate.backup
+```
+
+### View resources
+```bash
+terraform state list
+terraform state show <resource-name>
+```
+
+### Import new resources
+```bash
+terraform import <resource-type>.<name> <resource-id>
+```
+
+## Backup Strategy
+
+### Automatic backups
+Run the backup script periodically (e.g., cron):
+```bash
+scripts/terraform-state-backup.sh
+```
+
+Backups are saved to: `~/.terraform-backups/homelab/`
+
+### Manual backup
+```bash
+cd terraform
+terraform state pull > /tmp/terraform-$(date +%s).tfstate
+cp /tmp/terraform-*.tfstate ~/.terraform-backups/homelab/
+```
+
+## Disaster Recovery
+
+If state is corrupted or lost:
+
+1. **Stop all infrastructure changes:**
+   ```bash
+   git revert <commit>  # Rollback infrastructure changes
+   ```
+
+2. **Restore from local backup:**
+   ```bash
+   BACKUP_FILE=~/.terraform-backups/homelab/<timestamp>-terraform.tfstate
+   cd terraform
+   terraform state push $BACKUP_FILE
+   ```
+
+3. **Verify state:**
+   ```bash
+   terraform state list
+   terraform plan
+   ```
+
+## S3 Bucket Setup
+
+If S3 bucket doesn't exist, create it:
+
+```bash
+kubectl exec -n storage <minio-pod> -- mc mb minio/terraform-state --region us-east-1
+```
+
+## State Lock (Optional)
+
+For multi-person teams, enable state locking via DynamoDB (not yet configured).
+
+## Best Practices
+
+- ✓ Never commit `*.tfstate` or `*.tfstate.*` to git
+- ✓ Back up state before major `terraform apply` operations
+- ✓ Always run `terraform plan` before `terraform apply`
+- ✓ Review diff carefully for destructive changes
+- ✓ Keep state backend secure (MinIO has authentication)
+
+## Monitoring
+
+Check S3 backend status:
+```bash
+kubectl get pods -n storage -l app=minio
+# Or
+scripts/terraform-state-backup.sh
+```
+
+## Related Files
+
+- `terraform/state.tf` — Backend configuration
+- `scripts/terraform-state-backup.sh` — Automated backup script
+- `.gitignore` — Excludes local state files from git

scripts/terraform-state-backup.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+# Terraform state backup script
+# Backs up Terraform state to local directory and verifies S3 backend accessibility
+
+set -e
+
+BACKUP_DIR="$HOME/.terraform-backups/homelab"
+TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+STATE_BACKUP="$BACKUP_DIR/$TIMESTAMP-terraform.tfstate"
+
+mkdir -p "$BACKUP_DIR"
+
+echo "=== Terraform State Backup ==="
+echo "Backup directory: $BACKUP_DIR"
+
+# Navigate to terraform directory
+cd "$(dirname "${BASH_SOURCE[0]}")/../terraform" || exit 1
+
+# Pull state from remote backend
+echo "Pulling Terraform state from S3..."
+terraform state pull > "$STATE_BACKUP" && \
+  echo "✓ State backed up to $STATE_BACKUP" || \
+  echo "⚠️ Failed to pull state (S3 backend may not be initialized)"
+
+# Verify S3 bucket
+echo ""
+echo "Verifying S3 backend accessibility..."
+if [ -z "$KUBECONFIG" ]; then
+  echo "⚠️ KUBECONFIG not set. Run: core auth login"
+  exit 1
+fi
+
+if kubectl get pods -n storage -l app=minio &>/dev/null; then
+  MINIO_POD=$(kubectl get pods -n storage -l app=minio -o jsonpath='{.items[0].metadata.name}')
+  echo "MinIO pod: $MINIO_POD"
+
+  # Check bucket
+  if kubectl exec -n storage "$MINIO_POD" -- mc ls minio/terraform-state &>/dev/null; then
+    echo "✓ S3 bucket 'terraform-state' accessible"
+  else
+    echo "⚠️ S3 bucket 'terraform-state' not found or inaccessible"
+    echo "   To create: kubectl exec -n storage $MINIO_POD -- mc mb minio/terraform-state"
+  fi
+else
+  echo "⚠️ MinIO not running in storage namespace"
+fi
+
+# Keep only last 30 backups
+echo ""
+echo "Cleaning up old backups (keeping last 30)..."
+ls -t "$BACKUP_DIR"/*.tfstate 2>/dev/null | tail -n +31 | xargs -r rm && \
+  echo "✓ Cleanup complete" || \
+  echo "ℹ️ No old backups to remove"
+
+echo ""
+echo "Backup complete. Location: $STATE_BACKUP"