🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit:
```bash
terraform fmt -recursive terraform/
```
Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise.
🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck:
🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer.
🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages.
🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear.
🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads.
🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
add` in Alpine-based images** — apk's working directories and most of
`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log:
Permission denied`. Use `/tmp` (world-writable) for any binary you need to
download/install at runtime, and extend `PATH` rather than writing to
`/usr/local/bin`.
🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or
wrong-schema key is silently a no-op, not an error.** Confirmed root cause
of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual
chart version pinned (`temporalio/helm-charts@0.74.0`) uses a flat
`server.config.persistence.<store>.driver/.sql` schema, but the values file
used the *newer* chart's `datastores:`-wrapped schema (introduced in a
later major version) — silently ignored, so persistence stayed on the
chart's Cassandra default the entire time. **Before assuming "this chart
doesn't support X," clone the chart at the exact pinned tag/version and run
`helm template` with your real values — diff the rendered output, don't
trust values.yaml comments/examples from the chart's current `main`
branch, which may not match your pinned version's schema at all.**
## Workflow
- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push**
- If fmt check fails in CI, fix locally, commit fmt changes, push again
- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize <dir>/` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`)
## Documentation Map
- `CLAUDE.md` (this file) — private, cluster-specific, always current
- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs)
- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a
deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that
does not reflect actual current practice (100% ArgoCD GitOps as of this
writing). Treat as historical reference only until rewritten; do not
follow their deployment procedures literally.
- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism
