→ `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each
`argocd.argoproj.io/sync-wave` annotated) → each Application points at either
a remote Helm chart (+ a second `ref: values` git source for the values file)
or a plain git directory of raw manifests.
**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except:
- Pure cleanup of stuck/dead state (delete a failed hook Job so the *next*
legitimate sync creates a fresh one — not a config change, just clearing
wreckage). Confirm with the user first if in doubt.
- One-time bootstrap actions with a genuine circular dependency (Vault
`operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md`
equivalent scripts).
## Hard Rules (Never Violate)
🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
node — all replicas are pinned to that node name. Renaming orphans its Longhorn
node CR and faults every volume (permanent data loss). Rename/reprovision only
the dedicated CPs (.163/.166), never the data node.
🔴 **Control-plane etcd must advertise on the LAN.** Keep
`cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane
template — without it Talos advertises on the WireGuard IP and new members hang
as non-promoting etcd learners.
🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit:
```bash
terraform fmt -recursive terraform/
```
Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise.
🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck:
🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer.
🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages.
🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear.
🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads.
🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
add` in Alpine-based images** — apk's working directories and most of
`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log:
Permission denied`. Use `/tmp` (world-writable) for any binary you need to
download/install at runtime, and extend `PATH` rather than writing to
`/usr/local/bin`.
🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or
wrong-schema key is silently a no-op, not an error.** Confirmed root cause
of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual
chart version pinned (`temporalio/helm-charts@0.74.0`) uses a flat
`server.config.persistence.<store>.driver/.sql` schema, but the values file
used the *newer* chart's `datastores:`-wrapped schema (introduced in a
later major version) — silently ignored, so persistence stayed on the
chart's Cassandra default the entire time. **Before assuming "this chart
doesn't support X," clone the chart at the exact pinned tag/version and run
`helm template` with your real values — diff the rendered output, don't
trust values.yaml comments/examples from the chart's current `main`
branch, which may not match your pinned version's schema at all.**
## Workflow
- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push**
- If fmt check fails in CI, fix locally, commit fmt changes, push again
- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize <dir>/` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`)
## Documentation Map
- `CLAUDE.md` (this file) — private, cluster-specific, always current
- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs)
- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a
deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that
does not reflect actual current practice (100% ArgoCD GitOps as of this
writing). Treat as historical reference only until rewritten; do not
follow their deployment procedures literally.
- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism
