diff --git a/.gitignore b/.gitignore index a38c7cb..5e8863a 100644 --- a/.gitignore +++ b/.gitignore @@ -38,8 +38,9 @@ k8s/storage/test/test *.key *.conf -# Allowed markdown: CLAUDE.example.md, README.md, TROUBLESHOOTING.md -CLAUDE.md +# CLAUDE.md is now version-controlled (was previously excluded as a +# private-notes file; contains no secrets - just architecture, IPs +# [private RFC1918 space], and operational lessons, same bar as README.md). # Terraform state and cache (local files, remote state in MinIO) .terraform/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f0a7fda --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,181 @@ +# CLAUDE.md — Homelab Project Reference + +## Cluster Topology (3 control-plane HA) + +| Node | IP | Zone | Scheduling | Storage | +|------|----|----|-----------|---------| +| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node | +| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none | +| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none | + +3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and +holds storage → stateful services are single-instance. Full detail + gotchas in +`USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`. + +## Deployment Model: ArgoCD GitOps (helmfile is retired) + +**helmfile.yaml.gotmpl and the `core` CLI's helmfile-era workflows described in +`USAGE.md`/`project-usage/*.md` are STALE.** Actual practice, 100% of the time: + +``` +git commit → git push (Forgejo) → ArgoCD auto-sync → cluster +``` + +App-of-apps structure: `k8s/argocd/root/homelab-root.yaml` (root Application) +→ `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each +`argocd.argoproj.io/sync-wave` annotated) → each Application points at either +a remote Helm chart (+ a second `ref: values` git source for the values file) +or a plain git directory of raw manifests. + +**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except: +- Pure cleanup of stuck/dead state (delete a failed hook Job so the *next* + legitimate sync creates a fresh one — not a config change, just clearing + wreckage). Confirm with the user first if in doubt. +- One-time bootstrap actions with a genuine circular dependency (Vault + `operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md` + equivalent scripts). + +## Hard Rules (Never Violate) + +🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage +node — all replicas are pinned to that node name. Renaming orphans its Longhorn +node CR and faults every volume (permanent data loss). Rename/reprovision only +the dedicated CPs (.163/.166), never the data node. + +🔴 **Control-plane etcd must advertise on the LAN.** Keep +`cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane +template — without it Talos advertises on the WireGuard IP and new members hang +as non-promoting etcd learners. + +🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit: +```bash +terraform fmt -recursive terraform/ +``` +Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise. + +🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck: +1. Update Terraform config (variables.tf, *.tf files) +2. Run `terraform apply` (locally or via CI/CD) +3. Never bypass with manual kubectl operations + +🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first. + +🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer. + +🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages. + +🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear. + +🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads. + +🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources. + +## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster) + +🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently +drops anything not listed — no error, no drift shown.** ArgoCD reports +`Synced/Healthy` against a manifest set that never included the missing +file at all. Symptom: you commit+push a new manifest, ArgoCD says +"Synced", but the resource never appears in-cluster. Fix: check every +`kustomization.yaml` along the app's source path actually lists your new +file. `kubectl kustomize /` locally reproduces exactly what ArgoCD +will apply — always verify with it before pushing. + +🟠 **A `kustomization.yaml` top-level `namespace:` transformer rewrites +`metadata.namespace` on *every* resource it builds — including RBAC +RoleBindings deliberately targeting a *different* namespace.** If any +manifest in that directory needs cross-namespace resources (e.g. a +RoleBinding granting access to Secrets in another namespace), either drop +the transformer (safe if every resource already sets its own explicit +namespace) or move that manifest to its own directory/Application. + +🟠 **PreSync hooks run *before* an Application's own normal (non-hook) +resources are synced.** A PreSync-hooked Job that depends on a +ServiceAccount/RBAC defined as plain resources in the *same* Application +deadlocks: the Job tries to start before its own ServiceAccount exists. +Confirmed live — Job sat "Running" for 14+ minutes producing zero pods, +`job-controller` event log showed `serviceaccount ... not found` on every +retry. Fix: use PostSync instead (runs after that app's own resources are +applied), or split the hook into its own earlier-sync-wave Application. + +🟠 **ArgoCD hooks (PreSync/PostSync) are NOT continuously reconciled by +`selfHeal` the way normal resources are.** Once a hook Job completes +(success or exhausts `backoffLimit` into Failed), it only gets +deleted+recreated (per `hook-delete-policy: BeforeHookCreation`) during an +**actual new Sync operation** — not from passive drift detection, even +with `automated.selfHeal: true`. If you fix a hook Job's spec (image, +command, RBAC) and push, `status.sync.revision` may show "caught up" while +the *live* hook resource is still running the old, broken spec — because +no new operation actually re-ran it. To force a real resync: delete the +stuck Job (clear the `argocd.argoproj.io/hook-finalizer` if it's stuck +`Terminating`), and if that alone doesn't trigger a fresh full sync, delete ++ `kubectl apply -f` the Application object itself (re-reads current git +HEAD, starts a genuinely new operation, no cascade-delete of underlying +resources since Applications don't carry a cascade finalizer by default — +confirm with `kubectl get app -o jsonpath='{.metadata.finalizers}'` +first). + +🟠 **ArgoCD's repo-server caches rendered manifests (~120s TTL by default, +`argocd-cm` → `timeout.reconciliation`).** If you change a values file and +`Application` fields both, sometimes the old rendering wins on the next +sync. Restart `argocd-repo-server` after big multi-source/values changes if +sync behavior looks stale. + +🟠 **ArgoCD `repoURL` pointing at a hostname that CoreDNS rewrites to the +nginx ingress controller (for TLS termination) breaks if the URL includes +a non-standard port.** nginx only listens on 80/443 — `http://host:3000/...` +silently times out (`context deadline exceeded`) if `host` resolves to the +ingress controller, not the actual backend service. This blocked **every +single Application's sync** cluster-wide simultaneously (all showed +`Unknown` sync status) because the repo-server couldn't fetch git refs at +all. Use `https://host/...` (no port) so nginx's default TLS cert + normal +443 routing handles it. + +🟠 **Bitnami Docker Hub images no longer publish versioned tags (2025 +policy change) — only `latest` and sha256-pinned digests remain for their +free tier.** A pinned tag like `bitnami/kubectl:1.30` will 404/ImagePullBackOff +forever. Verify tags exist first: `curl -s "https://hub.docker.com/v2/repositories///tags?page_size=25" | jq -r '.results[].name'`. +Prefer avoiding third-party utility images entirely where possible — e.g. +`python:3.12-alpine` + stdlib `urllib.request` to fetch a static binary +(kubectl) avoids depending on any registry's tagging policy at all. + +🟠 **Non-root containers (`runAsNonRoot: true`, non-zero UID) can't `apk +add` in Alpine-based images** — apk's working directories and most of +`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log: +Permission denied`. Use `/tmp` (world-writable) for any binary you need to +download/install at runtime, and extend `PATH` rather than writing to +`/usr/local/bin`. + +🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or +wrong-schema key is silently a no-op, not an error.** Confirmed root cause +of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual +chart version pinned (`temporalio/helm-charts@0.74.0`) uses a flat +`server.config.persistence..driver/.sql` schema, but the values file +used the *newer* chart's `datastores:`-wrapped schema (introduced in a +later major version) — silently ignored, so persistence stayed on the +chart's Cassandra default the entire time. **Before assuming "this chart +doesn't support X," clone the chart at the exact pinned tag/version and run +`helm template` with your real values — diff the rendered output, don't +trust values.yaml comments/examples from the chart's current `main` +branch, which may not match your pinned version's schema at all.** + +## Workflow + +- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push** +- If fmt check fails in CI, fix locally, commit fmt changes, push again +- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize /` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`) + +## Documentation Map + +- `CLAUDE.md` (this file) — private, cluster-specific, always current +- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs) +- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a + deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that + does not reflect actual current practice (100% ArgoCD GitOps as of this + writing). Treat as historical reference only until rewritten; do not + follow their deployment procedures literally. +- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism + +--- + +**Last updated:** 2026-07-22