docs: rewrite CLAUDE.md for ArgoCD GitOps, track in git

.gitignore

@@ -38,8 +38,9 @@ k8s/storage/test/test
 *.key
 *.conf
 
-# Allowed markdown: CLAUDE.example.md, README.md, TROUBLESHOOTING.md
+# CLAUDE.md is now version-controlled (was previously excluded as a
-CLAUDE.md
+# private-notes file; contains no secrets - just architecture, IPs
+# [private RFC1918 space], and operational lessons, same bar as README.md).
 
 # Terraform state and cache (local files, remote state in MinIO)
 .terraform/

CLAUDE.example.md

@@ -1,86 +1,210 @@
-# CLAUDE.md — Homelab Integration Guide
+# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template)
 
-**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed).
+> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own
+> node IPs/hostnames/secrets, and delete this notice. Nothing in this file
+> should contain real credentials, real IPs beyond illustrative examples, or
+> anything that would matter if this file became public. It's meant to be
+> shared across homelabs running similar hardware/topology (3-node bare-metal
+> Talos Kubernetes + ArgoCD GitOps), not just this one.
 
-## Cluster Topology (3 control-plane HA, since 2026-07-20)
+**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a
+full observability stack, SSO via Authentik, secret management via Vault, and
+CI/CD infrastructure (self-hosted git forge + Argo CD).
+
+## Cluster Topology (adjust to your hardware)
 
 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
-| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
+| `<node-1>` | `<ip>` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) |
-| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
+| `<node-2>` | `<ip>` | az-b | dedicated (`NoSchedule`) | none |
-| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+| `<node-3>` | `<ip>` | az-c | dedicated (`NoSchedule`) | none |
 
-3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
+If your storage layer (Longhorn, local-path, etc.) only runs on one node,
-holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster`
+every stateful workload pinned to that storage class is effectively
-= 1 instance). API endpoint is single-homed to `.213` (no VIP yet).
+single-instance regardless of your control-plane HA count — document that
+explicitly here, it changes your failure-mode assumptions everywhere else.
+
+## Deployment Model: ArgoCD GitOps (app-of-apps)
+
+```
+git commit → git push (your git forge) → ArgoCD auto-sync → cluster
+```
+
+Structure:
+- One root `Application` (`k8s/argocd/root/`) pointing at a directory of
+  child `Application` manifests (`k8s/argocd/apps/*.yaml`)
+- Each child Application is either:
+  - A remote Helm chart + a **second** git source (`ref: values`) supplying
+    just the values file — lets you pin an upstream chart version while
+    keeping your values under normal git history/review
+  - A plain git directory of raw manifests (optionally with a
+    `kustomization.yaml`)
+- `argocd.argoproj.io/sync-wave` annotations control ordering across
+  Applications (lower number syncs first)
+
+**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except:
+- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook
+  Job so the next real sync can create a fresh one) — this is not a config
+  change, just clearing wreckage that GitOps itself won't clean up
+  automatically (see Gotchas below)
+- Genuine one-time bootstrap circular dependencies (e.g. Vault
+  `operator init`/unseal — nothing can configure Vault's own unseal keys
+  before Vault has generated them)
+
+## Hard Rules (adapt freely, but keep something like these)
+
+🔴 **Identify and document your storage/stateful-singleton node explicitly.**
+Whatever node holds your CSI driver's data (Longhorn, local-path, etc.),
+renaming or wiping it orphans every PVC pinned there. Name it here, in
+caps, so nobody "cleans up" it by accident.
+
+🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay,
+pin the advertised subnet explicitly** (e.g. Talos's
+`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the
+wrong interface and new members hang as non-promoting learners.
+
+🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit
+that touches infra code.** Wire this into CI as a hard gate, not a
+suggestion.
+
+🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never
+manually mutate resources it manages.** State drift is the single most
+common cause of "why did my last apply undo my manual fix" confusion.
+Fix the source, re-apply/re-sync, never bypass.
+
+🔴 **Never delete a PVC without confirming replica count / backup
+freshness first.** This is always a one-way door.
+
+🔴 **Decide your commit message convention up front and enforce it.**
+(This template's origin project uses: no co-authored-by footers, single-line
+commit summarizing what/why, solo-authorship assumption — adjust to your
+team's norms.)
+
+🔴 **Decide your git workflow (rebase vs merge) up front and stick to it**
+cluster-wide, across every contributor/agent working in the repo.
+
+🔴 **Long-running commands should not block a synchronous session** — run
+them in the background and poll, especially anything that waits on a
+Kubernetes rollout, an image pull, or a Terraform apply.
+
+## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab)
+
+🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist
+silently drops anything you forget to list.** No error, no drift shown in
+ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that
+never included your new file. Always run `kubectl kustomize <dir>/`
+locally before pushing to confirm exactly what ArgoCD will build.
+
+🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites
+`metadata.namespace` on every resource it builds** — including RBAC
+bindings deliberately targeting a *different* namespace (e.g. granting a
+ServiceAccount in namespace A read access to Secrets in namespace B). If
+any manifest needs cross-namespace RBAC, either drop the transformer
+(safe if every resource already sets its own explicit namespace) or give
+that manifest its own Application/directory.
+
+🟠 **PreSync hooks run before an Application's own normal resources are
+synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as
+plain (non-hook) resources in the *same* Application will deadlock — it
+tries to start before its own permissions exist. Use PostSync instead if
+the hook needs resources from its own Application, or move the
+prerequisite RBAC into an earlier sync-wave Application.
+
+🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a
+hook Job finishes (success, or exhausts `backoffLimit`), it's only
+deleted+recreated during an *actual new Sync operation* — not by passive
+drift detection, even with `automated.selfHeal: true` on. If you fix a
+broken hook's spec and push, the Application's `status.sync.revision` can
+show "caught up" while the live hook resource is still the old, broken
+one, because no new operation actually re-ran it. To force it: delete the
+stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's
+stuck `Terminating`), and if that alone doesn't trigger a fresh full sync,
+delete + re-`kubectl apply -f` the Application object itself.
+
+🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy
+hostname that only listens on 80/443, don't use a non-standard port in the
+URL** — it'll silently time out trying to reach a port the proxy never
+opened, and depending on your setup this can block *every* Application's
+sync simultaneously (repo-server can't fetch git refs for anything).
+
+🟠 **Don't pin exact version tags for images from registries that don't
+guarantee tag retention** (Bitnami stopped publishing versioned tags for
+free-tier images in 2025 — only `latest` + sha256 digests remain). Verify
+a tag actually exists before pinning it, or prefer minimal base images +
+a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a
+static binary) to avoid depending on any third party's tagging policy.
+
+🟠 **Non-root containers can't `apk add`/`apt install` in most default
+base images** — package manager directories are root-owned. Use a
+world-writable scratch dir (`/tmp`) for anything you need to
+download/install at runtime instead.
+
+🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a
+values schema copied from the wrong chart *version's* docs/examples, is
+silently a no-op — not an error. Before concluding "this chart doesn't
+support X," clone the chart at your exact pinned version/tag and run
+`helm template` against your real values file, then diff the rendered
+output. Don't trust a chart's current `main`-branch example values file
+if you're pinned to an older release — schemas do change between major
+versions without warning in your own values file.
 
 ## Service Integration Routes
 
-**New service? Pick your stack below:**
+**New service? Pick your stack below** (adjust doc paths to match your repo):
 
 | Need | Doc | Example |
 |------|-----|---------|
 | **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens |
-| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ |
+| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ |
 | **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend |
-| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync |
+| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync |
 | **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines |
 | **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas |
 | **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts |
 | **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal |
 | **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS |
 
-## Cluster Essentials
+## Cluster Essentials (fill in your own inventory)
 
-**22 namespaces, 18 releases:**
+**Architecture principles (adjust to taste, but these travel well):**
-```
+- Immutable OS (Talos, or similar — no SSH, fully declarative config)
-Core: cert-manager, ingress-nginx, kube-system, cilium
+- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in
-Storage: longhorn-system, storage (MinIO)
+  git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env`
-Data: ddb (PostgreSQL), iam (Authentik + Vault)
+- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap
-Observability: logging (Loki + Grafana), monitoring (Prometheus)
+  documented separately (chicken-and-egg: ArgoCD needs to exist before it
-Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend
+  can deploy itself declaratively — document your exact bootstrap steps)
-```
+- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI
+  runner; the runner only needs push access to git, ArgoCD does the rest
+- Federated OIDC (one identity provider fronting every service that
+  supports it)
 
-**Architecture principles:**
+## Deployment Checklist (per new service)
-- Immutable OS (Talos — no SSH, declarative config)
-- Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env`
-- ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md`
-- Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync
-- Federated OIDC (Authentik provider for all services)
 
-
+- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics
-## Deployment Checklist
+- [ ] All credentials in your secrets backend (never in plain values.yaml,
-
+      pod env directly, or committed anywhere in cleartext)
-- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor
+- [ ] Ingress rule with TLS, if externally reachable
-- [ ] All credentials in Vault (never in pod env, ConfigMap, or code)
+- [ ] Dashboard + alert rules, if metrics are exposed
-- [ ] Ingress rule in `k8s/ingress/` with TLS cert
+- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file,
-- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-<name>.yaml`
+      **not** a standalone `helm install`/`kubectl apply` run by hand
-- [ ] Alert rules in `k8s/monitoring/alerts/svc-<name>-rules.yaml` (if needed)
+- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`,
-- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies
+      `kubectl kustomize <dir>/` (if applicable), or `helm template` against
-
+      the exact pinned chart version (if Helm-sourced)
-## Hard Rules
+- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches
-
+      your new commit — not just that `status.sync.status` says `Synced`
-1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner)
+      (see Gotchas — a stale hook can hide behind an otherwise-current app)
-2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."`
-3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`)
-4. **External services via Ingress** — All public endpoints via TLS (homelab-ca)
-5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault
-6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs.
-7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners.
 
 ## Git & Release
 
-**Multi-remote push:**
+**Incremental commits (service-layer grouped) tend to age well:**
-```bash
-git push origin main
-```
-
-**Incremental commits (service-layer grouped):**
 - Foundation & Docs
-- Helmfile & Core Infra
+- Core Infra (CNI, ingress, cert management, storage)
-- Storage Layer
 - Observability Stack
 - IAM & Secrets
 - CI/CD & GitOps
-- Messaging Infrastructure
+- Messaging / Data Infrastructure
 - Applications & Utilities
+
+Grouping by layer (rather than by day or by "misc fixes") makes it much
+easier to `git log --oneline -- <path>` your way back to *why* a given
+piece of config looks the way it does, months later.

CLAUDE.md

@@ -0,0 +1,181 @@
+# CLAUDE.md — Homelab Project Reference
+
+## Cluster Topology (3 control-plane HA)
+
+| Node | IP | Zone | Scheduling | Storage |
+|------|----|----|-----------|---------|
+| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
+| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
+| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+
+3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
+holds storage → stateful services are single-instance. Full detail + gotchas in
+`USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`.
+
+## Deployment Model: ArgoCD GitOps (helmfile is retired)
+
+**helmfile.yaml.gotmpl and the `core` CLI's helmfile-era workflows described in
+`USAGE.md`/`project-usage/*.md` are STALE.** Actual practice, 100% of the time:
+
+```
+git commit → git push (Forgejo) → ArgoCD auto-sync → cluster
+```
+
+App-of-apps structure: `k8s/argocd/root/homelab-root.yaml` (root Application)
+→ `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each
+`argocd.argoproj.io/sync-wave` annotated) → each Application points at either
+a remote Helm chart (+ a second `ref: values` git source for the values file)
+or a plain git directory of raw manifests.
+
+**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except:
+- Pure cleanup of stuck/dead state (delete a failed hook Job so the *next*
+  legitimate sync creates a fresh one — not a config change, just clearing
+  wreckage). Confirm with the user first if in doubt.
+- One-time bootstrap actions with a genuine circular dependency (Vault
+  `operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md`
+  equivalent scripts).
+
+## Hard Rules (Never Violate)
+
+🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
+node — all replicas are pinned to that node name. Renaming orphans its Longhorn
+node CR and faults every volume (permanent data loss). Rename/reprovision only
+the dedicated CPs (.163/.166), never the data node.
+
+🔴 **Control-plane etcd must advertise on the LAN.** Keep
+`cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane
+template — without it Talos advertises on the WireGuard IP and new members hang
+as non-promoting etcd learners.
+
+🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit:
+```bash
+terraform fmt -recursive terraform/
+```
+Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise.
+
+🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck:
+1. Update Terraform config (variables.tf, *.tf files)
+2. Run `terraform apply` (locally or via CI/CD)
+3. Never bypass with manual kubectl operations
+
+🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
+
+🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer.
+
+🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages.
+
+🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear.
+
+🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads.
+
+🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
+
+## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster)
+
+🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently
+drops anything not listed — no error, no drift shown.** ArgoCD reports
+`Synced/Healthy` against a manifest set that never included the missing
+file at all. Symptom: you commit+push a new manifest, ArgoCD says
+"Synced", but the resource never appears in-cluster. Fix: check every
+`kustomization.yaml` along the app's source path actually lists your new
+file. `kubectl kustomize <dir>/` locally reproduces exactly what ArgoCD
+will apply — always verify with it before pushing.
+
+🟠 **A `kustomization.yaml` top-level `namespace:` transformer rewrites
+`metadata.namespace` on *every* resource it builds — including RBAC
+RoleBindings deliberately targeting a *different* namespace.** If any
+manifest in that directory needs cross-namespace resources (e.g. a
+RoleBinding granting access to Secrets in another namespace), either drop
+the transformer (safe if every resource already sets its own explicit
+namespace) or move that manifest to its own directory/Application.
+
+🟠 **PreSync hooks run *before* an Application's own normal (non-hook)
+resources are synced.** A PreSync-hooked Job that depends on a
+ServiceAccount/RBAC defined as plain resources in the *same* Application
+deadlocks: the Job tries to start before its own ServiceAccount exists.
+Confirmed live — Job sat "Running" for 14+ minutes producing zero pods,
+`job-controller` event log showed `serviceaccount ... not found` on every
+retry. Fix: use PostSync instead (runs after that app's own resources are
+applied), or split the hook into its own earlier-sync-wave Application.
+
+🟠 **ArgoCD hooks (PreSync/PostSync) are NOT continuously reconciled by
+`selfHeal` the way normal resources are.** Once a hook Job completes
+(success or exhausts `backoffLimit` into Failed), it only gets
+deleted+recreated (per `hook-delete-policy: BeforeHookCreation`) during an
+**actual new Sync operation** — not from passive drift detection, even
+with `automated.selfHeal: true`. If you fix a hook Job's spec (image,
+command, RBAC) and push, `status.sync.revision` may show "caught up" while
+the *live* hook resource is still running the old, broken spec — because
+no new operation actually re-ran it. To force a real resync: delete the
+stuck Job (clear the `argocd.argoproj.io/hook-finalizer` if it's stuck
+`Terminating`), and if that alone doesn't trigger a fresh full sync, delete
++ `kubectl apply -f` the Application object itself (re-reads current git
+HEAD, starts a genuinely new operation, no cascade-delete of underlying
+resources since Applications don't carry a cascade finalizer by default —
+confirm with `kubectl get app <name> -o jsonpath='{.metadata.finalizers}'`
+first).
+
+🟠 **ArgoCD's repo-server caches rendered manifests (~120s TTL by default,
+`argocd-cm` → `timeout.reconciliation`).** If you change a values file and
+`Application` fields both, sometimes the old rendering wins on the next
+sync. Restart `argocd-repo-server` after big multi-source/values changes if
+sync behavior looks stale.
+
+🟠 **ArgoCD `repoURL` pointing at a hostname that CoreDNS rewrites to the
+nginx ingress controller (for TLS termination) breaks if the URL includes
+a non-standard port.** nginx only listens on 80/443 — `http://host:3000/...`
+silently times out (`context deadline exceeded`) if `host` resolves to the
+ingress controller, not the actual backend service. This blocked **every
+single Application's sync** cluster-wide simultaneously (all showed
+`Unknown` sync status) because the repo-server couldn't fetch git refs at
+all. Use `https://host/...` (no port) so nginx's default TLS cert + normal
+443 routing handles it.
+
+🟠 **Bitnami Docker Hub images no longer publish versioned tags (2025
+policy change) — only `latest` and sha256-pinned digests remain for their
+free tier.** A pinned tag like `bitnami/kubectl:1.30` will 404/ImagePullBackOff
+forever. Verify tags exist first: `curl -s "https://hub.docker.com/v2/repositories/<org>/<image>/tags?page_size=25" | jq -r '.results[].name'`.
+Prefer avoiding third-party utility images entirely where possible — e.g.
+`python:3.12-alpine` + stdlib `urllib.request` to fetch a static binary
+(kubectl) avoids depending on any registry's tagging policy at all.
+
+🟠 **Non-root containers (`runAsNonRoot: true`, non-zero UID) can't `apk
+add` in Alpine-based images** — apk's working directories and most of
+`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log:
+Permission denied`. Use `/tmp` (world-writable) for any binary you need to
+download/install at runtime, and extend `PATH` rather than writing to
+`/usr/local/bin`.
+
+🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or
+wrong-schema key is silently a no-op, not an error.** Confirmed root cause
+of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual
+chart version pinned (`temporalio/[email protected]`) uses a flat
+`server.config.persistence.<store>.driver/.sql` schema, but the values file
+used the *newer* chart's `datastores:`-wrapped schema (introduced in a
+later major version) — silently ignored, so persistence stayed on the
+chart's Cassandra default the entire time. **Before assuming "this chart
+doesn't support X," clone the chart at the exact pinned tag/version and run
+`helm template` with your real values — diff the rendered output, don't
+trust values.yaml comments/examples from the chart's current `main`
+branch, which may not match your pinned version's schema at all.**
+
+## Workflow
+
+- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push**
+- If fmt check fails in CI, fix locally, commit fmt changes, push again
+- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize <dir>/` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`)
+
+## Documentation Map
+
+- `CLAUDE.md` (this file) — private, cluster-specific, always current
+- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs)
+- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a
+  deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that
+  does not reflect actual current practice (100% ArgoCD GitOps as of this
+  writing). Treat as historical reference only until rewritten; do not
+  follow their deployment procedures literally.
+- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism
+
+---
+
+**Last updated:** 2026-07-22

project-usage/gitops-workflow.md

@@ -0,0 +1,200 @@
+# GitOps Workflow (ArgoCD) — Current Practice
+
+**This is the actual, current deployment workflow.** Every other doc in this
+directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
+`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
+retired. This file exists to be the accurate replacement for "how do I
+actually deploy/change something" until those older docs get a full rewrite.
+
+## The Loop
+
+```
+1. Edit files under k8s/
+2. Validate locally (see "Validation" below — do not skip this)
+3. git add -A && git commit -m "..." && git push
+4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
+   up on its next reconciliation cycle (default ~3 min, or immediately if
+   you trigger a refresh)
+5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
+   matches your new commit hash - not just that status.sync.status says
+   "Synced" (a stale hook resource can hide behind an otherwise-current app,
+   see "ArgoCD Hooks" below)
+```
+
+There is no `helmfile apply`, no `core iam create-app`, no manual `helm
+install` in the current workflow. If you find yourself reaching for any of
+those, stop — figure out the ArgoCD-native equivalent instead.
+
+## App-of-Apps Structure
+
+```
+k8s/argocd/root/homelab-root.yaml       # the one Application ArgoCD bootstraps by hand
+  → source: k8s/argocd/apps/            # directory of child Application manifests
+      00-substrate.yaml                 # sync-wave 0: cert-manager, ingress-nginx, CRDs
+      00-secrets.yaml                   # sync-wave 0: SOPS secrets plugin
+      05-networking.yaml                # sync-wave... etc, ascending
+      10-storage-observability.yaml
+      20-logging.yaml
+      30-security.yaml                  # iam (Authentik, Vault)
+      40-data.yaml                      # CNPG postgres
+      50-messaging.yaml                 # Kafka/sqs
+      60-applications.yaml              # end-user workloads (Temporal, Portainer, etc.)
+```
+
+Each child `Application` either:
+- Points at a **remote Helm chart** with a **second git source** (`ref:
+  values`) supplying just the values file — lets you pin the chart version
+  independently while your values live under normal review/history, e.g.:
+  ```yaml
+  spec:
+    sources:
+      - repoURL: https://example.com/helm-charts
+        chart: some-chart
+        targetRevision: "1.2.3"
+        helm:
+          valueFiles:
+            - $values/k8s/applications/foo/foo-values.yaml
+      - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+        targetRevision: main
+        ref: values
+  ```
+- Or points at a **plain git directory** of raw manifests, optionally with
+  its own `kustomization.yaml`.
+
+`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
+controls ordering — lower N syncs first. Within a single Application,
+resource-level sync-wave annotations further order that app's own
+resources.
+
+## Validation Before Every Push
+
+Pick whichever applies:
+
+```bash
+# Plain manifests, no kustomization.yaml in the directory
+kubectl apply --dry-run=client -f path/to/file.yaml
+
+# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
+# a plain `apply --dry-run` on individual files will NOT catch a resource
+# missing from the resources: allowlist
+kubectl kustomize path/to/dir/
+
+# Helm-sourced Application - render against the EXACT pinned chart version,
+# not whatever `helm repo` happens to have cached, and not the chart's
+# current main-branch example values (schemas can differ between versions)
+git clone --depth 1 <chart-repo-url> /tmp/chart-check
+cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
+helm dependency build  # if the chart has subchart dependencies
+helm template . --values /path/to/your/values.yaml --namespace <ns>
+```
+
+## ArgoCD Hooks (PreSync / PostSync / Sync)
+
+Used for one-shot Jobs that need to run at a specific point in an
+Application's sync (schema migrations, secret-copy jobs, provisioning
+scripts). Two properties that bite people:
+
+1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
+   PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
+   defined as *plain* (non-hook) resources in the same Application will
+   deadlock — those get created during the Sync phase, which is *after*
+   PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
+   describe job` shows `FailedCreate ... serviceaccount ... not found`
+   repeating forever. **If your hook needs resources from its own
+   Application, make it PostSync, not PreSync.**
+
+2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
+   failed hook Job only gets deleted+recreated (per
+   `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
+   operation* — not from ambient drift detection. If you fix a broken
+   hook's spec (image, RBAC, command) and push, the Application's
+   `status.sync.revision` can update to show "caught up" (because the
+   *normal*, non-hook resources genuinely match) while the *live* hook Job
+   is still running the old broken spec, because no fresh sync operation
+   actually re-executed it.
+
+   **To force it, in order of preference:**
+   ```bash
+   # 1. Just delete the stuck/failed job - if a legitimate future sync
+   #    happens for any other reason, this clears the way for it
+   kubectl -n <ns> delete job <hook-job-name>
+
+   # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
+   #    deletion because the operation tracking it is itself stuck):
+   kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
+   kubectl -n <ns> patch job <hook-job-name> --type json \
+     -p '[{"op":"remove","path":"/metadata/finalizers"}]'
+
+   # 3. If deleting the job alone doesn't trigger a genuinely fresh sync
+   #    (check: does a new job appear using your LATEST commit's spec?
+   #    compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
+   #    against what your latest commit says), the Application's own
+   #    operation state is stuck. Confirm it has no cascade finalizer first
+   #    (Applications don't carry one by default - only delete if this is
+   #    empty):
+   kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
+   # then:
+   kubectl -n argocd delete application <app-name>
+   kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
+   ```
+   Step 3 re-reads current git HEAD from scratch and starts a genuinely new
+   operation — this is the reliable way to actually pick up a hook fix when
+   steps 1-2 don't unstick it.
+
+## kustomization.yaml Pitfalls
+
+- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
+  manifest file to a directory that has a `kustomization.yaml`, forget to
+  list it under `resources:`, and ArgoCD will silently never apply it — no
+  error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
+  <dir>/` locally before pushing to see exactly what will be built.
+- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
+  every resource in the build**, including RBAC bindings that deliberately
+  target a *different* namespace (e.g. a RoleBinding granting cross-
+  namespace Secret access for a sync job's ServiceAccount). If any manifest
+  in the directory needs to live in a different namespace than the
+  transformer specifies, either remove the transformer (safe if every
+  manifest already sets its own explicit `namespace:`) or move that
+  manifest to its own directory/Application entirely.
+
+## Cross-Namespace Secrets
+
+Kubernetes Secrets are strictly namespace-scoped — a Deployment in
+namespace A cannot reference a Secret living in namespace B via
+`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
+CNPG auto-generates DB credentials in its own operator namespace) but a
+consumer lives in a different namespace, you need an explicit copy
+mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
+`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
+example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
+(one per namespace involved) that reads the source Secret and re-creates it
+in the target namespace. Give this its own Application/sync-wave (earlier
+than whatever consumes the copied Secret) rather than folding it into an
+existing Application that has a namespace-transforming `kustomization.yaml`
+(see above).
+
+## Common Failure: Everything Shows "Unknown" Sync Status At Once
+
+If *every* Application (not just one) suddenly shows `Unknown` sync status
+simultaneously, check the Application controller's logs for the actual
+`repoURL` fetch error before assuming something is wrong with any
+individual app's manifests:
+
+```bash
+kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
+```
+
+Common cause in a homelab with CoreDNS rewriting your git host to the
+ingress controller for internal traffic: if `repoURL` uses a non-standard
+port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
+hostname to the ingress controller service (which only listens on 80/443),
+every fetch attempt times out. Fix: use the default-port HTTPS URL
+(`https://forgejo.example.com/...`) so it actually reaches the ingress
+controller correctly.
+
+## See Also
+
+- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
+  cross-referenced from the main project reference
+- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
+  applicable regardless of deployment mechanism