docs: rewrite CLAUDE.md/CLAUDE.example.md for ArgoCD GitOps, add gitops-workflow.md

CLAUDE.md and the entire project-usage/ tree were written for a helmfile + 'core iam'/'core secrets' CLI workflow that has been fully retired - actual practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD sync), confirmed by an extended live debugging session that touched Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and multiple ArgoCD Applications, none of which involved helmfile or core at any point. CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every hard-won lesson from this session with live evidence for each: - kustomization.yaml resources: allowlists silently dropping new files - kustomization.yaml namespace: transformers clobbering cross-namespace RBAC - PreSync hooks deadlocking on same-Application RBAC dependencies - ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely new sync operation to pick up fixes - repo-server manifest caching - repoURL port mismatches breaking every Application's sync simultaneously when routed through an ingress-rewriting CoreDNS rule - Bitnami's 2025 versioned-tag retirement - apk-as-non-root permission failures - Helm's lack of values.yaml schema validation (root cause of the Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was a schema mismatch between the pinned chart version and a newer chart's values.yaml example, silently a no-op) CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic template (explicit notice at top) - same lessons, genericized away from this specific homelab's IPs/hostnames/secrets, intended to be reusable by anyone running a similar bare-metal Talos + ArgoCD topology. project-usage/gitops-workflow.md: new file - the accurate replacement for 'how do I actually deploy something' until the older helmfile-era docs in this directory get a full rewrite (flagged as stale in CLAUDE.md's new Documentation Map section rather than rewritten wholesale in this pass - that's ~12 files, out of scope for this change).
2026-07-21 20:16:44 -07:00
parent 0a323fc039
commit af00467b2b
2 changed files with 379 additions and 55 deletions
@@ -0,0 +1,200 @@
+# GitOps Workflow (ArgoCD) — Current Practice
+
+**This is the actual, current deployment workflow.** Every other doc in this
+directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
+`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
+retired. This file exists to be the accurate replacement for "how do I
+actually deploy/change something" until those older docs get a full rewrite.
+
+## The Loop
+
+```
+1. Edit files under k8s/
+2. Validate locally (see "Validation" below — do not skip this)
+3. git add -A && git commit -m "..." && git push
+4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
+   up on its next reconciliation cycle (default ~3 min, or immediately if
+   you trigger a refresh)
+5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
+   matches your new commit hash - not just that status.sync.status says
+   "Synced" (a stale hook resource can hide behind an otherwise-current app,
+   see "ArgoCD Hooks" below)
+```
+
+There is no `helmfile apply`, no `core iam create-app`, no manual `helm
+install` in the current workflow. If you find yourself reaching for any of
+those, stop — figure out the ArgoCD-native equivalent instead.
+
+## App-of-Apps Structure
+
+```
+k8s/argocd/root/homelab-root.yaml       # the one Application ArgoCD bootstraps by hand
+  → source: k8s/argocd/apps/            # directory of child Application manifests
+      00-substrate.yaml                 # sync-wave 0: cert-manager, ingress-nginx, CRDs
+      00-secrets.yaml                   # sync-wave 0: SOPS secrets plugin
+      05-networking.yaml                # sync-wave... etc, ascending
+      10-storage-observability.yaml
+      20-logging.yaml
+      30-security.yaml                  # iam (Authentik, Vault)
+      40-data.yaml                      # CNPG postgres
+      50-messaging.yaml                 # Kafka/sqs
+      60-applications.yaml              # end-user workloads (Temporal, Portainer, etc.)
+```
+
+Each child `Application` either:
+- Points at a **remote Helm chart** with a **second git source** (`ref:
+  values`) supplying just the values file — lets you pin the chart version
+  independently while your values live under normal review/history, e.g.:
+  ```yaml
+  spec:
+    sources:
+      - repoURL: https://example.com/helm-charts
+        chart: some-chart
+        targetRevision: "1.2.3"
+        helm:
+          valueFiles:
+            - $values/k8s/applications/foo/foo-values.yaml
+      - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
+        targetRevision: main
+        ref: values
+  ```
+- Or points at a **plain git directory** of raw manifests, optionally with
+  its own `kustomization.yaml`.
+
+`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
+controls ordering — lower N syncs first. Within a single Application,
+resource-level sync-wave annotations further order that app's own
+resources.
+
+## Validation Before Every Push
+
+Pick whichever applies:
+
+```bash
+# Plain manifests, no kustomization.yaml in the directory
+kubectl apply --dry-run=client -f path/to/file.yaml
+
+# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
+# a plain `apply --dry-run` on individual files will NOT catch a resource
+# missing from the resources: allowlist
+kubectl kustomize path/to/dir/
+
+# Helm-sourced Application - render against the EXACT pinned chart version,
+# not whatever `helm repo` happens to have cached, and not the chart's
+# current main-branch example values (schemas can differ between versions)
+git clone --depth 1 <chart-repo-url> /tmp/chart-check
+cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
+helm dependency build  # if the chart has subchart dependencies
+helm template . --values /path/to/your/values.yaml --namespace <ns>
+```
+
+## ArgoCD Hooks (PreSync / PostSync / Sync)
+
+Used for one-shot Jobs that need to run at a specific point in an
+Application's sync (schema migrations, secret-copy jobs, provisioning
+scripts). Two properties that bite people:
+
+1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
+   PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
+   defined as *plain* (non-hook) resources in the same Application will
+   deadlock — those get created during the Sync phase, which is *after*
+   PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
+   describe job` shows `FailedCreate ... serviceaccount ... not found`
+   repeating forever. **If your hook needs resources from its own
+   Application, make it PostSync, not PreSync.**
+
+2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
+   failed hook Job only gets deleted+recreated (per
+   `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
+   operation* — not from ambient drift detection. If you fix a broken
+   hook's spec (image, RBAC, command) and push, the Application's
+   `status.sync.revision` can update to show "caught up" (because the
+   *normal*, non-hook resources genuinely match) while the *live* hook Job
+   is still running the old broken spec, because no fresh sync operation
+   actually re-executed it.
+
+   **To force it, in order of preference:**
+   ```bash
+   # 1. Just delete the stuck/failed job - if a legitimate future sync
+   #    happens for any other reason, this clears the way for it
+   kubectl -n <ns> delete job <hook-job-name>
+
+   # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
+   #    deletion because the operation tracking it is itself stuck):
+   kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
+   kubectl -n <ns> patch job <hook-job-name> --type json \
+     -p '[{"op":"remove","path":"/metadata/finalizers"}]'
+
+   # 3. If deleting the job alone doesn't trigger a genuinely fresh sync
+   #    (check: does a new job appear using your LATEST commit's spec?
+   #    compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
+   #    against what your latest commit says), the Application's own
+   #    operation state is stuck. Confirm it has no cascade finalizer first
+   #    (Applications don't carry one by default - only delete if this is
+   #    empty):
+   kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
+   # then:
+   kubectl -n argocd delete application <app-name>
+   kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
+   ```
+   Step 3 re-reads current git HEAD from scratch and starts a genuinely new
+   operation — this is the reliable way to actually pick up a hook fix when
+   steps 1-2 don't unstick it.
+
+## kustomization.yaml Pitfalls
+
+- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
+  manifest file to a directory that has a `kustomization.yaml`, forget to
+  list it under `resources:`, and ArgoCD will silently never apply it — no
+  error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
+  <dir>/` locally before pushing to see exactly what will be built.
+- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
+  every resource in the build**, including RBAC bindings that deliberately
+  target a *different* namespace (e.g. a RoleBinding granting cross-
+  namespace Secret access for a sync job's ServiceAccount). If any manifest
+  in the directory needs to live in a different namespace than the
+  transformer specifies, either remove the transformer (safe if every
+  manifest already sets its own explicit `namespace:`) or move that
+  manifest to its own directory/Application entirely.
+
+## Cross-Namespace Secrets
+
+Kubernetes Secrets are strictly namespace-scoped — a Deployment in
+namespace A cannot reference a Secret living in namespace B via
+`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
+CNPG auto-generates DB credentials in its own operator namespace) but a
+consumer lives in a different namespace, you need an explicit copy
+mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
+`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
+example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
+(one per namespace involved) that reads the source Secret and re-creates it
+in the target namespace. Give this its own Application/sync-wave (earlier
+than whatever consumes the copied Secret) rather than folding it into an
+existing Application that has a namespace-transforming `kustomization.yaml`
+(see above).
+
+## Common Failure: Everything Shows "Unknown" Sync Status At Once
+
+If *every* Application (not just one) suddenly shows `Unknown` sync status
+simultaneously, check the Application controller's logs for the actual
+`repoURL` fetch error before assuming something is wrong with any
+individual app's manifests:
+
+```bash
+kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
+```
+
+Common cause in a homelab with CoreDNS rewriting your git host to the
+ingress controller for internal traffic: if `repoURL` uses a non-standard
+port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
+hostname to the ingress controller service (which only listens on 80/443),
+every fetch attempt times out. Fix: use the default-port HTTPS URL
+(`https://forgejo.example.com/...`) so it actually reaches the ingress
+controller correctly.
+
+## See Also
+
+- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
+  cross-referenced from the main project reference
+- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
+  applicable regardless of deployment mechanism