docs: rewrite CLAUDE.md/CLAUDE.example.md for ArgoCD GitOps, add gitops-workflow.md

CLAUDE.md and the entire project-usage/ tree were written for a helmfile + 'core iam'/'core secrets' CLI workflow that has been fully retired - actual practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD sync), confirmed by an extended live debugging session that touched Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and multiple ArgoCD Applications, none of which involved helmfile or core at any point. CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every hard-won lesson from this session with live evidence for each: - kustomization.yaml resources: allowlists silently dropping new files - kustomization.yaml namespace: transformers clobbering cross-namespace RBAC - PreSync hooks deadlocking on same-Application RBAC dependencies - ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely new sync operation to pick up fixes - repo-server manifest caching - repoURL port mismatches breaking every Application's sync simultaneously when routed through an ingress-rewriting CoreDNS rule - Bitnami's 2025 versioned-tag retirement - apk-as-non-root permission failures - Helm's lack of values.yaml schema validation (root cause of the Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was a schema mismatch between the pinned chart version and a newer chart's values.yaml example, silently a no-op) CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic template (explicit notice at top) - same lessons, genericized away from this specific homelab's IPs/hostnames/secrets, intended to be reusable by anyone running a similar bare-metal Talos + ArgoCD topology. project-usage/gitops-workflow.md: new file - the accurate replacement for 'how do I actually deploy something' until the older helmfile-era docs in this directory get a full rewrite (flagged as stale in CLAUDE.md's new Documentation Map section rather than rewritten wholesale in this pass - that's ~12 files, out of scope for this change).
2026-07-21 20:16:44 -07:00
parent 0a323fc039
commit af00467b2b
2 changed files with 379 additions and 55 deletions
@@ -1,86 +1,210 @@
-# CLAUDE.md — Homelab Integration Guide
+# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template)
-**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed).
+> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own
 > node IPs/hostnames/secrets, and delete this notice. Nothing in this file
 > should contain real credentials, real IPs beyond illustrative examples, or
 > anything that would matter if this file became public. It's meant to be
 > shared across homelabs running similar hardware/topology (3-node bare-metal
 > Talos Kubernetes + ArgoCD GitOps), not just this one.
-## Cluster Topology (3 control-plane HA, since 2026-07-20)
+**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a
 full observability stack, SSO via Authentik, secret management via Vault, and
 CI/CD infrastructure (self-hosted git forge + Argo CD).
 ## Cluster Topology (adjust to your hardware)
 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
-| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
+| `<node-1>` | `<ip>` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) |
-| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
+| `<node-2>` | `<ip>` | az-b | dedicated (`NoSchedule`) | none |
-| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+| `<node-3>` | `<ip>` | az-c | dedicated (`NoSchedule`) | none |
-3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
+If your storage layer (Longhorn, local-path, etc.) only runs on one node,
-holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster`
+every stateful workload pinned to that storage class is effectively
-= 1 instance). API endpoint is single-homed to `.213` (no VIP yet).
+single-instance regardless of your control-plane HA count — document that
 explicitly here, it changes your failure-mode assumptions everywhere else.
 ## Deployment Model: ArgoCD GitOps (app-of-apps)
 ```
 git commit → git push (your git forge) → ArgoCD auto-sync → cluster
 ```
 Structure:
 - One root `Application` (`k8s/argocd/root/`) pointing at a directory of
  child `Application` manifests (`k8s/argocd/apps/*.yaml`)
 - Each child Application is either:
  - A remote Helm chart + a **second** git source (`ref: values`) supplying
    just the values file — lets you pin an upstream chart version while
    keeping your values under normal git history/review
  - A plain git directory of raw manifests (optionally with a
    `kustomization.yaml`)
 - `argocd.argoproj.io/sync-wave` annotations control ordering across
  Applications (lower number syncs first)
 **Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except:
 - Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook
  Job so the next real sync can create a fresh one) — this is not a config
  change, just clearing wreckage that GitOps itself won't clean up
  automatically (see Gotchas below)
 - Genuine one-time bootstrap circular dependencies (e.g. Vault
  `operator init`/unseal — nothing can configure Vault's own unseal keys
  before Vault has generated them)
 ## Hard Rules (adapt freely, but keep something like these)
 🔴 **Identify and document your storage/stateful-singleton node explicitly.**
 Whatever node holds your CSI driver's data (Longhorn, local-path, etc.),
 renaming or wiping it orphans every PVC pinned there. Name it here, in
 caps, so nobody "cleans up" it by accident.
 🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay,
 pin the advertised subnet explicitly** (e.g. Talos's
 `cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the
 wrong interface and new members hang as non-promoting learners.
 🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit
 that touches infra code.** Wire this into CI as a hard gate, not a
 suggestion.
 🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never
 manually mutate resources it manages.** State drift is the single most
 common cause of "why did my last apply undo my manual fix" confusion.
 Fix the source, re-apply/re-sync, never bypass.
 🔴 **Never delete a PVC without confirming replica count / backup
 freshness first.** This is always a one-way door.
 🔴 **Decide your commit message convention up front and enforce it.**
 (This template's origin project uses: no co-authored-by footers, single-line
 commit summarizing what/why, solo-authorship assumption — adjust to your
 team's norms.)
 🔴 **Decide your git workflow (rebase vs merge) up front and stick to it**
 cluster-wide, across every contributor/agent working in the repo.
 🔴 **Long-running commands should not block a synchronous session** — run
 them in the background and poll, especially anything that waits on a
 Kubernetes rollout, an image pull, or a Terraform apply.
 ## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab)
 🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist
 silently drops anything you forget to list.** No error, no drift shown in
 ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that
 never included your new file. Always run `kubectl kustomize <dir>/`
 locally before pushing to confirm exactly what ArgoCD will build.
 🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites
 `metadata.namespace` on every resource it builds** — including RBAC
 bindings deliberately targeting a *different* namespace (e.g. granting a
 ServiceAccount in namespace A read access to Secrets in namespace B). If
 any manifest needs cross-namespace RBAC, either drop the transformer
 (safe if every resource already sets its own explicit namespace) or give
 that manifest its own Application/directory.
 🟠 **PreSync hooks run before an Application's own normal resources are
 synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as
 plain (non-hook) resources in the *same* Application will deadlock — it
 tries to start before its own permissions exist. Use PostSync instead if
 the hook needs resources from its own Application, or move the
 prerequisite RBAC into an earlier sync-wave Application.
 🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a
 hook Job finishes (success, or exhausts `backoffLimit`), it's only
 deleted+recreated during an *actual new Sync operation* — not by passive
 drift detection, even with `automated.selfHeal: true` on. If you fix a
 broken hook's spec and push, the Application's `status.sync.revision` can
 show "caught up" while the live hook resource is still the old, broken
 one, because no new operation actually re-ran it. To force it: delete the
 stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's
 stuck `Terminating`), and if that alone doesn't trigger a fresh full sync,
 delete + re-`kubectl apply -f` the Application object itself.
 🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy
 hostname that only listens on 80/443, don't use a non-standard port in the
 URL** — it'll silently time out trying to reach a port the proxy never
 opened, and depending on your setup this can block *every* Application's
 sync simultaneously (repo-server can't fetch git refs for anything).
 🟠 **Don't pin exact version tags for images from registries that don't
 guarantee tag retention** (Bitnami stopped publishing versioned tags for
 free-tier images in 2025 — only `latest` + sha256 digests remain). Verify
 a tag actually exists before pinning it, or prefer minimal base images +
 a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a
 static binary) to avoid depending on any third party's tagging policy.
 🟠 **Non-root containers can't `apk add`/`apt install` in most default
 base images** — package manager directories are root-owned. Use a
 world-writable scratch dir (`/tmp`) for anything you need to
 download/install at runtime instead.
 🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a
 values schema copied from the wrong chart *version's* docs/examples, is
 silently a no-op — not an error. Before concluding "this chart doesn't
 support X," clone the chart at your exact pinned version/tag and run
 `helm template` against your real values file, then diff the rendered
 output. Don't trust a chart's current `main`-branch example values file
 if you're pinned to an older release — schemas do change between major
 versions without warning in your own values file.
 ## Service Integration Routes
-**New service? Pick your stack below:**
+**New service? Pick your stack below** (adjust doc paths to match your repo):
 | Need | Doc | Example |
 |------|-----|---------|
 | **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens |
-| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ |
+| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ |
 | **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend |
-| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync |
+| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync |
 | **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines |
 | **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas |
 | **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts |
 | **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal |
 | **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS |
-## Cluster Essentials
+## Cluster Essentials (fill in your own inventory)
-**22 namespaces, 18 releases:**
+**Architecture principles (adjust to taste, but these travel well):**
-```
+- Immutable OS (Talos, or similar — no SSH, fully declarative config)
-Core: cert-manager, ingress-nginx, kube-system, cilium
+- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in
-Storage: longhorn-system, storage (MinIO)
+  git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env`
-Data: ddb (PostgreSQL), iam (Authentik + Vault)
+- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap
-Observability: logging (Loki + Grafana), monitoring (Prometheus)
+  documented separately (chicken-and-egg: ArgoCD needs to exist before it
-Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend
+  can deploy itself declaratively — document your exact bootstrap steps)
-```
+- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI
  runner; the runner only needs push access to git, ArgoCD does the rest
 - Federated OIDC (one identity provider fronting every service that
  supports it)
-**Architecture principles:**
+## Deployment Checklist (per new service)
 - Immutable OS (Talos — no SSH, declarative config)
 - Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env`
 - ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md`
 - Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync
 - Federated OIDC (Authentik provider for all services)
-
+- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics
-## Deployment Checklist
+- [ ] All credentials in your secrets backend (never in plain values.yaml,
-
+      pod env directly, or committed anywhere in cleartext)
- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor
+- [ ] Ingress rule with TLS, if externally reachable
- [ ] All credentials in Vault (never in pod env, ConfigMap, or code)
+- [ ] Dashboard + alert rules, if metrics are exposed
- [ ] Ingress rule in `k8s/ingress/` with TLS cert
+- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file,
- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-<name>.yaml`
+      **not** a standalone `helm install`/`kubectl apply` run by hand
- [ ] Alert rules in `k8s/monitoring/alerts/svc-<name>-rules.yaml` (if needed)
+- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`,
- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies
+      `kubectl kustomize <dir>/` (if applicable), or `helm template` against
-
+      the exact pinned chart version (if Helm-sourced)
-## Hard Rules
+- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches
-
+      your new commit — not just that `status.sync.status` says `Synced`
-1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner)
+      (see Gotchas — a stale hook can hide behind an otherwise-current app)
 2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."`
 3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`)
 4. **External services via Ingress** — All public endpoints via TLS (homelab-ca)
 5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault
 6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs.
 7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners.
 ## Git & Release
-**Multi-remote push:**
+**Incremental commits (service-layer grouped) tend to age well:**
 ```bash
 git push origin main
 ```
 **Incremental commits (service-layer grouped):**
 - Foundation & Docs
- Helmfile & Core Infra
+- Core Infra (CNI, ingress, cert management, storage)
 - Storage Layer
 - Observability Stack
 - IAM & Secrets
 - CI/CD & GitOps
- Messaging Infrastructure
+- Messaging / Data Infrastructure
 - Applications & Utilities
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
@@ -0,0 +1,200 @@
 # GitOps Workflow (ArgoCD) — Current Practice
 **This is the actual, current deployment workflow.** Every other doc in this
 directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
 `helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
 retired. This file exists to be the accurate replacement for "how do I
 actually deploy/change something" until those older docs get a full rewrite.
 ## The Loop
 ```
 1. Edit files under k8s/
 2. Validate locally (see "Validation" below — do not skip this)
 3. git add -A && git commit -m "..." && git push
 4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
   up on its next reconciliation cycle (default ~3 min, or immediately if
   you trigger a refresh)
 5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
   matches your new commit hash - not just that status.sync.status says
   "Synced" (a stale hook resource can hide behind an otherwise-current app,
   see "ArgoCD Hooks" below)
 ```
 There is no `helmfile apply`, no `core iam create-app`, no manual `helm
 install` in the current workflow. If you find yourself reaching for any of
 those, stop — figure out the ArgoCD-native equivalent instead.
 ## App-of-Apps Structure
 ```
 k8s/argocd/root/homelab-root.yaml       # the one Application ArgoCD bootstraps by hand
  → source: k8s/argocd/apps/            # directory of child Application manifests
      00-substrate.yaml                 # sync-wave 0: cert-manager, ingress-nginx, CRDs
      00-secrets.yaml                   # sync-wave 0: SOPS secrets plugin
      05-networking.yaml                # sync-wave... etc, ascending
      10-storage-observability.yaml
      20-logging.yaml
      30-security.yaml                  # iam (Authentik, Vault)
      40-data.yaml                      # CNPG postgres
      50-messaging.yaml                 # Kafka/sqs
      60-applications.yaml              # end-user workloads (Temporal, Portainer, etc.)
 ```
 Each child `Application` either:
 - Points at a **remote Helm chart** with a **second git source** (`ref:
  values`) supplying just the values file — lets you pin the chart version
  independently while your values live under normal review/history, e.g.:
  ```yaml
  spec:
    sources:
      - repoURL: https://example.com/helm-charts
        chart: some-chart
        targetRevision: "1.2.3"
        helm:
          valueFiles:
            - $values/k8s/applications/foo/foo-values.yaml
      - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
        targetRevision: main
        ref: values
  ```
 - Or points at a **plain git directory** of raw manifests, optionally with
  its own `kustomization.yaml`.
 `argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
 controls ordering — lower N syncs first. Within a single Application,
 resource-level sync-wave annotations further order that app's own
 resources.
 ## Validation Before Every Push
 Pick whichever applies:
 ```bash
 # Plain manifests, no kustomization.yaml in the directory
 kubectl apply --dry-run=client -f path/to/file.yaml
 # Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
 # a plain `apply --dry-run` on individual files will NOT catch a resource
 # missing from the resources: allowlist
 kubectl kustomize path/to/dir/
 # Helm-sourced Application - render against the EXACT pinned chart version,
 # not whatever `helm repo` happens to have cached, and not the chart's
 # current main-branch example values (schemas can differ between versions)
 git clone --depth 1 <chart-repo-url> /tmp/chart-check
 cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
 helm dependency build  # if the chart has subchart dependencies
 helm template . --values /path/to/your/values.yaml --namespace <ns>
 ```
 ## ArgoCD Hooks (PreSync / PostSync / Sync)
 Used for one-shot Jobs that need to run at a specific point in an
 Application's sync (schema migrations, secret-copy jobs, provisioning
 scripts). Two properties that bite people:
 1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
   PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
   defined as *plain* (non-hook) resources in the same Application will
   deadlock — those get created during the Sync phase, which is *after*
   PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
   describe job` shows `FailedCreate ... serviceaccount ... not found`
   repeating forever. **If your hook needs resources from its own
   Application, make it PostSync, not PreSync.**
 2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
   failed hook Job only gets deleted+recreated (per
   `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
   operation* — not from ambient drift detection. If you fix a broken
   hook's spec (image, RBAC, command) and push, the Application's
   `status.sync.revision` can update to show "caught up" (because the
   *normal*, non-hook resources genuinely match) while the *live* hook Job
   is still running the old broken spec, because no fresh sync operation
   actually re-executed it.
   **To force it, in order of preference:**
   ```bash
   # 1. Just delete the stuck/failed job - if a legitimate future sync
   #    happens for any other reason, this clears the way for it
   kubectl -n <ns> delete job <hook-job-name>
   # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
   #    deletion because the operation tracking it is itself stuck):
   kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
   kubectl -n <ns> patch job <hook-job-name> --type json \
     -p '[{"op":"remove","path":"/metadata/finalizers"}]'
   # 3. If deleting the job alone doesn't trigger a genuinely fresh sync
   #    (check: does a new job appear using your LATEST commit's spec?
   #    compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
   #    against what your latest commit says), the Application's own
   #    operation state is stuck. Confirm it has no cascade finalizer first
   #    (Applications don't carry one by default - only delete if this is
   #    empty):
   kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
   # then:
   kubectl -n argocd delete application <app-name>
   kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
   ```
   Step 3 re-reads current git HEAD from scratch and starts a genuinely new
   operation — this is the reliable way to actually pick up a hook fix when
   steps 1-2 don't unstick it.
 ## kustomization.yaml Pitfalls
 - **`resources:` is an explicit allowlist, not a directory scan.** Add a new
  manifest file to a directory that has a `kustomization.yaml`, forget to
  list it under `resources:`, and ArgoCD will silently never apply it — no
  error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
  <dir>/` locally before pushing to see exactly what will be built.
 - **A top-level `namespace:` transformer rewrites `metadata.namespace` on
  every resource in the build**, including RBAC bindings that deliberately
  target a *different* namespace (e.g. a RoleBinding granting cross-
  namespace Secret access for a sync job's ServiceAccount). If any manifest
  in the directory needs to live in a different namespace than the
  transformer specifies, either remove the transformer (safe if every
  manifest already sets its own explicit `namespace:`) or move that
  manifest to its own directory/Application entirely.
 ## Cross-Namespace Secrets
 Kubernetes Secrets are strictly namespace-scoped — a Deployment in
 namespace A cannot reference a Secret living in namespace B via
 `secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
 CNPG auto-generates DB credentials in its own operator namespace) but a
 consumer lives in a different namespace, you need an explicit copy
 mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
 `k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
 example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
 (one per namespace involved) that reads the source Secret and re-creates it
 in the target namespace. Give this its own Application/sync-wave (earlier
 than whatever consumes the copied Secret) rather than folding it into an
 existing Application that has a namespace-transforming `kustomization.yaml`
 (see above).
 ## Common Failure: Everything Shows "Unknown" Sync Status At Once
 If *every* Application (not just one) suddenly shows `Unknown` sync status
 simultaneously, check the Application controller's logs for the actual
 `repoURL` fetch error before assuming something is wrong with any
 individual app's manifests:
 ```bash
 kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
 ```
 Common cause in a homelab with CoreDNS rewriting your git host to the
 ingress controller for internal traffic: if `repoURL` uses a non-standard
 port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
 hostname to the ingress controller service (which only listens on 80/443),
 every fetch attempt times out. Fix: use the default-port HTTPS URL
 (`https://forgejo.example.com/...`) so it actually reaches the ingress
 controller correctly.
 ## See Also
 - `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
  cross-referenced from the main project reference
 - Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
  applicable regardless of deployment mechanism