From af00467b2b974ee66e0a4a0f9a97520c54faac25 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Tue, 21 Jul 2026 20:16:44 -0700 Subject: [PATCH] docs: rewrite CLAUDE.md/CLAUDE.example.md for ArgoCD GitOps, add gitops-workflow.md CLAUDE.md and the entire project-usage/ tree were written for a helmfile + 'core iam'/'core secrets' CLI workflow that has been fully retired - actual practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD sync), confirmed by an extended live debugging session that touched Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and multiple ArgoCD Applications, none of which involved helmfile or core at any point. CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every hard-won lesson from this session with live evidence for each: - kustomization.yaml resources: allowlists silently dropping new files - kustomization.yaml namespace: transformers clobbering cross-namespace RBAC - PreSync hooks deadlocking on same-Application RBAC dependencies - ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely new sync operation to pick up fixes - repo-server manifest caching - repoURL port mismatches breaking every Application's sync simultaneously when routed through an ingress-rewriting CoreDNS rule - Bitnami's 2025 versioned-tag retirement - apk-as-non-root permission failures - Helm's lack of values.yaml schema validation (root cause of the Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was a schema mismatch between the pinned chart version and a newer chart's values.yaml example, silently a no-op) CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic template (explicit notice at top) - same lessons, genericized away from this specific homelab's IPs/hostnames/secrets, intended to be reusable by anyone running a similar bare-metal Talos + ArgoCD topology. project-usage/gitops-workflow.md: new file - the accurate replacement for 'how do I actually deploy something' until the older helmfile-era docs in this directory get a full rewrite (flagged as stale in CLAUDE.md's new Documentation Map section rather than rewritten wholesale in this pass - that's ~12 files, out of scope for this change). --- CLAUDE.example.md | 234 +++++++++++++++++++++++-------- project-usage/gitops-workflow.md | 200 ++++++++++++++++++++++++++ 2 files changed, 379 insertions(+), 55 deletions(-) create mode 100644 project-usage/gitops-workflow.md diff --git a/CLAUDE.example.md b/CLAUDE.example.md index 34b03f5..4870b5e 100644 --- a/CLAUDE.example.md +++ b/CLAUDE.example.md @@ -1,86 +1,210 @@ -# CLAUDE.md — Homelab Integration Guide +# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template) -**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed). +> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own +> node IPs/hostnames/secrets, and delete this notice. Nothing in this file +> should contain real credentials, real IPs beyond illustrative examples, or +> anything that would matter if this file became public. It's meant to be +> shared across homelabs running similar hardware/topology (3-node bare-metal +> Talos Kubernetes + ArgoCD GitOps), not just this one. -## Cluster Topology (3 control-plane HA, since 2026-07-20) +**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a +full observability stack, SSO via Authentik, secret management via Vault, and +CI/CD infrastructure (self-hosted git forge + Argo CD). + +## Cluster Topology (adjust to your hardware) | Node | IP | Zone | Scheduling | Storage | |------|----|----|-----------|---------| -| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node | -| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none | -| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none | +| `` | `` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) | +| `` | `` | az-b | dedicated (`NoSchedule`) | none | +| `` | `` | az-c | dedicated (`NoSchedule`) | none | -3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and -holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster` -= 1 instance). API endpoint is single-homed to `.213` (no VIP yet). +If your storage layer (Longhorn, local-path, etc.) only runs on one node, +every stateful workload pinned to that storage class is effectively +single-instance regardless of your control-plane HA count — document that +explicitly here, it changes your failure-mode assumptions everywhere else. + +## Deployment Model: ArgoCD GitOps (app-of-apps) + +``` +git commit → git push (your git forge) → ArgoCD auto-sync → cluster +``` + +Structure: +- One root `Application` (`k8s/argocd/root/`) pointing at a directory of + child `Application` manifests (`k8s/argocd/apps/*.yaml`) +- Each child Application is either: + - A remote Helm chart + a **second** git source (`ref: values`) supplying + just the values file — lets you pin an upstream chart version while + keeping your values under normal git history/review + - A plain git directory of raw manifests (optionally with a + `kustomization.yaml`) +- `argocd.argoproj.io/sync-wave` annotations control ordering across + Applications (lower number syncs first) + +**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except: +- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook + Job so the next real sync can create a fresh one) — this is not a config + change, just clearing wreckage that GitOps itself won't clean up + automatically (see Gotchas below) +- Genuine one-time bootstrap circular dependencies (e.g. Vault + `operator init`/unseal — nothing can configure Vault's own unseal keys + before Vault has generated them) + +## Hard Rules (adapt freely, but keep something like these) + +🔴 **Identify and document your storage/stateful-singleton node explicitly.** +Whatever node holds your CSI driver's data (Longhorn, local-path, etc.), +renaming or wiping it orphans every PVC pinned there. Name it here, in +caps, so nobody "cleans up" it by accident. + +🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay, +pin the advertised subnet explicitly** (e.g. Talos's +`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the +wrong interface and new members hang as non-promoting learners. + +🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit +that touches infra code.** Wire this into CI as a hard gate, not a +suggestion. + +🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never +manually mutate resources it manages.** State drift is the single most +common cause of "why did my last apply undo my manual fix" confusion. +Fix the source, re-apply/re-sync, never bypass. + +🔴 **Never delete a PVC without confirming replica count / backup +freshness first.** This is always a one-way door. + +🔴 **Decide your commit message convention up front and enforce it.** +(This template's origin project uses: no co-authored-by footers, single-line +commit summarizing what/why, solo-authorship assumption — adjust to your +team's norms.) + +🔴 **Decide your git workflow (rebase vs merge) up front and stick to it** +cluster-wide, across every contributor/agent working in the repo. + +🔴 **Long-running commands should not block a synchronous session** — run +them in the background and poll, especially anything that waits on a +Kubernetes rollout, an image pull, or a Terraform apply. + +## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab) + +🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist +silently drops anything you forget to list.** No error, no drift shown in +ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that +never included your new file. Always run `kubectl kustomize /` +locally before pushing to confirm exactly what ArgoCD will build. + +🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites +`metadata.namespace` on every resource it builds** — including RBAC +bindings deliberately targeting a *different* namespace (e.g. granting a +ServiceAccount in namespace A read access to Secrets in namespace B). If +any manifest needs cross-namespace RBAC, either drop the transformer +(safe if every resource already sets its own explicit namespace) or give +that manifest its own Application/directory. + +🟠 **PreSync hooks run before an Application's own normal resources are +synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as +plain (non-hook) resources in the *same* Application will deadlock — it +tries to start before its own permissions exist. Use PostSync instead if +the hook needs resources from its own Application, or move the +prerequisite RBAC into an earlier sync-wave Application. + +🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a +hook Job finishes (success, or exhausts `backoffLimit`), it's only +deleted+recreated during an *actual new Sync operation* — not by passive +drift detection, even with `automated.selfHeal: true` on. If you fix a +broken hook's spec and push, the Application's `status.sync.revision` can +show "caught up" while the live hook resource is still the old, broken +one, because no new operation actually re-ran it. To force it: delete the +stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's +stuck `Terminating`), and if that alone doesn't trigger a fresh full sync, +delete + re-`kubectl apply -f` the Application object itself. + +🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy +hostname that only listens on 80/443, don't use a non-standard port in the +URL** — it'll silently time out trying to reach a port the proxy never +opened, and depending on your setup this can block *every* Application's +sync simultaneously (repo-server can't fetch git refs for anything). + +🟠 **Don't pin exact version tags for images from registries that don't +guarantee tag retention** (Bitnami stopped publishing versioned tags for +free-tier images in 2025 — only `latest` + sha256 digests remain). Verify +a tag actually exists before pinning it, or prefer minimal base images + +a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a +static binary) to avoid depending on any third party's tagging policy. + +🟠 **Non-root containers can't `apk add`/`apt install` in most default +base images** — package manager directories are root-owned. Use a +world-writable scratch dir (`/tmp`) for anything you need to +download/install at runtime instead. + +🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a +values schema copied from the wrong chart *version's* docs/examples, is +silently a no-op — not an error. Before concluding "this chart doesn't +support X," clone the chart at your exact pinned version/tag and run +`helm template` against your real values file, then diff the rendered +output. Don't trust a chart's current `main`-branch example values file +if you're pinned to an older release — schemas do change between major +versions without warning in your own values file. ## Service Integration Routes -**New service? Pick your stack below:** +**New service? Pick your stack below** (adjust doc paths to match your repo): | Need | Doc | Example | |------|-----|---------| | **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens | -| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ | +| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ | | **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend | -| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync | +| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync | | **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines | | **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas | | **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts | | **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal | | **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS | -## Cluster Essentials +## Cluster Essentials (fill in your own inventory) -**22 namespaces, 18 releases:** -``` -Core: cert-manager, ingress-nginx, kube-system, cilium -Storage: longhorn-system, storage (MinIO) -Data: ddb (PostgreSQL), iam (Authentik + Vault) -Observability: logging (Loki + Grafana), monitoring (Prometheus) -Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend -``` +**Architecture principles (adjust to taste, but these travel well):** +- Immutable OS (Talos, or similar — no SSH, fully declarative config) +- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in + git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env` +- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap + documented separately (chicken-and-egg: ArgoCD needs to exist before it + can deploy itself declaratively — document your exact bootstrap steps) +- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI + runner; the runner only needs push access to git, ArgoCD does the rest +- Federated OIDC (one identity provider fronting every service that + supports it) -**Architecture principles:** -- Immutable OS (Talos — no SSH, declarative config) -- Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env` -- ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md` -- Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync -- Federated OIDC (Authentik provider for all services) +## Deployment Checklist (per new service) - -## Deployment Checklist - -- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor -- [ ] All credentials in Vault (never in pod env, ConfigMap, or code) -- [ ] Ingress rule in `k8s/ingress/` with TLS cert -- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-.yaml` -- [ ] Alert rules in `k8s/monitoring/alerts/svc--rules.yaml` (if needed) -- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies - -## Hard Rules - -1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner) -2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."` -3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`) -4. **External services via Ingress** — All public endpoints via TLS (homelab-ca) -5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault -6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs. -7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners. +- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics +- [ ] All credentials in your secrets backend (never in plain values.yaml, + pod env directly, or committed anywhere in cleartext) +- [ ] Ingress rule with TLS, if externally reachable +- [ ] Dashboard + alert rules, if metrics are exposed +- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file, + **not** a standalone `helm install`/`kubectl apply` run by hand +- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`, + `kubectl kustomize /` (if applicable), or `helm template` against + the exact pinned chart version (if Helm-sourced) +- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches + your new commit — not just that `status.sync.status` says `Synced` + (see Gotchas — a stale hook can hide behind an otherwise-current app) ## Git & Release -**Multi-remote push:** -```bash -git push origin main -``` - -**Incremental commits (service-layer grouped):** +**Incremental commits (service-layer grouped) tend to age well:** - Foundation & Docs -- Helmfile & Core Infra -- Storage Layer +- Core Infra (CNI, ingress, cert management, storage) - Observability Stack - IAM & Secrets - CI/CD & GitOps -- Messaging Infrastructure +- Messaging / Data Infrastructure - Applications & Utilities + +Grouping by layer (rather than by day or by "misc fixes") makes it much +easier to `git log --oneline -- ` your way back to *why* a given +piece of config looks the way it does, months later. diff --git a/project-usage/gitops-workflow.md b/project-usage/gitops-workflow.md new file mode 100644 index 0000000..482ac1d --- /dev/null +++ b/project-usage/gitops-workflow.md @@ -0,0 +1,200 @@ +# GitOps Workflow (ArgoCD) — Current Practice + +**This is the actual, current deployment workflow.** Every other doc in this +directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older +`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been +retired. This file exists to be the accurate replacement for "how do I +actually deploy/change something" until those older docs get a full rewrite. + +## The Loop + +``` +1. Edit files under k8s/ +2. Validate locally (see "Validation" below — do not skip this) +3. git add -A && git commit -m "..." && git push +4. ArgoCD (automated sync + selfHeal on almost every Application) picks it + up on its next reconciliation cycle (default ~3 min, or immediately if + you trigger a refresh) +5. Confirm: kubectl -n argocd get app -o jsonpath='{.status.sync.revision}' + matches your new commit hash - not just that status.sync.status says + "Synced" (a stale hook resource can hide behind an otherwise-current app, + see "ArgoCD Hooks" below) +``` + +There is no `helmfile apply`, no `core iam create-app`, no manual `helm +install` in the current workflow. If you find yourself reaching for any of +those, stop — figure out the ArgoCD-native equivalent instead. + +## App-of-Apps Structure + +``` +k8s/argocd/root/homelab-root.yaml # the one Application ArgoCD bootstraps by hand + → source: k8s/argocd/apps/ # directory of child Application manifests + 00-substrate.yaml # sync-wave 0: cert-manager, ingress-nginx, CRDs + 00-secrets.yaml # sync-wave 0: SOPS secrets plugin + 05-networking.yaml # sync-wave... etc, ascending + 10-storage-observability.yaml + 20-logging.yaml + 30-security.yaml # iam (Authentik, Vault) + 40-data.yaml # CNPG postgres + 50-messaging.yaml # Kafka/sqs + 60-applications.yaml # end-user workloads (Temporal, Portainer, etc.) +``` + +Each child `Application` either: +- Points at a **remote Helm chart** with a **second git source** (`ref: + values`) supplying just the values file — lets you pin the chart version + independently while your values live under normal review/history, e.g.: + ```yaml + spec: + sources: + - repoURL: https://example.com/helm-charts + chart: some-chart + targetRevision: "1.2.3" + helm: + valueFiles: + - $values/k8s/applications/foo/foo-values.yaml + - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git + targetRevision: main + ref: values + ``` +- Or points at a **plain git directory** of raw manifests, optionally with + its own `kustomization.yaml`. + +`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata +controls ordering — lower N syncs first. Within a single Application, +resource-level sync-wave annotations further order that app's own +resources. + +## Validation Before Every Push + +Pick whichever applies: + +```bash +# Plain manifests, no kustomization.yaml in the directory +kubectl apply --dry-run=client -f path/to/file.yaml + +# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds, +# a plain `apply --dry-run` on individual files will NOT catch a resource +# missing from the resources: allowlist +kubectl kustomize path/to/dir/ + +# Helm-sourced Application - render against the EXACT pinned chart version, +# not whatever `helm repo` happens to have cached, and not the chart's +# current main-branch example values (schemas can differ between versions) +git clone --depth 1 /tmp/chart-check +cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout +helm dependency build # if the chart has subchart dependencies +helm template . --values /path/to/your/values.yaml --namespace +``` + +## ArgoCD Hooks (PreSync / PostSync / Sync) + +Used for one-shot Jobs that need to run at a specific point in an +Application's sync (schema migrations, secret-copy jobs, provisioning +scripts). Two properties that bite people: + +1. **Ordering is PreSync → Sync (normal resources, by sync-wave) → + PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC + defined as *plain* (non-hook) resources in the same Application will + deadlock — those get created during the Sync phase, which is *after* + PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl + describe job` shows `FailedCreate ... serviceaccount ... not found` + repeating forever. **If your hook needs resources from its own + Application, make it PostSync, not PreSync.** + +2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or + failed hook Job only gets deleted+recreated (per + `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync + operation* — not from ambient drift detection. If you fix a broken + hook's spec (image, RBAC, command) and push, the Application's + `status.sync.revision` can update to show "caught up" (because the + *normal*, non-hook resources genuinely match) while the *live* hook Job + is still running the old broken spec, because no fresh sync operation + actually re-executed it. + + **To force it, in order of preference:** + ```bash + # 1. Just delete the stuck/failed job - if a legitimate future sync + # happens for any other reason, this clears the way for it + kubectl -n delete job + + # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking + # deletion because the operation tracking it is itself stuck): + kubectl -n get job -o jsonpath='{.metadata.finalizers}' + kubectl -n patch job --type json \ + -p '[{"op":"remove","path":"/metadata/finalizers"}]' + + # 3. If deleting the job alone doesn't trigger a genuinely fresh sync + # (check: does a new job appear using your LATEST commit's spec? + # compare kubectl -n get job -o jsonpath='{.spec.template.spec.containers[0].image}' + # against what your latest commit says), the Application's own + # operation state is stuck. Confirm it has no cascade finalizer first + # (Applications don't carry one by default - only delete if this is + # empty): + kubectl -n argocd get app -o jsonpath='{.metadata.finalizers}' + # then: + kubectl -n argocd delete application + kubectl apply -f k8s/argocd/apps/.yaml + ``` + Step 3 re-reads current git HEAD from scratch and starts a genuinely new + operation — this is the reliable way to actually pick up a hook fix when + steps 1-2 don't unstick it. + +## kustomization.yaml Pitfalls + +- **`resources:` is an explicit allowlist, not a directory scan.** Add a new + manifest file to a directory that has a `kustomization.yaml`, forget to + list it under `resources:`, and ArgoCD will silently never apply it — no + error, `Synced/Healthy` shown regardless. Always `kubectl kustomize + /` locally before pushing to see exactly what will be built. +- **A top-level `namespace:` transformer rewrites `metadata.namespace` on + every resource in the build**, including RBAC bindings that deliberately + target a *different* namespace (e.g. a RoleBinding granting cross- + namespace Secret access for a sync job's ServiceAccount). If any manifest + in the directory needs to live in a different namespace than the + transformer specifies, either remove the transformer (safe if every + manifest already sets its own explicit `namespace:`) or move that + manifest to its own directory/Application entirely. + +## Cross-Namespace Secrets + +Kubernetes Secrets are strictly namespace-scoped — a Deployment in +namespace A cannot reference a Secret living in namespace B via +`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g. +CNPG auto-generates DB credentials in its own operator namespace) but a +consumer lives in a different namespace, you need an explicit copy +mechanism. Pattern used in this repo: a small PostSync-hooked Job (see +`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked +example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings +(one per namespace involved) that reads the source Secret and re-creates it +in the target namespace. Give this its own Application/sync-wave (earlier +than whatever consumes the copied Secret) rather than folding it into an +existing Application that has a namespace-transforming `kustomization.yaml` +(see above). + +## Common Failure: Everything Shows "Unknown" Sync Status At Once + +If *every* Application (not just one) suddenly shows `Unknown` sync status +simultaneously, check the Application controller's logs for the actual +`repoURL` fetch error before assuming something is wrong with any +individual app's manifests: + +```bash +kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline" +``` + +Common cause in a homelab with CoreDNS rewriting your git host to the +ingress controller for internal traffic: if `repoURL` uses a non-standard +port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that +hostname to the ingress controller service (which only listens on 80/443), +every fetch attempt times out. Fix: use the default-port HTTPS URL +(`https://forgejo.example.com/...`) so it actually reaches the ingress +controller correctly. + +## See Also + +- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above, + cross-referenced from the main project reference +- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still + applicable regardless of deployment mechanism