rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite CLAUDE.md/CLAUDE.example.md for ArgoCD GitOps, add gitops-workflow.md

Browse Source 
CLAUDE.md and the entire project-usage/ tree were written for a helmfile +
'core iam'/'core secrets' CLI workflow that has been fully retired - actual
practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD
sync), confirmed by an extended live debugging session that touched
Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and
multiple ArgoCD Applications, none of which involved helmfile or core at
any point.

CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps
loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every
hard-won lesson from this session with live evidence for each:
  - kustomization.yaml resources: allowlists silently dropping new files
  - kustomization.yaml namespace: transformers clobbering cross-namespace
    RBAC
  - PreSync hooks deadlocking on same-Application RBAC dependencies
  - ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely
    new sync operation to pick up fixes
  - repo-server manifest caching
  - repoURL port mismatches breaking every Application's sync
    simultaneously when routed through an ingress-rewriting CoreDNS rule
  - Bitnami's 2025 versioned-tag retirement
  - apk-as-non-root permission failures
  - Helm's lack of values.yaml schema validation (root cause of the
    Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was
    a schema mismatch between the pinned chart version and a newer
    chart's values.yaml example, silently a no-op)

CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic
template (explicit notice at top) - same lessons, genericized away from
this specific homelab's IPs/hostnames/secrets, intended to be reusable by
anyone running a similar bare-metal Talos + ArgoCD topology.

project-usage/gitops-workflow.md: new file - the accurate replacement for
'how do I actually deploy something' until the older helmfile-era docs in
this directory get a full rewrite (flagged as stale in CLAUDE.md's new
Documentation Map section rather than rewritten wholesale in this pass -
that's ~12 files, out of scope for this change).
This commit is contained in:
Story Crater Bot
2026-07-21 20:16:44 -07:00
parent 0a323fc039
commit af00467b2b
2 changed files with 379 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.example.md
+179 -55
View File
@@ -1,86 +1,210 @@
# CLAUDE.md — Homelab Integration Guide
# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template)
**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed).
> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own
> node IPs/hostnames/secrets, and delete this notice. Nothing in this file
> should contain real credentials, real IPs beyond illustrative examples, or
> anything that would matter if this file became public. It's meant to be
> shared across homelabs running similar hardware/topology (3-node bare-metal
> Talos Kubernetes + ArgoCD GitOps), not just this one.
## Cluster Topology (3 control-plane HA, since 2026-07-20)
**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a
full observability stack, SSO via Authentik, secret management via Vault, and
CI/CD infrastructure (self-hosted git forge + Argo CD).
## Cluster Topology (adjust to your hardware)
| Node | IP | Zone | Scheduling | Storage |
|------|----|----|-----------|---------|
| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
| `<node-1>` | `<ip>` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) |
| `<node-2>` | `<ip>` | az-b | dedicated (`NoSchedule`) | none |
| `<node-3>` | `<ip>` | az-c | dedicated (`NoSchedule`) | none |
3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster`
= 1 instance). API endpoint is single-homed to `.213` (no VIP yet).
If your storage layer (Longhorn, local-path, etc.) only runs on one node,
every stateful workload pinned to that storage class is effectively
single-instance regardless of your control-plane HA count — document that
explicitly here, it changes your failure-mode assumptions everywhere else.
## Deployment Model: ArgoCD GitOps (app-of-apps)
```
git commit → git push (your git forge) → ArgoCD auto-sync → cluster
```
Structure:
- One root `Application` (`k8s/argocd/root/`) pointing at a directory of
child `Application` manifests (`k8s/argocd/apps/*.yaml`)
- Each child Application is either:
- A remote Helm chart + a **second** git source (`ref: values`) supplying
just the values file — lets you pin an upstream chart version while
keeping your values under normal git history/review
- A plain git directory of raw manifests (optionally with a
`kustomization.yaml`)
- `argocd.argoproj.io/sync-wave` annotations control ordering across
Applications (lower number syncs first)
**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except:
- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook
Job so the next real sync can create a fresh one) — this is not a config
change, just clearing wreckage that GitOps itself won't clean up
automatically (see Gotchas below)
- Genuine one-time bootstrap circular dependencies (e.g. Vault
`operator init`/unseal — nothing can configure Vault's own unseal keys
before Vault has generated them)
## Hard Rules (adapt freely, but keep something like these)
🔴 **Identify and document your storage/stateful-singleton node explicitly.**
Whatever node holds your CSI driver's data (Longhorn, local-path, etc.),
renaming or wiping it orphans every PVC pinned there. Name it here, in
caps, so nobody "cleans up" it by accident.
🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay,
pin the advertised subnet explicitly** (e.g. Talos's
`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the
wrong interface and new members hang as non-promoting learners.
🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit
that touches infra code.** Wire this into CI as a hard gate, not a
suggestion.
🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never
manually mutate resources it manages.** State drift is the single most
common cause of "why did my last apply undo my manual fix" confusion.
Fix the source, re-apply/re-sync, never bypass.
🔴 **Never delete a PVC without confirming replica count / backup
freshness first.** This is always a one-way door.
🔴 **Decide your commit message convention up front and enforce it.**
(This template's origin project uses: no co-authored-by footers, single-line
commit summarizing what/why, solo-authorship assumption — adjust to your
team's norms.)
🔴 **Decide your git workflow (rebase vs merge) up front and stick to it**
cluster-wide, across every contributor/agent working in the repo.
🔴 **Long-running commands should not block a synchronous session** — run
them in the background and poll, especially anything that waits on a
Kubernetes rollout, an image pull, or a Terraform apply.
## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab)
🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist
silently drops anything you forget to list.** No error, no drift shown in
ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that
never included your new file. Always run `kubectl kustomize <dir>/`
locally before pushing to confirm exactly what ArgoCD will build.
🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites
`metadata.namespace` on every resource it builds** — including RBAC
bindings deliberately targeting a *different* namespace (e.g. granting a
ServiceAccount in namespace A read access to Secrets in namespace B). If
any manifest needs cross-namespace RBAC, either drop the transformer
(safe if every resource already sets its own explicit namespace) or give
that manifest its own Application/directory.
🟠 **PreSync hooks run before an Application's own normal resources are
synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as
plain (non-hook) resources in the *same* Application will deadlock — it
tries to start before its own permissions exist. Use PostSync instead if
the hook needs resources from its own Application, or move the
prerequisite RBAC into an earlier sync-wave Application.
🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a
hook Job finishes (success, or exhausts `backoffLimit`), it's only
deleted+recreated during an *actual new Sync operation* — not by passive
drift detection, even with `automated.selfHeal: true` on. If you fix a
broken hook's spec and push, the Application's `status.sync.revision` can
show "caught up" while the live hook resource is still the old, broken
one, because no new operation actually re-ran it. To force it: delete the
stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's
stuck `Terminating`), and if that alone doesn't trigger a fresh full sync,
delete + re-`kubectl apply -f` the Application object itself.
🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy
hostname that only listens on 80/443, don't use a non-standard port in the
URL** — it'll silently time out trying to reach a port the proxy never
opened, and depending on your setup this can block *every* Application's
sync simultaneously (repo-server can't fetch git refs for anything).
🟠 **Don't pin exact version tags for images from registries that don't
guarantee tag retention** (Bitnami stopped publishing versioned tags for
free-tier images in 2025 — only `latest` + sha256 digests remain). Verify
a tag actually exists before pinning it, or prefer minimal base images +
a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a
static binary) to avoid depending on any third party's tagging policy.
🟠 **Non-root containers can't `apk add`/`apt install` in most default
base images** — package manager directories are root-owned. Use a
world-writable scratch dir (`/tmp`) for anything you need to
download/install at runtime instead.
🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a
values schema copied from the wrong chart *version's* docs/examples, is
silently a no-op — not an error. Before concluding "this chart doesn't
support X," clone the chart at your exact pinned version/tag and run
`helm template` against your real values file, then diff the rendered
output. Don't trust a chart's current `main`-branch example values file
if you're pinned to an older release — schemas do change between major
versions without warning in your own values file.
## Service Integration Routes
**New service? Pick your stack below:**
**New service? Pick your stack below** (adjust doc paths to match your repo):
| Need | Doc | Example |
|------|-----|---------|
| **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens |
| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ |
| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ |
| **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend |
| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync |
| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync |
| **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines |
| **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas |
| **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts |
| **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal |
| **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS |
## Cluster Essentials
## Cluster Essentials (fill in your own inventory)
**22 namespaces, 18 releases:**
```
Core: cert-manager, ingress-nginx, kube-system, cilium
Storage: longhorn-system, storage (MinIO)
Data: ddb (PostgreSQL), iam (Authentik + Vault)
Observability: logging (Loki + Grafana), monitoring (Prometheus)
Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend
```
**Architecture principles (adjust to taste, but these travel well):**
- Immutable OS (Talos, or similar — no SSH, fully declarative config)
- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in
git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env`
- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap
documented separately (chicken-and-egg: ArgoCD needs to exist before it
can deploy itself declaratively — document your exact bootstrap steps)
- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI
runner; the runner only needs push access to git, ArgoCD does the rest
- Federated OIDC (one identity provider fronting every service that
supports it)
**Architecture principles:**
- Immutable OS (Talos — no SSH, declarative config)
- Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env`
- ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md`
- Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync
- Federated OIDC (Authentik provider for all services)
## Deployment Checklist (per new service)
## Deployment Checklist
- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor
- [ ] All credentials in Vault (never in pod env, ConfigMap, or code)
- [ ] Ingress rule in `k8s/ingress/` with TLS cert
- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-<name>.yaml`
- [ ] Alert rules in `k8s/monitoring/alerts/svc-<name>-rules.yaml` (if needed)
- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies
## Hard Rules
1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner)
2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."`
3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`)
4. **External services via Ingress** — All public endpoints via TLS (homelab-ca)
5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault
6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs.
7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners.
- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics
- [ ] All credentials in your secrets backend (never in plain values.yaml,
pod env directly, or committed anywhere in cleartext)
- [ ] Ingress rule with TLS, if externally reachable
- [ ] Dashboard + alert rules, if metrics are exposed
- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file,
**not** a standalone `helm install`/`kubectl apply` run by hand
- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`,
`kubectl kustomize <dir>/` (if applicable), or `helm template` against
the exact pinned chart version (if Helm-sourced)
- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches
your new commit — not just that `status.sync.status` says `Synced`
(see Gotchas — a stale hook can hide behind an otherwise-current app)
## Git & Release
**Multi-remote push:**
```bash
git push origin main
```
**Incremental commits (service-layer grouped):**
**Incremental commits (service-layer grouped) tend to age well:**
- Foundation & Docs
- Helmfile & Core Infra
- Storage Layer
- Core Infra (CNI, ingress, cert management, storage)
- Observability Stack
- IAM & Secrets
- CI/CD & GitOps
- Messaging Infrastructure
- Messaging / Data Infrastructure
- Applications & Utilities
Grouping by layer (rather than by day or by "misc fixes") makes it much
easier to `git log --oneline -- <path>` your way back to *why* a given
piece of config looks the way it does, months later.
project-usage/gitops-workflow.md
+200
View File
@@ -0,0 +1,200 @@
# GitOps Workflow (ArgoCD) — Current Practice
**This is the actual, current deployment workflow.** Every other doc in this
directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
retired. This file exists to be the accurate replacement for "how do I
actually deploy/change something" until those older docs get a full rewrite.
## The Loop
```
1. Edit files under k8s/
2. Validate locally (see "Validation" below — do not skip this)
3. git add -A && git commit -m "..." && git push
4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
up on its next reconciliation cycle (default ~3 min, or immediately if
you trigger a refresh)
5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
matches your new commit hash - not just that status.sync.status says
"Synced" (a stale hook resource can hide behind an otherwise-current app,
see "ArgoCD Hooks" below)
```
There is no `helmfile apply`, no `core iam create-app`, no manual `helm
install` in the current workflow. If you find yourself reaching for any of
those, stop — figure out the ArgoCD-native equivalent instead.
## App-of-Apps Structure
```
k8s/argocd/root/homelab-root.yaml # the one Application ArgoCD bootstraps by hand
→ source: k8s/argocd/apps/ # directory of child Application manifests
00-substrate.yaml # sync-wave 0: cert-manager, ingress-nginx, CRDs
00-secrets.yaml # sync-wave 0: SOPS secrets plugin
05-networking.yaml # sync-wave... etc, ascending
10-storage-observability.yaml
20-logging.yaml
30-security.yaml # iam (Authentik, Vault)
40-data.yaml # CNPG postgres
50-messaging.yaml # Kafka/sqs
60-applications.yaml # end-user workloads (Temporal, Portainer, etc.)
```
Each child `Application` either:
- Points at a **remote Helm chart** with a **second git source** (`ref:
values`) supplying just the values file — lets you pin the chart version
independently while your values live under normal review/history, e.g.:
```yaml
spec:
sources:
- repoURL: https://example.com/helm-charts
chart: some-chart
targetRevision: "1.2.3"
helm:
valueFiles:
- $values/k8s/applications/foo/foo-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
ref: values
```
- Or points at a **plain git directory** of raw manifests, optionally with
its own `kustomization.yaml`.
`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
controls ordering — lower N syncs first. Within a single Application,
resource-level sync-wave annotations further order that app's own
resources.
## Validation Before Every Push
Pick whichever applies:
```bash
# Plain manifests, no kustomization.yaml in the directory
kubectl apply --dry-run=client -f path/to/file.yaml
# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
# a plain `apply --dry-run` on individual files will NOT catch a resource
# missing from the resources: allowlist
kubectl kustomize path/to/dir/
# Helm-sourced Application - render against the EXACT pinned chart version,
# not whatever `helm repo` happens to have cached, and not the chart's
# current main-branch example values (schemas can differ between versions)
git clone --depth 1 <chart-repo-url> /tmp/chart-check
cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
helm dependency build # if the chart has subchart dependencies
helm template . --values /path/to/your/values.yaml --namespace <ns>
```
## ArgoCD Hooks (PreSync / PostSync / Sync)
Used for one-shot Jobs that need to run at a specific point in an
Application's sync (schema migrations, secret-copy jobs, provisioning
scripts). Two properties that bite people:
1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
defined as *plain* (non-hook) resources in the same Application will
deadlock — those get created during the Sync phase, which is *after*
PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
describe job` shows `FailedCreate ... serviceaccount ... not found`
repeating forever. **If your hook needs resources from its own
Application, make it PostSync, not PreSync.**
2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
failed hook Job only gets deleted+recreated (per
`hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
operation* — not from ambient drift detection. If you fix a broken
hook's spec (image, RBAC, command) and push, the Application's
`status.sync.revision` can update to show "caught up" (because the
*normal*, non-hook resources genuinely match) while the *live* hook Job
is still running the old broken spec, because no fresh sync operation
actually re-executed it.
**To force it, in order of preference:**
```bash
# 1. Just delete the stuck/failed job - if a legitimate future sync
# happens for any other reason, this clears the way for it
kubectl -n <ns> delete job <hook-job-name>
# 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
# deletion because the operation tracking it is itself stuck):
kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
kubectl -n <ns> patch job <hook-job-name> --type json \
-p '[{"op":"remove","path":"/metadata/finalizers"}]'
# 3. If deleting the job alone doesn't trigger a genuinely fresh sync
# (check: does a new job appear using your LATEST commit's spec?
# compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
# against what your latest commit says), the Application's own
# operation state is stuck. Confirm it has no cascade finalizer first
# (Applications don't carry one by default - only delete if this is
# empty):
kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
# then:
kubectl -n argocd delete application <app-name>
kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
```
Step 3 re-reads current git HEAD from scratch and starts a genuinely new
operation — this is the reliable way to actually pick up a hook fix when
steps 1-2 don't unstick it.
## kustomization.yaml Pitfalls
- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
manifest file to a directory that has a `kustomization.yaml`, forget to
list it under `resources:`, and ArgoCD will silently never apply it — no
error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
<dir>/` locally before pushing to see exactly what will be built.
- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
every resource in the build**, including RBAC bindings that deliberately
target a *different* namespace (e.g. a RoleBinding granting cross-
namespace Secret access for a sync job's ServiceAccount). If any manifest
in the directory needs to live in a different namespace than the
transformer specifies, either remove the transformer (safe if every
manifest already sets its own explicit `namespace:`) or move that
manifest to its own directory/Application entirely.
## Cross-Namespace Secrets
Kubernetes Secrets are strictly namespace-scoped — a Deployment in
namespace A cannot reference a Secret living in namespace B via
`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
CNPG auto-generates DB credentials in its own operator namespace) but a
consumer lives in a different namespace, you need an explicit copy
mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
(one per namespace involved) that reads the source Secret and re-creates it
in the target namespace. Give this its own Application/sync-wave (earlier
than whatever consumes the copied Secret) rather than folding it into an
existing Application that has a namespace-transforming `kustomization.yaml`
(see above).
## Common Failure: Everything Shows "Unknown" Sync Status At Once
If *every* Application (not just one) suddenly shows `Unknown` sync status
simultaneously, check the Application controller's logs for the actual
`repoURL` fetch error before assuming something is wrong with any
individual app's manifests:
```bash
kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
```
Common cause in a homelab with CoreDNS rewriting your git host to the
ingress controller for internal traffic: if `repoURL` uses a non-standard
port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
hostname to the ingress controller service (which only listens on 80/443),
every fetch attempt times out. Fix: use the default-port HTTPS URL
(`https://forgejo.example.com/...`) so it actually reaches the ingress
controller correctly.
## See Also
- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
cross-referenced from the main project reference
- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
applicable regardless of deployment mechanism