docs: rewrite CLAUDE.md for ArgoCD GitOps, track in git
This commit is contained in:
Story Crater Bot
+3
-2
@@ -38,8 +38,9 @@ k8s/storage/test/test
|
||||
*.key
|
||||
*.conf
|
||||
|
||||
# Allowed markdown: CLAUDE.example.md, README.md, TROUBLESHOOTING.md
|
||||
CLAUDE.md
|
||||
# CLAUDE.md is now version-controlled (was previously excluded as a
|
||||
# private-notes file; contains no secrets - just architecture, IPs
|
||||
# [private RFC1918 space], and operational lessons, same bar as README.md).
|
||||
|
||||
# Terraform state and cache (local files, remote state in MinIO)
|
||||
.terraform/
|
||||
|
||||
+179
-55
@@ -1,86 +1,210 @@
|
||||
# CLAUDE.md — Homelab Integration Guide
|
||||
# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template)
|
||||
|
||||
**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed).
|
||||
> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own
|
||||
> node IPs/hostnames/secrets, and delete this notice. Nothing in this file
|
||||
> should contain real credentials, real IPs beyond illustrative examples, or
|
||||
> anything that would matter if this file became public. It's meant to be
|
||||
> shared across homelabs running similar hardware/topology (3-node bare-metal
|
||||
> Talos Kubernetes + ArgoCD GitOps), not just this one.
|
||||
|
||||
## Cluster Topology (3 control-plane HA, since 2026-07-20)
|
||||
**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a
|
||||
full observability stack, SSO via Authentik, secret management via Vault, and
|
||||
CI/CD infrastructure (self-hosted git forge + Argo CD).
|
||||
|
||||
## Cluster Topology (adjust to your hardware)
|
||||
|
||||
| Node | IP | Zone | Scheduling | Storage |
|
||||
|------|----|----|-----------|---------|
|
||||
| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
|
||||
| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
|
||||
| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
|
||||
| `<node-1>` | `<ip>` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) |
|
||||
| `<node-2>` | `<ip>` | az-b | dedicated (`NoSchedule`) | none |
|
||||
| `<node-3>` | `<ip>` | az-c | dedicated (`NoSchedule`) | none |
|
||||
|
||||
3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
|
||||
holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster`
|
||||
= 1 instance). API endpoint is single-homed to `.213` (no VIP yet).
|
||||
If your storage layer (Longhorn, local-path, etc.) only runs on one node,
|
||||
every stateful workload pinned to that storage class is effectively
|
||||
single-instance regardless of your control-plane HA count — document that
|
||||
explicitly here, it changes your failure-mode assumptions everywhere else.
|
||||
|
||||
## Deployment Model: ArgoCD GitOps (app-of-apps)
|
||||
|
||||
```
|
||||
git commit → git push (your git forge) → ArgoCD auto-sync → cluster
|
||||
```
|
||||
|
||||
Structure:
|
||||
- One root `Application` (`k8s/argocd/root/`) pointing at a directory of
|
||||
child `Application` manifests (`k8s/argocd/apps/*.yaml`)
|
||||
- Each child Application is either:
|
||||
- A remote Helm chart + a **second** git source (`ref: values`) supplying
|
||||
just the values file — lets you pin an upstream chart version while
|
||||
keeping your values under normal git history/review
|
||||
- A plain git directory of raw manifests (optionally with a
|
||||
`kustomization.yaml`)
|
||||
- `argocd.argoproj.io/sync-wave` annotations control ordering across
|
||||
Applications (lower number syncs first)
|
||||
|
||||
**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except:
|
||||
- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook
|
||||
Job so the next real sync can create a fresh one) — this is not a config
|
||||
change, just clearing wreckage that GitOps itself won't clean up
|
||||
automatically (see Gotchas below)
|
||||
- Genuine one-time bootstrap circular dependencies (e.g. Vault
|
||||
`operator init`/unseal — nothing can configure Vault's own unseal keys
|
||||
before Vault has generated them)
|
||||
|
||||
## Hard Rules (adapt freely, but keep something like these)
|
||||
|
||||
🔴 **Identify and document your storage/stateful-singleton node explicitly.**
|
||||
Whatever node holds your CSI driver's data (Longhorn, local-path, etc.),
|
||||
renaming or wiping it orphans every PVC pinned there. Name it here, in
|
||||
caps, so nobody "cleans up" it by accident.
|
||||
|
||||
🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay,
|
||||
pin the advertised subnet explicitly** (e.g. Talos's
|
||||
`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the
|
||||
wrong interface and new members hang as non-promoting learners.
|
||||
|
||||
🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit
|
||||
that touches infra code.** Wire this into CI as a hard gate, not a
|
||||
suggestion.
|
||||
|
||||
🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never
|
||||
manually mutate resources it manages.** State drift is the single most
|
||||
common cause of "why did my last apply undo my manual fix" confusion.
|
||||
Fix the source, re-apply/re-sync, never bypass.
|
||||
|
||||
🔴 **Never delete a PVC without confirming replica count / backup
|
||||
freshness first.** This is always a one-way door.
|
||||
|
||||
🔴 **Decide your commit message convention up front and enforce it.**
|
||||
(This template's origin project uses: no co-authored-by footers, single-line
|
||||
commit summarizing what/why, solo-authorship assumption — adjust to your
|
||||
team's norms.)
|
||||
|
||||
🔴 **Decide your git workflow (rebase vs merge) up front and stick to it**
|
||||
cluster-wide, across every contributor/agent working in the repo.
|
||||
|
||||
🔴 **Long-running commands should not block a synchronous session** — run
|
||||
them in the background and poll, especially anything that waits on a
|
||||
Kubernetes rollout, an image pull, or a Terraform apply.
|
||||
|
||||
## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab)
|
||||
|
||||
🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist
|
||||
silently drops anything you forget to list.** No error, no drift shown in
|
||||
ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that
|
||||
never included your new file. Always run `kubectl kustomize <dir>/`
|
||||
locally before pushing to confirm exactly what ArgoCD will build.
|
||||
|
||||
🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites
|
||||
`metadata.namespace` on every resource it builds** — including RBAC
|
||||
bindings deliberately targeting a *different* namespace (e.g. granting a
|
||||
ServiceAccount in namespace A read access to Secrets in namespace B). If
|
||||
any manifest needs cross-namespace RBAC, either drop the transformer
|
||||
(safe if every resource already sets its own explicit namespace) or give
|
||||
that manifest its own Application/directory.
|
||||
|
||||
🟠 **PreSync hooks run before an Application's own normal resources are
|
||||
synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as
|
||||
plain (non-hook) resources in the *same* Application will deadlock — it
|
||||
tries to start before its own permissions exist. Use PostSync instead if
|
||||
the hook needs resources from its own Application, or move the
|
||||
prerequisite RBAC into an earlier sync-wave Application.
|
||||
|
||||
🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a
|
||||
hook Job finishes (success, or exhausts `backoffLimit`), it's only
|
||||
deleted+recreated during an *actual new Sync operation* — not by passive
|
||||
drift detection, even with `automated.selfHeal: true` on. If you fix a
|
||||
broken hook's spec and push, the Application's `status.sync.revision` can
|
||||
show "caught up" while the live hook resource is still the old, broken
|
||||
one, because no new operation actually re-ran it. To force it: delete the
|
||||
stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's
|
||||
stuck `Terminating`), and if that alone doesn't trigger a fresh full sync,
|
||||
delete + re-`kubectl apply -f` the Application object itself.
|
||||
|
||||
🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy
|
||||
hostname that only listens on 80/443, don't use a non-standard port in the
|
||||
URL** — it'll silently time out trying to reach a port the proxy never
|
||||
opened, and depending on your setup this can block *every* Application's
|
||||
sync simultaneously (repo-server can't fetch git refs for anything).
|
||||
|
||||
🟠 **Don't pin exact version tags for images from registries that don't
|
||||
guarantee tag retention** (Bitnami stopped publishing versioned tags for
|
||||
free-tier images in 2025 — only `latest` + sha256 digests remain). Verify
|
||||
a tag actually exists before pinning it, or prefer minimal base images +
|
||||
a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a
|
||||
static binary) to avoid depending on any third party's tagging policy.
|
||||
|
||||
🟠 **Non-root containers can't `apk add`/`apt install` in most default
|
||||
base images** — package manager directories are root-owned. Use a
|
||||
world-writable scratch dir (`/tmp`) for anything you need to
|
||||
download/install at runtime instead.
|
||||
|
||||
🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a
|
||||
values schema copied from the wrong chart *version's* docs/examples, is
|
||||
silently a no-op — not an error. Before concluding "this chart doesn't
|
||||
support X," clone the chart at your exact pinned version/tag and run
|
||||
`helm template` against your real values file, then diff the rendered
|
||||
output. Don't trust a chart's current `main`-branch example values file
|
||||
if you're pinned to an older release — schemas do change between major
|
||||
versions without warning in your own values file.
|
||||
|
||||
## Service Integration Routes
|
||||
|
||||
**New service? Pick your stack below:**
|
||||
**New service? Pick your stack below** (adjust doc paths to match your repo):
|
||||
|
||||
| Need | Doc | Example |
|
||||
|------|-----|---------|
|
||||
| **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens |
|
||||
| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ |
|
||||
| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ |
|
||||
| **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend |
|
||||
| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync |
|
||||
| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync |
|
||||
| **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines |
|
||||
| **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas |
|
||||
| **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts |
|
||||
| **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal |
|
||||
| **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS |
|
||||
|
||||
## Cluster Essentials
|
||||
## Cluster Essentials (fill in your own inventory)
|
||||
|
||||
**22 namespaces, 18 releases:**
|
||||
```
|
||||
Core: cert-manager, ingress-nginx, kube-system, cilium
|
||||
Storage: longhorn-system, storage (MinIO)
|
||||
Data: ddb (PostgreSQL), iam (Authentik + Vault)
|
||||
Observability: logging (Loki + Grafana), monitoring (Prometheus)
|
||||
Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend
|
||||
```
|
||||
**Architecture principles (adjust to taste, but these travel well):**
|
||||
- Immutable OS (Talos, or similar — no SSH, fully declarative config)
|
||||
- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in
|
||||
git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env`
|
||||
- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap
|
||||
documented separately (chicken-and-egg: ArgoCD needs to exist before it
|
||||
can deploy itself declaratively — document your exact bootstrap steps)
|
||||
- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI
|
||||
runner; the runner only needs push access to git, ArgoCD does the rest
|
||||
- Federated OIDC (one identity provider fronting every service that
|
||||
supports it)
|
||||
|
||||
**Architecture principles:**
|
||||
- Immutable OS (Talos — no SSH, declarative config)
|
||||
- Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env`
|
||||
- ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md`
|
||||
- Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync
|
||||
- Federated OIDC (Authentik provider for all services)
|
||||
## Deployment Checklist (per new service)
|
||||
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor
|
||||
- [ ] All credentials in Vault (never in pod env, ConfigMap, or code)
|
||||
- [ ] Ingress rule in `k8s/ingress/` with TLS cert
|
||||
- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-<name>.yaml`
|
||||
- [ ] Alert rules in `k8s/monitoring/alerts/svc-<name>-rules.yaml` (if needed)
|
||||
- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies
|
||||
|
||||
## Hard Rules
|
||||
|
||||
1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner)
|
||||
2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."`
|
||||
3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`)
|
||||
4. **External services via Ingress** — All public endpoints via TLS (homelab-ca)
|
||||
5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault
|
||||
6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs.
|
||||
7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners.
|
||||
- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics
|
||||
- [ ] All credentials in your secrets backend (never in plain values.yaml,
|
||||
pod env directly, or committed anywhere in cleartext)
|
||||
- [ ] Ingress rule with TLS, if externally reachable
|
||||
- [ ] Dashboard + alert rules, if metrics are exposed
|
||||
- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file,
|
||||
**not** a standalone `helm install`/`kubectl apply` run by hand
|
||||
- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`,
|
||||
`kubectl kustomize <dir>/` (if applicable), or `helm template` against
|
||||
the exact pinned chart version (if Helm-sourced)
|
||||
- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches
|
||||
your new commit — not just that `status.sync.status` says `Synced`
|
||||
(see Gotchas — a stale hook can hide behind an otherwise-current app)
|
||||
|
||||
## Git & Release
|
||||
|
||||
**Multi-remote push:**
|
||||
```bash
|
||||
git push origin main
|
||||
```
|
||||
|
||||
**Incremental commits (service-layer grouped):**
|
||||
**Incremental commits (service-layer grouped) tend to age well:**
|
||||
- Foundation & Docs
|
||||
- Helmfile & Core Infra
|
||||
- Storage Layer
|
||||
- Core Infra (CNI, ingress, cert management, storage)
|
||||
- Observability Stack
|
||||
- IAM & Secrets
|
||||
- CI/CD & GitOps
|
||||
- Messaging Infrastructure
|
||||
- Messaging / Data Infrastructure
|
||||
- Applications & Utilities
|
||||
|
||||
Grouping by layer (rather than by day or by "misc fixes") makes it much
|
||||
easier to `git log --oneline -- <path>` your way back to *why* a given
|
||||
piece of config looks the way it does, months later.
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
# CLAUDE.md — Homelab Project Reference
|
||||
|
||||
## Cluster Topology (3 control-plane HA)
|
||||
|
||||
| Node | IP | Zone | Scheduling | Storage |
|
||||
|------|----|----|-----------|---------|
|
||||
| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
|
||||
| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
|
||||
| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
|
||||
|
||||
3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
|
||||
holds storage → stateful services are single-instance. Full detail + gotchas in
|
||||
`USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`.
|
||||
|
||||
## Deployment Model: ArgoCD GitOps (helmfile is retired)
|
||||
|
||||
**helmfile.yaml.gotmpl and the `core` CLI's helmfile-era workflows described in
|
||||
`USAGE.md`/`project-usage/*.md` are STALE.** Actual practice, 100% of the time:
|
||||
|
||||
```
|
||||
git commit → git push (Forgejo) → ArgoCD auto-sync → cluster
|
||||
```
|
||||
|
||||
App-of-apps structure: `k8s/argocd/root/homelab-root.yaml` (root Application)
|
||||
→ `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each
|
||||
`argocd.argoproj.io/sync-wave` annotated) → each Application points at either
|
||||
a remote Helm chart (+ a second `ref: values` git source for the values file)
|
||||
or a plain git directory of raw manifests.
|
||||
|
||||
**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except:
|
||||
- Pure cleanup of stuck/dead state (delete a failed hook Job so the *next*
|
||||
legitimate sync creates a fresh one — not a config change, just clearing
|
||||
wreckage). Confirm with the user first if in doubt.
|
||||
- One-time bootstrap actions with a genuine circular dependency (Vault
|
||||
`operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md`
|
||||
equivalent scripts).
|
||||
|
||||
## Hard Rules (Never Violate)
|
||||
|
||||
🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
|
||||
node — all replicas are pinned to that node name. Renaming orphans its Longhorn
|
||||
node CR and faults every volume (permanent data loss). Rename/reprovision only
|
||||
the dedicated CPs (.163/.166), never the data node.
|
||||
|
||||
🔴 **Control-plane etcd must advertise on the LAN.** Keep
|
||||
`cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane
|
||||
template — without it Talos advertises on the WireGuard IP and new members hang
|
||||
as non-promoting etcd learners.
|
||||
|
||||
🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit:
|
||||
```bash
|
||||
terraform fmt -recursive terraform/
|
||||
```
|
||||
Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise.
|
||||
|
||||
🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck:
|
||||
1. Update Terraform config (variables.tf, *.tf files)
|
||||
2. Run `terraform apply` (locally or via CI/CD)
|
||||
3. Never bypass with manual kubectl operations
|
||||
|
||||
🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
|
||||
|
||||
🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer.
|
||||
|
||||
🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages.
|
||||
|
||||
🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear.
|
||||
|
||||
🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads.
|
||||
|
||||
🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
|
||||
|
||||
## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster)
|
||||
|
||||
🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently
|
||||
drops anything not listed — no error, no drift shown.** ArgoCD reports
|
||||
`Synced/Healthy` against a manifest set that never included the missing
|
||||
file at all. Symptom: you commit+push a new manifest, ArgoCD says
|
||||
"Synced", but the resource never appears in-cluster. Fix: check every
|
||||
`kustomization.yaml` along the app's source path actually lists your new
|
||||
file. `kubectl kustomize <dir>/` locally reproduces exactly what ArgoCD
|
||||
will apply — always verify with it before pushing.
|
||||
|
||||
🟠 **A `kustomization.yaml` top-level `namespace:` transformer rewrites
|
||||
`metadata.namespace` on *every* resource it builds — including RBAC
|
||||
RoleBindings deliberately targeting a *different* namespace.** If any
|
||||
manifest in that directory needs cross-namespace resources (e.g. a
|
||||
RoleBinding granting access to Secrets in another namespace), either drop
|
||||
the transformer (safe if every resource already sets its own explicit
|
||||
namespace) or move that manifest to its own directory/Application.
|
||||
|
||||
🟠 **PreSync hooks run *before* an Application's own normal (non-hook)
|
||||
resources are synced.** A PreSync-hooked Job that depends on a
|
||||
ServiceAccount/RBAC defined as plain resources in the *same* Application
|
||||
deadlocks: the Job tries to start before its own ServiceAccount exists.
|
||||
Confirmed live — Job sat "Running" for 14+ minutes producing zero pods,
|
||||
`job-controller` event log showed `serviceaccount ... not found` on every
|
||||
retry. Fix: use PostSync instead (runs after that app's own resources are
|
||||
applied), or split the hook into its own earlier-sync-wave Application.
|
||||
|
||||
🟠 **ArgoCD hooks (PreSync/PostSync) are NOT continuously reconciled by
|
||||
`selfHeal` the way normal resources are.** Once a hook Job completes
|
||||
(success or exhausts `backoffLimit` into Failed), it only gets
|
||||
deleted+recreated (per `hook-delete-policy: BeforeHookCreation`) during an
|
||||
**actual new Sync operation** — not from passive drift detection, even
|
||||
with `automated.selfHeal: true`. If you fix a hook Job's spec (image,
|
||||
command, RBAC) and push, `status.sync.revision` may show "caught up" while
|
||||
the *live* hook resource is still running the old, broken spec — because
|
||||
no new operation actually re-ran it. To force a real resync: delete the
|
||||
stuck Job (clear the `argocd.argoproj.io/hook-finalizer` if it's stuck
|
||||
`Terminating`), and if that alone doesn't trigger a fresh full sync, delete
|
||||
+ `kubectl apply -f` the Application object itself (re-reads current git
|
||||
HEAD, starts a genuinely new operation, no cascade-delete of underlying
|
||||
resources since Applications don't carry a cascade finalizer by default —
|
||||
confirm with `kubectl get app <name> -o jsonpath='{.metadata.finalizers}'`
|
||||
first).
|
||||
|
||||
🟠 **ArgoCD's repo-server caches rendered manifests (~120s TTL by default,
|
||||
`argocd-cm` → `timeout.reconciliation`).** If you change a values file and
|
||||
`Application` fields both, sometimes the old rendering wins on the next
|
||||
sync. Restart `argocd-repo-server` after big multi-source/values changes if
|
||||
sync behavior looks stale.
|
||||
|
||||
🟠 **ArgoCD `repoURL` pointing at a hostname that CoreDNS rewrites to the
|
||||
nginx ingress controller (for TLS termination) breaks if the URL includes
|
||||
a non-standard port.** nginx only listens on 80/443 — `http://host:3000/...`
|
||||
silently times out (`context deadline exceeded`) if `host` resolves to the
|
||||
ingress controller, not the actual backend service. This blocked **every
|
||||
single Application's sync** cluster-wide simultaneously (all showed
|
||||
`Unknown` sync status) because the repo-server couldn't fetch git refs at
|
||||
all. Use `https://host/...` (no port) so nginx's default TLS cert + normal
|
||||
443 routing handles it.
|
||||
|
||||
🟠 **Bitnami Docker Hub images no longer publish versioned tags (2025
|
||||
policy change) — only `latest` and sha256-pinned digests remain for their
|
||||
free tier.** A pinned tag like `bitnami/kubectl:1.30` will 404/ImagePullBackOff
|
||||
forever. Verify tags exist first: `curl -s "https://hub.docker.com/v2/repositories/<org>/<image>/tags?page_size=25" | jq -r '.results[].name'`.
|
||||
Prefer avoiding third-party utility images entirely where possible — e.g.
|
||||
`python:3.12-alpine` + stdlib `urllib.request` to fetch a static binary
|
||||
(kubectl) avoids depending on any registry's tagging policy at all.
|
||||
|
||||
🟠 **Non-root containers (`runAsNonRoot: true`, non-zero UID) can't `apk
|
||||
add` in Alpine-based images** — apk's working directories and most of
|
||||
`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log:
|
||||
Permission denied`. Use `/tmp` (world-writable) for any binary you need to
|
||||
download/install at runtime, and extend `PATH` rather than writing to
|
||||
`/usr/local/bin`.
|
||||
|
||||
🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or
|
||||
wrong-schema key is silently a no-op, not an error.** Confirmed root cause
|
||||
of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual
|
||||
chart version pinned (`temporalio/[email protected]`) uses a flat
|
||||
`server.config.persistence.<store>.driver/.sql` schema, but the values file
|
||||
used the *newer* chart's `datastores:`-wrapped schema (introduced in a
|
||||
later major version) — silently ignored, so persistence stayed on the
|
||||
chart's Cassandra default the entire time. **Before assuming "this chart
|
||||
doesn't support X," clone the chart at the exact pinned tag/version and run
|
||||
`helm template` with your real values — diff the rendered output, don't
|
||||
trust values.yaml comments/examples from the chart's current `main`
|
||||
branch, which may not match your pinned version's schema at all.**
|
||||
|
||||
## Workflow
|
||||
|
||||
- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push**
|
||||
- If fmt check fails in CI, fix locally, commit fmt changes, push again
|
||||
- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize <dir>/` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`)
|
||||
|
||||
## Documentation Map
|
||||
|
||||
- `CLAUDE.md` (this file) — private, cluster-specific, always current
|
||||
- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs)
|
||||
- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a
|
||||
deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that
|
||||
does not reflect actual current practice (100% ArgoCD GitOps as of this
|
||||
writing). Treat as historical reference only until rewritten; do not
|
||||
follow their deployment procedures literally.
|
||||
- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism
|
||||
|
||||
---
|
||||
|
||||
**Last updated:** 2026-07-22
|
||||
@@ -0,0 +1,200 @@
|
||||
# GitOps Workflow (ArgoCD) — Current Practice
|
||||
|
||||
**This is the actual, current deployment workflow.** Every other doc in this
|
||||
directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
|
||||
`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
|
||||
retired. This file exists to be the accurate replacement for "how do I
|
||||
actually deploy/change something" until those older docs get a full rewrite.
|
||||
|
||||
## The Loop
|
||||
|
||||
```
|
||||
1. Edit files under k8s/
|
||||
2. Validate locally (see "Validation" below — do not skip this)
|
||||
3. git add -A && git commit -m "..." && git push
|
||||
4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
|
||||
up on its next reconciliation cycle (default ~3 min, or immediately if
|
||||
you trigger a refresh)
|
||||
5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
|
||||
matches your new commit hash - not just that status.sync.status says
|
||||
"Synced" (a stale hook resource can hide behind an otherwise-current app,
|
||||
see "ArgoCD Hooks" below)
|
||||
```
|
||||
|
||||
There is no `helmfile apply`, no `core iam create-app`, no manual `helm
|
||||
install` in the current workflow. If you find yourself reaching for any of
|
||||
those, stop — figure out the ArgoCD-native equivalent instead.
|
||||
|
||||
## App-of-Apps Structure
|
||||
|
||||
```
|
||||
k8s/argocd/root/homelab-root.yaml # the one Application ArgoCD bootstraps by hand
|
||||
→ source: k8s/argocd/apps/ # directory of child Application manifests
|
||||
00-substrate.yaml # sync-wave 0: cert-manager, ingress-nginx, CRDs
|
||||
00-secrets.yaml # sync-wave 0: SOPS secrets plugin
|
||||
05-networking.yaml # sync-wave... etc, ascending
|
||||
10-storage-observability.yaml
|
||||
20-logging.yaml
|
||||
30-security.yaml # iam (Authentik, Vault)
|
||||
40-data.yaml # CNPG postgres
|
||||
50-messaging.yaml # Kafka/sqs
|
||||
60-applications.yaml # end-user workloads (Temporal, Portainer, etc.)
|
||||
```
|
||||
|
||||
Each child `Application` either:
|
||||
- Points at a **remote Helm chart** with a **second git source** (`ref:
|
||||
values`) supplying just the values file — lets you pin the chart version
|
||||
independently while your values live under normal review/history, e.g.:
|
||||
```yaml
|
||||
spec:
|
||||
sources:
|
||||
- repoURL: https://example.com/helm-charts
|
||||
chart: some-chart
|
||||
targetRevision: "1.2.3"
|
||||
helm:
|
||||
valueFiles:
|
||||
- $values/k8s/applications/foo/foo-values.yaml
|
||||
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
||||
targetRevision: main
|
||||
ref: values
|
||||
```
|
||||
- Or points at a **plain git directory** of raw manifests, optionally with
|
||||
its own `kustomization.yaml`.
|
||||
|
||||
`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
|
||||
controls ordering — lower N syncs first. Within a single Application,
|
||||
resource-level sync-wave annotations further order that app's own
|
||||
resources.
|
||||
|
||||
## Validation Before Every Push
|
||||
|
||||
Pick whichever applies:
|
||||
|
||||
```bash
|
||||
# Plain manifests, no kustomization.yaml in the directory
|
||||
kubectl apply --dry-run=client -f path/to/file.yaml
|
||||
|
||||
# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
|
||||
# a plain `apply --dry-run` on individual files will NOT catch a resource
|
||||
# missing from the resources: allowlist
|
||||
kubectl kustomize path/to/dir/
|
||||
|
||||
# Helm-sourced Application - render against the EXACT pinned chart version,
|
||||
# not whatever `helm repo` happens to have cached, and not the chart's
|
||||
# current main-branch example values (schemas can differ between versions)
|
||||
git clone --depth 1 <chart-repo-url> /tmp/chart-check
|
||||
cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
|
||||
helm dependency build # if the chart has subchart dependencies
|
||||
helm template . --values /path/to/your/values.yaml --namespace <ns>
|
||||
```
|
||||
|
||||
## ArgoCD Hooks (PreSync / PostSync / Sync)
|
||||
|
||||
Used for one-shot Jobs that need to run at a specific point in an
|
||||
Application's sync (schema migrations, secret-copy jobs, provisioning
|
||||
scripts). Two properties that bite people:
|
||||
|
||||
1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
|
||||
PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
|
||||
defined as *plain* (non-hook) resources in the same Application will
|
||||
deadlock — those get created during the Sync phase, which is *after*
|
||||
PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
|
||||
describe job` shows `FailedCreate ... serviceaccount ... not found`
|
||||
repeating forever. **If your hook needs resources from its own
|
||||
Application, make it PostSync, not PreSync.**
|
||||
|
||||
2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
|
||||
failed hook Job only gets deleted+recreated (per
|
||||
`hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
|
||||
operation* — not from ambient drift detection. If you fix a broken
|
||||
hook's spec (image, RBAC, command) and push, the Application's
|
||||
`status.sync.revision` can update to show "caught up" (because the
|
||||
*normal*, non-hook resources genuinely match) while the *live* hook Job
|
||||
is still running the old broken spec, because no fresh sync operation
|
||||
actually re-executed it.
|
||||
|
||||
**To force it, in order of preference:**
|
||||
```bash
|
||||
# 1. Just delete the stuck/failed job - if a legitimate future sync
|
||||
# happens for any other reason, this clears the way for it
|
||||
kubectl -n <ns> delete job <hook-job-name>
|
||||
|
||||
# 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
|
||||
# deletion because the operation tracking it is itself stuck):
|
||||
kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
|
||||
kubectl -n <ns> patch job <hook-job-name> --type json \
|
||||
-p '[{"op":"remove","path":"/metadata/finalizers"}]'
|
||||
|
||||
# 3. If deleting the job alone doesn't trigger a genuinely fresh sync
|
||||
# (check: does a new job appear using your LATEST commit's spec?
|
||||
# compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
|
||||
# against what your latest commit says), the Application's own
|
||||
# operation state is stuck. Confirm it has no cascade finalizer first
|
||||
# (Applications don't carry one by default - only delete if this is
|
||||
# empty):
|
||||
kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
|
||||
# then:
|
||||
kubectl -n argocd delete application <app-name>
|
||||
kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
|
||||
```
|
||||
Step 3 re-reads current git HEAD from scratch and starts a genuinely new
|
||||
operation — this is the reliable way to actually pick up a hook fix when
|
||||
steps 1-2 don't unstick it.
|
||||
|
||||
## kustomization.yaml Pitfalls
|
||||
|
||||
- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
|
||||
manifest file to a directory that has a `kustomization.yaml`, forget to
|
||||
list it under `resources:`, and ArgoCD will silently never apply it — no
|
||||
error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
|
||||
<dir>/` locally before pushing to see exactly what will be built.
|
||||
- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
|
||||
every resource in the build**, including RBAC bindings that deliberately
|
||||
target a *different* namespace (e.g. a RoleBinding granting cross-
|
||||
namespace Secret access for a sync job's ServiceAccount). If any manifest
|
||||
in the directory needs to live in a different namespace than the
|
||||
transformer specifies, either remove the transformer (safe if every
|
||||
manifest already sets its own explicit `namespace:`) or move that
|
||||
manifest to its own directory/Application entirely.
|
||||
|
||||
## Cross-Namespace Secrets
|
||||
|
||||
Kubernetes Secrets are strictly namespace-scoped — a Deployment in
|
||||
namespace A cannot reference a Secret living in namespace B via
|
||||
`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
|
||||
CNPG auto-generates DB credentials in its own operator namespace) but a
|
||||
consumer lives in a different namespace, you need an explicit copy
|
||||
mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
|
||||
`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
|
||||
example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
|
||||
(one per namespace involved) that reads the source Secret and re-creates it
|
||||
in the target namespace. Give this its own Application/sync-wave (earlier
|
||||
than whatever consumes the copied Secret) rather than folding it into an
|
||||
existing Application that has a namespace-transforming `kustomization.yaml`
|
||||
(see above).
|
||||
|
||||
## Common Failure: Everything Shows "Unknown" Sync Status At Once
|
||||
|
||||
If *every* Application (not just one) suddenly shows `Unknown` sync status
|
||||
simultaneously, check the Application controller's logs for the actual
|
||||
`repoURL` fetch error before assuming something is wrong with any
|
||||
individual app's manifests:
|
||||
|
||||
```bash
|
||||
kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
|
||||
```
|
||||
|
||||
Common cause in a homelab with CoreDNS rewriting your git host to the
|
||||
ingress controller for internal traffic: if `repoURL` uses a non-standard
|
||||
port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
|
||||
hostname to the ingress controller service (which only listens on 80/443),
|
||||
every fetch attempt times out. Fix: use the default-port HTTPS URL
|
||||
(`https://forgejo.example.com/...`) so it actually reaches the ingress
|
||||
controller correctly.
|
||||
|
||||
## See Also
|
||||
|
||||
- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
|
||||
cross-referenced from the main project reference
|
||||
- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
|
||||
applicable regardless of deployment mechanism
|
||||
Reference in New Issue
Block a user