diff --git a/.gitignore b/.gitignore index a38c7cb..5e8863a 100644 --- a/.gitignore +++ b/.gitignore @@ -38,8 +38,9 @@ k8s/storage/test/test *.key *.conf -# Allowed markdown: CLAUDE.example.md, README.md, TROUBLESHOOTING.md -CLAUDE.md +# CLAUDE.md is now version-controlled (was previously excluded as a +# private-notes file; contains no secrets - just architecture, IPs +# [private RFC1918 space], and operational lessons, same bar as README.md). # Terraform state and cache (local files, remote state in MinIO) .terraform/ diff --git a/CLAUDE.example.md b/CLAUDE.example.md index 34b03f5..4870b5e 100644 --- a/CLAUDE.example.md +++ b/CLAUDE.example.md @@ -1,86 +1,210 @@ -# CLAUDE.md — Homelab Integration Guide +# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template) -**Homelab:** A bare-metal three-node Kubernetes cluster running Talos Linux with a full observability stack, SSO via Authentik, secret management via Vault, and CI/CD infrastructure (Forgejo + Argo CD, deployed). +> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own +> node IPs/hostnames/secrets, and delete this notice. Nothing in this file +> should contain real credentials, real IPs beyond illustrative examples, or +> anything that would matter if this file became public. It's meant to be +> shared across homelabs running similar hardware/topology (3-node bare-metal +> Talos Kubernetes + ArgoCD GitOps), not just this one. -## Cluster Topology (3 control-plane HA, since 2026-07-20) +**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a +full observability stack, SSO via Authentik, secret management via Vault, and +CI/CD infrastructure (self-hosted git forge + Argo CD). + +## Cluster Topology (adjust to your hardware) | Node | IP | Zone | Scheduling | Storage | |------|----|----|-----------|---------| -| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node | -| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none | -| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none | +| `` | `` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) | +| `` | `` | az-b | dedicated (`NoSchedule`) | none | +| `` | `` | az-c | dedicated (`NoSchedule`) | none | -3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and -holds storage → stateful services are single-instance (e.g. CNPG `ddb-cluster` -= 1 instance). API endpoint is single-homed to `.213` (no VIP yet). +If your storage layer (Longhorn, local-path, etc.) only runs on one node, +every stateful workload pinned to that storage class is effectively +single-instance regardless of your control-plane HA count — document that +explicitly here, it changes your failure-mode assumptions everywhere else. + +## Deployment Model: ArgoCD GitOps (app-of-apps) + +``` +git commit → git push (your git forge) → ArgoCD auto-sync → cluster +``` + +Structure: +- One root `Application` (`k8s/argocd/root/`) pointing at a directory of + child `Application` manifests (`k8s/argocd/apps/*.yaml`) +- Each child Application is either: + - A remote Helm chart + a **second** git source (`ref: values`) supplying + just the values file — lets you pin an upstream chart version while + keeping your values under normal git history/review + - A plain git directory of raw manifests (optionally with a + `kustomization.yaml`) +- `argocd.argoproj.io/sync-wave` annotations control ordering across + Applications (lower number syncs first) + +**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except: +- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook + Job so the next real sync can create a fresh one) — this is not a config + change, just clearing wreckage that GitOps itself won't clean up + automatically (see Gotchas below) +- Genuine one-time bootstrap circular dependencies (e.g. Vault + `operator init`/unseal — nothing can configure Vault's own unseal keys + before Vault has generated them) + +## Hard Rules (adapt freely, but keep something like these) + +🔴 **Identify and document your storage/stateful-singleton node explicitly.** +Whatever node holds your CSI driver's data (Longhorn, local-path, etc.), +renaming or wiping it orphans every PVC pinned there. Name it here, in +caps, so nobody "cleans up" it by accident. + +🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay, +pin the advertised subnet explicitly** (e.g. Talos's +`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the +wrong interface and new members hang as non-promoting learners. + +🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit +that touches infra code.** Wire this into CI as a hard gate, not a +suggestion. + +🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never +manually mutate resources it manages.** State drift is the single most +common cause of "why did my last apply undo my manual fix" confusion. +Fix the source, re-apply/re-sync, never bypass. + +🔴 **Never delete a PVC without confirming replica count / backup +freshness first.** This is always a one-way door. + +🔴 **Decide your commit message convention up front and enforce it.** +(This template's origin project uses: no co-authored-by footers, single-line +commit summarizing what/why, solo-authorship assumption — adjust to your +team's norms.) + +🔴 **Decide your git workflow (rebase vs merge) up front and stick to it** +cluster-wide, across every contributor/agent working in the repo. + +🔴 **Long-running commands should not block a synchronous session** — run +them in the background and poll, especially anything that waits on a +Kubernetes rollout, an image pull, or a Terraform apply. + +## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab) + +🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist +silently drops anything you forget to list.** No error, no drift shown in +ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that +never included your new file. Always run `kubectl kustomize /` +locally before pushing to confirm exactly what ArgoCD will build. + +🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites +`metadata.namespace` on every resource it builds** — including RBAC +bindings deliberately targeting a *different* namespace (e.g. granting a +ServiceAccount in namespace A read access to Secrets in namespace B). If +any manifest needs cross-namespace RBAC, either drop the transformer +(safe if every resource already sets its own explicit namespace) or give +that manifest its own Application/directory. + +🟠 **PreSync hooks run before an Application's own normal resources are +synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as +plain (non-hook) resources in the *same* Application will deadlock — it +tries to start before its own permissions exist. Use PostSync instead if +the hook needs resources from its own Application, or move the +prerequisite RBAC into an earlier sync-wave Application. + +🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a +hook Job finishes (success, or exhausts `backoffLimit`), it's only +deleted+recreated during an *actual new Sync operation* — not by passive +drift detection, even with `automated.selfHeal: true` on. If you fix a +broken hook's spec and push, the Application's `status.sync.revision` can +show "caught up" while the live hook resource is still the old, broken +one, because no new operation actually re-ran it. To force it: delete the +stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's +stuck `Terminating`), and if that alone doesn't trigger a fresh full sync, +delete + re-`kubectl apply -f` the Application object itself. + +🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy +hostname that only listens on 80/443, don't use a non-standard port in the +URL** — it'll silently time out trying to reach a port the proxy never +opened, and depending on your setup this can block *every* Application's +sync simultaneously (repo-server can't fetch git refs for anything). + +🟠 **Don't pin exact version tags for images from registries that don't +guarantee tag retention** (Bitnami stopped publishing versioned tags for +free-tier images in 2025 — only `latest` + sha256 digests remain). Verify +a tag actually exists before pinning it, or prefer minimal base images + +a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a +static binary) to avoid depending on any third party's tagging policy. + +🟠 **Non-root containers can't `apk add`/`apt install` in most default +base images** — package manager directories are root-owned. Use a +world-writable scratch dir (`/tmp`) for anything you need to +download/install at runtime instead. + +🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a +values schema copied from the wrong chart *version's* docs/examples, is +silently a no-op — not an error. Before concluding "this chart doesn't +support X," clone the chart at your exact pinned version/tag and run +`helm template` against your real values file, then diff the rendered +output. Don't trust a chart's current `main`-branch example values file +if you're pinned to an older release — schemas do change between major +versions without warning in your own values file. ## Service Integration Routes -**New service? Pick your stack below:** +**New service? Pick your stack below** (adjust doc paths to match your repo): | Need | Doc | Example | |------|-----|---------| | **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens | -| **Async messaging** | `project-usage/sqs-messaging.md` | Kafka topic consumers, fire-and-forget, DLQ | +| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ | | **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend | -| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | GitHub Actions syntax, image push, Argo CD sync | +| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync | | **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines | | **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas | | **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts | | **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal | | **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS | -## Cluster Essentials +## Cluster Essentials (fill in your own inventory) -**22 namespaces, 18 releases:** -``` -Core: cert-manager, ingress-nginx, kube-system, cilium -Storage: longhorn-system, storage (MinIO) -Data: ddb (PostgreSQL), iam (Authentik + Vault) -Observability: logging (Loki + Grafana), monitoring (Prometheus) -Apps: cicd (Forgejo + Argo CD), sqs (Kafka + kmsvc), temporal, story-crater-backend -``` +**Architecture principles (adjust to taste, but these travel well):** +- Immutable OS (Talos, or similar — no SSH, fully declarative config) +- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in + git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env` +- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap + documented separately (chicken-and-egg: ArgoCD needs to exist before it + can deploy itself declaratively — document your exact bootstrap steps) +- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI + runner; the runner only needs push access to git, ArgoCD does the rest +- Federated OIDC (one identity provider fronting every service that + supports it) -**Architecture principles:** -- Immutable OS (Talos — no SSH, declarative config) -- Secrets in Vault + SOPS-encrypted (`*.enc.yaml`, age); never commit `.env` -- ArgoCD app-of-apps = CD source of truth (`k8s/argocd/root` → `k8s/argocd/apps/*`); helmfile is deprecated. Two-phase bootstrap in `k8s/argocd/bootstrap/BOOTSTRAP.md` -- Pull-based GitOps (Argo CD, no kubeconfig in CI); iterate = `git push` to Forgejo → auto-sync -- Federated OIDC (Authentik provider for all services) +## Deployment Checklist (per new service) - -## Deployment Checklist - -- [ ] Service has Prometheus `/metrics` endpoint or ServiceMonitor -- [ ] All credentials in Vault (never in pod env, ConfigMap, or code) -- [ ] Ingress rule in `k8s/ingress/` with TLS cert -- [ ] Grafana dashboard in `k8s/monitoring/dashboards/svc-.yaml` -- [ ] Alert rules in `k8s/monitoring/alerts/svc--rules.yaml` (if needed) -- [ ] Helm release in `helmfile.yaml.gotmpl` with correct `needs:` dependencies - -## Hard Rules - -1. **No kubeconfig in CI** — Argo CD bridges gap (pull-based, never push secrets to runner) -2. **Field name = variable name** — In Vault: `talos put cluster/KAFKA_BOOTSTRAP KAFKA_BOOTSTRAP="..."` -3. **Secrets via volumes** — Never `--env` flag in pod specs (exposes in `kubectl describe`) -4. **External services via Ingress** — All public endpoints via TLS (homelab-ca) -5. **Never commit `.env`** — Only `.env.example` in git; real secrets in Vault -6. **Never rename or wipe `talos-cp-1` (.213)** — sole Longhorn storage node; renaming orphans its node CR and faults every volume (permanent data loss). Rename/reprovision only the dedicated CPs. -7. **Control-plane etcd advertises on the LAN** — keep `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]`, else Talos advertises on WireGuard and new members hang as etcd learners. +- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics +- [ ] All credentials in your secrets backend (never in plain values.yaml, + pod env directly, or committed anywhere in cleartext) +- [ ] Ingress rule with TLS, if externally reachable +- [ ] Dashboard + alert rules, if metrics are exposed +- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file, + **not** a standalone `helm install`/`kubectl apply` run by hand +- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`, + `kubectl kustomize /` (if applicable), or `helm template` against + the exact pinned chart version (if Helm-sourced) +- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches + your new commit — not just that `status.sync.status` says `Synced` + (see Gotchas — a stale hook can hide behind an otherwise-current app) ## Git & Release -**Multi-remote push:** -```bash -git push origin main -``` - -**Incremental commits (service-layer grouped):** +**Incremental commits (service-layer grouped) tend to age well:** - Foundation & Docs -- Helmfile & Core Infra -- Storage Layer +- Core Infra (CNI, ingress, cert management, storage) - Observability Stack - IAM & Secrets - CI/CD & GitOps -- Messaging Infrastructure +- Messaging / Data Infrastructure - Applications & Utilities + +Grouping by layer (rather than by day or by "misc fixes") makes it much +easier to `git log --oneline -- ` your way back to *why* a given +piece of config looks the way it does, months later. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f0a7fda --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,181 @@ +# CLAUDE.md — Homelab Project Reference + +## Cluster Topology (3 control-plane HA) + +| Node | IP | Zone | Scheduling | Storage | +|------|----|----|-----------|---------| +| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node | +| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none | +| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none | + +3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and +holds storage → stateful services are single-instance. Full detail + gotchas in +`USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`. + +## Deployment Model: ArgoCD GitOps (helmfile is retired) + +**helmfile.yaml.gotmpl and the `core` CLI's helmfile-era workflows described in +`USAGE.md`/`project-usage/*.md` are STALE.** Actual practice, 100% of the time: + +``` +git commit → git push (Forgejo) → ArgoCD auto-sync → cluster +``` + +App-of-apps structure: `k8s/argocd/root/homelab-root.yaml` (root Application) +→ `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each +`argocd.argoproj.io/sync-wave` annotated) → each Application points at either +a remote Helm chart (+ a second `ref: values` git source for the values file) +or a plain git directory of raw manifests. + +**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except: +- Pure cleanup of stuck/dead state (delete a failed hook Job so the *next* + legitimate sync creates a fresh one — not a config change, just clearing + wreckage). Confirm with the user first if in doubt. +- One-time bootstrap actions with a genuine circular dependency (Vault + `operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md` + equivalent scripts). + +## Hard Rules (Never Violate) + +🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage +node — all replicas are pinned to that node name. Renaming orphans its Longhorn +node CR and faults every volume (permanent data loss). Rename/reprovision only +the dedicated CPs (.163/.166), never the data node. + +🔴 **Control-plane etcd must advertise on the LAN.** Keep +`cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane +template — without it Talos advertises on the WireGuard IP and new members hang +as non-promoting etcd learners. + +🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit: +```bash +terraform fmt -recursive terraform/ +``` +Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise. + +🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck: +1. Update Terraform config (variables.tf, *.tf files) +2. Run `terraform apply` (locally or via CI/CD) +3. Never bypass with manual kubectl operations + +🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first. + +🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer. + +🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages. + +🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear. + +🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads. + +🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources. + +## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster) + +🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently +drops anything not listed — no error, no drift shown.** ArgoCD reports +`Synced/Healthy` against a manifest set that never included the missing +file at all. Symptom: you commit+push a new manifest, ArgoCD says +"Synced", but the resource never appears in-cluster. Fix: check every +`kustomization.yaml` along the app's source path actually lists your new +file. `kubectl kustomize /` locally reproduces exactly what ArgoCD +will apply — always verify with it before pushing. + +🟠 **A `kustomization.yaml` top-level `namespace:` transformer rewrites +`metadata.namespace` on *every* resource it builds — including RBAC +RoleBindings deliberately targeting a *different* namespace.** If any +manifest in that directory needs cross-namespace resources (e.g. a +RoleBinding granting access to Secrets in another namespace), either drop +the transformer (safe if every resource already sets its own explicit +namespace) or move that manifest to its own directory/Application. + +🟠 **PreSync hooks run *before* an Application's own normal (non-hook) +resources are synced.** A PreSync-hooked Job that depends on a +ServiceAccount/RBAC defined as plain resources in the *same* Application +deadlocks: the Job tries to start before its own ServiceAccount exists. +Confirmed live — Job sat "Running" for 14+ minutes producing zero pods, +`job-controller` event log showed `serviceaccount ... not found` on every +retry. Fix: use PostSync instead (runs after that app's own resources are +applied), or split the hook into its own earlier-sync-wave Application. + +🟠 **ArgoCD hooks (PreSync/PostSync) are NOT continuously reconciled by +`selfHeal` the way normal resources are.** Once a hook Job completes +(success or exhausts `backoffLimit` into Failed), it only gets +deleted+recreated (per `hook-delete-policy: BeforeHookCreation`) during an +**actual new Sync operation** — not from passive drift detection, even +with `automated.selfHeal: true`. If you fix a hook Job's spec (image, +command, RBAC) and push, `status.sync.revision` may show "caught up" while +the *live* hook resource is still running the old, broken spec — because +no new operation actually re-ran it. To force a real resync: delete the +stuck Job (clear the `argocd.argoproj.io/hook-finalizer` if it's stuck +`Terminating`), and if that alone doesn't trigger a fresh full sync, delete ++ `kubectl apply -f` the Application object itself (re-reads current git +HEAD, starts a genuinely new operation, no cascade-delete of underlying +resources since Applications don't carry a cascade finalizer by default — +confirm with `kubectl get app -o jsonpath='{.metadata.finalizers}'` +first). + +🟠 **ArgoCD's repo-server caches rendered manifests (~120s TTL by default, +`argocd-cm` → `timeout.reconciliation`).** If you change a values file and +`Application` fields both, sometimes the old rendering wins on the next +sync. Restart `argocd-repo-server` after big multi-source/values changes if +sync behavior looks stale. + +🟠 **ArgoCD `repoURL` pointing at a hostname that CoreDNS rewrites to the +nginx ingress controller (for TLS termination) breaks if the URL includes +a non-standard port.** nginx only listens on 80/443 — `http://host:3000/...` +silently times out (`context deadline exceeded`) if `host` resolves to the +ingress controller, not the actual backend service. This blocked **every +single Application's sync** cluster-wide simultaneously (all showed +`Unknown` sync status) because the repo-server couldn't fetch git refs at +all. Use `https://host/...` (no port) so nginx's default TLS cert + normal +443 routing handles it. + +🟠 **Bitnami Docker Hub images no longer publish versioned tags (2025 +policy change) — only `latest` and sha256-pinned digests remain for their +free tier.** A pinned tag like `bitnami/kubectl:1.30` will 404/ImagePullBackOff +forever. Verify tags exist first: `curl -s "https://hub.docker.com/v2/repositories///tags?page_size=25" | jq -r '.results[].name'`. +Prefer avoiding third-party utility images entirely where possible — e.g. +`python:3.12-alpine` + stdlib `urllib.request` to fetch a static binary +(kubectl) avoids depending on any registry's tagging policy at all. + +🟠 **Non-root containers (`runAsNonRoot: true`, non-zero UID) can't `apk +add` in Alpine-based images** — apk's working directories and most of +`/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log: +Permission denied`. Use `/tmp` (world-writable) for any binary you need to +download/install at runtime, and extend `PATH` rather than writing to +`/usr/local/bin`. + +🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or +wrong-schema key is silently a no-op, not an error.** Confirmed root cause +of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual +chart version pinned (`temporalio/helm-charts@0.74.0`) uses a flat +`server.config.persistence..driver/.sql` schema, but the values file +used the *newer* chart's `datastores:`-wrapped schema (introduced in a +later major version) — silently ignored, so persistence stayed on the +chart's Cassandra default the entire time. **Before assuming "this chart +doesn't support X," clone the chart at the exact pinned tag/version and run +`helm template` with your real values — diff the rendered output, don't +trust values.yaml comments/examples from the chart's current `main` +branch, which may not match your pinned version's schema at all.** + +## Workflow + +- **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push** +- If fmt check fails in CI, fix locally, commit fmt changes, push again +- For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize /` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`) + +## Documentation Map + +- `CLAUDE.md` (this file) — private, cluster-specific, always current +- `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs) +- `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a + deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that + does not reflect actual current practice (100% ArgoCD GitOps as of this + writing). Treat as historical reference only until rewritten; do not + follow their deployment procedures literally. +- `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism + +--- + +**Last updated:** 2026-07-22 diff --git a/project-usage/gitops-workflow.md b/project-usage/gitops-workflow.md new file mode 100644 index 0000000..482ac1d --- /dev/null +++ b/project-usage/gitops-workflow.md @@ -0,0 +1,200 @@ +# GitOps Workflow (ArgoCD) — Current Practice + +**This is the actual, current deployment workflow.** Every other doc in this +directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older +`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been +retired. This file exists to be the accurate replacement for "how do I +actually deploy/change something" until those older docs get a full rewrite. + +## The Loop + +``` +1. Edit files under k8s/ +2. Validate locally (see "Validation" below — do not skip this) +3. git add -A && git commit -m "..." && git push +4. ArgoCD (automated sync + selfHeal on almost every Application) picks it + up on its next reconciliation cycle (default ~3 min, or immediately if + you trigger a refresh) +5. Confirm: kubectl -n argocd get app -o jsonpath='{.status.sync.revision}' + matches your new commit hash - not just that status.sync.status says + "Synced" (a stale hook resource can hide behind an otherwise-current app, + see "ArgoCD Hooks" below) +``` + +There is no `helmfile apply`, no `core iam create-app`, no manual `helm +install` in the current workflow. If you find yourself reaching for any of +those, stop — figure out the ArgoCD-native equivalent instead. + +## App-of-Apps Structure + +``` +k8s/argocd/root/homelab-root.yaml # the one Application ArgoCD bootstraps by hand + → source: k8s/argocd/apps/ # directory of child Application manifests + 00-substrate.yaml # sync-wave 0: cert-manager, ingress-nginx, CRDs + 00-secrets.yaml # sync-wave 0: SOPS secrets plugin + 05-networking.yaml # sync-wave... etc, ascending + 10-storage-observability.yaml + 20-logging.yaml + 30-security.yaml # iam (Authentik, Vault) + 40-data.yaml # CNPG postgres + 50-messaging.yaml # Kafka/sqs + 60-applications.yaml # end-user workloads (Temporal, Portainer, etc.) +``` + +Each child `Application` either: +- Points at a **remote Helm chart** with a **second git source** (`ref: + values`) supplying just the values file — lets you pin the chart version + independently while your values live under normal review/history, e.g.: + ```yaml + spec: + sources: + - repoURL: https://example.com/helm-charts + chart: some-chart + targetRevision: "1.2.3" + helm: + valueFiles: + - $values/k8s/applications/foo/foo-values.yaml + - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git + targetRevision: main + ref: values + ``` +- Or points at a **plain git directory** of raw manifests, optionally with + its own `kustomization.yaml`. + +`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata +controls ordering — lower N syncs first. Within a single Application, +resource-level sync-wave annotations further order that app's own +resources. + +## Validation Before Every Push + +Pick whichever applies: + +```bash +# Plain manifests, no kustomization.yaml in the directory +kubectl apply --dry-run=client -f path/to/file.yaml + +# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds, +# a plain `apply --dry-run` on individual files will NOT catch a resource +# missing from the resources: allowlist +kubectl kustomize path/to/dir/ + +# Helm-sourced Application - render against the EXACT pinned chart version, +# not whatever `helm repo` happens to have cached, and not the chart's +# current main-branch example values (schemas can differ between versions) +git clone --depth 1 /tmp/chart-check +cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout +helm dependency build # if the chart has subchart dependencies +helm template . --values /path/to/your/values.yaml --namespace +``` + +## ArgoCD Hooks (PreSync / PostSync / Sync) + +Used for one-shot Jobs that need to run at a specific point in an +Application's sync (schema migrations, secret-copy jobs, provisioning +scripts). Two properties that bite people: + +1. **Ordering is PreSync → Sync (normal resources, by sync-wave) → + PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC + defined as *plain* (non-hook) resources in the same Application will + deadlock — those get created during the Sync phase, which is *after* + PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl + describe job` shows `FailedCreate ... serviceaccount ... not found` + repeating forever. **If your hook needs resources from its own + Application, make it PostSync, not PreSync.** + +2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or + failed hook Job only gets deleted+recreated (per + `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync + operation* — not from ambient drift detection. If you fix a broken + hook's spec (image, RBAC, command) and push, the Application's + `status.sync.revision` can update to show "caught up" (because the + *normal*, non-hook resources genuinely match) while the *live* hook Job + is still running the old broken spec, because no fresh sync operation + actually re-executed it. + + **To force it, in order of preference:** + ```bash + # 1. Just delete the stuck/failed job - if a legitimate future sync + # happens for any other reason, this clears the way for it + kubectl -n delete job + + # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking + # deletion because the operation tracking it is itself stuck): + kubectl -n get job -o jsonpath='{.metadata.finalizers}' + kubectl -n patch job --type json \ + -p '[{"op":"remove","path":"/metadata/finalizers"}]' + + # 3. If deleting the job alone doesn't trigger a genuinely fresh sync + # (check: does a new job appear using your LATEST commit's spec? + # compare kubectl -n get job -o jsonpath='{.spec.template.spec.containers[0].image}' + # against what your latest commit says), the Application's own + # operation state is stuck. Confirm it has no cascade finalizer first + # (Applications don't carry one by default - only delete if this is + # empty): + kubectl -n argocd get app -o jsonpath='{.metadata.finalizers}' + # then: + kubectl -n argocd delete application + kubectl apply -f k8s/argocd/apps/.yaml + ``` + Step 3 re-reads current git HEAD from scratch and starts a genuinely new + operation — this is the reliable way to actually pick up a hook fix when + steps 1-2 don't unstick it. + +## kustomization.yaml Pitfalls + +- **`resources:` is an explicit allowlist, not a directory scan.** Add a new + manifest file to a directory that has a `kustomization.yaml`, forget to + list it under `resources:`, and ArgoCD will silently never apply it — no + error, `Synced/Healthy` shown regardless. Always `kubectl kustomize + /` locally before pushing to see exactly what will be built. +- **A top-level `namespace:` transformer rewrites `metadata.namespace` on + every resource in the build**, including RBAC bindings that deliberately + target a *different* namespace (e.g. a RoleBinding granting cross- + namespace Secret access for a sync job's ServiceAccount). If any manifest + in the directory needs to live in a different namespace than the + transformer specifies, either remove the transformer (safe if every + manifest already sets its own explicit `namespace:`) or move that + manifest to its own directory/Application entirely. + +## Cross-Namespace Secrets + +Kubernetes Secrets are strictly namespace-scoped — a Deployment in +namespace A cannot reference a Secret living in namespace B via +`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g. +CNPG auto-generates DB credentials in its own operator namespace) but a +consumer lives in a different namespace, you need an explicit copy +mechanism. Pattern used in this repo: a small PostSync-hooked Job (see +`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked +example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings +(one per namespace involved) that reads the source Secret and re-creates it +in the target namespace. Give this its own Application/sync-wave (earlier +than whatever consumes the copied Secret) rather than folding it into an +existing Application that has a namespace-transforming `kustomization.yaml` +(see above). + +## Common Failure: Everything Shows "Unknown" Sync Status At Once + +If *every* Application (not just one) suddenly shows `Unknown` sync status +simultaneously, check the Application controller's logs for the actual +`repoURL` fetch error before assuming something is wrong with any +individual app's manifests: + +```bash +kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline" +``` + +Common cause in a homelab with CoreDNS rewriting your git host to the +ingress controller for internal traffic: if `repoURL` uses a non-standard +port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that +hostname to the ingress controller service (which only listens on 80/443), +every fetch attempt times out. Fix: use the default-port HTTPS URL +(`https://forgejo.example.com/...`) so it actually reaches the ingress +controller correctly. + +## See Also + +- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above, + cross-referenced from the main project reference +- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still + applicable regardless of deployment mechanism