homelab/CLAUDE.example.md

# CLAUDE.md — Homelab Integration Guide (Example / Reusable Template)

> **This is a sanitized template.** Copy to `CLAUDE.md`, fill in your own
> node IPs/hostnames/secrets, and delete this notice. Nothing in this file
> should contain real credentials, real IPs beyond illustrative examples, or
> anything that would matter if this file became public. It's meant to be
> shared across homelabs running similar hardware/topology (3-node bare-metal
> Talos Kubernetes + ArgoCD GitOps), not just this one.

**Homelab:** A bare-metal N-node Kubernetes cluster running Talos Linux with a
full observability stack, SSO via Authentik, secret management via Vault, and
CI/CD infrastructure (self-hosted git forge + Argo CD).

## Cluster Topology (adjust to your hardware)

| Node | IP | Zone | Scheduling | Storage |
|------|----|----|-----------|---------|
| `<node-1>` | `<ip>` | az-a | schedulable (all workloads) | sole storage node (if single-node storage) |
| `<node-2>` | `<ip>` | az-b | dedicated (`NoSchedule`) | none |
| `<node-3>` | `<ip>` | az-c | dedicated (`NoSchedule`) | none |

If your storage layer (Longhorn, local-path, etc.) only runs on one node,
every stateful workload pinned to that storage class is effectively
single-instance regardless of your control-plane HA count — document that
explicitly here, it changes your failure-mode assumptions everywhere else.

## Deployment Model: ArgoCD GitOps (app-of-apps)

```
git commit → git push (your git forge) → ArgoCD auto-sync → cluster
```

Structure:
- One root `Application` (`k8s/argocd/root/`) pointing at a directory of
  child `Application` manifests (`k8s/argocd/apps/*.yaml`)
- Each child Application is either:
  - A remote Helm chart + a **second** git source (`ref: values`) supplying
    just the values file — lets you pin an upstream chart version while
    keeping your values under normal git history/review
  - A plain git directory of raw manifests (optionally with a
    `kustomization.yaml`)
- `argocd.argoproj.io/sync-wave` annotations control ordering across
  Applications (lower number syncs first)

**Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages**, except:
- Pure cleanup of stuck/dead state (e.g. deleting a permanently-failed hook
  Job so the next real sync can create a fresh one) — this is not a config
  change, just clearing wreckage that GitOps itself won't clean up
  automatically (see Gotchas below)
- Genuine one-time bootstrap circular dependencies (e.g. Vault
  `operator init`/unseal — nothing can configure Vault's own unseal keys
  before Vault has generated them)

## Hard Rules (adapt freely, but keep something like these)

🔴 **Identify and document your storage/stateful-singleton node explicitly.**
Whatever node holds your CSI driver's data (Longhorn, local-path, etc.),
renaming or wiping it orphans every PVC pinned there. Name it here, in
caps, so nobody "cleans up" it by accident.

🔴 **If you run multi-member etcd across a LAN + VPN/WireGuard overlay,
pin the advertised subnet explicitly** (e.g. Talos's
`cluster.etcd.advertisedSubnets`). Without it, etcd may advertise on the
wrong interface and new members hang as non-promoting learners.

🔴 **Run your IaC formatter (terraform fmt, etc.) before every commit
that touches infra code.** Wire this into CI as a hard gate, not a
suggestion.

🔴 **Whatever your source of truth is (Terraform, ArgoCD, both) — never
manually mutate resources it manages.** State drift is the single most
common cause of "why did my last apply undo my manual fix" confusion.
Fix the source, re-apply/re-sync, never bypass.

🔴 **Never delete a PVC without confirming replica count / backup
freshness first.** This is always a one-way door.

🔴 **Decide your commit message convention up front and enforce it.**
(This template's origin project uses: no co-authored-by footers, single-line
commit summarizing what/why, solo-authorship assumption — adjust to your
team's norms.)

🔴 **Decide your git workflow (rebase vs merge) up front and stick to it**
cluster-wide, across every contributor/agent working in the repo.

🔴 **Long-running commands should not block a synchronous session** — run
them in the background and poll, especially anything that waits on a
Kubernetes rollout, an image pull, or a Terraform apply.

## GitOps / ArgoCD Gotchas (transferable to any ArgoCD-based homelab)

🟠 **A `kustomization.yaml` with an explicit `resources:` allowlist
silently drops anything you forget to list.** No error, no drift shown in
ArgoCD's UI — it just reports `Synced/Healthy` against a manifest set that
never included your new file. Always run `kubectl kustomize <dir>/`
locally before pushing to confirm exactly what ArgoCD will build.

🟠 **A top-level `namespace:` transformer in `kustomization.yaml` rewrites
`metadata.namespace` on every resource it builds** — including RBAC
bindings deliberately targeting a *different* namespace (e.g. granting a
ServiceAccount in namespace A read access to Secrets in namespace B). If
any manifest needs cross-namespace RBAC, either drop the transformer
(safe if every resource already sets its own explicit namespace) or give
that manifest its own Application/directory.

🟠 **PreSync hooks run before an Application's own normal resources are
synced.** A PreSync Job that depends on RBAC/ServiceAccounts defined as
plain (non-hook) resources in the *same* Application will deadlock — it
tries to start before its own permissions exist. Use PostSync instead if
the hook needs resources from its own Application, or move the
prerequisite RBAC into an earlier sync-wave Application.

🟠 **ArgoCD hooks are not continuously reconciled by `selfHeal`.** Once a
hook Job finishes (success, or exhausts `backoffLimit`), it's only
deleted+recreated during an *actual new Sync operation* — not by passive
drift detection, even with `automated.selfHeal: true` on. If you fix a
broken hook's spec and push, the Application's `status.sync.revision` can
show "caught up" while the live hook resource is still the old, broken
one, because no new operation actually re-ran it. To force it: delete the
stuck hook (clear `argocd.argoproj.io/hook-finalizer` manually if it's
stuck `Terminating`), and if that alone doesn't trigger a fresh full sync,
delete + re-`kubectl apply -f` the Application object itself.

🟠 **If you route ArgoCD's own `repoURL` through an ingress/reverse-proxy
hostname that only listens on 80/443, don't use a non-standard port in the
URL** — it'll silently time out trying to reach a port the proxy never
opened, and depending on your setup this can block *every* Application's
sync simultaneously (repo-server can't fetch git refs for anything).

🟠 **Don't pin exact version tags for images from registries that don't
guarantee tag retention** (Bitnami stopped publishing versioned tags for
free-tier images in 2025 — only `latest` + sha256 digests remain). Verify
a tag actually exists before pinning it, or prefer minimal base images +
a stdlib-only runtime download (e.g. Python's `urllib.request` to fetch a
static binary) to avoid depending on any third party's tagging policy.

🟠 **Non-root containers can't `apk add`/`apt install` in most default
base images** — package manager directories are root-owned. Use a
world-writable scratch dir (`/tmp`) for anything you need to
download/install at runtime instead.

🟠 **Helm does not validate unknown `values.yaml` keys.** A typo, or a
values schema copied from the wrong chart *version's* docs/examples, is
silently a no-op — not an error. Before concluding "this chart doesn't
support X," clone the chart at your exact pinned version/tag and run
`helm template` against your real values file, then diff the rendered
output. Don't trust a chart's current `main`-branch example values file
if you're pinned to an older release — schemas do change between major
versions without warning in your own values file.

## Service Integration Routes

**New service? Pick your stack below** (adjust doc paths to match your repo):

| Need | Doc | Example |
|------|-----|---------|
| **Authentication** | `project-usage/authentik-oidc.md` | OAuth2 login, RBAC groups, JWT tokens |
| **Async messaging** | `project-usage/sqs-messaging.md` | Queue consumers, fire-and-forget, DLQ |
| **Object storage** | `project-usage/minio-s3.md` | File uploads, backups, log backend |
| **CI/CD pipeline** | `project-usage/cicd-workflow.md` | Pipeline syntax, image push, ArgoCD sync |
| **Workflows** | `project-usage/temporal-workflows.md` | Long-running jobs, retries, state machines |
| **Database** | `project-usage/database-postgres.md` | CloudNativePG setup, schema migrations, replicas |
| **Monitoring** | `project-usage/monitoring-metrics.md` | Prometheus scrape, Grafana dashboard, alerts |
| **Secrets** | `project-usage/vault-secrets.md` | Store credentials, rotate tokens, seal/unseal |
| **Networking** | `project-usage/networking-ingress.md` | Public HTTPS, hostname routing, TLS |

## Cluster Essentials (fill in your own inventory)

**Architecture principles (adjust to taste, but these travel well):**
- Immutable OS (Talos, or similar — no SSH, fully declarative config)
- Secrets in a proper secrets backend (Vault) + SOPS-encrypted manifests in
  git (`*.enc.yaml`, age-encrypted); never commit plaintext secrets or `.env`
- ArgoCD app-of-apps as the single CD source of truth; two-phase bootstrap
  documented separately (chicken-and-egg: ArgoCD needs to exist before it
  can deploy itself declaratively — document your exact bootstrap steps)
- Pull-based GitOps — no kubeconfig/cluster credentials ever touch your CI
  runner; the runner only needs push access to git, ArgoCD does the rest
- Federated OIDC (one identity provider fronting every service that
  supports it)

## Deployment Checklist (per new service)

- [ ] Prometheus `/metrics` endpoint or ServiceMonitor, if it exposes metrics
- [ ] All credentials in your secrets backend (never in plain values.yaml,
      pod env directly, or committed anywhere in cleartext)
- [ ] Ingress rule with TLS, if externally reachable
- [ ] Dashboard + alert rules, if metrics are exposed
- [ ] ArgoCD `Application` manifest added to the appropriate sync-wave file,
      **not** a standalone `helm install`/`kubectl apply` run by hand
- [ ] Validated locally before push: `kubectl apply --dry-run=client -f`,
      `kubectl kustomize <dir>/` (if applicable), or `helm template` against
      the exact pinned chart version (if Helm-sourced)
- [ ] After push: confirmed ArgoCD's `status.sync.revision` actually matches
      your new commit — not just that `status.sync.status` says `Synced`
      (see Gotchas — a stale hook can hide behind an otherwise-current app)

## Git & Release

**Incremental commits (service-layer grouped) tend to age well:**
- Foundation & Docs
- Core Infra (CNI, ingress, cert management, storage)
- Observability Stack
- IAM & Secrets
- CI/CD & GitOps
- Messaging / Data Infrastructure
- Applications & Utilities

Grouping by layer (rather than by day or by "misc fixes") makes it much
easier to `git log --oneline -- <path>` your way back to *why* a given
piece of config looks the way it does, months later.