homelab/k8s/argocd/bootstrap/BOOTSTRAP.md

Cluster bootstrap & CD — two scopes

The GitOps repo is hosted on the cluster's own Forgejo, so ArgoCD cannot pull it until Forgejo exists. There are two distinct workflows — pick the one that matches your situation.

Scope 1 — from scratch     cold cluster, nothing running yet   → full bootstrap
Scope 2 — iterating CD     cluster + Forgejo + ArgoCD are up   → just git push

---

Scope 1 — Cluster from scratch (cold bootstrap)

Strict ordering, because each layer depends on the previous one existing.

Talos + Cilium CNI          Terraform (nodes)                     — cluster reachable
Substrate                   cert-manager, ingress-nginx, reloader — Terraform helm_releases
                            Longhorn, ArgoCD                      — imperative install
Phase 0 (git-independent)   CNPG + ddb-cluster + redis + Forgejo  — manual, below
Seed                        git push repo into Forgejo
Phase 1                     app-of-apps root                      — ArgoCD deploys the rest

Run every command from a local checkout with KUBECONFIG pointed at the cluster.

Phase 0 — bootstrap Forgejo and its data plane (git-independent)

# 1. CNPG operator (public Helm) + its CRDs.
kubectl apply -f k8s/argocd/bootstrap/cnpg-operator.yaml
argocd app sync cnpg-operator
# CNPG ships CRDs in the chart's crds/ folder, which ArgoCD's helm-template does
# NOT install. If the ddb-cluster apply below fails with "no matches for kind
# Cluster", install them once:
#   kubectl apply --server-side -f \
#     https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.1.yaml

# 2. Namespaces first. cicd MUST be labelled privileged (DinD/runner needs it);
#    the Forgejo app deliberately does NOT create it.
kubectl create namespace ddb  --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace cicd --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace cicd pod-security.kubernetes.io/enforce=privileged --overwrite

# 3. Postgres cluster + Forgejo database + Redis (raw manifests, local checkout).
kubectl apply -f k8s/data/ddb-cluster.yaml
kubectl apply -f k8s/data/forgejo-database.yaml
kubectl apply -f k8s/security/ci-cd/forgejo-redis.yaml

# 4. Copy the CNPG-generated app password into cicd (Secrets are ns-scoped).
kubectl get secret ddb-cluster-app -n ddb -o yaml \
  | sed 's/namespace: ddb/namespace: cicd/' \
  | kubectl apply -n cicd -f -

# 5. Forgejo (public Helm, values inlined — git-independent).
kubectl apply -f k8s/argocd/bootstrap/forgejo.yaml
argocd app sync forgejo

Seed + Phase 1

# 6. Push this repo to the freshly-created Forgejo so Phase 1 has a source.
git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git
git push forgejo main

# 7. (Optional) Cloudflared tunnel secret — if the SOPS CMP plugin is NOT yet
#    wired up: create the cloudflared-token Secret manually before wave 8 syncs.
#    Skip this step if the CMP plugin is already live and handling SOPS decryption.
kubectl create namespace cloudflared --dry-run=client -o yaml | kubectl apply -f -
sops -d k8s/applications/cloudflared/cloudflared-secrets.enc.yaml \
  | yq '.cloudflared.tunnelToken' \
  | kubectl create secret generic cloudflared-token -n cloudflared \
      --from-literal=token=- --dry-run=client -o yaml | kubectl apply -f -
# Once the SOPS CMP plugin is deployed (Phase 1 wave 0), migrate this Secret
# into the plugin's script output and remove this manual step.

# 8. Deploy everything else from git — one sync brings up the whole cluster.
kubectl apply -f k8s/argocd/projects/homelab-project.yaml   # AppProject
kubectl apply -k k8s/argocd/root                            # app-of-apps root
argocd app sync homelab-root

homelab-root renders every Application under k8s/argocd/apps/ and syncs them in sync-wave order (networking → storage/observability → logging → security → data → messaging → applications).

---

Scope 2 — Iterating CD (day-2, cluster already up)

The app-of-apps and its children already exist and reconcile from Forgejo. To change anything, you do NOT re-run the bootstrap — you just push:

# edit manifests under k8s/**
git add -A && git commit -m "..."
git push                      # to Forgejo main (origin)
# ArgoCD auto-syncs (child apps have syncPolicy.automated); to force it:
argocd app sync homelab-root

• Adding a new service = add an Application manifest under k8s/argocd/apps/ and its manifests under the path it references, then push. The app-of-apps picks it up on next sync.
• Changing a Helm value = edit the values file the Application points at ($values/...), push.
• Forgejo itself is Phase-0 / bootstrap (manual sync only) — a bad CI commit must not be able to break the system CI depends on.

---

Notes / prerequisites

• Secrets: several values files (loki S3, grafana admin, authentik, vault) expect secrets that helmfile used to inject via --set. Under ArgoCD these come from the *.enc.yaml SOPS files via the ArgoCD SOPS plugin — confirm the plugin is configured before syncing waves 2–3, or those charts render with empty secrets.
• Substrate (cert-manager, ingress-nginx, reloader) is installed by Terraform as bootstrap Helm releases, not ArgoCD. Cilium CNI, Longhorn, and ArgoCD itself are cluster-bootstrap installs.
• Single storage node: only talos-cp-1 runs workloads/storage, so stateful services are single-instance (ddb-cluster = 1). See the repo USAGE.md and root CLAUDE.md topology section.