homelab/k8s/argocd/bootstrap.archived/BOOTSTRAP.md

# Cluster bootstrap & CD — two scopes

The GitOps repo is hosted on the cluster's own Forgejo, so ArgoCD cannot pull it
until Forgejo exists. There are two distinct workflows — pick the one that
matches your situation.

```
Scope 1 — from scratch     cold cluster, nothing running yet   → full bootstrap
Scope 2 — iterating CD     cluster + Forgejo + ArgoCD are up   → just git push
```

---

## Scope 1 — Cluster from scratch (cold bootstrap)

Strict ordering, because each layer depends on the previous one existing.

```
Talos + Cilium CNI          Terraform (nodes)                     — cluster reachable
Substrate                   cert-manager, ingress-nginx, reloader — Terraform helm_releases
                            Longhorn, ArgoCD                      — imperative install
Phase 0 (git-independent)   CNPG + ddb-cluster + redis + Forgejo  — manual, below
Seed                        git push repo into Forgejo
Phase 1                     app-of-apps root                      — ArgoCD deploys the rest
```

Run every command from a local checkout with `KUBECONFIG` pointed at the cluster.

### Phase 0 — bootstrap Forgejo and its data plane (git-independent)

```bash
# 1. CNPG operator (public Helm) + its CRDs.
kubectl apply -f k8s/argocd/bootstrap/cnpg-operator.yaml
argocd app sync cnpg-operator
# CNPG ships CRDs in the chart's crds/ folder, which ArgoCD's helm-template does
# NOT install. If the ddb-cluster apply below fails with "no matches for kind
# Cluster", install them once:
#   kubectl apply --server-side -f \
#     https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.1.yaml

# 2. Namespaces first. cicd MUST be labelled privileged (DinD/runner needs it);
#    the Forgejo app deliberately does NOT create it.
kubectl create namespace ddb  --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace cicd --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace cicd pod-security.kubernetes.io/enforce=privileged --overwrite

# 3. Postgres cluster + Forgejo database + Redis (raw manifests, local checkout).
kubectl apply -f k8s/data/ddb-cluster.yaml
kubectl apply -f k8s/data/forgejo-database.yaml
kubectl apply -f k8s/security/ci-cd/forgejo-redis.yaml

# 4. Copy the CNPG-generated app password into cicd (Secrets are ns-scoped).
kubectl get secret ddb-cluster-app -n ddb -o yaml \
  | sed 's/namespace: ddb/namespace: cicd/' \
  | kubectl apply -n cicd -f -

# 5. Forgejo (public Helm, values inlined — git-independent).
kubectl apply -f k8s/argocd/bootstrap/forgejo.yaml
argocd app sync forgejo
```

### Seed + Phase 1

```bash
# 6. Push this repo to the freshly-created Forgejo so Phase 1 has a source.
git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git
git push forgejo main

# 7. (Optional) Cloudflared tunnel secret — if the SOPS CMP plugin is NOT yet
#    wired up: create the cloudflared-token Secret manually before wave 8 syncs.
#    Skip this step if the CMP plugin is already live and handling SOPS decryption.
kubectl create namespace cloudflared --dry-run=client -o yaml | kubectl apply -f -
sops -d k8s/applications/cloudflared/cloudflared-secrets.enc.yaml \
  | yq '.cloudflared.tunnelToken' \
  | kubectl create secret generic cloudflared-token -n cloudflared \
      --from-literal=token=- --dry-run=client -o yaml | kubectl apply -f -
# Once the SOPS CMP plugin is deployed (Phase 1 wave 0), migrate this Secret
# into the plugin's script output and remove this manual step.

# 8. Deploy everything else from git — one sync brings up the whole cluster.
kubectl apply -f k8s/argocd/projects/homelab-project.yaml   # AppProject
kubectl apply -k k8s/argocd/root                            # app-of-apps root
argocd app sync homelab-root
```

`homelab-root` renders every Application under `k8s/argocd/apps/` and syncs them
in sync-wave order (networking → storage/observability → logging → security →
data → messaging → applications).

---

## Scope 2 — Iterating CD (day-2, cluster already up)

The app-of-apps and its children already exist and reconcile from Forgejo. To
change anything, you do NOT re-run the bootstrap — you just push:

```bash
# edit manifests under k8s/**
git add -A && git commit -m "..."
git push                      # to Forgejo main (origin)
# ArgoCD auto-syncs (child apps have syncPolicy.automated); to force it:
argocd app sync homelab-root
```

- Adding a new service = add an Application manifest under `k8s/argocd/apps/`
  and its manifests under the path it references, then push. The app-of-apps
  picks it up on next sync.
- Changing a Helm value = edit the values file the Application points at
  (`$values/...`), push.
- Forgejo itself is Phase-0 / bootstrap (manual sync only) — a bad CI commit
  must not be able to break the system CI depends on.

---

## Notes / prerequisites

- **Secrets:** several values files (loki S3, grafana admin, authentik, vault)
  expect secrets that helmfile used to inject via `--set`. Under ArgoCD these
  come from the `*.enc.yaml` SOPS files via the ArgoCD SOPS plugin — confirm the
  plugin is configured before syncing waves 2–3, or those charts render with
  empty secrets.
- **Substrate** (cert-manager, ingress-nginx, reloader) is installed by Terraform
  as bootstrap Helm releases, not ArgoCD. Cilium CNI, Longhorn, and ArgoCD
  itself are cluster-bootstrap installs.
- **Single storage node:** only `talos-cp-1` runs workloads/storage, so stateful
  services are single-instance (`ddb-cluster` = 1). See the repo `USAGE.md` and
  root `CLAUDE.md` topology section.