# ADR 0001 — GitOps Bootstrap, CD, and Coverage

- **Status:** Proposed
- **Date:** 2026-07-23
- **Deciders:** Homelab platform owner
- **Context tags:** bootstrap, gitops, argocd, forgejo, cicd, supply-chain, disaster-recovery

One delivery system, three coupled parts:

- **Part A — Bootstrap:** bare cluster → self-hosted GitOps control plane (breaks the ArgoCD ↔ Forgejo circle).
- **Part B — Steady-state CD:** how app changes flow to the cluster once it's up.
- **Part C — Coverage & organization:** every resource captured by exactly one Application; no orphans, no duplicates.

Single 3-control-plane Talos cluster. GitOps via ArgoCD (app-of-apps under `k8s/argocd/apps/`, waves 0–8); CI via Forgejo Actions on a self-hosted runner.

---

## Part A — Bootstrap

### Problem — the circle

```
ArgoCD ──syncs from──▶ git repo ──hosted by──▶ Forgejo ──deployed by──▶ (the repo)
```

Forgejo hosts the repo ArgoCD reconciles. Today the circle is broken by installing Forgejo + Longhorn + CNPG **by hand** (`bootstrap-local/`, `bootstrap/phase1-5/`) — imperative, off-git, never reconciled, human-ordered rebuild.

### Decision — GitHub seed + Forgejo pull-mirror + cutover

GitHub is the seed and permanent DR source; Forgejo is steady-state truth after cutover and stays synced via a native pull-mirror.

**No adoption.** Longhorn, CNPG, and Forgejo are **pure Phase-0 manual bootstrap** — ArgoCD never manages them (this is the deliberate divergence from the "ArgoCD adopts hand-installed infra" pattern). They must exist *before* ArgoCD can sync from a git host at all, so making them Apps would re-introduce the circle. ArgoCD (seeded from GitHub) deploys only everything *after* the control plane.

```
Phase 0 (manual, once — the whole control plane):
  Talos ─ Cilium CNI ─ Longhorn ─ CNPG operator ─ forgejo-db Cluster (wait Ready)
        ─ Forgejo + Redis ─ push repo ─ helm install argocd ─ apply root-app→GitHub
Phase 1 (ArgoCD from GitHub seed): waves 0–8 = monitoring · logging · iam · messaging ·
  apps. Does NOT manage Longhorn / CNPG / Forgejo (those stay bootstrap-owned).
Phase 2 (cutover, once): flip root-app repoURL GitHub→Forgejo; Forgejo pull-mirrors
  GitHub (live DR seed).
```

Manual order matters: CNPG operator + `forgejo-db` Cluster must be **Ready** (`.status.phase: "Cluster in healthy state"`, not just created) before `helm install forgejo`. Wait on it. `forgejo-db` = dedicated CNPG Cluster (`owner: forgejo`), preferred anti-affinity + control-plane tolerations so it can't deadlock on a partly-schedulable 3-CP.

**Options:** (A) one-time seed + manual cutover — mirror drifts. (B) **seed + pull-mirror + cutover — chosen** — safe cutover, always-current DR. (C) external-permanent — no self-hosted control plane. Rejected.

### Repository credentials (private GitHub seed)

Two read-only creds, neither needs write:

1. **ArgoCD → GitHub: SSH deploy key.** `ssh-keygen -t ed25519 -f argocd_seed -N ""`; add public key to repo → Deploy keys (read-only). Private key → ArgoCD repo-credential Secret at Phase 0 (before any secret controller):
   ```yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: seed-github-repo
     namespace: argocd
     labels: { argocd.argoproj.io/secret-type: repository }
   stringData:
     type: git
     url: git@github.com:<org>/homelab.git   # SSH; must match root-app repoURL
     sshPrivateKey: |
       -----BEGIN OPENSSH PRIVATE KEY-----
       ...
   ```
2. **Forgejo → GitHub: fine-grained PAT (Contents: Read).** HTTPS token for the pull-mirror; SOPS-encrypted in-repo (consumed at w1 once SOPS plugin is up).

Rotation: replace GitHub-side key/token, re-apply Secret / re-encrypt SOPS. Cutover keeps both (mirror stays live).

---

## Part B — Steady-state CD

**CI does build + push + sign only.** Dedicated controllers own promotion, reconciliation, safe rollout. Git = single source; nothing mutates cluster or CD repo by string-replace.

### Current problems

1. **`sed` image-bump** (`ci.yml:72`) — fragile, non-atomic, races, `git revert`-only rollback. "CI mutates CD repo" anti-pattern.
2. **Registry hostname split-brain** — one Forgejo addressed 4 ways; in-cluster pods pull via public ingress name (nginx hairpin, CLAUDE.md trap). `api` defaults to `:latest`.
3. **Fragmented truth** — `project: default` vs `homelab`; `api` App points at external `rock/deploy.git`.
4. **DinD runner** — privileged; RWO cache pins it to cp-1 (SPOF).
5. **No progressive delivery / supply-chain gate** — all `prune+selfHeal`, no canary/analysis/rollback, unsigned images.

### Decision

| Concern | From | To |
|---|---|---|
| Image promotion | CI `sed` → deploy repo | **ArgoCD Image Updater**, git write-back, digest-pinned |
| Build | DinD (privileged, cp-1) | **rootless BuildKit** (or Kaniko) + registry cache |
| Registry addressing | 4 hostnames | in-cluster `*.svc.cluster.local:3000`; one external push name |
| Delivery | raw auto-sync | **Argo Rollouts** canary + Prometheus `AnalysisTemplate` (app tier) |
| Supply chain | none | **cosign sign + Kyverno `verifyImages`** |
| Success signal | pod `Running` | **Playwright E2E** (video/trace) gate |
| Projects/repos | `default`+`homelab`, 2 repos | one `homelab` project, one convention |

```
push→main → runner(BuildKit): test · build·push <in-cluster-registry>/<repo>@sha256 · cosign sign
         → Image Updater watches registry, writes digest back to git
         → app-of-apps: infra/platform = plain sync ; app tier = Rollouts canary + Kyverno gate
```

**Options:** promotion — Image Updater (chosen) over `kustomize edit` (still coupled) / Flux (2nd engine). Build — BuildKit (chosen) over DinD / Kaniko. Delivery — Rollouts app-tier-only (chosen) over none / Flagger (needs mesh).

### Deployment verification (synthetic E2E)

`Running` proves scheduling, not the user path (ingress/TLS/OIDC/session/UI). Signal ladder: readiness → synthetic HTTP → **Playwright browser E2E** (gate on the top rung). Suite hits **public ingress hostnames**, records video+trace → MinIO `e2e-artifacts/` (private, lifecycle-expired). Journeys:

- **Portainer** — login page renders.
- **Authentik** — sign in (bootstrap `akadmin` from secret store), dashboard loads.
- **OAuth/OIDC federation** — each app (ArgoCD, Forgejo, Grafana…): click SSO → complete Authentik → land back authenticated. Catches client mis-registration / redirect-URI / broken provider that a green pod hides.

Runs two ways: **deploy gate** (Rollouts `AnalysisTemplate` / PostSync Job → fails sync/rollout) and **continuous smoke** (`CronJob` → Alertmanager). Creds from SOPS/Vault as env, never hardcoded. Constraint: **WebKit only** (Safari engine) per requirement. Suite lives at `tests/e2e/`; manifests at `k8s/platform/e2e/` (unwired until phase 7).

---

## Part C — GitOps coverage & organization

**Principle: every deployable directory is the source of exactly one Application, grouped by wave.** No dead aggregator kustomizations, no orphan manifests, no second app-of-apps. Layer folders stay; the tree is reconciled to the map. An App uses a git **`path:`** for manifests/kustomize bases, a **`$values` valueFile** when it only parameterizes a remote Helm chart. Secrets are the exception — one `sops-secrets` App globs all `*.enc.yaml` from repo root.

### Folder ↔ Application map (as-built)

Consolidated to two folders — `k8s/infra` (platform/infra tier) and `k8s/apps`
(workloads) — replacing the old `infrastructure/platform/security/applications`
layer folders (deleted).

```
k8s/
├── argocd/          root/ + apps/ (waves) + projects/            # engine, not an App target
├── bootstrap/       cert-manager w1 · cilium w0 · coredns w0 · ingress w1   # Phase-0 substrate
├── infra/
│   ├── longhorn/          →longhorn-config w1
│   ├── minio/             →minio-tenant (+minio-operator-values for operator) w1
│   ├── monitoring/        →monitoring-config w2 · crds/ →prometheus-crds w0
│   │                       *-values →prometheus/blackbox w1
│   ├── logging/*-values   →grafana/loki/promtail w2
│   ├── data-schemas/      →data-schemas w6      (CNPG operator + forgejo-db → Phase-0 manual bootstrap, NOT an App)
│   ├── iam/               →iam-jobs w3 (+authentik/vault values)
│   ├── forgejo-runner/    →forgejo-runner w3
│   └── cicd/              →cicd-config w3 (Forgejo OAuth job)
└── apps/
    ├── cloudflared/ duckdns/ homarr/ portainer/ temporal/  →w8
    └── messaging/{kafka-cluster,queue-crd,management-service} →w5–7
```

### Orphans — declared in git, reconciled by NOTHING

| Path | Count | Kind | Action |
|---|---|---|---|
| `platform/monitoring/{alerts,servicemonitors,dashboards}` | 27 | PrometheusRule / ServiceMonitor / dashboard ConfigMap | wire → **`monitoring-config` App (done)** |
| `security/ci-cd/forgejo-oauth-setup-job.yaml` | 1 | Job+SA (PostSync) | wire → `cicd-config` App |
| CNPG operator + `forgejo-db` Cluster | — | CNPG install + Cluster | **Phase-0 manual bootstrap** (`bootstrap/phase2-cnpg`, `bootstrap/phase3-forgejo`), never an App — making it an App re-introduces the circle |

Root cause (monitoring): no App used `path: k8s/platform/monitoring` — only crds + valueFiles; the aggregator kustomization chain was dead. Now fixed by one `monitoring-config` App (one kustomization, **no `namespace:` transformer** — PrometheusRules live in per-app namespaces; `ServerSideApply=true` for large dashboard JSON).

### Stale / dead — delete, do NOT wire

- `applications/sqs/argocd/*` — **second competing app-of-apps** (repo `rock/kafaka-management-service.git`, project `kmsvc`).
- `applications/sqs/{queues,environments,helmfile.yaml.gotmpl,kustomization.yaml}` — helmfile-era.
- **`applications/temporal/elasticsearch.yaml`** — superseded (see Temporal datastore below).
- `applications/temporal/queues/`, root `applications/kustomization.yaml` — example / legacy overlay.
- `security/iam/authentik-migrations-job.yaml` — dead/broken (server self-migrates).
- Dead aggregator kustomizations (`infrastructure/`, `platform/`, `security/`, `ci-cd/`), `logging/minio-values.yaml`, `infrastructure/minio/test/`, scripts / `.env.example`.

**Experimental — leave unmanaged** (per `60-applications.yaml`): `applications/{dev-tools,forge,llm,shadowsocks}`.

### 🔴 Temporal datastore = PostgreSQL (hard constraint)

Temporal runs on the shared CNPG **PostgreSQL** (`ddb-cluster`), for BOTH the main store and **visibility**. **NOT Cassandra** (the Helm chart's default) and **NOT Elasticsearch.** `temporal-values.yaml` sets `elasticsearch.enabled: false` and points visibility at Postgres (`temporal-visibility-database`). Therefore `applications/temporal/elasticsearch.yaml` is dead → delete. When touching Temporal values, verify persistence stays on the pinned chart's SQL schema (chart schema drift silently reverts to Cassandra — see CLAUDE.md).

### Reconcile actions (tree → map)

1. **Restore** live-consumed paths currently staged-deleted: `infrastructure/{longhorn,minio}`, `security/iam`, `security/ci-cd/charts/forgejo-runner`, `platform/{logging,monitoring}` values + crds. Without them their Apps have no source.
2. **Add Apps:** `monitoring-config` (done), `cicd-config` (oauth job).
3. **Delete** the stale/dead list above.
4. **CNPG** (operator + `forgejo-db` Cluster) stays **Phase-0 manual bootstrap** — never an App (circular).

⚠️ The current staged deletion is **over-broad** — it wiped live GitOps sources (longhorn, minio, iam, monitoring/logging values). Committing as-is breaks those Apps' sync. Curate: restore live, delete only confirmed orphans.

ApplicationSet git-generator considered, rejected: the hand-tuned per-App settings (multi-source helm+values, `managedNamespaceMetadata`, `ServerSideApply`, waves) don't fit one template.

---

## Consequences

**Positive:** Forgejo/Longhorn/CNPG fully declarative; Phase-0 shrinks to "CNI + ArgoCD + root-app"; repeatable rebuild seeded from always-current GitHub (free off-site DR); CI shrinks to build/push/sign; digest-pinned auditable promotions; no privileged DinD; CI survives cp-1 loss; auto-rollback on SLO breach; only signed images admit; monitoring finally reconciled.

**Costs:** GitHub egress + deploy key needed at Phase 0; one cutover flip remains; two read creds (deploy key + SOPS PAT); three new controllers (Image Updater, Rollouts, Kyverno) + `Deployment`→`Rollout` migration; one-time registry-hostname normalization.

---

## Migration phases (one PR each, impact-over-risk)

0. **Consolidate bootstrap + coverage.** Single-source bootstrap; convert phase1–3 to ArgoCD Apps (seed repo); Forgejo pull-mirror + SOPS PAT; script = Phase-0 + cutover only. Curate the staged deletion (Part C reconcile): restore live paths, add `monitoring-config`(done)+`cicd-config`, delete stale/dead + `data/cluster`. CNPG (operator + `forgejo-db`) stays Phase-0 manual bootstrap.
1. **Kill `sed`.** `argocd-image-updater` (~w4); annotate `image-list` + git write-back; drop ci.yml step 3.
2. **Normalize registry.** In-cluster pulls via `forgejo-gitea-http.cicd.svc.cluster.local:3000`; retire `forgejo.forge.*`; pin digests.
3. **Rootless BuildKit.** Drop `az-a` nodeSelector; `--cache-to type=registry`.
4. **Argo Rollouts.** Controller (~w4); `api`/app tier → `Rollout` + canary `AnalysisTemplate`.
5. **Supply chain.** cosign (Forgejo OIDC) + Kyverno `verifyImages`.
6. **Consolidate projects.** One `homelab` project; monorepo-only vs single `deploy` repo.
7. **Synthetic E2E.** Wire `tests/e2e` suite as Rollouts/PostSync gate + `CronJob`; Portainer + Authentik + OAuth journeys.

---

## Open questions

1. ArgoCD install — `helm install` (leaning) vs Talos `extraManifests`.
2. Post-cutover primary — push GitHub (mirror pulls) vs push Forgejo (push-mirror out). Pull-from-GitHub keeps GitHub authoritative.
3. Cutover — scripted `kubectl patch` of root repoURL vs PostSync hook gated on Forgejo health.

---

## References

- `CLAUDE.md` — ArgoCD gotchas (SkipDryRunOnMissingResource, wait-for-Ready, repoURL/ingress hairpin, Helm schema drift → Temporal/Cassandra)
- `k8s/argocd/{root,apps,projects}/` · `k8s/bootstrap/` (to be superseded)
- `k8s/security/ci-cd/{example-workflows/ci.yml,charts/forgejo-runner,deploy-scaffold}`
- `tests/e2e/` (Playwright WebKit suite) · `k8s/platform/monitoring/` (monitoring-config)