Files
homelab/k8s/bootstrap
rock e93d4ba86c fix(cert-manager): restore wildcard renewal and drop per-host TLS
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".

Changes:

- cert-manager: pin the self-check to the public resolvers with
  --dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
  Verified against Let's Encrypt staging: the challenge went valid and the
  certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
  cert-manager.io/cluster-issuer annotations. The wildcard covers both
  hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
  with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
  and the zero-downtime renewal path.

Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
2026-10-24 02:30:48 -07:00
..

Homelab Bootstrap — Single-Cluster, GitOps-Ready

Run once manually, GitOps forever after.

This bootstrap breaks the ArgoCD ↔ Forgejo circular dependency by:

  1. Installing infrastructure in correct dependency order
  2. Pointing ArgoCD at a GitHub mirror initially
  3. Cutting over to Forgejo once healthy
  4. Using Helm for reproducible installs
  5. Ensuring ArgoCD adopts (not duplicates) bootstrap resources

Prerequisites

  • Talos cluster running (terraform applied)
  • kubectl configured (KUBECONFIG points at cluster)
  • Helm 3 installed
  • SOPS age key at ~/.sops/homelab-age.key
  • GitHub mirror of this repo (for initial ArgoCD source)

Directory Structure

bootstrap/
├── phase1-storage/         # Longhorn via Helm
├── phase2-cnpg/           # CNPG operator via Helm
├── phase3-forgejo/        # Forgejo DB + Forgejo via Helm
├── phase4-argocd/         # ArgoCD via Helm → GitHub initially
└── phase5-cutover/        # Switch ArgoCD source to Forgejo

Usage

# From repo root:
./bootstrap.sh

# Or step-by-step:
./bootstrap.sh phase1  # Storage
./bootstrap.sh phase2  # CNPG
./bootstrap.sh phase3  # Forgejo
./bootstrap.sh phase4  # ArgoCD (GitHub mirror)
./bootstrap.sh phase5  # Cut over to Forgejo

Design Principles

  1. DRY: Helm values used by both bootstrap and ArgoCD
  2. Single Source of Truth: Manifests match what ArgoCD will manage
  3. Idempotent: Can re-run phases safely
  4. Adoption Ready: Resources have argocd.argoproj.io/sync-options: Prune=false
  5. Dependency Ordered: Each phase waits for previous to be Ready

Phase Details

Phase 1: Storage (Longhorn)

Installs Longhorn with:

  • 3-node HA configuration
  • Unified longhorn StorageClass (default)
  • Special longhorn-cnpg StorageClass with postgres UID/GID mount options
  • CSI plugin tolerations for control-plane nodes

Source of Truth: phase1-storage/longhorn-values.yaml

Phase 2: CNPG Operator

Installs CloudNativePG operator with:

  • CRD registration (blocks until CRD available)
  • Webhook configuration
  • Monitoring enabled

Source of Truth: phase2-cnpg/cnpg-values.yaml

Phase 3: Forgejo Database + Forgejo

  1. Creates forgejo-db CNPG Cluster
  2. Waits for cluster Ready (PostgreSQL accepting connections)
  3. Installs Forgejo via Helm pointing at forgejo-db-rw service
  4. Waits for Forgejo healthy

Source of Truth:

  • phase3-forgejo/forgejo-db.yaml (CNPG Cluster CR)
  • phase3-forgejo/forgejo-values.yaml (Helm values)

Phase 4: ArgoCD (GitHub Mirror)

Installs ArgoCD via Helm, then applies root app-of-apps pointing at GitHub mirror.

This is the circle-breaker: ArgoCD syncs from GitHub (not Forgejo) initially.

Source of Truth:

  • phase4-argocd/argocd-values.yaml
  • phase4-argocd/root-app-github.yaml (repoURL = GitHub)

ArgoCD adopts Phases 1-3 resources (no duplication) because manifests match.

Phase 5: Cut Over to Forgejo

  1. Push repo to Forgejo
  2. Update root app repoURL from GitHub → Forgejo
  3. ArgoCD re-syncs from Forgejo

The circle is broken. GitHub mirror is now disaster recovery only.

Post-Bootstrap

All changes via Git:

git commit -m "feat(app): add new service"
git push forgejo main
# ArgoCD auto-syncs

Troubleshooting

  • Phase stuck? Check kubectl get events -n <namespace> --sort-by='.lastTimestamp'
  • ArgoCD duplicating? Verify manifests match exactly (Helm values ↔ ArgoCD Application)
  • Forgejo won't start? Check CNPG cluster Ready: kubectl get cluster forgejo-db -n cicd
  • Can't push to Forgejo? Verify ingress-nginx healthy, DNS resolves forgejo.riotpiao.com

Migration from Old Bootstrap

If you have existing k8s/bootstrap-local/:

  1. DO NOT delete existing resources (Longhorn data!)
  2. Run refined bootstrap in "adoption mode" (no delete, just apply)
  3. Verify ArgoCD shows "Synced" for all apps
  4. Archive old bootstrap: git mv k8s/bootstrap-local k8s/archive/bootstrap-local-v1