rock
f81db119f4
The Actions queue was stalling because Gitea had no CPU, not because the database was slow. Measured on the running pod before this change: nr_throttled 189528 of 1169014 CFS periods (16.2%) throttled_usec 4424898602 us (~74 minutes lost) actual usage 951m against a 1000m limit The user-visible symptom was slow Actions traffic in Gitea's own log: slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms So runners waited, timed out, and retried, which loaded the saturated process further. The database is exonerated. On the primary, a real action_runner UPDATE measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran longer than 3 seconds. Raising any database resource would not have helped. Why these numbers: - requests.cpu 200m -> 750m. A request below real usage earns the pod little scheduling weight during contention, which is exactly when it matters. - limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a burst finish instead of being chopped into 100ms slices. - requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old request. - limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never the problem and does not need room. Caution recorded in the file: this pod runs on talos-cp-3, a control-plane node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near this limit, move the workload off the control plane instead of raising the limit again. ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so this must reach main to take effect. A live patch is reverted within seconds.
Homelab Bootstrap — Single-Cluster, GitOps-Ready
Run once manually, GitOps forever after.
This bootstrap breaks the ArgoCD ↔ Forgejo circular dependency by:
- Installing infrastructure in correct dependency order
- Pointing ArgoCD at a GitHub mirror initially
- Cutting over to Forgejo once healthy
- Using Helm for reproducible installs
- Ensuring ArgoCD adopts (not duplicates) bootstrap resources
Prerequisites
- Talos cluster running (terraform applied)
- kubectl configured (
KUBECONFIGpoints at cluster) - Helm 3 installed
- SOPS age key at
~/.sops/homelab-age.key - GitHub mirror of this repo (for initial ArgoCD source)
Directory Structure
bootstrap/
├── phase1-storage/ # Longhorn via Helm
├── phase2-cnpg/ # CNPG operator via Helm
├── phase3-forgejo/ # Forgejo DB + Forgejo via Helm
├── phase4-argocd/ # ArgoCD via Helm → GitHub initially
└── phase5-cutover/ # Switch ArgoCD source to Forgejo
Usage
# From repo root:
./bootstrap.sh
# Or step-by-step:
./bootstrap.sh phase1 # Storage
./bootstrap.sh phase2 # CNPG
./bootstrap.sh phase3 # Forgejo
./bootstrap.sh phase4 # ArgoCD (GitHub mirror)
./bootstrap.sh phase5 # Cut over to Forgejo
Design Principles
- DRY: Helm values used by both bootstrap and ArgoCD
- Single Source of Truth: Manifests match what ArgoCD will manage
- Idempotent: Can re-run phases safely
- Adoption Ready: Resources have
argocd.argoproj.io/sync-options: Prune=false - Dependency Ordered: Each phase waits for previous to be Ready
Phase Details
Phase 1: Storage (Longhorn)
Installs Longhorn with:
- 3-node HA configuration
- Unified
longhornStorageClass (default) - Special
longhorn-cnpgStorageClass with postgres UID/GID mount options - CSI plugin tolerations for control-plane nodes
Source of Truth: phase1-storage/longhorn-values.yaml
Phase 2: CNPG Operator
Installs CloudNativePG operator with:
- CRD registration (blocks until CRD available)
- Webhook configuration
- Monitoring enabled
Source of Truth: phase2-cnpg/cnpg-values.yaml
Phase 3: Forgejo Database + Forgejo
- Creates
forgejo-dbCNPG Cluster - Waits for cluster Ready (PostgreSQL accepting connections)
- Installs Forgejo via Helm pointing at
forgejo-db-rwservice - Waits for Forgejo healthy
Source of Truth:
phase3-forgejo/forgejo-db.yaml(CNPG Cluster CR)phase3-forgejo/forgejo-values.yaml(Helm values)
Phase 4: ArgoCD (GitHub Mirror)
Installs ArgoCD via Helm, then applies root app-of-apps pointing at GitHub mirror.
This is the circle-breaker: ArgoCD syncs from GitHub (not Forgejo) initially.
Source of Truth:
phase4-argocd/argocd-values.yamlphase4-argocd/root-app-github.yaml(repoURL = GitHub)
ArgoCD adopts Phases 1-3 resources (no duplication) because manifests match.
Phase 5: Cut Over to Forgejo
- Push repo to Forgejo
- Update root app
repoURLfrom GitHub → Forgejo - ArgoCD re-syncs from Forgejo
The circle is broken. GitHub mirror is now disaster recovery only.
Post-Bootstrap
All changes via Git:
git commit -m "feat(app): add new service"
git push forgejo main
# ArgoCD auto-syncs
Troubleshooting
- Phase stuck? Check
kubectl get events -n <namespace> --sort-by='.lastTimestamp' - ArgoCD duplicating? Verify manifests match exactly (Helm values ↔ ArgoCD Application)
- Forgejo won't start? Check CNPG cluster Ready:
kubectl get cluster forgejo-db -n cicd - Can't push to Forgejo? Verify ingress-nginx healthy, DNS resolves
forgejo.riotpiao.com
Migration from Old Bootstrap
If you have existing k8s/bootstrap-local/:
- DO NOT delete existing resources (Longhorn data!)
- Run refined bootstrap in "adoption mode" (no delete, just apply)
- Verify ArgoCD shows "Synced" for all apps
- Archive old bootstrap:
git mv k8s/bootstrap-local k8s/archive/bootstrap-local-v1