From f81db119f4514689a75ab3991515f8fbd788653f Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 20:25:04 -0700 Subject: [PATCH] fix(forgejo): raise Gitea CPU limit, it was throttled 16% of the time The Actions queue was stalling because Gitea had no CPU, not because the database was slow. Measured on the running pod before this change: nr_throttled 189528 of 1169014 CFS periods (16.2%) throttled_usec 4424898602 us (~74 minutes lost) actual usage 951m against a 1000m limit The user-visible symptom was slow Actions traffic in Gitea's own log: slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms So runners waited, timed out, and retried, which loaded the saturated process further. The database is exonerated. On the primary, a real action_runner UPDATE measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran longer than 3 seconds. Raising any database resource would not have helped. Why these numbers: - requests.cpu 200m -> 750m. A request below real usage earns the pod little scheduling weight during contention, which is exactly when it matters. - limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a burst finish instead of being chopped into 100ms slices. - requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old request. - limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never the problem and does not need room. Caution recorded in the file: this pod runs on talos-cp-3, a control-plane node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near this limit, move the workload off the control plane instead of raising the limit again. ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so this must reach main to take effect. A live patch is reverted within seconds. --- .../phase3-forgejo/forgejo-values.yaml | 25 ++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml b/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml index 5df79e3..106785f 100644 --- a/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml +++ b/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml @@ -134,12 +134,31 @@ ingress: # wildcard. The wildcard covers this host. # Resources +# +# The 1000m ceiling throttled Gitea badly. Measured on a running pod: +# nr_throttled 189528 of 1169014 CFS periods (16.2%) +# throttled_usec 4424898602 us (~74 minutes lost) +# actual usage 951m against the 1000m limit +# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and +# repository compare endpoints took 3091-3811ms, so runners timed out and +# retried, which added more load to an already saturated process. +# +# The database was never the bottleneck. A real action_runner UPDATE measured +# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself. +# +# requests.cpu rises to 750m because a request below real usage earns the pod +# little scheduling weight during contention. limits.cpu rises to 3000m +# because diff computation is bursty and needs room to run without throttling. +# +# Caution: this pod runs on talos-cp-3, a control-plane node. 3000m is 25% of +# that node's 11950m allocatable CPU. If Gitea sustains near this limit, move +# the workload off the control plane rather than raising the limit again. resources: requests: - cpu: 200m - memory: 512Mi + cpu: 750m + memory: 768Mi limits: - cpu: 1000m + cpu: 3000m memory: 2Gi # Tolerations for control-plane -- 2.54.0