fix(forgejo): raise Gitea CPU limit — it was throttled 16% of the time #93

2026-11-01T03:25:17Z

rock commented

2026-11-01 03:25:17 +00:00

Root cause

The Actions queue was not stalling because of the database. It was stalling because Gitea had no CPU.

Measured on the running pod before the change:

Metric	Value
`nr_throttled`	189,528 of 1,169,014 CFS periods (16.2%)
`throttled_usec`	4,424,898,602 µs (~74 minutes of lost CPU)
Actual usage	951m against a 1000m limit

Gitea's own log shows the consequence:

slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms

Runners waited, timed out, retried, and loaded the saturated process further.

The database is exonerated

Check	Result
`action_runner` UPDATE	2.133 ms
Lock waits	0
Queries > 3s	none

Raising any database resource would not have helped. The forgejo-db cluster is a separate problem, tracked in #92.

The change

Setting	Before	After	Why
`requests.cpu`	200m	750m	A request below real usage earns little scheduling weight during contention
`limits.cpu`	1000m	3000m	Diff computation is bursty and needs room to finish
`requests.memory`	512Mi	768Mi	Observed usage was 621Mi, above the old request
`limits.memory`	2Gi	2Gi	Memory was never the problem, 621Mi observed

Caution

This pod runs on talos-cp-3, a control-plane node. 3000m is 25% of that node's 11950m allocatable CPU. If Gitea sustains near this limit, the correct next step is to move the workload off the control plane, not to raise the limit again.

I recorded that in the values file so the next person reads it before bumping the number.

Effect

ArgoCD owns this file (forgejo, chart gitea 12.7.0, selfHeal: true), so the change takes effect on merge. A direct live patch is reverted within seconds, which I verified earlier when a ConfigMap edit reverted on me.

Structural gate: 0 error, 0 warning.

## Root cause The Actions queue was not stalling because of the database. It was stalling because Gitea had no CPU. Measured on the running pod before the change: | Metric | Value | |---|---| | `nr_throttled` | **189,528** of 1,169,014 CFS periods (**16.2%**) | | `throttled_usec` | **4,424,898,602 µs** (~74 minutes of lost CPU) | | Actual usage | **951m** against a **1000m** limit | Gitea's own log shows the consequence: ``` slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms ``` Runners waited, timed out, retried, and loaded the saturated process further. ## The database is exonerated | Check | Result | |---|---| | `action_runner` UPDATE | **2.133 ms** | | Lock waits | **0** | | Queries > 3s | **none** | Raising any database resource would not have helped. The `forgejo-db` cluster is a separate problem, tracked in #92. ## The change | Setting | Before | After | Why | |---|---|---|---| | `requests.cpu` | 200m | **750m** | A request below real usage earns little scheduling weight during contention | | `limits.cpu` | 1000m | **3000m** | Diff computation is bursty and needs room to finish | | `requests.memory` | 512Mi | **768Mi** | Observed usage was 621Mi, above the old request | | `limits.memory` | 2Gi | 2Gi | Memory was never the problem, 621Mi observed | ## Caution This pod runs on **talos-cp-3, a control-plane node**. 3000m is 25% of that node's 11950m allocatable CPU. If Gitea sustains near this limit, the correct next step is to move the workload off the control plane, not to raise the limit again. I recorded that in the values file so the next person reads it before bumping the number. ## Effect ArgoCD owns this file (`forgejo`, chart gitea 12.7.0, `selfHeal: true`), so the change takes effect on merge. A direct live patch is reverted within seconds, which I verified earlier when a ConfigMap edit reverted on me. Structural gate: `0 error, 0 warning`.

rock added 1 commit 2026-11-01 03:25:17 +00:00

fix(forgejo): raise Gitea CPU limit, it was throttled 16% of the time f81db119f4

The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:

  nr_throttled    189528 of 1169014 CFS periods   (16.2%)
  throttled_usec  4424898602 us                    (~74 minutes lost)
  actual usage    951m against a 1000m limit

The user-visible symptom was slow Actions traffic in Gitea's own log:

  slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
  slow GET .../compare/...  elapsed 3091.5ms / 3520.8ms / 3811.5ms

So runners waited, timed out, and retried, which loaded the saturated process
further.

The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.

Why these numbers:

- requests.cpu 200m -> 750m. A request below real usage earns the pod little
  scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
  burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
  request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
  the problem and does not need room.

Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.

ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.

rock merged commit ce4434286c into main

2026-11-01 03:27:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#93