Compare commits

..
Author SHA1 Message Date
rock f81db119f4 fix(forgejo): raise Gitea CPU limit, it was throttled 16% of the time
The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:

  nr_throttled    189528 of 1169014 CFS periods   (16.2%)
  throttled_usec  4424898602 us                    (~74 minutes lost)
  actual usage    951m against a 1000m limit

The user-visible symptom was slow Actions traffic in Gitea's own log:

  slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
  slow GET .../compare/...  elapsed 3091.5ms / 3520.8ms / 3811.5ms

So runners waited, timed out, and retried, which loaded the saturated process
further.

The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.

Why these numbers:

- requests.cpu 200m -> 750m. A request below real usage earns the pod little
  scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
  burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
  request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
  the problem and does not need room.

Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.

ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.
2026-10-31 20:25:04 -07:00
@@ -97,25 +97,10 @@ gitea:
# USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD) # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
# Persistence (shared storage for repos) # Persistence (shared storage for repos)
#
# 60Gi, not the original 20Gi. The container registry stores its blobs here,
# and a single candle-vllm sm_70 image is 14GB uncompressed. At 20Gi the volume
# filled, and every blob upload failed with:
# Package registry API internal error: 500 write /data/tmp/package-upload/...:
# no space left on device
# The push client surfaces that as a bare `received unexpected HTTP status: 500`
# after retrying every layer, which looks like a network fault and is not one.
#
# Note for whoever expands this again: Longhorn's admission webhook checks the
# PHYSICAL free space of the disk holding each replica, not the thin volume
# size. A request for 100Gi was rejected while the disk had 23.5Gi free:
# disk ... does not have sufficient physical space for expansion:
# physical free space would drop below minimal: left=14596177920 < minimal=49931236556
# 60Gi fit. Raise it in steps and watch the webhook message.
persistence: persistence:
enabled: true enabled: true
storageClass: longhorn storageClass: longhorn
size: 60Gi size: 20Gi
accessModes: accessModes:
- ReadWriteOnce - ReadWriteOnce