Compare commits

...
Author SHA1 Message Date
rock 747a6fb9cb fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi
The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:

  Package registry API internal error: 500 write
  /data/tmp/package-upload/<id>: no space left on device

The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.

Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.

This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.

NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:

  disk ... does not have sufficient physical space for expansion:
  physical free space would drop below minimal:
  left=14596177920 < minimal=49931236556

60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.
2026-11-01 13:54:41 -07:00
rock ce4434286c fix(forgejo): raise Gitea CPU limit — it was throttled 16% of the time (#93)
Co-authored-by: rock <[email protected]>
2026-11-01 03:27:56 +00:00
@@ -97,10 +97,25 @@ gitea:
# USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD) # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD)
# Persistence (shared storage for repos) # Persistence (shared storage for repos)
#
# 60Gi, not the original 20Gi. The container registry stores its blobs here,
# and a single candle-vllm sm_70 image is 14GB uncompressed. At 20Gi the volume
# filled, and every blob upload failed with:
# Package registry API internal error: 500 write /data/tmp/package-upload/...:
# no space left on device
# The push client surfaces that as a bare `received unexpected HTTP status: 500`
# after retrying every layer, which looks like a network fault and is not one.
#
# Note for whoever expands this again: Longhorn's admission webhook checks the
# PHYSICAL free space of the disk holding each replica, not the thin volume
# size. A request for 100Gi was rejected while the disk had 23.5Gi free:
# disk ... does not have sufficient physical space for expansion:
# physical free space would drop below minimal: left=14596177920 < minimal=49931236556
# 60Gi fit. Raise it in steps and watch the webhook message.
persistence: persistence:
enabled: true enabled: true
storageClass: longhorn storageClass: longhorn
size: 20Gi size: 60Gi
accessModes: accessModes:
- ReadWriteOnce - ReadWriteOnce
@@ -134,12 +149,31 @@ ingress:
# wildcard. The wildcard covers this host. # wildcard. The wildcard covers this host.
# Resources # Resources
#
# The 1000m ceiling throttled Gitea badly. Measured on a running pod:
# nr_throttled 189528 of 1169014 CFS periods (16.2%)
# throttled_usec 4424898602 us (~74 minutes lost)
# actual usage 951m against the 1000m limit
# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and
# repository compare endpoints took 3091-3811ms, so runners timed out and
# retried, which added more load to an already saturated process.
#
# The database was never the bottleneck. A real action_runner UPDATE measured
# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself.
#
# requests.cpu rises to 750m because a request below real usage earns the pod
# little scheduling weight during contention. limits.cpu rises to 3000m
# because diff computation is bursty and needs room to run without throttling.
#
# Caution: this pod runs on talos-cp-3, a control-plane node. 3000m is 25% of
# that node's 11950m allocatable CPU. If Gitea sustains near this limit, move
# the workload off the control plane rather than raising the limit again.
resources: resources:
requests: requests:
cpu: 200m cpu: 750m
memory: 512Mi memory: 768Mi
limits: limits:
cpu: 1000m cpu: 3000m
memory: 2Gi memory: 2Gi
# Tolerations for control-plane # Tolerations for control-plane