homelab

riotpiao-poimen/homelab

Fork 0

Author	SHA1	Message	Date
rock	8f9bf6ee8d	fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi The container registry stores blobs on this volume. A single candle-vllm sm_70 image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob upload failed: Package registry API internal error: 500 write /data/tmp/package-upload/<id>: no space left on device The push client reports that as `received unexpected HTTP status: 500` after retrying every layer, which reads like a network or auth fault. It is neither. Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports 58.9G total with 39.3G free, and the sm70 image pushed and verified by digest sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27. This commit reconciles the chart with the live PVC, which is why the Application currently reads OutOfSync. NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL free space of the disk holding each replica, not the thin volume size. A request for 100Gi was rejected outright: disk ... does not have sufficient physical space for expansion: physical free space would drop below minimal: left=14596177920 < minimal=49931236556 60Gi fit on the same disk. Raise it in steps, and read the webhook message rather than retrying the same number.	2026-11-01 13:54:17 -07:00
rock	f81db119f4	fix(forgejo): raise Gitea CPU limit, it was throttled 16% of the time The Actions queue was stalling because Gitea had no CPU, not because the database was slow. Measured on the running pod before this change: nr_throttled 189528 of 1169014 CFS periods (16.2%) throttled_usec 4424898602 us (~74 minutes lost) actual usage 951m against a 1000m limit The user-visible symptom was slow Actions traffic in Gitea's own log: slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms So runners waited, timed out, and retried, which loaded the saturated process further. The database is exonerated. On the primary, a real action_runner UPDATE measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran longer than 3 seconds. Raising any database resource would not have helped. Why these numbers: - requests.cpu 200m -> 750m. A request below real usage earns the pod little scheduling weight during contention, which is exactly when it matters. - limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a burst finish instead of being chopped into 100ms slices. - requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old request. - limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never the problem and does not need room. Caution recorded in the file: this pod runs on talos-cp-3, a control-plane node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near this limit, move the workload off the control plane instead of raising the limit again. ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so this must reach main to take effect. A live patch is reverted within seconds.	2026-10-31 20:25:04 -07:00

Author

SHA1

Message

Date

rock

8f9bf6ee8d

fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi

The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:

  Package registry API internal error: 500 write
  /data/tmp/package-upload/<id>: no space left on device

The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.

Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.

This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.

NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:

  disk ... does not have sufficient physical space for expansion:
  physical free space would drop below minimal:
  left=14596177920 < minimal=49931236556

60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.

2026-11-01 13:54:17 -07:00

rock

f81db119f4

fix(forgejo): raise Gitea CPU limit, it was throttled 16% of the time

The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:

  nr_throttled    189528 of 1169014 CFS periods   (16.2%)
  throttled_usec  4424898602 us                    (~74 minutes lost)
  actual usage    951m against a 1000m limit

The user-visible symptom was slow Actions traffic in Gitea's own log:

  slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
  slow GET .../compare/...  elapsed 3091.5ms / 3520.8ms / 3811.5ms

So runners waited, timed out, and retried, which loaded the saturated process
further.

The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.

Why these numbers:

- requests.cpu 200m -> 750m. A request below real usage earns the pod little
  scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
  burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
  request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
  the problem and does not need room.

Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.

ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.

2026-10-31 20:25:04 -07:00

Compare commits

Diff Content Not Available