fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi #94

2026-11-01T20:54:46Z

rock commented

2026-11-01 20:54:46 +00:00

Why

The container registry stores its blobs on gitea-shared-storage. A single candle-vllm sm_70 image is 14GB. The volume was 20Gi, so it filled, and every blob upload failed:

Package registry API internal error: 500 write
/data/tmp/package-upload/<id>: no space left on device

The push client reports that as received unexpected HTTP status: 500 after retrying every layer. That reads like a network or auth fault. It is neither — it is a full disk.

Verified live before this commit

Expanded 20Gi → 40Gi → 60Gi, then pushed the image successfully:

forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70
digest sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27

/data   58.9G total   19.6G used   39.3G free

This commit reconciles the chart with the live PVC. That is why the forgejo Application currently reads OutOfSync.

Note for whoever expands this again

Longhorn's admission webhook checks the physical free space of the disk holding each replica, not the thin volume size. A request for 100Gi was rejected outright:

disk ... does not have sufficient physical space for expansion:
physical free space would drop below minimal:
left=14596177920 < minimal=49931236556

60Gi fit on the same disk. Raise it in steps and read the webhook message instead of retrying the same number.

## Why The container registry stores its blobs on `gitea-shared-storage`. A single candle-vllm sm_70 image is **14GB**. The volume was 20Gi, so it filled, and every blob upload failed: ``` Package registry API internal error: 500 write /data/tmp/package-upload/<id>: no space left on device ``` The push client reports that as `received unexpected HTTP status: 500` after retrying every layer. That reads like a network or auth fault. It is neither — it is a full disk. ## Verified live before this commit Expanded 20Gi → 40Gi → 60Gi, then pushed the image successfully: ``` forgejo.riotpiao.com/riotpiao-poimen/candle-vllm:sm70 digest sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27 ``` ``` /data 58.9G total 19.6G used 39.3G free ``` This commit reconciles the chart with the live PVC. That is why the `forgejo` Application currently reads OutOfSync. ## Note for whoever expands this again Longhorn's admission webhook checks the **physical** free space of the disk holding each replica, not the thin volume size. A request for 100Gi was rejected outright: ``` disk ... does not have sufficient physical space for expansion: physical free space would drop below minimal: left=14596177920 < minimal=49931236556 ``` 60Gi fit on the same disk. Raise it in steps and read the webhook message instead of retrying the same number.

rock added 1 commit 2026-11-01 20:54:47 +00:00

fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi 747a6fb9cb

The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:

  Package registry API internal error: 500 write
  /data/tmp/package-upload/<id>: no space left on device

The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.

Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.

This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.

NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:

  disk ... does not have sufficient physical space for expansion:
  physical free space would drop below minimal:
  left=14596177920 < minimal=49931236556

60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.

rock referenced this pull request

2026-11-02 05:40:31 +00:00

fix(forgejo): keep 60Gi storage, revert CPU limit to 1000m #96

rock closed this pull request

2026-11-02 05:40:41 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#94