From 8f9bf6ee8daeeeba14caedb0334dcc16c69ec00e Mon Sep 17 00:00:00 2001 From: rock Date: Sun, 1 Nov 2026 13:54:17 -0700 Subject: [PATCH] fix(forgejo): raise the shared storage to 60Gi, the registry outgrew 20Gi The container registry stores blobs on this volume. A single candle-vllm sm_70 image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob upload failed: Package registry API internal error: 500 write /data/tmp/package-upload/: no space left on device The push client reports that as `received unexpected HTTP status: 500` after retrying every layer, which reads like a network or auth fault. It is neither. Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports 58.9G total with 39.3G free, and the sm70 image pushed and verified by digest sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27. This commit reconciles the chart with the live PVC, which is why the Application currently reads OutOfSync. NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL free space of the disk holding each replica, not the thin volume size. A request for 100Gi was rejected outright: disk ... does not have sufficient physical space for expansion: physical free space would drop below minimal: left=14596177920 < minimal=49931236556 60Gi fit on the same disk. Raise it in steps, and read the webhook message rather than retrying the same number. --- .../phase3-forgejo/forgejo-values.yaml | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml b/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml index 106785f..2bcf109 100644 --- a/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml +++ b/k8s/bootstrap/phase3-forgejo/forgejo-values.yaml @@ -97,10 +97,25 @@ gitea: # USER and PASSWD injected via env vars below (GITEA__MAILER__USER, GITEA__MAILER__PASSWD) # Persistence (shared storage for repos) +# +# 60Gi, not the original 20Gi. The container registry stores its blobs here, +# and a single candle-vllm sm_70 image is 14GB uncompressed. At 20Gi the volume +# filled, and every blob upload failed with: +# Package registry API internal error: 500 write /data/tmp/package-upload/...: +# no space left on device +# The push client surfaces that as a bare `received unexpected HTTP status: 500` +# after retrying every layer, which looks like a network fault and is not one. +# +# Note for whoever expands this again: Longhorn's admission webhook checks the +# PHYSICAL free space of the disk holding each replica, not the thin volume +# size. A request for 100Gi was rejected while the disk had 23.5Gi free: +# disk ... does not have sufficient physical space for expansion: +# physical free space would drop below minimal: left=14596177920 < minimal=49931236556 +# 60Gi fit. Raise it in steps and watch the webhook message. persistence: enabled: true storageClass: longhorn - size: 20Gi + size: 60Gi accessModes: - ReadWriteOnce