The container registry stores blobs on this volume. A single candle-vllm sm_70
image is 14GB, so a 20Gi volume cannot hold one. It filled, and every blob
upload failed:
Package registry API internal error: 500 write
/data/tmp/package-upload/<id>: no space left on device
The push client reports that as `received unexpected HTTP status: 500` after
retrying every layer, which reads like a network or auth fault. It is neither.
Applied live first (20Gi -> 40Gi -> 60Gi) and verified: /data now reports
58.9G total with 39.3G free, and the sm70 image pushed and verified by digest
sha256:8533d742dfe99c926f42cf99db33e057acb1aafc4b74b0ac22f697f4a45cba27.
This commit reconciles the chart with the live PVC, which is why the Application
currently reads OutOfSync.
NOTE FOR THE NEXT EXPANSION: Longhorn's admission webhook checks the PHYSICAL
free space of the disk holding each replica, not the thin volume size. A request
for 100Gi was rejected outright:
disk ... does not have sufficient physical space for expansion:
physical free space would drop below minimal:
left=14596177920 < minimal=49931236556
60Gi fit on the same disk. Raise it in steps, and read the webhook message
rather than retrying the same number.
The Actions queue was stalling because Gitea had no CPU, not because the
database was slow. Measured on the running pod before this change:
nr_throttled 189528 of 1169014 CFS periods (16.2%)
throttled_usec 4424898602 us (~74 minutes lost)
actual usage 951m against a 1000m limit
The user-visible symptom was slow Actions traffic in Gitea's own log:
slow POST /api/actions/runner.v1.RunnerService/FetchTask ... 3469.9ms
slow GET .../compare/... elapsed 3091.5ms / 3520.8ms / 3811.5ms
So runners waited, timed out, and retried, which loaded the saturated process
further.
The database is exonerated. On the primary, a real action_runner UPDATE
measured 2.133ms, pg_stat_activity showed zero lock waits, and no query ran
longer than 3 seconds. Raising any database resource would not have helped.
Why these numbers:
- requests.cpu 200m -> 750m. A request below real usage earns the pod little
scheduling weight during contention, which is exactly when it matters.
- limits.cpu 1000m -> 3000m. Diff computation is bursty; a 3x ceiling lets a
burst finish instead of being chopped into 100ms slices.
- requests.memory 512Mi -> 768Mi. Observed usage was 621Mi, above the old
request.
- limits.memory stays at 2Gi. Observed usage was 621Mi, so memory was never
the problem and does not need room.
Caution recorded in the file: this pod runs on talos-cp-3, a control-plane
node, and 3000m is 25% of its 11950m allocatable CPU. If Gitea sustains near
this limit, move the workload off the control plane instead of raising the
limit again.
ArgoCD owns this file (app `forgejo`, chart gitea 12.7.0, selfHeal true), so
this must reach main to take effect. A live patch is reverted within seconds.
The gitea chart reads pod annotations from gitea.podAnnotations (values.yaml line 583).
A top-level key is silently ignored, so the previous commit changed nothing — the
Deployment came back without the annotation and Reloader still had no trigger.
GITEA__DATABASE_PASSWD comes from forgejo-db-app, which the CNPG Cluster CR owns and
generates. There is no bootstrap secret, so nothing but cloudnative-pg knows that
password. The Deployment reads it into the environment at start.
So a regenerated password leaves Forgejo authenticating with a value the database no
longer accepts: the pod stays Ready, every query fails, and the symptom looks like a
network problem rather than a credential problem. The cnpg.io/reload label on the
Secret is only a hint; this annotation is what makes Reloader act on it.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
GET /user/sign_up returned 200 — anyone reaching forgejo.riotpiao.com
could create an account. Add [service] DISABLE_REGISTRATION: true.
Admin-created accounts (UI/API) are unaffected.
GITEA__WEBHOOK__ALLOWED_HOST_LIST=external,api.riotpiao.com
api.riotpiao.com resolves to cluster ingress ClusterIP (private range)
which is blocked by Forgejo's default SSRF protection. Safe to allowlist
as it is our own api-gw.
Every workflow in the cluster has been silently dead. app.ini carried no
[actions] section, so Forgejo never created a run: the API returns
total_count: 0 for rock/homelab and rock/homelab-frontend alike, despite both
repos reporting has_actions: true, cluster-ci.yaml and build.yaml sitting on
their default branches, and forgejo-runner having registered successfully.
Registration does not go through the dispatcher, which is why the runner looks
healthy -- it logs "declared successfully" and "[poller 0] launched" and then
picks up nothing, forever. That reads like a runner or label problem and is
neither.
This also explains why the api-gateway images in the registry were all built
by hand: the pipeline that was supposed to build them has never once run.
Forgejo restarts on this values change; git and the container registry are
briefly unavailable.
Two Ingresses claim forgejo.riotpiao.com and nginx honours the older chart one,
so the annotation on the other never applied and OCI pushes over 1m got 413.
rock
Story Crater Bot