forgejo-db: CNPG cluster stuck failing over, Actions runners timeout #92

New Issue

2026-11-01T03:13:09Z

rock commented

2026-11-01 03:13:09 +00:00

Symptom

Forgejo Actions runners cannot fetch tasks. Every FetchTask call times out (504). The
UPDATE action_runner SET last_online ... query takes 7-9 seconds per call.

Root cause

The forgejo-db CNPG cluster is stuck in Failing over for ~5 hours.

forgejo-db-2 crashed 21 times (last ~5h ago), which triggered the failover.
forgejo-db-3 is stuck in crash recovery: FATAL: the database system is starting up (57P03), retrying every 1s.
forgejo-db-1 (current primary) has a FailedMount on its Longhorn PVC: no Pending workload pods for volume pvc-c5aa5b7a-991a-4063-acd2-87275d3f1df3 to be mounted.
Operator event: PrimaryStatusCheckFailed — primary forgejo-db-1 is Ready but the operator's /pg/status check failed; failover deferred until Kubernetes marks the primary as not Ready.

Impact

All Forgejo Actions are blocked (runners time out fetching tasks).
The new candle-vllm sm_70 build is queued but cannot run.
General Forgejo reads/writes still work but are slow.

Evidence

$ kubectl get cluster -n cicd
NAME         AGE   INSTANCES   READY   STATUS         PRIMARY
forgejo-db   98d   3           2       Failing over   forgejo-db-1

$ kubectl get pods -n cicd -l cnpg.io/cluster=forgejo-db
forgejo-db-1   1/1   Running   0    46h
forgejo-db-2   1/1   Running   21 (5h2m ago)   5d21h
forgejo-db-3   0/1   Running   0    46h

Ask

Restore the forgejo-db cluster to a healthy state (2 or 3 ready instances, status
Cluster in healthy state). Investigate why forgejo-db-3 is stuck in recovery and why
forgejo-db-1 has a FailedMount on its PVC.

## Symptom Forgejo Actions runners cannot fetch tasks. Every `FetchTask` call times out (504). The `UPDATE action_runner SET last_online ...` query takes 7-9 seconds per call. ## Root cause The `forgejo-db` CNPG cluster is stuck in `Failing over` for ~5 hours. - `forgejo-db-2` crashed 21 times (last ~5h ago), which triggered the failover. - `forgejo-db-3` is stuck in crash recovery: `FATAL: the database system is starting up` (57P03), retrying every 1s. - `forgejo-db-1` (current primary) has a `FailedMount` on its Longhorn PVC: `no Pending workload pods for volume pvc-c5aa5b7a-991a-4063-acd2-87275d3f1df3 to be mounted`. - Operator event: `PrimaryStatusCheckFailed` — `primary forgejo-db-1 is Ready but the operator's /pg/status check failed; failover deferred until Kubernetes marks the primary as not Ready`. ## Impact - All Forgejo Actions are blocked (runners time out fetching tasks). - The new `candle-vllm` sm_70 build is queued but cannot run. - General Forgejo reads/writes still work but are slow. ## Evidence ``` $ kubectl get cluster -n cicd NAME AGE INSTANCES READY STATUS PRIMARY forgejo-db 98d 3 2 Failing over forgejo-db-1 $ kubectl get pods -n cicd -l cnpg.io/cluster=forgejo-db forgejo-db-1 1/1 Running 0 46h forgejo-db-2 1/1 Running 21 (5h2m ago) 5d21h forgejo-db-3 0/1 Running 0 46h ``` ## Ask Restore the `forgejo-db` cluster to a healthy state (2 or 3 ready instances, status `Cluster in healthy state`). Investigate why `forgejo-db-3` is stuck in recovery and why `forgejo-db-1` has a `FailedMount` on its PVC.

rock referenced this issue

2026-11-01 03:25:17 +00:00

fix(forgejo): raise Gitea CPU limit — it was throttled 16% of the time #93

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#92