Compare commits

..
Author SHA1 Message Date
rock ce4434286c fix(forgejo): raise Gitea CPU limit — it was throttled 16% of the time (#93)
Co-authored-by: rock <[email protected]>
2026-11-01 03:27:56 +00:00
2 changed files with 23 additions and 16 deletions
+1 -13
View File
@@ -33,19 +33,7 @@ metadata:
app: clef-serve app: clef-serve
app.kubernetes.io/part-of: llm-serving app.kubernetes.io/part-of: llm-serving
spec: spec:
# Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can replicas: 1
# run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
# this 1) and a new GPU workload had nowhere to land.
#
# This is the only GPU consumer in the namespace that is not vLLM, which is why it is
# the first card released: the standing instruction is to keep the vLLM pods healthy
# until candle-vllm answers, and this pod is not one of them.
#
# It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
# /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
# in git selects this service. Restore it once the rollout leaves a card free:
# 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
replicas: 0
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
# the new pod Pending forever waiting for a card the old pod still holds. # the new pod Pending forever waiting for a card the old pod still holds.
strategy: strategy:
@@ -134,12 +134,31 @@ ingress:
# wildcard. The wildcard covers this host. # wildcard. The wildcard covers this host.
# Resources # Resources
#
# The 1000m ceiling throttled Gitea badly. Measured on a running pod:
# nr_throttled 189528 of 1169014 CFS periods (16.2%)
# throttled_usec 4424898602 us (~74 minutes lost)
# actual usage 951m against the 1000m limit
# The visible symptom was slow Actions traffic: FetchTask took 3469.9ms and
# repository compare endpoints took 3091-3811ms, so runners timed out and
# retried, which added more load to an already saturated process.
#
# The database was never the bottleneck. A real action_runner UPDATE measured
# 2.133ms with zero lock waits, so this is CPU starvation in Gitea itself.
#
# requests.cpu rises to 750m because a request below real usage earns the pod
# little scheduling weight during contention. limits.cpu rises to 3000m
# because diff computation is bursty and needs room to run without throttling.
#
# Caution: this pod runs on talos-cp-3, a control-plane node. 3000m is 25% of
# that node's 11950m allocatable CPU. If Gitea sustains near this limit, move
# the workload off the control plane rather than raising the limit again.
resources: resources:
requests: requests:
cpu: 200m cpu: 750m
memory: 512Mi memory: 768Mi
limits: limits:
cpu: 1000m cpu: 3000m
memory: 2Gi memory: 2Gi
# Tolerations for control-plane # Tolerations for control-plane