From 6e2b681b25243c58057ae15b892bfe6e102d1f1f Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 16:16:17 -0700 Subject: [PATCH] chore(comfyui): scale to 0 to free the 4th V100 for llm-serving comfyui has served nothing for 42h while holding the 4th GPU on worker-1. Its oauth2-proxy roll-out is stuck: the pod requests client-id, client-secret and cookie-secret from Secret comfyui/comfyui-oauth2-proxy, and that Secret carries only 1 key. It failed 11621 times. The old ReplicaSet keeps running, so the Deployment reports 1/1 while every request is refused at the auth step. ArgoCD reports the app Degraded. The card sat at 0% utilisation the whole time. llm-serving has no spare card. worker-1 has 4 and all 4 are allocated: clef-serve, coder-predictor, reasoning-predictor, and this one. Freeing it is the cheapest way to give an LLM workload headroom, because nothing else on the node can yield a card without a service interruption. This change is one field: replicas 1 -> 0. The Deployment, Service, Ingress and the 50Gi comfyui-models PVC are untouched, so no data is at risk and restoring is a one-line revert plus a fix to the Secret. --- k8s/apps/comfyui/deployment.yaml | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/k8s/apps/comfyui/deployment.yaml b/k8s/apps/comfyui/deployment.yaml index 017b757..af1dbbf 100644 --- a/k8s/apps/comfyui/deployment.yaml +++ b/k8s/apps/comfyui/deployment.yaml @@ -8,7 +8,18 @@ metadata: labels: app: comfyui spec: - replicas: 1 + # Retired on purpose. This Deployment held the 4th V100 on worker-1 while + # serving nothing: its oauth2-proxy roll-out has been stuck for 42h and + # failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries + # only 1 key while the pod requests client-id, client-secret and + # cookie-secret. Every request was therefore refused, so the card sat at 0% + # utilisation and blocked a GPU that llm-serving can use. + # + # Scaling to 0 is the whole change. The Deployment, the Service, the Ingress + # and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is + # touched. To bring it back: set replicas to 1 here, then fix the Secret to + # carry all three keys, otherwise the same stuck roll-out returns. + replicas: 0 strategy: type: Recreate selector: