chore(comfyui): retire to 0 replicas and free the 4th V100 #89

2026-10-31T23:59:53Z

rock commented

2026-10-31 23:59:53 +00:00

comfyui has held a V100 on worker-1 for 42h while serving nothing.

Why it serves nothing

Its oauth2-proxy roll-out is stuck. The pod requests client-id, client-secret and cookie-secret from Secret comfyui/comfyui-oauth2-proxy, and that Secret carries only 1 key. Kubelet reports:

couldn't find key client-id in Secret comfyui/comfyui-oauth2-proxy

It failed 11621 times. The old ReplicaSet keeps running, so the Deployment reports 1/1 Ready while every request is refused at the auth step. ArgoCD reports the app Degraded. The card sat at 0% utilisation.

Why this matters

worker-1 has 4 GPUs and llm-serving holds 3 of them:

Card	Holder	Used
1	clef-serve	18654 MiB
2	coder-predictor	29600 MiB
3	reasoning-predictor	29464 MiB
4	comfyui	idle

Node GPU requests were 4/4. Nothing in llm-serving could grow without this card. It is now needed: coder runs TP=2 at 256K context, which needs a second card.

What this changes

One field: replicas: 1 -> 0.

The Deployment, Service, Ingress and the 50Gi comfyui-models PVC are untouched, so no data is at risk.

What this deliberately does not do

It does not delete the Application or the PVC. The PVC is declared inside the same file as the Deployment, so removing the Application with prune: true would delete the 50Gi models volume. If the volume should also go, that is a separate, explicit step.

Restore

Set replicas: 1 here, then fix the Secret to carry all three keys. Otherwise the same stuck roll-out returns.

comfyui has held a V100 on worker-1 for 42h while serving nothing. ## Why it serves nothing Its oauth2-proxy roll-out is stuck. The pod requests `client-id`, `client-secret` and `cookie-secret` from Secret `comfyui/comfyui-oauth2-proxy`, and that Secret carries only 1 key. Kubelet reports: ``` couldn't find key client-id in Secret comfyui/comfyui-oauth2-proxy ``` It failed 11621 times. The old ReplicaSet keeps running, so the Deployment reports 1/1 Ready while every request is refused at the auth step. ArgoCD reports the app Degraded. The card sat at 0% utilisation. ## Why this matters worker-1 has 4 GPUs and llm-serving holds 3 of them: | Card | Holder | Used | |---|---|---| | 1 | clef-serve | 18654 MiB | | 2 | coder-predictor | 29600 MiB | | 3 | reasoning-predictor | 29464 MiB | | 4 | comfyui | idle | Node GPU requests were 4/4. Nothing in llm-serving could grow without this card. It is now needed: coder runs TP=2 at 256K context, which needs a second card. ## What this changes One field: `replicas: 1` -> `0`. The Deployment, Service, Ingress and the 50Gi `comfyui-models` PVC are untouched, so **no data is at risk**. ## What this deliberately does not do It does not delete the Application or the PVC. The PVC is declared inside the same file as the Deployment, so removing the Application with `prune: true` would delete the 50Gi models volume. If the volume should also go, that is a separate, explicit step. ## Restore Set `replicas: 1` here, then fix the Secret to carry all three keys. Otherwise the same stuck roll-out returns.

rock added 1 commit 2026-10-31 23:59:56 +00:00

chore(comfyui): scale to 0 to free the 4th V100 for llm-serving 6e2b681b25

comfyui has served nothing for 42h while holding the 4th GPU on worker-1.
Its oauth2-proxy roll-out is stuck: the pod requests client-id,
client-secret and cookie-secret from Secret comfyui/comfyui-oauth2-proxy,
and that Secret carries only 1 key. It failed 11621 times. The old
ReplicaSet keeps running, so the Deployment reports 1/1 while every
request is refused at the auth step. ArgoCD reports the app Degraded.

The card sat at 0% utilisation the whole time.

llm-serving has no spare card. worker-1 has 4 and all 4 are allocated:
clef-serve, coder-predictor, reasoning-predictor, and this one. Freeing
it is the cheapest way to give an LLM workload headroom, because nothing
else on the node can yield a card without a service interruption.

This change is one field: replicas 1 -> 0. The Deployment, Service,
Ingress and the 50Gi comfyui-models PVC are untouched, so no data is at
risk and restoring is a one-line revert plus a fix to the Secret.

rock merged commit 3afb7b1a65 into main

2026-11-01 00:01:27 +00:00

rock deleted branch chore/retire-comfyui

2026-11-01 00:01:32 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#89