rock
6e2b681b25
comfyui has served nothing for 42h while holding the 4th GPU on worker-1. Its oauth2-proxy roll-out is stuck: the pod requests client-id, client-secret and cookie-secret from Secret comfyui/comfyui-oauth2-proxy, and that Secret carries only 1 key. It failed 11621 times. The old ReplicaSet keeps running, so the Deployment reports 1/1 while every request is refused at the auth step. ArgoCD reports the app Degraded. The card sat at 0% utilisation the whole time. llm-serving has no spare card. worker-1 has 4 and all 4 are allocated: clef-serve, coder-predictor, reasoning-predictor, and this one. Freeing it is the cheapest way to give an LLM workload headroom, because nothing else on the node can yield a card without a service interruption. This change is one field: replicas 1 -> 0. The Deployment, Service, Ingress and the 50Gi comfyui-models PVC are untouched, so no data is at risk and restoring is a one-line revert plus a fix to the Secret.
94 lines
2.8 KiB
YAML
94 lines
2.8 KiB
YAML
# ComfyUI — GPU-accelerated image generation on worker-1.
|
|
# Uses 1x V100 32GB (sm70). Freed by scaling ornith 2→1.
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: comfyui
|
|
namespace: comfyui
|
|
labels:
|
|
app: comfyui
|
|
spec:
|
|
# Retired on purpose. This Deployment held the 4th V100 on worker-1 while
|
|
# serving nothing: its oauth2-proxy roll-out has been stuck for 42h and
|
|
# failing 11621 times, because the Secret comfyui/comfyui-oauth2-proxy carries
|
|
# only 1 key while the pod requests client-id, client-secret and
|
|
# cookie-secret. Every request was therefore refused, so the card sat at 0%
|
|
# utilisation and blocked a GPU that llm-serving can use.
|
|
#
|
|
# Scaling to 0 is the whole change. The Deployment, the Service, the Ingress
|
|
# and the 50Gi comfyui-models PVC all stay, so nothing is lost and no data is
|
|
# touched. To bring it back: set replicas to 1 here, then fix the Secret to
|
|
# carry all three keys, otherwise the same stuck roll-out returns.
|
|
replicas: 0
|
|
strategy:
|
|
type: Recreate
|
|
selector:
|
|
matchLabels:
|
|
app: comfyui
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: comfyui
|
|
spec:
|
|
nodeSelector:
|
|
kubernetes.io/hostname: worker-1
|
|
runtimeClassName: nvidia
|
|
# k8s Service named 'comfyui' injects COMFYUI_PORT=tcp://... into pod env,
|
|
# which clobbers ai-dock's own COMFYUI_PORT variable (expects a port number).
|
|
# Disable service link injection to avoid the collision.
|
|
enableServiceLinks: false
|
|
containers:
|
|
- name: comfyui
|
|
image: ghcr.io/ai-dock/comfyui:v2-cuda-12.1.1-base-22.04
|
|
ports:
|
|
- containerPort: 8188
|
|
protocol: TCP
|
|
env:
|
|
- name: NVIDIA_VISIBLE_DEVICES
|
|
value: "all"
|
|
resources:
|
|
requests:
|
|
cpu: "4"
|
|
memory: 8Gi
|
|
nvidia.com/gpu: "1"
|
|
limits:
|
|
cpu: "8"
|
|
memory: 16Gi
|
|
nvidia.com/gpu: "1"
|
|
volumeMounts:
|
|
- mountPath: /workspace/ComfyUI/models
|
|
name: models
|
|
- mountPath: /workspace/ComfyUI/output
|
|
name: output
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /
|
|
port: 8188
|
|
periodSeconds: 10
|
|
initialDelaySeconds: 30
|
|
startupProbe:
|
|
httpGet:
|
|
path: /
|
|
port: 8188
|
|
failureThreshold: 120
|
|
periodSeconds: 10
|
|
volumes:
|
|
- name: models
|
|
persistentVolumeClaim:
|
|
claimName: comfyui-models
|
|
- name: output
|
|
emptyDir: {}
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: comfyui-models
|
|
namespace: comfyui
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 50Gi
|