Compare commits

..
Author SHA1 Message Date
rock 6d02f59a9b k8s(llm-serving): stop managing the coder InferenceService
coder.yaml requests 2 V100s and never reaches Ready, so it permanently
occupies half the GPU pool. candle-vllm serves the coder model on the
other two cards; the unmanaged predictor starves clef-serve and blocks
its own rollout on every sync.

ArgoCD has prune:false, so this removes management only. The live object
is deleted manually after the sync, otherwise selfHeal recreates it.
2026-11-04 06:07:20 +08:00
3 changed files with 11 additions and 50 deletions
+11 -9
View File
@@ -33,17 +33,19 @@ metadata:
app: clef-serve app: clef-serve
app.kubernetes.io/part-of: llm-serving app.kubernetes.io/part-of: llm-serving
spec: spec:
# One card. worker-1 holds four V100s: candle-vllm takes one for the coder, and # Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
# this pod takes one, so two stay free for a TP=2 candle run. # run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
# this 1) and a new GPU workload had nowhere to land.
# #
# Enabled 2026-11-04 to serve /v1/systemone, which the gateway already routes to # This is the only GPU consumer in the namespace that is not vLLM, which is why it is
# this Service. It was parked at 0 while candle-vllm was brought up on the same # the first card released: the standing instruction is to keep the vLLM pods healthy
# cards, and it served no inference request between 2026-10-30T06:23:40Z and then. # until candle-vllm answers, and this pod is not one of them.
# #
# Cold start is about four minutes: the pod pip-installs on every boot because the # It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
# torch base image cannot be pushed to the registry (Cloudflare answers 413 above # /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
# 100 MB per layer). startupProbe allows for that. # in git selects this service. Restore it once the rollout leaves a card free:
replicas: 1 # 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
replicas: 0
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves # Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
# the new pod Pending forever waiting for a card the old pod still holds. # the new pod Pending forever waiting for a card the old pod still holds.
strategy: strategy:
-40
View File
@@ -1,40 +0,0 @@
# Stable client-facing name for the coder chat model, deliberately decoupled
# from the process that serves it.
#
# poimen-memory addresses a Service, not a pod and not an InferenceService. The
# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
# else, so the client config never changes again and there is no window in which
# the host name is wrong.
#
# The previous KTML predictor requested two V100s and never reached Ready, so it
# held half the GPU pool while serving nothing. candle-vllm replaces it.
#
# Two ports on purpose:
# - 80 matches every other predictor Service in this namespace, and matches
# a client base URL that carries no port number.
# - 8080 is the container port. Publishing it directly avoids a port
# translation that is easy to get wrong in a curl.
#
# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
# client that cached the name during the gap fails until it re-resolves.
apiVersion: v1
kind: Service
metadata:
name: coder-predictor
namespace: llm-serving
labels:
app.kubernetes.io/part-of: llm-serving
app: coder-predictor
spec:
type: ClusterIP
selector:
app: candle-vllm
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
- name: http-8080
protocol: TCP
port: 8080
targetPort: 8080
-1
View File
@@ -16,7 +16,6 @@ resources:
- clef-serve.yaml - clef-serve.yaml
# Stable client-facing name for the chat model. poimen-memory points at this Service, # Stable client-facing name for the chat model. poimen-memory points at this Service,
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change. # so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
- coder-service.yaml
- reasoning-long-service.yaml - reasoning-long-service.yaml
# ornith is retired: its InferenceService still reports Ready=True while running zero # ornith is retired: its InferenceService still reports Ready=True while running zero
# replicas, because desired equals available at 0. Its manifest was never committed, # replicas, because desired equals available at 0. Its manifest was never committed,