k8s(llm-serving): client-facing Service for candle-vllm coder
Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so clients keep one stable host name across the vLLM -> candle-vllm handoff and only this selector changes from now on. The retired KTML predictor requested two V100s and never reached Ready.
This commit is contained in:
@@ -0,0 +1,40 @@
|
|||||||
|
# Stable client-facing name for the coder chat model, deliberately decoupled
|
||||||
|
# from the process that serves it.
|
||||||
|
#
|
||||||
|
# poimen-memory addresses a Service, not a pod and not an InferenceService. The
|
||||||
|
# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
|
||||||
|
# else, so the client config never changes again and there is no window in which
|
||||||
|
# the host name is wrong.
|
||||||
|
#
|
||||||
|
# The previous KTML predictor requested two V100s and never reached Ready, so it
|
||||||
|
# held half the GPU pool while serving nothing. candle-vllm replaces it.
|
||||||
|
#
|
||||||
|
# Two ports on purpose:
|
||||||
|
# - 80 matches every other predictor Service in this namespace, and matches
|
||||||
|
# a client base URL that carries no port number.
|
||||||
|
# - 8080 is the container port. Publishing it directly avoids a port
|
||||||
|
# translation that is easy to get wrong in a curl.
|
||||||
|
#
|
||||||
|
# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
|
||||||
|
# client that cached the name during the gap fails until it re-resolves.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: coder-predictor
|
||||||
|
namespace: llm-serving
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: llm-serving
|
||||||
|
app: coder-predictor
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: candle-vllm
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
protocol: TCP
|
||||||
|
port: 80
|
||||||
|
targetPort: 8080
|
||||||
|
- name: http-8080
|
||||||
|
protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
targetPort: 8080
|
||||||
@@ -16,6 +16,7 @@ resources:
|
|||||||
- clef-serve.yaml
|
- clef-serve.yaml
|
||||||
# Stable client-facing name for the chat model. poimen-memory points at this Service,
|
# Stable client-facing name for the chat model. poimen-memory points at this Service,
|
||||||
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
|
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
|
||||||
|
- coder-service.yaml
|
||||||
- reasoning-long-service.yaml
|
- reasoning-long-service.yaml
|
||||||
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
||||||
# replicas, because desired equals available at 0. Its manifest was never committed,
|
# replicas, because desired equals available at 0. Its manifest was never committed,
|
||||||
|
|||||||
Reference in New Issue
Block a user