rock
67760720fc
Creates coder-predictor selecting app=candle-vllm on ports 80 and 8080, so clients keep one stable host name across the vLLM -> candle-vllm handoff and only this selector changes from now on. The retired KTML predictor requested two V100s and never reached Ready.
41 lines
1.3 KiB
YAML
41 lines
1.3 KiB
YAML
# Stable client-facing name for the coder chat model, deliberately decoupled
|
|
# from the process that serves it.
|
|
#
|
|
# poimen-memory addresses a Service, not a pod and not an InferenceService. The
|
|
# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
|
|
# else, so the client config never changes again and there is no window in which
|
|
# the host name is wrong.
|
|
#
|
|
# The previous KTML predictor requested two V100s and never reached Ready, so it
|
|
# held half the GPU pool while serving nothing. candle-vllm replaces it.
|
|
#
|
|
# Two ports on purpose:
|
|
# - 80 matches every other predictor Service in this namespace, and matches
|
|
# a client base URL that carries no port number.
|
|
# - 8080 is the container port. Publishing it directly avoids a port
|
|
# translation that is easy to get wrong in a curl.
|
|
#
|
|
# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
|
|
# client that cached the name during the gap fails until it re-resolves.
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: coder-predictor
|
|
namespace: llm-serving
|
|
labels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
app: coder-predictor
|
|
spec:
|
|
type: ClusterIP
|
|
selector:
|
|
app: candle-vllm
|
|
ports:
|
|
- name: http
|
|
protocol: TCP
|
|
port: 80
|
|
targetPort: 8080
|
|
- name: http-8080
|
|
protocol: TCP
|
|
port: 8080
|
|
targetPort: 8080
|