Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db97159905 | ||
|
|
67760720fc | ||
|
|
c5099e3a8f | ||
|
|
6b609eb4c2 |
@@ -51,25 +51,4 @@ data:
|
||||
# even though the DNS CNAME points at the tunnel.
|
||||
- hostname: zitadel.riotpiao.com
|
||||
service: https://zitadel.riotpiao.com
|
||||
# KUBERNETES API, for kubectl from outside the LAN.
|
||||
#
|
||||
# Origin is the in-cluster Service, not the node IP: that survives a
|
||||
# control-plane address change and lets the connectors reach whichever
|
||||
# apiserver is local-healthy.
|
||||
#
|
||||
# noTLSVerify is required. The apiserver serves a certificate for the
|
||||
# cluster CA plus its node/LB addresses, so its SANs cannot match this
|
||||
# public hostname. Cloudflare terminates the public TLS with the
|
||||
# *.riotpiao.com edge certificate, and the connector hop is already
|
||||
# inside the cluster, so skipping verification on that short hop is the
|
||||
# least-bad option. Do not widen this to other rules.
|
||||
#
|
||||
# SECURITY: this exposes the API to the internet. Authentication is still
|
||||
# ZITADEL OIDC plus client certificates, but put a Cloudflare Access
|
||||
# application in front of this hostname as well -- see the note in the
|
||||
# PR. Without Access, every scanner on the internet reaches the apiserver.
|
||||
- hostname: k8s.riotpiao.com
|
||||
service: https://kubernetes.default.svc.cluster.local:443
|
||||
originRequest:
|
||||
noTLSVerify: true
|
||||
- service: http_status:404
|
||||
|
||||
@@ -33,19 +33,17 @@ metadata:
|
||||
app: clef-serve
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
spec:
|
||||
# Parked at 0 to free the only V100 that was not serving anything, so candle-vllm can
|
||||
# run its Phase 0 on one card. All four cards were allocated (coder 2, reasoning 1,
|
||||
# this 1) and a new GPU workload had nowhere to land.
|
||||
# One card. worker-1 holds four V100s: candle-vllm takes one for the coder, and
|
||||
# this pod takes one, so two stay free for a TP=2 candle run.
|
||||
#
|
||||
# This is the only GPU consumer in the namespace that is not vLLM, which is why it is
|
||||
# the first card released: the standing instruction is to keep the vLLM pods healthy
|
||||
# until candle-vllm answers, and this pod is not one of them.
|
||||
# Enabled 2026-11-04 to serve /v1/systemone, which the gateway already routes to
|
||||
# this Service. It was parked at 0 while candle-vllm was brought up on the same
|
||||
# cards, and it served no inference request between 2026-10-30T06:23:40Z and then.
|
||||
#
|
||||
# It was serving nothing. The last 30 log lines at park time were 30/30 kubelet
|
||||
# /health probes and no inference request since start (2026-10-30T06:23:40Z). Nothing
|
||||
# in git selects this service. Restore it once the rollout leaves a card free:
|
||||
# 30B-A3B takes 1, Coder-Next TP=2 takes 2, so card 4 is this one.
|
||||
replicas: 0
|
||||
# Cold start is about four minutes: the pod pip-installs on every boot because the
|
||||
# torch base image cannot be pushed to the registry (Cloudflare answers 413 above
|
||||
# 100 MB per layer). startupProbe allows for that.
|
||||
replicas: 1
|
||||
# Recreate, not RollingUpdate. Every GPU card is allocated, so a rolling update leaves
|
||||
# the new pod Pending forever waiting for a card the old pod still holds.
|
||||
strategy:
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Stable client-facing name for the coder chat model, deliberately decoupled
|
||||
# from the process that serves it.
|
||||
#
|
||||
# poimen-memory addresses a Service, not a pod and not an InferenceService. The
|
||||
# vLLM -> candle-vllm handoff therefore changes this file's selector and nothing
|
||||
# else, so the client config never changes again and there is no window in which
|
||||
# the host name is wrong.
|
||||
#
|
||||
# The previous KTML predictor requested two V100s and never reached Ready, so it
|
||||
# held half the GPU pool while serving nothing. candle-vllm replaces it.
|
||||
#
|
||||
# Two ports on purpose:
|
||||
# - 80 matches every other predictor Service in this namespace, and matches
|
||||
# a client base URL that carries no port number.
|
||||
# - 8080 is the container port. Publishing it directly avoids a port
|
||||
# translation that is easy to get wrong in a curl.
|
||||
#
|
||||
# Do not delete and recreate this Service: a delete drops the ClusterIP, and a
|
||||
# client that cached the name during the gap fails until it re-resolves.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: coder-predictor
|
||||
namespace: llm-serving
|
||||
labels:
|
||||
app.kubernetes.io/part-of: llm-serving
|
||||
app: coder-predictor
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: candle-vllm
|
||||
ports:
|
||||
- name: http
|
||||
protocol: TCP
|
||||
port: 80
|
||||
targetPort: 8080
|
||||
- name: http-8080
|
||||
protocol: TCP
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
@@ -14,13 +14,9 @@ resources:
|
||||
# v0.5.22+, published only on CUDA 13, and CUDA 13 dropped Volta kernels. The model
|
||||
# instead runs the implementation it ships (joint_schema_model.py) in fp16 on one V100.
|
||||
- clef-serve.yaml
|
||||
# coder = Qwen3-Coder-30B-A3B-Instruct, 4-bit GPTQ, TP=2 at 262,144 tokens. It serves
|
||||
# both `coder` and `reasoning`, so poimen-memory needs a host change only, never a
|
||||
# model-name change. reasoning.yaml below can be dropped once that host change lands.
|
||||
- coder.yaml
|
||||
- reasoning.yaml
|
||||
# Stable client-facing name for the chat model. poimen-memory points at this Service,
|
||||
# so the vLLM -> candle-vllm handoff is a selector edit here and no client change.
|
||||
- coder-service.yaml
|
||||
- reasoning-long-service.yaml
|
||||
# ornith is retired: its InferenceService still reports Ready=True while running zero
|
||||
# replicas, because desired equals available at 0. Its manifest was never committed,
|
||||
|
||||
Reference in New Issue
Block a user