rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm-serving): retire verifier-predictor, add grm (qwen2.5:3b)

Browse Source 
Frees verifier's GPU from an underused vLLM PRM deployment. qwen2.5:3b-instruct moves off ornith-predictor's shared pod onto its own dedicated GPU (grm.yaml), so verification/judge traffic stops contending with ornith:35b's agent traffic. /v1/qwen/chat/completions now points at grm-predictor; path unchanged.
This commit is contained in:
Story Crater Bot
2026-08-18 18:18:31 -07:00
parent e6ada95b39
commit 50d00ae350
5 changed files with 115 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/apps/api/llm-routes.yaml
+12 -40
View File
@@ -9,10 +9,14 @@
# ── Model -> upstream map (verified live) ───────────────────────────────────
# reasoning -> reasoning-predictor vLLM, DeepSeek-R1-Distill-32B
# ornith:35b -> ornith-predictor Ollama
# qwen2.5:3b-instruct -> ornith-predictor Ollama (same pod!)
# qwen2.5:3b-instruct -> grm-predictor Ollama (dedicated GPU --
# retired verifier-predictor's
# vLLM PRM slot; verification/
# judge traffic no longer
# contends with ornith:35b's
# planner/implementer traffic)
# nomic-embed-text-v2 -> embeddings-predictor TEI
# bge-reranker-base -> reranker-predictor TEI
# Qwen2.5-Math-PRM-7B -> verifier-predictor vLLM pooling
#
# ── Why path-per-model, and why the body is rewritten ───────────────────────
# Kong matches routes on host, path, method and headers — never on the request
@@ -20,12 +24,10 @@
# `model` field is not expressible in Kong OSS (`ai-proxy-advanced`, which does
# multi-target model routing, is Enterprise-only).
#
# Hence the model is in the path. But `ornith:35b` and `qwen2.5:3b-instruct`
# share ONE Ollama pod, and Ollama still reads which model to load from the
# body's `model` field. If only the path selected the route, a client calling
# /v1/qwen/... with `"model": "ornith:35b"` in the body would silently get the
# 35B model. So each chat route force-overwrites `model` in the body, making the
# path the single source of truth. Callers may omit `model` entirely.
# Hence the model is in the path, and each chat route force-overwrites `model`
# in the body regardless, so a client calling /v1/qwen/... with some other
# `model` value in the body can't silently get routed to the wrong weights.
# Callers may omit `model` entirely.
#
# ── Timeouts ───────────────────────────────────────────────────────────────
# Kong's upstream timeouts default to 60000ms. A 32B model generating a long
@@ -56,8 +58,7 @@ config:
{"id":"ornith:35b","object":"model","owned_by":"homelab","created":0},
{"id":"qwen2.5:3b-instruct","object":"model","owned_by":"homelab","created":0},
{"id":"nomic-ai/nomic-embed-text-v2-moe","object":"model","owned_by":"homelab","created":0},
{"id":"BAAI/bge-reranker-base","object":"model","owned_by":"homelab","created":0},
{"id":"Qwen/Qwen2.5-Math-PRM-7B","object":"model","owned_by":"homelab","created":0}
{"id":"BAAI/bge-reranker-base","object":"model","owned_by":"homelab","created":0}
]}
---
apiVersion: networking.k8s.io/v1
@@ -213,7 +214,7 @@ spec:
pathType: Prefix
backend:
service:
name: ornith-predictor
name: grm-predictor
port:
number: 80
---
@@ -286,32 +287,3 @@ spec:
name: reranker-predictor
port:
number: 80
---
# ── POST /v1/score ──────────────────────────────────────────────────────────
# The process reward model. Returns scores, not tokens, so it is deliberately
# not under /chat/completions. vLLM serves /v1/score natively (verified), so no
# rewrite is needed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: llm-score
namespace: llm-serving
annotations:
konghq.com/strip-path: "false"
konghq.com/methods: "POST"
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "600000"
konghq.com/write-timeout: "600000"
spec:
ingressClassName: kong
rules:
- host: api.riotpiao.com
http:
paths:
- path: /v1/score
pathType: Prefix
backend:
service:
name: verifier-predictor
port:
number: 80