rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm-serving): scale ornith to 2 replicas instead of a dedicated grm GPU

Browse Source 
reasoning keeps its 2 GPUs untouched. verifier's freed GPU goes to a second ornith replica instead of a standalone qwen-only pod -- both replicas load ornith:35b + qwen2.5:3b-instruct, k8s Service load-balances across them, so 2 concurrent implementer-style calls get independent instances.
This commit is contained in:
Story Crater Bot
2026-08-18 18:25:12 -07:00
parent 50d00ae350
commit 5d7b9acd43
4 changed files with 24 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/apps/api/llm-routes.yaml
+8 -9
View File
@@ -7,14 +7,13 @@
# has a `path: k8s/apps/api` source) so all gateway config stays in one place.
#
# ── Model -> upstream map (verified live) ───────────────────────────────────
# reasoning -> reasoning-predictor vLLM, DeepSeek-R1-Distill-32B
# ornith:35b -> ornith-predictor Ollama
# qwen2.5:3b-instruct -> grm-predictor Ollama (dedicated GPU --
# retired verifier-predictor's
# vLLM PRM slot; verification/
# judge traffic no longer
# contends with ornith:35b's
# planner/implementer traffic)
# reasoning -> reasoning-predictor vLLM, DeepSeek-R1-Distill-32B, 2 replicas
# ornith:35b -> ornith-predictor Ollama, 2 replicas (retired verifier-
# qwen2.5:3b-instruct -> ornith-predictor Ollama predictor's vLLM PRM slot to get
# the 2nd GPU) -- k8s Service load-balances
# across both, each replica loads both
# models, so 2 concurrent implementer-style
# calls each land on an independent instance
# nomic-embed-text-v2 -> embeddings-predictor TEI
# bge-reranker-base -> reranker-predictor TEI
#
@@ -214,7 +213,7 @@ spec:
pathType: Prefix
backend:
service:
name: grm-predictor
name: ornith-predictor
port:
number: 80
---