fix(llm-serving): ornith 504 on cold start
Two bugs. Kong timeouts were on the Ingress; it reads them from the Service, so its 60s default applied. Moved to the isvc, which KServe propagates. Probes ran 'ollama list' — models on disk, not in VRAM — so the pod went Ready before it could serve. Now 'ollama ps', and both models are warmed at startup.
This commit is contained in:
@@ -3,6 +3,20 @@ kind: InferenceService
|
|||||||
metadata:
|
metadata:
|
||||||
annotations:
|
annotations:
|
||||||
serving.kserve.io/deploymentMode: RawDeployment
|
serving.kserve.io/deploymentMode: RawDeployment
|
||||||
|
# Kong reads its timeouts from the Kubernetes Service, not the Ingress —
|
||||||
|
# Ingress annotations configure Route entities (strip-path, methods,
|
||||||
|
# plugins), these configure the Service entity. They were on
|
||||||
|
# llm-chat-ornith's Ingress and therefore ignored, leaving Kong's 60s
|
||||||
|
# default in force. KServe propagates InferenceService annotations to the
|
||||||
|
# Service it generates, which is how they reach Kong from here.
|
||||||
|
#
|
||||||
|
# This was invisible while OLLAMA_KEEP_ALIVE=-1 kept the model resident: no
|
||||||
|
# request ever waited on a cold load. A pod restart flushes VRAM, and
|
||||||
|
# loading ornith:35b takes longer than 60s, so the first request after any
|
||||||
|
# restart returned 504.
|
||||||
|
konghq.com/connect-timeout: "10000"
|
||||||
|
konghq.com/read-timeout: "3600000"
|
||||||
|
konghq.com/write-timeout: "3600000"
|
||||||
labels:
|
labels:
|
||||||
app.kubernetes.io/name: llm-ornith
|
app.kubernetes.io/name: llm-ornith
|
||||||
app.kubernetes.io/part-of: llm-serving
|
app.kubernetes.io/part-of: llm-serving
|
||||||
@@ -26,6 +40,10 @@ spec:
|
|||||||
|
|
||||||
ollama pull qwen2.5:3b-instruct
|
ollama pull qwen2.5:3b-instruct
|
||||||
|
|
||||||
|
ollama run ornith:35b "ok" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
wait $SERVE_PID
|
wait $SERVE_PID
|
||||||
|
|
||||||
'
|
'
|
||||||
@@ -52,7 +70,7 @@ spec:
|
|||||||
command:
|
command:
|
||||||
- /bin/sh
|
- /bin/sh
|
||||||
- -c
|
- -c
|
||||||
- ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null |
|
- ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
|
||||||
grep -q qwen2.5
|
grep -q qwen2.5
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
resources:
|
resources:
|
||||||
@@ -69,7 +87,7 @@ spec:
|
|||||||
command:
|
command:
|
||||||
- /bin/sh
|
- /bin/sh
|
||||||
- -c
|
- -c
|
||||||
- ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null |
|
- ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
|
||||||
grep -q qwen2.5
|
grep -q qwen2.5
|
||||||
failureThreshold: 120
|
failureThreshold: 120
|
||||||
periodSeconds: 15
|
periodSeconds: 15
|
||||||
|
|||||||
Reference in New Issue
Block a user