fix(llm-serving): ornith 504 on cold start

Two bugs. Kong timeouts were on the Ingress; it reads them from the Service, so
its 60s default applied. Moved to the isvc, which KServe propagates.

Probes ran 'ollama list' — models on disk, not in VRAM — so the pod went Ready
before it could serve. Now 'ollama ps', and both models are warmed at startup.
This commit is contained in:
Story Crater Bot
2026-08-15 22:58:24 -07:00
parent 5b7d22c90d
commit a005a07648
+20 -2
View File
@@ -3,6 +3,20 @@ kind: InferenceService
metadata: metadata:
annotations: annotations:
serving.kserve.io/deploymentMode: RawDeployment serving.kserve.io/deploymentMode: RawDeployment
# Kong reads its timeouts from the Kubernetes Service, not the Ingress —
# Ingress annotations configure Route entities (strip-path, methods,
# plugins), these configure the Service entity. They were on
# llm-chat-ornith's Ingress and therefore ignored, leaving Kong's 60s
# default in force. KServe propagates InferenceService annotations to the
# Service it generates, which is how they reach Kong from here.
#
# This was invisible while OLLAMA_KEEP_ALIVE=-1 kept the model resident: no
# request ever waited on a cold load. A pod restart flushes VRAM, and
# loading ornith:35b takes longer than 60s, so the first request after any
# restart returned 504.
konghq.com/connect-timeout: "10000"
konghq.com/read-timeout: "3600000"
konghq.com/write-timeout: "3600000"
labels: labels:
app.kubernetes.io/name: llm-ornith app.kubernetes.io/name: llm-ornith
app.kubernetes.io/part-of: llm-serving app.kubernetes.io/part-of: llm-serving
@@ -26,6 +40,10 @@ spec:
ollama pull qwen2.5:3b-instruct ollama pull qwen2.5:3b-instruct
ollama run ornith:35b "ok" >/dev/null 2>&1 || true
ollama run qwen2.5:3b-instruct "ok" >/dev/null 2>&1 || true
wait $SERVE_PID wait $SERVE_PID
' '
@@ -52,7 +70,7 @@ spec:
command: command:
- /bin/sh - /bin/sh
- -c - -c
- ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
grep -q qwen2.5 grep -q qwen2.5
periodSeconds: 10 periodSeconds: 10
resources: resources:
@@ -69,7 +87,7 @@ spec:
command: command:
- /bin/sh - /bin/sh
- -c - -c
- ollama list 2>/dev/null | grep -q ornith && ollama list 2>/dev/null | - ollama ps 2>/dev/null | grep -q ornith && ollama ps 2>/dev/null |
grep -q qwen2.5 grep -q qwen2.5
failureThreshold: 120 failureThreshold: 120
periodSeconds: 15 periodSeconds: 15