Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
Kong matches routes on host/path/method/header, never on the request body, so a
single /v1/chat/completions dispatching on body.model is not expressible in Kong
OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise).
Model therefore goes in the path:
GET /v1/models static list (request-termination)
POST /v1/reasoning/chat/completions reasoning-predictor (vLLM)
POST /v1/ornith/chat/completions ornith-predictor (Ollama)
POST /v1/qwen/chat/completions ornith-predictor (Ollama, same pod)
POST /v1/embeddings embeddings-predictor (TEI)
POST /v1/rerank reranker-predictor (TEI)
POST /v1/score verifier-predictor (vLLM pooling)
- each chat route force-overwrites body.model via request-transformer add+replace:
ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a
client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B
- routes live in ns llm-serving, not api: an Ingress can only reference a Service
in its own namespace, and KIC watches all namespaces
- embeddings and score need no rewrite (TEI/vLLM already serve the canonical
paths); rerank does, since /v1/rerank 404s and only /rerank exists
- read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta
exceeds mid-generation and returns 504
- nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both
hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins
- no auth for now, per decision; api.riotpiao.com is reachable through nginx, so
GPU time is currently unauthenticated
Story Crater Bot