LLM API Gateway Request Flow

Client sends model name in request body, gateway dispatches to correct predictor

LLM API Gateway Request Flow Client sends model name in request body, gateway dispatches to correct predictor POST /v1/chat/completions (model=reasoning) proxy_pass :8080 verify JWT claims ok lookup model upstream endpoint request streaming tokens stream HTTPS stream Auth Dispatch Stream Client · api.riotpiao.com · Sequence participant Client api.riotpiao.com Nginx Ingress · TLS termination · Sequence participant Nginx Ingress TLS termination API Gateway · Go, 3 replicas · Sequence participant API Gateway Go, 3 replicas Authentik · JWT verify · Sequence participant Authentik JWT verify Model Registry · configured routes · Sequence participant Model Registry configured routes vLLM Predictor · reasoning endpoint · Sequence participant vLLM Predictor reasoning endpoint Ollama Predictor · ornith endpoint · Sequence participant Ollama Predictor ornith endpoint Legend request return security default message

Request Entry

  • • Client sends model name in POST body (not path)
  • • HTTPS hits nginx, proxied to gateway:8080
  • • Gateway validates JWT via Authentik

Route Dispatch

  • • Gateway looks up model in in-memory ConfigMap
  • • Registry returns upstream endpoint (reasoning-predictor.llm-serving:80)
  • • No per-model ingress paths; one endpoint routes all models

Streaming Response

  • • Predictor streams tokens back to gateway
  • • Gateway forwards stream to nginx
  • • nginx sends to client with kept-alive proxy-buffering=off

Model Routing

  • • reasoning → reasoning-predictor:80 (vLLM, Qwen3-32B)
  • • ornith:35b → ornith-predictor:80 (Ollama)
  • • qwen2.5:3b-instruct → ornith-predictor:80 (Ollama, same pod)
  • • nomic-ai/nomic-embed-text-v2-moe → embeddings-predictor:80 (TEI)