Story Crater Bot
b7809cb58b
Kong matches routes on host/path/method/header, never on the request body, so a single /v1/chat/completions dispatching on body.model is not expressible in Kong OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise). Model therefore goes in the path: GET /v1/models static list (request-termination) POST /v1/reasoning/chat/completions reasoning-predictor (vLLM) POST /v1/ornith/chat/completions ornith-predictor (Ollama) POST /v1/qwen/chat/completions ornith-predictor (Ollama, same pod) POST /v1/embeddings embeddings-predictor (TEI) POST /v1/rerank reranker-predictor (TEI) POST /v1/score verifier-predictor (vLLM pooling) - each chat route force-overwrites body.model via request-transformer add+replace: ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B - routes live in ns llm-serving, not api: an Ingress can only reference a Service in its own namespace, and KIC watches all namespaces - embeddings and score need no rewrite (TEI/vLLM already serve the canonical paths); rerank does, since /v1/rerank 404s and only /rerank exists - read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta exceeds mid-generation and returns 504 - nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins - no auth for now, per decision; api.riotpiao.com is reachable through nginx, so GPU time is currently unauthenticated
13 lines
635 B
YAML
13 lines
635 B
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# Explicit allowlist so kong-values.yaml in this directory is NOT treated as a
|
|
# manifest — it is Helm input consumed by the chart source of the `kong`
|
|
# Application, not a Kubernetes object. Anything new added here must be listed
|
|
# or it is silently dropped with no error and no drift shown.
|
|
resources:
|
|
- ingress.yaml
|
|
- llm-routes.yaml
|
|
# No top-level `namespace:` transformer on purpose: ingress.yaml sets its own
|
|
# namespace, and the transformer rewrites metadata.namespace on every resource
|
|
# it builds, which is a trap for anything cross-namespace added later.
|