Story Crater Bot
b7809cb58b
Kong matches routes on host/path/method/header, never on the request body, so a single /v1/chat/completions dispatching on body.model is not expressible in Kong OSS (ai-proxy-advanced, which does multi-target model routing, is Enterprise). Model therefore goes in the path: GET /v1/models static list (request-termination) POST /v1/reasoning/chat/completions reasoning-predictor (vLLM) POST /v1/ornith/chat/completions ornith-predictor (Ollama) POST /v1/qwen/chat/completions ornith-predictor (Ollama, same pod) POST /v1/embeddings embeddings-predictor (TEI) POST /v1/rerank reranker-predictor (TEI) POST /v1/score verifier-predictor (vLLM pooling) - each chat route force-overwrites body.model via request-transformer add+replace: ornith:35b and qwen2.5:3b-instruct share one Ollama pod, so without this a client hitting /v1/qwen with "model":"ornith:35b" would silently get the 35B - routes live in ns llm-serving, not api: an Ingress can only reference a Service in its own namespace, and KIC watches all namespaces - embeddings and score need no rewrite (TEI/vLLM already serve the canonical paths); rerank does, since /v1/rerank 404s and only /rerank exists - read/write timeouts 1h: Kong defaults to 60s, which a 32B model on Volta exceeds mid-generation and returns 504 - nginx_proxy_proxy_buffering=off: buffered responses lump or stall SSE, and both hops (nginx Ingress and Kong) must be unbuffered or the buffered one wins - no auth for now, per decision; api.riotpiao.com is reachable through nginx, so GPU time is currently unauthenticated
108 lines
4.6 KiB
YAML
108 lines
4.6 KiB
YAML
# Kong Gateway — cluster-internal API gateway (namespace `api`).
|
|
#
|
|
# Chart: kong/kong 3.4.1 (appVersion 3.9). Only overrides are listed; every key
|
|
# here was checked against `helm show values kong/kong --version 3.4.1`, because
|
|
# Helm silently ignores unknown keys — a typo is a no-op, not an error.
|
|
#
|
|
# ── Topology ────────────────────────────────────────────────────────────────
|
|
# external: client -> nginx (TLS, wildcard *.riotpiao.com) -> kong-proxy:80
|
|
# internal: pod -> kong-proxy.api.svc.cluster.local:80
|
|
#
|
|
# nginx stays the single edge and the only LoadBalancer (192.168.1.160). Kong is
|
|
# the policy/routing layer behind it, so it needs no LB IP and no TLS of its own
|
|
# — hence ClusterIP and proxy.tls disabled. Giving Kong its own IP from
|
|
# homelab-pool would mean duplicating cert-manager wiring and diverging from the
|
|
# CoreDNS convention that sends every *.riotpiao.com host to nginx.
|
|
#
|
|
# ── Routing model ───────────────────────────────────────────────────────────
|
|
# Consumers publish an Ingress with `ingressClassName: kong`; the controller
|
|
# turns it into a Kong route. `nginx` remains the default IngressClass, so this
|
|
# is strictly opt-in and no existing Ingress changes behaviour.
|
|
|
|
# Without this the release name is prefixed onto everything (`kong-kong-proxy`).
|
|
# Pinning it keeps the Service name stable and independent of the release name,
|
|
# which matters because the nginx Ingress in k8s/bootstrap/ingress/ingress.yaml
|
|
# references it by name.
|
|
fullnameOverride: kong
|
|
|
|
# Two replicas so a node drain or rollout doesn't take the gateway down. Kong is
|
|
# stateless in DB-less mode, so replicas are pure redundancy.
|
|
replicaCount: 2
|
|
|
|
env:
|
|
# DB-less. Config comes from Kubernetes objects via the ingress controller, so
|
|
# git stays the source of truth. A Postgres-backed Kong would put live routing
|
|
# config in a database mutated through the Admin API — state outside git, plus
|
|
# migration Jobs on every upgrade.
|
|
database: "off"
|
|
# `nginx_proxy_<directive>` injects a directive into the proxy location block;
|
|
# this renders `proxy_buffering off;`.
|
|
#
|
|
# Required for LLM streaming. With buffering on (the default) nginx accumulates
|
|
# the upstream response before forwarding, so an SSE stream from
|
|
# `"stream": true` arrives in lumps or stalls until the generation finishes —
|
|
# which defeats the point of streaming. The matching setting is already on the
|
|
# nginx Ingress in ingress.yaml; both hops have to be unbuffered or the
|
|
# buffered one dominates.
|
|
nginx_proxy_proxy_buffering: "off"
|
|
|
|
ingressController:
|
|
enabled: true
|
|
ingressClass: kong
|
|
# The chart's ingress-class template is gated on
|
|
# `.Capabilities.APIVersions.Has "networking.k8s.io/v1/IngressClass"`, so a
|
|
# bare `helm template` renders nothing. ArgoCD passes --api-versions from the
|
|
# live cluster, so it does render there — verify `kubectl get ingressclass
|
|
# kong` after the first sync rather than assuming it.
|
|
createIngressClass: true
|
|
# Deliberately empty: setting is-default-class here would hijack every Ingress
|
|
# in the cluster that omits ingressClassName. nginx keeps that role.
|
|
ingressClassAnnotations: {}
|
|
|
|
proxy:
|
|
enabled: true
|
|
# Chart default is LoadBalancer, which would claim an IP from homelab-pool.
|
|
type: ClusterIP
|
|
http:
|
|
enabled: true
|
|
servicePort: 80
|
|
containerPort: 8000
|
|
# nginx already terminated TLS; a second handshake to the same cluster buys
|
|
# nothing and would need Kong to hold its own certificate.
|
|
tls:
|
|
enabled: false
|
|
|
|
# No Service for the Admin API. The controller reaches it over localhost inside
|
|
# the pod, so exposing it would only create an unauthenticated write path to the
|
|
# gateway's entire configuration.
|
|
admin:
|
|
enabled: false
|
|
|
|
# Kong Manager UI — chart default is `enabled: true` with type NodePort, which
|
|
# would open a port on every node. Not wanted.
|
|
manager:
|
|
enabled: false
|
|
|
|
resources:
|
|
requests:
|
|
cpu: 200m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "2"
|
|
memory: 1Gi
|
|
|
|
podDisruptionBudget:
|
|
enabled: true
|
|
minAvailable: 1
|
|
|
|
# Spread the two replicas across nodes; `ScheduleAnyway` so a single-node
|
|
# situation degrades to co-location instead of leaving a pod Pending.
|
|
topologySpreadConstraints:
|
|
- maxSkew: 1
|
|
topologyKey: kubernetes.io/hostname
|
|
whenUnsatisfiable: ScheduleAnyway
|
|
labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: kong
|
|
app.kubernetes.io/instance: kong
|