# ADR-0001 — Retire Kong OSS in favour of a Go API gateway Status: Accepted Date: 2026-08-19 Deciders: rock ## Context `api.riotpiao.com` is currently served by Kong OSS 3.4.1 (Helm, DB-less, namespace `api`, Argo wave 7), sitting behind ingress-nginx which owns TLS. Kong routes to the KServe model predictors in `llm-serving` via seven `ingressClassName: kong` Ingresses and six `KongPlugin` CRs. Three separate capabilities were attempted on Kong OSS. All three failed, and each failure is already documented in-repo by the person who hit it: **1. Body-based model dispatch is not expressible.** From `k8s/apps/api/llm-routes.yaml`: > a single `/v1/chat/completions` endpoint that dispatches on the body's `model` field is > not expressible in Kong OSS (`ai-proxy-advanced`, which does multi-target model routing, > is Enterprise-only). The workaround is a path-per-model surface (`/v1/reasoning/chat/completions`, `/v1/ornith/...`, `/v1/qwen/...`) with a `request-transformer` force-overwriting the body's `model` field. This is not OpenAI-standard, so every client needs bespoke configuration — visible today in `~/.pi/agent/models.json`, which carries three separate provider entries for what should be one endpoint. **2. OIDC is Enterprise-only.** `k8s/apps/api/AUTH-PLAN.md` routes around the missing `openid-connect` plugin using the built-in `jwt` plugin, which requires pinning Authentik's RSA public key onto a KongConsumer. That plan lists its own consequence: > Pinning `rsa_public_key`: Authentik key rotation would break it — document a rotation > runbook, or have the provision script re-export the cert PEM into the Kong credential on > each run. A rotation runbook is a standing operational liability accepted only because the gateway cannot fetch JWKS itself. **3. `key-auth` cannot read `Authorization: Bearer`.** From `k8s/apps/api/model-auth.yaml`: > a raw `apikey: ` header succeeds (200), the same request with only > `Authorization: Bearer ` fails (401). No OpenAI-SDK-compatible client (pi included) > sends a raw apikey header or lets you customize the header name, so every such client was > hard-blocked. Consequence: authentication on the model routes is **currently disabled**. Verified live 2026-08-19 — `api.riotpiao.com/v1/reasoning/chat/completions` answers unauthenticated. Separately, the intended surface has grown beyond LLM routing. The target is a capability-per-subdomain API over cluster services — `sqs.riotpiao.com` for queue operations, `workflow.riotpiao.com` for Temporal, `cluster.riotpiao.com` for atlas — each needing request shaping, per-caller budgets and streaming semantics that are application concerns, not gateway-plugin concerns. ## Decision Retire Kong OSS entirely. Replace it with a purpose-built Go service, `homelab-frontend`, which owns north-south routing, authentication, and request shaping for every public capability on `*.riotpiao.com`. ingress-nginx keeps the edge and TLS. It forwards to the gateway instead of `kong-proxy`. Authentication is Authentik OIDC, validated by fetching JWKS from `https://authentik.riotpiao.com` at runtime. ## Options considered **A. Stay on Kong OSS, accept the workarounds.** Keeps a battle-tested proxy and its Prometheus plugin. But the path-per-model surface stays non-standard, the RSA pinning runbook stays, and auth stays off until someone writes a `request-transformer` shim to copy Bearer into an `apikey` header. Every new capability (`sqs`, `workflow`) inherits the same constraints. **B. Buy Kong Enterprise.** `ai-proxy-advanced` and `openid-connect` solve 1 and 2. Does not solve the genuinely application-level requirements at all — signed session cookies, per-session daily message budgets, a 6-of-8 GPU sequence-slot semaphore with a bounded queue, and disconnect-cancels-upstream are not gateway features in any tier. Cost for a homelab is not justifiable. **C. Go gateway, Kong retained for LLM paths only.** Gradual migration, lower risk. But it means running two gateways indefinitely, splitting the routing table across Kong CRDs and Go code, and keeping the Kong Helm release and its CRDs. The split is the thing most likely to drift. **D. Go gateway, Kong retired entirely.** — chosen One routing table, one auth implementation, one place to reason about timeouts. The logic being replaced is small: four `request-transformer` plugins that set a body field and rewrite a URI, one `request-termination` serving a static JSON model list, and one `prometheus` plugin. That is on the order of a hundred lines of Go, against roughly 480 lines of YAML it retires. ## Consequences ### Gained - **Standard OpenAI surface.** One `POST /v1/chat/completions`, model selected from the request body. Any OpenAI SDK works unmodified. The three pi provider entries collapse to one. - **Working authentication.** Bearer tokens are read from the header, because it is our code. JWKS is fetched and cached with automatic rotation handling, so the AUTH-PLAN.md rotation runbook is deleted rather than written. - **Application-level policy becomes possible.** GPU slot semaphore, per-session budgets, disconnect propagation and SSE handling live where the state is. - **One timeout story.** Kong currently sets `read-timeout: 3600000` (1 hour) on chat routes, which silently defeats any shorter server-side cap. Retiring Kong removes the conflicting layer. - **~480 lines of gateway YAML deleted**, plus the Kong CRDs, the Helm release, and its `ServerSideApply` workaround for oversized CRD annotations. ### Lost / assumed - **We now own proxy correctness.** Connection pooling, retries, timeout propagation, streaming passthrough, header hygiene, graceful shutdown. `net/http/httputil.ReverseProxy` covers most of it, but it is our bug surface now. - **Kong's Prometheus plugin goes away.** The gateway must expose equivalent metrics itself (bandwidth, latency, status codes, upstream health) or observability regresses. - **Migration touches live traffic.** pi depends on `api.riotpiao.com` today. Cutover must be reversible — see `docs/MIGRATION-kong.md`. - **`agent-pod/console` is a kong-class Ingress** exposing `/console` (WebSocket), `/run` and `/sessions`. It must migrate too, and it is currently unauthenticated and publicly routed while accepting free-form prompts into a shell-capable container. Putting it behind the gateway's Authentik auth is a security improvement, not just a port. ### Risks - Enabling Authentik auth will break any client currently relying on the unauthenticated surface — including pi, until its `models.json` is updated. Auth must ship behind a flag and be enabled deliberately. - Kong's `request-termination` for `/v1/models` returns a **static** list that can drift from what the engines actually serve. Porting it verbatim ports the bug; the gateway should derive the list from configured upstreams instead.