Story Crater BotandClaude Opus 5
058f11cf2b
Baseline for the Kong replacement on api.riotpiao.com. Brings the working tree under version control for the first time: gateway source, the task board that drives the agent runs, test fixtures, and K8s manifests. Anchor the gateway ignore rule to the repo root. Unanchored, "gateway" also matched the cmd/gateway/ source directory, so the program entrypoint was excluded from every commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
144 lines
6.9 KiB
Markdown
144 lines
6.9 KiB
Markdown
# Kong retirement — inventory and cutover
|
||
|
||
Everything Kong does on `api.riotpiao.com` today, and where it goes. Inventory
|
||
verified live against context `admin@homelab-cluster` on 2026-08-19.
|
||
|
||
Source of the objects being retired: `~/workplace/homelab/k8s/apps/api/` and
|
||
`k8s/argocd/apps/55-api-gateway.yaml`.
|
||
|
||
## What is running now
|
||
|
||
Kong OSS 3.4.1, Helm chart from `https://charts.konghq.com`, DB-less, namespace
|
||
`api`, Argo Application `kong` at sync wave 7. Two replicas. Fronted by
|
||
`ingress-nginx` via Ingress `api/api`, which catch-alls `/` on `api.riotpiao.com`
|
||
to `kong-proxy:80`.
|
||
|
||
Eleven ReplicaSets exist on the Kong Deployment, the newest minutes old — this
|
||
config is being actively iterated, so re-verify the inventory immediately before
|
||
cutover.
|
||
|
||
## Routing table to port
|
||
|
||
Seven `ingressClassName: kong` Ingresses. Six in `llm-serving`, one in `agent-pod`.
|
||
|
||
| Method | Path | Upstream | Transform applied by Kong |
|
||
|---|---|---|---|
|
||
| GET | `/v1/models` | — | `request-termination`: static 200 JSON, upstream never contacted |
|
||
| POST | `/v1/reasoning/chat/completions` | `reasoning-predictor:80` | force body `model=reasoning`, rewrite URI to `/v1/chat/completions` |
|
||
| POST | `/v1/ornith/chat/completions` | `ornith-predictor:80` | force body `model=ornith:35b`, rewrite URI |
|
||
| POST | `/v1/qwen/chat/completions` | `ornith-predictor:80` | force body `model=qwen2.5:3b-instruct`, rewrite URI |
|
||
| POST | `/v1/embeddings` | `embeddings-predictor:80` | none — TEI already serves the canonical path |
|
||
| POST | `/v1/rerank` | `reranker-predictor:80` | rewrite URI to `/rerank` (TEI does not serve `/v1/rerank`) |
|
||
| GET/WS | `/console`, `/run`, `/sessions` | `agent-hub:9090` (`agent-pod` ns) | none, `strip-path: false` |
|
||
|
||
Upstream model map, from the manifest comments and confirmed live:
|
||
|
||
- `reasoning` → `reasoning-predictor` — vLLM, DeepSeek-R1-Distill-Qwen-32B, 2 replicas,
|
||
`--max-num-seqs=4`, `--max-model-len=16384`, `--reasoning-parser=deepseek_r1`,
|
||
`--enable-auto-tool-choice --tool-call-parser=hermes`
|
||
- `ornith:35b` → `ornith-predictor` — Ollama, 2 replicas
|
||
- `qwen2.5:3b-instruct` → `ornith-predictor` — same pods; both models stay resident via
|
||
`OLLAMA_MAX_LOADED_MODELS=2`, `OLLAMA_KEEP_ALIVE=-1`
|
||
- `nomic-ai/nomic-embed-text-v2-moe` → `embeddings-predictor` — TEI
|
||
- `BAAI/bge-reranker-base` → `reranker-predictor` — TEI
|
||
|
||
### The path-per-model surface goes away
|
||
|
||
The three chat paths exist only because Kong OSS cannot dispatch on the request
|
||
body. The gateway serves a single `POST /v1/chat/completions` and selects the
|
||
upstream from the body's `model` field.
|
||
|
||
Keep the old paths as aliases during cutover so live clients do not break, then
|
||
remove them once callers have migrated. pi is a live caller today.
|
||
|
||
### `/v1/models` should not be ported verbatim
|
||
|
||
Kong serves a hardcoded list via `request-termination`. The manifest already flags
|
||
that it can drift from what the engines actually serve. Derive the response from
|
||
the gateway's configured upstream map instead, so the list cannot disagree with
|
||
what routing will accept.
|
||
|
||
## Plugins being retired
|
||
|
||
| Plugin | Scope | Replacement |
|
||
|---|---|---|
|
||
| `llm-rewrite-reasoning` / `-ornith` / `-qwen` | llm-serving | body-based dispatch in `internal/llm` |
|
||
| `llm-rewrite-rerank` | llm-serving | per-upstream path rewrite in the route table |
|
||
| `llm-models-list` | llm-serving | derived from the upstream map |
|
||
| `prometheus` | **cluster-wide** | `internal/observability` — must expose bandwidth, latency, status codes, upstream health or observability regresses |
|
||
|
||
No `rate-limiting` plugin exists anywhere in the cluster. REQUIREMENTS.md §4 Tier 2
|
||
describes it as an existing layer; it is not built. Nothing to migrate — it is net
|
||
new work, and it now belongs in the gateway rather than in Kong.
|
||
|
||
## Auth: currently off, must land on
|
||
|
||
`KongConsumer model-invoker` exists in namespace `api` and stays defined, but the
|
||
`key-auth` plugin is commented out and every route has `model-key-auth` stripped
|
||
from its `konghq.com/plugins` annotation.
|
||
|
||
**The model API is unauthenticated right now.** Confirmed live 2026-08-19: a request
|
||
to `/v1/reasoning/chat/completions` with no credentials returns 200.
|
||
|
||
The reason is recorded in `model-auth.yaml` — Kong's `key-auth` accepts a raw
|
||
`apikey:` header but rejects `Authorization: Bearer`, which blocks every
|
||
OpenAI-compatible client. That is why `~/.pi/agent/models.json` carries a
|
||
`customHeaders: {apikey: ...}` block.
|
||
|
||
The gateway reads Bearer tokens directly and validates them against Authentik via
|
||
JWKS. `AUTH-PLAN.md`'s pinned-RSA-key approach and its rotation runbook are not
|
||
needed and should not be carried over.
|
||
|
||
Ship auth behind a flag. Turning it on breaks every current caller until they hold
|
||
a token — pi included.
|
||
|
||
## Timeouts
|
||
|
||
Kong today:
|
||
|
||
| Route class | connect | read | write |
|
||
|---|---|---|---|
|
||
| chat | 10s | **1h** | 1h |
|
||
| embeddings / rerank | 10s | 10m | 10m |
|
||
|
||
nginx in front sets `proxy-read-timeout: 3600`, `proxy-send-timeout: 3600`,
|
||
`proxy-buffering: off`, `proxy-body-size: 0`. Those stay — they are what makes token
|
||
streaming work, and the gateway needs the same treatment from nginx.
|
||
|
||
The 1-hour read timeout is deliberate: a 32B model on a Volta GPU routinely exceeds
|
||
60s. Any shorter server-side cap must be enforced *in the gateway*, not by shortening
|
||
the proxy timeout, or long legitimate generations get truncated mid-stream.
|
||
|
||
## Cutover
|
||
|
||
Reversible at every step. Kong keeps serving until the last step.
|
||
|
||
1. Deploy the gateway alongside Kong, unexposed. Verify in-cluster against
|
||
`http://homelab-frontend.api.svc.cluster.local`.
|
||
2. Compare gateway and Kong responses for every route in the table above, including
|
||
a streaming chat request and a client disconnect mid-stream.
|
||
3. Repoint Ingress `api/api` from `kong-proxy:80` to the gateway Service. **This is
|
||
the cutover.** Reverting is a one-line change to the same Ingress.
|
||
4. Soak. Watch gateway metrics and pi traffic.
|
||
5. Delete the seven kong-class Ingresses and the six KongPlugin CRs.
|
||
6. Remove the `kong` Application from `k8s/argocd/apps/55-api-gateway.yaml`; let Argo
|
||
prune the Helm release, the CRDs and namespace leftovers.
|
||
|
||
Steps 1–4 are reversible in seconds. Step 5 onward is not — do not start it until the
|
||
soak is clean.
|
||
|
||
All of this flows through git and Argo. No `kubectl apply`, no `helm upgrade`.
|
||
|
||
## Loose ends
|
||
|
||
- `agent-pod/console` is publicly routed, unauthenticated, accepts free-form prompts
|
||
into a shell-capable container, and exposes a WebSocket. Migrating it behind the
|
||
gateway's auth is a security fix, not merely a port. Treat WebSocket upgrade as an
|
||
explicit requirement of the proxy layer.
|
||
- Eight `*.example.com` hosts exist on istio-class Ingresses in `llm-serving`
|
||
(`{embeddings,ornith,reasoning,reranker}[-predictor]-llm-serving.example.com`).
|
||
KServe defaults, not public, not Kong's — out of scope here, but they exist and
|
||
should not be mistaken for gateway routes.
|
||
- Ingress class split across the cluster is 7 kong / 17 nginx / 4 istio. Only the 7
|
||
kong ones are in scope.
|