Story Crater BotandClaude Opus 5
058f11cf2b
Baseline for the Kong replacement on api.riotpiao.com. Brings the working tree under version control for the first time: gateway source, the task board that drives the agent runs, test fixtures, and K8s manifests. Anchor the gateway ignore rule to the repo root. Unanchored, "gateway" also matched the cmd/gateway/ source directory, so the program entrypoint was excluded from every commit. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2.4 KiB
2.4 KiB
2.2 — Body-based model dispatch (GREEN)
Phase: 2 — LLM surface Stage: GREEN Depends on: 2.1, 1.1, 1.2
POST /v1/chat/completionsselects its upstream from themodelfield of the JSON request body- The body reaching the upstream is byte-identical to the body received,
modelincluded — dispatch reads, it does not rewrite - The upstream sees the canonical path
/v1/chat/completions "model":"reasoning"reachesreasoning-predictor.llm-serving:80;"model":"ornith:35b"and"model":"qwen2.5:3b-instruct"both reachornith-predictor.llm-serving:80"stream":truestreams unbuffered — chunks reach the client as the upstream emits them, and are not accumulated in order to inspect the body- A client disconnect mid-stream cancels the upstream request rather than orphaning it
reasoning_contentis passed through untouched alongsidecontent; the gateway does not merge, reorder or strip either- Reading the body to find
modelrespects the route's body size cap and does not load an unbounded request into memory
This is the single capability Kong OSS lacked — ai-proxy-advanced is Enterprise-only —
and the entire reason this gateway exists. Everything else in this phase is a
consequence of it. Ports are 80, not 8080.
An orphaned generation holds one of only eight vLLM sequence slots in the cluster, which is why disconnect cancellation belongs in the acceptance criteria of this task and not only in the proxy layer.
Verify
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
# expected: 200, stub for reasoning-predictor recorded the hit with path /v1/chat/completions and an unmodified body
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"hi"}]}'
# expected: 200, ornith-predictor stub hit, reasoning-predictor stub not hit
curl -N -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"hi"}]}'
# expected: 200, content-type text/event-stream, first data: chunk arrives before the stub finishes emitting