40 lines
2.4 KiB
Markdown
40 lines
2.4 KiB
Markdown
|
# 2.2 — Body-based model dispatch (GREEN)
|
||
|
|
|
||
|
|
Phase: 2 — LLM surface
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [2.1](2.1-model-registry.md), [1.1](1.1-reverse-proxy.md), [1.2](1.2-streaming-passthrough.md)
|
||
|
|
|
||
|
|
- [ ] `POST /v1/chat/completions` selects its upstream from the `model` field of the JSON request body
|
||
|
|
- [ ] The body reaching the upstream is byte-identical to the body received, `model` included — dispatch reads, it does not rewrite
|
||
|
|
- [ ] The upstream sees the canonical path `/v1/chat/completions`
|
||
|
|
- [ ] `"model":"reasoning"` reaches `reasoning-predictor.llm-serving:80`; `"model":"ornith:35b"` and `"model":"qwen2.5:3b-instruct"` both reach `ornith-predictor.llm-serving:80`
|
||
|
|
- [ ] `"stream":true` streams unbuffered — chunks reach the client as the upstream emits them, and are not accumulated in order to inspect the body
|
||
|
|
- [ ] A client disconnect mid-stream cancels the upstream request rather than orphaning it
|
||
|
|
- [ ] `reasoning_content` is passed through untouched alongside `content`; the gateway does not merge, reorder or strip either
|
||
|
|
- [ ] Reading the body to find `model` respects the route's body size cap and does not load an unbounded request into memory
|
||
|
|
|
||
|
|
This is the single capability Kong OSS lacked — `ai-proxy-advanced` is Enterprise-only —
|
||
|
|
and the entire reason this gateway exists. Everything else in this phase is a
|
||
|
|
consequence of it. Ports are 80, not 8080.
|
||
|
|
|
||
|
|
An orphaned generation holds one of only eight vLLM sequence slots in the cluster,
|
||
|
|
which is why disconnect cancellation belongs in the acceptance criteria of this task
|
||
|
|
and not only in the proxy layer.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||
|
|
-H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
|
||
|
|
# expected: 200, stub for reasoning-predictor recorded the hit with path /v1/chat/completions and an unmodified body
|
||
|
|
|
||
|
|
curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||
|
|
-d '{"model":"qwen2.5:3b-instruct","messages":[{"role":"user","content":"hi"}]}'
|
||
|
|
# expected: 200, ornith-predictor stub hit, reasoning-predictor stub not hit
|
||
|
|
|
||
|
|
curl -N -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","stream":true,"messages":[{"role":"user","content":"hi"}]}'
|
||
|
|
# expected: 200, content-type text/event-stream, first data: chunk arrives before the stub finishes emitting
|
||
|
|
```
|