49 lines
2.5 KiB
Markdown
49 lines
2.5 KiB
Markdown
|
# 2.9 — Dialect-neutral canonical request (GREEN)
|
||
|
|
|
||
|
|
Phase: 2 — LLM surfaces
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [2.1](2.1-model-registry.md), [2.2](2.2-body-based-dispatch.md)
|
||
|
|
|
||
|
|
The gateway serves two permanent protocol dialects: `/v1/*` is OpenAI-compatible and
|
||
|
|
`/llm/*` is Anthropic Messages. Both translate into one internal shape before anything
|
||
|
|
else happens to them, and both translate back out of it on the way to the client.
|
||
|
|
|
||
|
|
- [ ] A single internal request representation exists that carries at minimum: the
|
||
|
|
resolved upstream, the model name as the client sent it, the ordered turns, an
|
||
|
|
optional system instruction, a maximum output token count, stop sequences, and a
|
||
|
|
streaming flag
|
||
|
|
- [ ] The representation names no dialect: nothing in it is called openai or anthropic,
|
||
|
|
and no field exists solely because one dialect happens to spell it that way
|
||
|
|
- [ ] Model dispatch, the shared slot controller, per-caller budgets, logging and
|
||
|
|
metrics all read the canonical request and never the raw dialect body
|
||
|
|
- [ ] The surface that received a request is recorded as one field on the canonical
|
||
|
|
request, used only to choose the response and error rendering, never to choose an
|
||
|
|
upstream or a slot
|
||
|
|
- [ ] An identical prompt sent to `/v1/chat/completions` and to `/llm/v1/messages`
|
||
|
|
produces the same resolved upstream, the same slot accounting and the same log
|
||
|
|
fields apart from that one surface label
|
||
|
|
- [ ] Adding a third dialect requires a new translator only; the slot controller,
|
||
|
|
dispatch and budget code are untouched
|
||
|
|
- [ ] Translation failure is a client error at the surface boundary, and no partially
|
||
|
|
populated canonical request ever reaches an upstream
|
||
|
|
|
||
|
|
`reasoning` maps to `reasoning-predictor.llm-serving:80`; `ornith:35b` and
|
||
|
|
`qwen2.5:3b-instruct` both map to `ornith-predictor.llm-serving:80`. Ports are 80.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/v1/chat/completions \
|
||
|
|
-H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","messages":[{"role":"user","content":"hi"}]}'
|
||
|
|
# expected: 200, reasoning stub hit once
|
||
|
|
|
||
|
|
curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/llm/v1/messages \
|
||
|
|
-H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'
|
||
|
|
# expected: 200, reasoning stub hit once, same upstream and same slot counter as the /v1 call
|
||
|
|
|
||
|
|
grep -h 'upstream=' /tmp/gateway.log | tail -2
|
||
|
|
# expected: both lines show upstream=reasoning-predictor and model=reasoning, differing only in the surface label
|
||
|
|
```
|