53 lines
2.8 KiB
Markdown
53 lines
2.8 KiB
Markdown
|
# 2.14 — Queue position event on `/llm/*` (GREEN)
|
||
|
|
|
||
|
|
Phase: 2 — LLM surfaces
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [2.12](2.12-anthropic-sse-state-machine.md), [4.1](4.1-gpu-slot-semaphore.md)
|
||
|
|
|
||
|
|
The Anthropic event set has no way to say "you are queued". Its stream implicitly
|
||
|
|
begins after a slot has been acquired, so a queued client sees nothing at all until
|
||
|
|
generation starts. The `reasoning` upstream has only 8 sequence slots cluster-wide and
|
||
|
|
the gateway caps below that, so waiting is normal and worth showing.
|
||
|
|
|
||
|
|
- [ ] When a streaming `/llm/v1/messages` request waits for a slot, the gateway emits a
|
||
|
|
frame with `event: queue` before any `message_start`
|
||
|
|
- [ ] The queue frame's data carries the caller's current position in the queue
|
||
|
|
- [ ] Position updates are emitted as the queue drains, each as another `event: queue`
|
||
|
|
frame, until a slot is acquired
|
||
|
|
- [ ] Once a slot is acquired the stream continues with the standard sequence beginning
|
||
|
|
at `message_start`, and no further queue frame is emitted for that request
|
||
|
|
- [ ] A request that acquires a slot immediately emits no queue frame at all
|
||
|
|
- [ ] Queue frames are flushed as they are produced, not buffered behind the first
|
||
|
|
upstream token
|
||
|
|
- [ ] A client that disconnects while still queued is removed from the queue, never
|
||
|
|
reaches the upstream and never consumes a slot
|
||
|
|
- [ ] Non-streaming requests emit no queue frames; they simply wait, then answer
|
||
|
|
- [ ] The extension is documented in the surface's own docs as non-standard, alongside
|
||
|
|
the fact that a strict Anthropic client ignoring unknown events degrades to
|
||
|
|
showing nothing while queued rather than erroring
|
||
|
|
|
||
|
|
This is a deliberate departure from the Anthropic contract. It is safe only because
|
||
|
|
the sole client of `/llm/*` is the first-party riotpiao frontend. It must never be
|
||
|
|
required for correctness: dropping every `event: queue` frame leaves a valid,
|
||
|
|
complete Anthropic stream.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Fill the slots first, with cap=2 configured against a stub that holds each request 3s.
|
||
|
|
seq 4 | xargs -P4 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
|
||
|
|
-H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' &
|
||
|
|
|
||
|
|
sleep 1
|
||
|
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||
|
|
-d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||
|
|
| grep -m3 '^event:'
|
||
|
|
# expected: event: queue arrives within a second, before any event: message_start
|
||
|
|
|
||
|
|
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
|
||
|
|
-d '{"model":"qwen2.5:3b-instruct","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
|
||
|
|
| head -1
|
||
|
|
# expected: event: message_start, no queue frame on an uncontended upstream
|
||
|
|
```
|