homelab-frontend/tasks/2.14-queue-position-event.md

2.14 — Queue position event on /llm/* (GREEN)

Phase: 2 — LLM surfaces Stage: GREEN Depends on: 2.12, 4.1

The Anthropic event set has no way to say "you are queued". Its stream implicitly begins after a slot has been acquired, so a queued client sees nothing at all until generation starts. The reasoning upstream has only 8 sequence slots cluster-wide and the gateway caps below that, so waiting is normal and worth showing.

• When a streaming /llm/v1/messages request waits for a slot, the gateway emits a frame with event: queue before any message_start
• The queue frame's data carries the caller's current position in the queue
• Position updates are emitted as the queue drains, each as another event: queue frame, until a slot is acquired
• Once a slot is acquired the stream continues with the standard sequence beginning at message_start, and no further queue frame is emitted for that request
• A request that acquires a slot immediately emits no queue frame at all
• Queue frames are flushed as they are produced, not buffered behind the first upstream token
• A client that disconnects while still queued is removed from the queue, never reaches the upstream and never consumes a slot
• Non-streaming requests emit no queue frames; they simply wait, then answer
• The extension is documented in the surface's own docs as non-standard, alongside the fact that a strict Anthropic client ignoring unknown events degrades to showing nothing while queued rather than erroring

This is a deliberate departure from the Anthropic contract. It is safe only because the sole client of /llm/* is the first-party riotpiao frontend. It must never be required for correctness: dropping every event: queue frame leaves a valid, complete Anthropic stream.

Verify

# Fill the slots first, with cap=2 configured against a stub that holds each request 3s.
seq 4 | xargs -P4 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
  -H 'content-type: application/json' \
  -d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' &

sleep 1
curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
  -d '{"model":"reasoning","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
  | grep -m3 '^event:'
# expected: event: queue arrives within a second, before any event: message_start

curl -N -s localhost:8080/llm/v1/messages -H 'content-type: application/json' \
  -d '{"model":"qwen2.5:3b-instruct","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}' \
  | head -1
# expected: event: message_start, no queue frame on an uncontended upstream