homelab-frontend/tasks/4.1-gpu-slot-semaphore.md

4.1 — GPU slot semaphore for reasoning (GREEN)

Phase: 4 — Limits and budgets Stage: GREEN Depends on: 4.3, 2.9

The reasoning upstream is 2 replicas x --max-num-seqs=4 = 8 concurrent sequence slots cluster-wide. That 8 is a vLLM concurrency setting, not a GPU count — worker-1 has 4 physical GPUs and the two numbers are unrelated. Exceeding 8 does not fail fast; it silently queues inside vLLM where the gateway has no visibility and no ability to cancel.

The gateway serves two permanent dialects over the same predictors: /v1/* is OpenAI-compatible and /llm/* is Anthropic Messages. There is ONE controller, keyed by upstream, sitting below both.

• Concurrent in-flight requests to reasoning are capped at a configured limit strictly below 8, leaving operator headroom
• The cap, the queue depth, and the queue wait timeout are all explicit in configuration with no silent defaults
• Requests over the cap wait in a bounded queue rather than being rejected immediately
• Once the queue is full, further requests are rejected with a retryable status and a Retry-After, as an application/problem+json document
• A slot is released when the response completes, when the upstream errors, and when the client disconnects mid-stream — no path leaks a slot
• A client that disconnects while still queued never reaches the upstream and never consumes a slot
• The cap applies only to reasoning; other upstreams are unaffected and are not blocked behind its queue
• The controller is keyed by UPSTREAM, never by route, path prefix or dialect — reasoning-predictor.llm-serving:80 has exactly one cap and one queue
• A request arriving on /v1/chat/completions and one arriving on /llm/v1/messages contend for the same slots and the same queue, admitted in arrival order
• Total in-flight requests against reasoning never exceed the configured cap regardless of which surface they arrived through, including when both surfaces are saturated at once
• Per-dialect semaphores are explicitly wrong and must not exist: the 8 sequence slots are physical, so two independent gates would each believe they were within budget while together exceeding it
• The controller reads the dialect-neutral canonical request and has no knowledge of which surface produced it; adding a third dialect requires no change here
• Slot occupancy and queue depth are reported per upstream, and requests are attributable to a surface for logging only, never for admission
• Rejection at a full queue renders per surface — application/problem+json on /v1, the Anthropic error shape on /llm — from one shared decision

Verify

# Fire 40 concurrent chat requests at a stub reasoning upstream that holds each for 2s,
# with cap=6 and queue=8 configured.
seq 40 | xargs -P40 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | sort | uniq -c
# expected: a mix of 200 and 503; zero 500s

# expected: the stub logged at most 6 simultaneous in-flight requests, never 7
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
# expected: 0

curl -s -D - -o /dev/null -X POST localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | grep -i retry-after
# expected: Retry-After present on the rejected request

# Same cap=6 stub, but split the load across both dialects: 20 on /v1 and 20 on /llm.
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' &
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
  -H 'content-type: application/json' \
  -d '{"model":"reasoning","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' &
wait
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
# expected: 0 — the two surfaces share one cap, they do not get 6 each