45 lines
2.5 KiB
Markdown
45 lines
2.5 KiB
Markdown
# 4.1 — GPU slot semaphore for `reasoning` (GREEN)
|
|
|
|
Phase: 4 — Limits and budgets
|
|
Stage: GREEN
|
|
Depends on: [4.3](4.3-problem-json-errors.md)
|
|
|
|
The `reasoning` upstream is 2 replicas x `--max-num-seqs=4` = 8 concurrent sequence
|
|
slots cluster-wide. That 8 is a vLLM concurrency setting, not a GPU count — worker-1
|
|
has 4 physical GPUs and the two numbers are unrelated. Exceeding 8 does not fail
|
|
fast; it silently queues inside vLLM where the gateway has no visibility and no
|
|
ability to cancel.
|
|
|
|
The gateway serves `/v1/*` (OpenAI-compatible) over the same predictors. The
|
|
Anthropic `/llm/*` dialect was dropped (see `tasks/INDEX.md` Phase 2) — this
|
|
controller only ever needs to know about `/v1/*`.
|
|
|
|
- [ ] Concurrent in-flight requests to `reasoning` are capped at a configured limit strictly below 8, leaving operator headroom
|
|
- [ ] The cap, the queue depth, and the queue wait timeout are all explicit in configuration with no silent defaults
|
|
- [ ] Requests over the cap wait in a bounded queue rather than being rejected immediately
|
|
- [ ] Once the queue is full, further requests are rejected with a retryable status and a `Retry-After`, as an `application/problem+json` document
|
|
- [ ] A slot is released when the response completes, when the upstream errors, and when the client disconnects mid-stream — no path leaks a slot
|
|
- [ ] A client that disconnects while still queued never reaches the upstream and never consumes a slot
|
|
- [ ] The cap applies only to `reasoning`; other upstreams are unaffected and are not blocked behind its queue
|
|
- [ ] The controller is keyed by UPSTREAM, never by route or path prefix — `reasoning-predictor.llm-serving:80` has exactly one cap and one queue
|
|
- [ ] Slot occupancy and queue depth are reported per upstream
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Fire 40 concurrent chat requests at a stub reasoning upstream that holds each for 2s,
|
|
# with cap=6 and queue=8 configured.
|
|
seq 40 | xargs -P40 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
|
|
-X POST localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | sort | uniq -c
|
|
# expected: a mix of 200 and 503; zero 500s
|
|
|
|
# expected: the stub logged at most 6 simultaneous in-flight requests, never 7
|
|
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
|
|
# expected: 0
|
|
|
|
curl -s -D - -o /dev/null -X POST localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | grep -i retry-after
|
|
# expected: Retry-After present on the rejected request
|
|
```
|