feat: phase 8 serviceadapter crd rollout (32/33 tasks)
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
Phase: 4 — Limits and budgets
|
||||
Stage: GREEN
|
||||
Depends on: [4.3](4.3-problem-json-errors.md), [2.9](2.9-canonical-request-model.md)
|
||||
Depends on: [4.3](4.3-problem-json-errors.md)
|
||||
|
||||
The `reasoning` upstream is 2 replicas x `--max-num-seqs=4` = 8 concurrent sequence
|
||||
slots cluster-wide. That 8 is a vLLM concurrency setting, not a GPU count — worker-1
|
||||
@@ -10,9 +10,9 @@ has 4 physical GPUs and the two numbers are unrelated. Exceeding 8 does not fail
|
||||
fast; it silently queues inside vLLM where the gateway has no visibility and no
|
||||
ability to cancel.
|
||||
|
||||
The gateway serves two permanent dialects over the same predictors: `/v1/*` is
|
||||
OpenAI-compatible and `/llm/*` is Anthropic Messages. There is ONE controller, keyed by
|
||||
upstream, sitting below both.
|
||||
The gateway serves `/v1/*` (OpenAI-compatible) over the same predictors. The
|
||||
Anthropic `/llm/*` dialect was dropped (see `tasks/INDEX.md` Phase 2) — this
|
||||
controller only ever needs to know about `/v1/*`.
|
||||
|
||||
- [ ] Concurrent in-flight requests to `reasoning` are capped at a configured limit strictly below 8, leaving operator headroom
|
||||
- [ ] The cap, the queue depth, and the queue wait timeout are all explicit in configuration with no silent defaults
|
||||
@@ -21,13 +21,8 @@ upstream, sitting below both.
|
||||
- [ ] A slot is released when the response completes, when the upstream errors, and when the client disconnects mid-stream — no path leaks a slot
|
||||
- [ ] A client that disconnects while still queued never reaches the upstream and never consumes a slot
|
||||
- [ ] The cap applies only to `reasoning`; other upstreams are unaffected and are not blocked behind its queue
|
||||
- [ ] The controller is keyed by UPSTREAM, never by route, path prefix or dialect — `reasoning-predictor.llm-serving:80` has exactly one cap and one queue
|
||||
- [ ] A request arriving on `/v1/chat/completions` and one arriving on `/llm/v1/messages` contend for the same slots and the same queue, admitted in arrival order
|
||||
- [ ] Total in-flight requests against `reasoning` never exceed the configured cap regardless of which surface they arrived through, including when both surfaces are saturated at once
|
||||
- [ ] Per-dialect semaphores are explicitly wrong and must not exist: the 8 sequence slots are physical, so two independent gates would each believe they were within budget while together exceeding it
|
||||
- [ ] The controller reads the dialect-neutral canonical request and has no knowledge of which surface produced it; adding a third dialect requires no change here
|
||||
- [ ] Slot occupancy and queue depth are reported per upstream, and requests are attributable to a surface for logging only, never for admission
|
||||
- [ ] Rejection at a full queue renders per surface — `application/problem+json` on `/v1`, the Anthropic error shape on `/llm` — from one shared decision
|
||||
- [ ] The controller is keyed by UPSTREAM, never by route or path prefix — `reasoning-predictor.llm-serving:80` has exactly one cap and one queue
|
||||
- [ ] Slot occupancy and queue depth are reported per upstream
|
||||
|
||||
## Verify
|
||||
|
||||
@@ -46,14 +41,4 @@ grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
|
||||
curl -s -D - -o /dev/null -X POST localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' | grep -i retry-after
|
||||
# expected: Retry-After present on the rejected request
|
||||
|
||||
# Same cap=6 stub, but split the load across both dialects: 20 on /v1 and 20 on /llm.
|
||||
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' -d '{"model":"reasoning","messages":[]}' &
|
||||
seq 20 | xargs -P20 -I{} curl -s -o /dev/null -X POST localhost:8080/llm/v1/messages \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"model":"reasoning","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' &
|
||||
wait
|
||||
grep -c 'max_concurrent=7' /tmp/stub-reasoning.log
|
||||
# expected: 0 — the two surfaces share one cap, they do not get 6 each
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user