33 lines
1.6 KiB
Markdown
33 lines
1.6 KiB
Markdown
|
# 4.2 — Per-caller request budgets (GREEN)
|
||
|
|
|
||
|
|
Phase: 4 — Limits and budgets
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [4.3](4.3-problem-json-errors.md)
|
||
|
|
|
||
|
|
No `rate-limiting` plugin exists anywhere in the cluster today. This is net-new work,
|
||
|
|
not a migration — there is no prior behaviour to preserve.
|
||
|
|
|
||
|
|
- [ ] An identified caller gets a bounded number of requests per configured time window
|
||
|
|
- [ ] Caller identity comes from the authenticated token's subject when auth is on, and from a documented fallback attribute when auth is off
|
||
|
|
- [ ] Budget size and window length are explicit in configuration, per caller class, with no silent defaults
|
||
|
|
- [ ] Exceeding the budget is rejected with a retryable status, an `application/problem+json` body, and a `Retry-After` naming when the window resets
|
||
|
|
- [ ] Remaining budget and reset time are observable to the caller on allowed requests, not only on rejections
|
||
|
|
- [ ] Budgets are enforced independently of the `reasoning` concurrency cap — a caller under budget can still be queued or rejected for slot pressure, and vice versa
|
||
|
|
- [ ] Two distinct callers do not consume each other's budget
|
||
|
|
- [ ] Budget state is per-replica-safe: the documented behaviour with 2 replicas is stated, not accidental
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# budget=5 per 60s window for the test caller
|
||
|
|
for i in $(seq 1 7); do
|
||
|
|
curl -s -o /dev/null -w '%{http_code} ' -H 'authorization: Bearer test-caller-a' \
|
||
|
|
localhost:8080/v1/models
|
||
|
|
done; echo
|
||
|
|
# expected: 200 200 200 200 200 429 429
|
||
|
|
|
||
|
|
curl -s -o /dev/null -w '%{http_code}\n' -H 'authorization: Bearer test-caller-b' \
|
||
|
|
localhost:8080/v1/models
|
||
|
|
# expected: 200 — caller B has its own budget
|
||
|
|
```
|