16 lines
541 B
Markdown
16 lines
541 B
Markdown
|
# 5.8 — Rate limiting Tier 3 (atlas) (GREEN)
|
||
|
|
|
||
|
|
Phase: 5 — Chat + Rate Limiter (Surface D)
|
||
|
|
Stage: GREEN
|
||
|
|
|
||
|
|
Depends on: [5.7](5.7-api-chat.md)
|
||
|
|
|
||
|
|
- [ ] Global chat semaphore = 6 (2 of 8 GPU slots kept as operator headroom); queue depth 20 then reject `429`; per-session budget 12 msg/24h; per-request timeout 120s hard server-side; disconnect cancels upstream immediately (`req.Context()` threaded to vLLM request)
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# load test at 20 concurrent clients
|
||
|
|
# GPU sequence usage never exceeds 6, no upstream 5xx, queue drains
|
||
|
|
```
|