26 lines
1.3 KiB
Markdown
26 lines
1.3 KiB
Markdown
# 1.3 — Client disconnect propagation (GREEN)
|
|
|
|
Phase: 1 — Proxy core
|
|
Stage: RED
|
|
Depends on: [1.2](1.2-streaming-passthrough.md)
|
|
|
|
- [ ] When a client closes the connection mid-response, the gateway cancels the upstream request immediately
|
|
- [ ] The stub upstream observes its request context cancelled, not a request that runs to completion
|
|
- [ ] Cancellation happens within a small bounded delay of the client close, not at the route's read timeout
|
|
- [ ] The same holds for a non-streaming request abandoned before the upstream replies
|
|
- [ ] The disconnect is logged with a reason distinguishing it from an upstream error
|
|
- [ ] No goroutine or upstream connection is left alive after the disconnect — the test asserts this, not just the response
|
|
|
|
This is load-bearing. The `reasoning` upstream runs 2 replicas at `--max-num-seqs=4`,
|
|
which is 8 concurrent sequence slots cluster-wide. An orphaned generation holds one of
|
|
those 8 until it finishes on its own, which for a 32B model on a Volta GPU can be
|
|
minutes. A handful of abandoned browser tabs can starve the entire cluster.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
go test ./internal/proxy/... -run TestClientDisconnectCancelsUpstream -race -v
|
|
# expected: passes — stub upstream reports context cancellation within 1s of the client
|
|
# closing mid-stream, and the post-test goroutine count returns to baseline
|
|
```
|