30 lines
1.6 KiB
Markdown
30 lines
1.6 KiB
Markdown
|
# 1.4 — Per-route timeouts (GREEN)
|
||
|
|
|
||
|
|
Phase: 1 — Proxy core
|
||
|
|
Stage: GREEN
|
||
|
|
Depends on: [0.2](0.2-route-configuration.md), [1.1](1.1-reverse-proxy.md)
|
||
|
|
|
||
|
|
- [ ] Connect, read and write timeouts are taken per route from configuration, never from a global default
|
||
|
|
- [ ] Chat routes use connect `10s`, read `1h`, write `1h`
|
||
|
|
- [ ] Embeddings and rerank routes use connect `10s`, read `10m`, write `10m`
|
||
|
|
- [ ] An upstream that never accepts a connection fails at the configured connect timeout, not later
|
||
|
|
- [ ] An upstream that accepts then stalls fails at the configured read timeout with a `5xx` and a logged reason
|
||
|
|
- [ ] A stream still emitting tokens is never cut by the read timeout — the timeout applies to inactivity, not total duration
|
||
|
|
- [ ] No code path shortens a configured proxy timeout to enforce an application-level cap
|
||
|
|
|
||
|
|
These are Kong's current values and they are deliberate. The 1-hour read timeout
|
||
|
|
exists because a 32B model on a Volta GPU routinely exceeds 60 seconds per request.
|
||
|
|
Any shorter application-level cap must be enforced by the gateway's own logic — a
|
||
|
|
budget, a slot limit, an explicit max-generation-time — and never by shortening the
|
||
|
|
proxy timeout, or long legitimate generations truncate mid-stream and callers see
|
||
|
|
corrupted output rather than an error.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
go test ./internal/proxy/... -run 'TestConnectTimeout|TestReadTimeout|TestLongStreamNotTruncated' -v
|
||
|
|
# expected: passes — stalled upstream errors at the configured read timeout, a stub
|
||
|
|
# emitting one event every 200ms for longer than the timeout window is not cut off,
|
||
|
|
# and a blackholed address fails at ~10s
|
||
|
|
```
|