fix(server): stop WriteTimeout from killing in-progress LLM streams
http.Server.WriteTimeout is an absolute deadline over the whole response, not an inactivity timeout -- 15s was cutting off SSE streams from the reasoning model mid-generation, surfacing to clients as a "terminated" error well before the model finished. Switch to ReadHeaderTimeout (protects against slow headers without capping body duration) and raise WriteTimeout to match the edge nginx Ingress's proxy-read/send-timeout of 3600s.
This commit is contained in:
@@ -21,11 +21,17 @@ type Server struct {
|
|||||||
func New(listenAddr string, shutdownTimeout time.Duration, handler http.Handler) *Server {
|
func New(listenAddr string, shutdownTimeout time.Duration, handler http.Handler) *Server {
|
||||||
return &Server{
|
return &Server{
|
||||||
httpServer: &http.Server{
|
httpServer: &http.Server{
|
||||||
Addr: listenAddr,
|
Addr: listenAddr,
|
||||||
Handler: handler,
|
Handler: handler,
|
||||||
ReadTimeout: 15 * time.Second,
|
// ReadHeaderTimeout (not ReadTimeout) and a long WriteTimeout: both
|
||||||
WriteTimeout: 15 * time.Second,
|
// ReadTimeout and WriteTimeout are absolute deadlines covering the
|
||||||
IdleTimeout: 60 * time.Second,
|
// whole request/response body, not inactivity timeouts -- a 15s
|
||||||
|
// WriteTimeout here was killing in-progress LLM SSE streams (proxy.go's
|
||||||
|
// outbound transport deliberately avoids this same mistake). Mirrors
|
||||||
|
// the edge nginx Ingress's proxy-read/send-timeout of 3600s.
|
||||||
|
ReadHeaderTimeout: 15 * time.Second,
|
||||||
|
WriteTimeout: 1 * time.Hour,
|
||||||
|
IdleTimeout: 60 * time.Second,
|
||||||
},
|
},
|
||||||
shutdownTimeout: shutdownTimeout,
|
shutdownTimeout: shutdownTimeout,
|
||||||
healthChecker: NewHealthChecker(false, false),
|
healthChecker: NewHealthChecker(false, false),
|
||||||
|
|||||||
Reference in New Issue
Block a user