fix(server): stop WriteTimeout from killing in-progress LLM streams
http.Server.WriteTimeout is an absolute deadline over the whole response, not an inactivity timeout -- 15s was cutting off SSE streams from the reasoning model mid-generation, surfacing to clients as a "terminated" error well before the model finished. Switch to ReadHeaderTimeout (protects against slow headers without capping body duration) and raise WriteTimeout to match the edge nginx Ingress's proxy-read/send-timeout of 3600s.
This commit is contained in:
@@ -21,11 +21,17 @@ type Server struct {
|
||||
func New(listenAddr string, shutdownTimeout time.Duration, handler http.Handler) *Server {
|
||||
return &Server{
|
||||
httpServer: &http.Server{
|
||||
Addr: listenAddr,
|
||||
Handler: handler,
|
||||
ReadTimeout: 15 * time.Second,
|
||||
WriteTimeout: 15 * time.Second,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
Addr: listenAddr,
|
||||
Handler: handler,
|
||||
// ReadHeaderTimeout (not ReadTimeout) and a long WriteTimeout: both
|
||||
// ReadTimeout and WriteTimeout are absolute deadlines covering the
|
||||
// whole request/response body, not inactivity timeouts -- a 15s
|
||||
// WriteTimeout here was killing in-progress LLM SSE streams (proxy.go's
|
||||
// outbound transport deliberately avoids this same mistake). Mirrors
|
||||
// the edge nginx Ingress's proxy-read/send-timeout of 3600s.
|
||||
ReadHeaderTimeout: 15 * time.Second,
|
||||
WriteTimeout: 1 * time.Hour,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
},
|
||||
shutdownTimeout: shutdownTimeout,
|
||||
healthChecker: NewHealthChecker(false, false),
|
||||
|
||||
Reference in New Issue
Block a user