feat(network): SSE optimization for local LLM streaming (#31 #32 #33)
CI / CI (pull_request) Successful in 3m4s

Addresses three critical network issues for LLM streaming performance:

**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery

**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking

**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly

**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively

Fixes all three streaming performance issues in one coherent change.
This commit is contained in:
Admin Bot
2026-09-14 08:12:03 +09:00
parent 7de71180b3
commit be4d33eb87
10 changed files with 528 additions and 13 deletions
+23
View File
@@ -11,6 +11,7 @@ import (
"net/url"
"sort"
"strings"
"syscall"
"time"
"forgejo.riotpiao.com/rock/homelab-frontend/internal/auth"
@@ -127,6 +128,15 @@ func (h *Handler) getOrCreateTransport(addr string, up *config.Upstream) *http.T
dialer := &net.Dialer{
Timeout: up.ConnectTimeout,
KeepAlive: 30 * time.Second,
// Issue #31: TCP_NODELAY disables Nagle's algorithm, reducing latency
// for streaming responses by sending small packets immediately instead of
// waiting for larger batches. Critical for low-latency LLM token streaming.
Control: func(network, address string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
// TCP_NODELAY disables Nagle's algorithm for immediate packet transmission
_ = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_TCP, syscall.TCP_NODELAY, 1)
})
},
}
transport := &http.Transport{
@@ -134,8 +144,15 @@ func (h *Handler) getOrCreateTransport(addr string, up *config.Upstream) *http.T
DialContext: dialer.DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
// Issue #32: Increase per-host connection limit to support HTTP/2 multiplexing.
// With HTTP/2, we can serve many concurrent streams over fewer connections,
// but we still allow more connections for better resource utilization.
MaxConnsPerHost: 10,
// Allow persistent connections
DisableKeepAlives: false,
// Issue #31: Enable HTTP/2 for client connections to support multiplexing.
// This allows concurrent requests to stream simultaneously with better flow control.
ForceAttemptHTTP2: true,
}
// Store the upstream config for use in the handler
@@ -429,6 +446,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// A streaming response that's continuously sending should not be cut off.
// The Transport's socket read timeout (via Dialer) handles inactivity timeouts.
// For streaming responses (SSE, chunked), disable buffering to ensure events
// reach clients immediately. Issue #33: X-Accel-Buffering:no tells nginx/Ingress
// to stream instead of buffer. ResponseController.Flush() in upstream handler
// pairs with this to deliver unbuffered chunks.
w.Header().Set("X-Accel-Buffering", "no")
// Serve the request through the proxy
proxy.ServeHTTP(w, r)
}