feat(network): SSE optimization for local LLM streaming (#31 #32 #33) (#26)

Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly --------- Co-authored-by: poison <[email protected]> Reviewed-on: #26 Co-authored-by: poimen <[email protected]>
2026-09-13 23:37:33 +00:00
parent 7de71180b3
commit 30e0a83a50
23 changed files with 1606 additions and 797 deletions
@@ -0,0 +1,4 @@
+(apply,CacheStats{hitCount=337, missCount=199, loadSuccessCount=199, loadExceptionCount=0, totalLoadTime=581291927, evictionCount=0})
+(tree,CacheStats{hitCount=986, missCount=352, loadSuccessCount=299, loadExceptionCount=0, totalLoadTime=821650758, evictionCount=0})
+(commit,CacheStats{hitCount=108, missCount=107, loadSuccessCount=107, loadExceptionCount=0, totalLoadTime=78983052, evictionCount=0})
+(tag,CacheStats{hitCount=0, missCount=2, loadSuccessCount=2, loadExceptionCount=0, totalLoadTime=319542, evictionCount=0})