feat(network): SSE optimization for local LLM streaming (#31 #32 #33) (#26)
CI / CI (push) Successful in 3m35s

Addresses three critical network issues for LLM streaming performance:

**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering

**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost to 10 for better concurrency
- Allows multiple concurrent LLM requests without blocking

**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load

**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly

---------

Co-authored-by: poison <[email protected]>
Reviewed-on: #26
Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #26.
This commit is contained in:
2026-09-13 23:37:33 +00:00
committed by rock
co-authored by poison
parent 7de71180b3
commit 30e0a83a50
23 changed files with 1606 additions and 797 deletions
+9
View File
@@ -6,6 +6,9 @@ namespace: api
resources:
- ci-rbac.yaml
- task-integration-test.yaml
- task-load-test.yaml
- task-workflow-visibility.yaml
- pipeline-sse-optimization.yaml
generatorOptions:
disableNameSuffixHash: true
@@ -14,3 +17,9 @@ configMapGenerator:
- name: integration-test-script
files:
- scripts/integration-test.sh
- name: load-test-script
files:
- scripts/load-test.sh
- name: workflow-visibility-test-script
files:
- scripts/workflow-visibility-test.sh