feat(network): SSE optimization for local LLM streaming (#31 #32 #33) (#26)
CI / CI (push) Successful in 3m35s
CI / CI (push) Successful in 3m35s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly --------- Co-authored-by: poison <[email protected]> Reviewed-on: #26 Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #26.
This commit is contained in:
@@ -6,6 +6,9 @@ namespace: api
|
||||
resources:
|
||||
- ci-rbac.yaml
|
||||
- task-integration-test.yaml
|
||||
- task-load-test.yaml
|
||||
- task-workflow-visibility.yaml
|
||||
- pipeline-sse-optimization.yaml
|
||||
|
||||
generatorOptions:
|
||||
disableNameSuffixHash: true
|
||||
@@ -14,3 +17,9 @@ configMapGenerator:
|
||||
- name: integration-test-script
|
||||
files:
|
||||
- scripts/integration-test.sh
|
||||
- name: load-test-script
|
||||
files:
|
||||
- scripts/load-test.sh
|
||||
- name: workflow-visibility-test-script
|
||||
files:
|
||||
- scripts/workflow-visibility-test.sh
|
||||
|
||||
Reference in New Issue
Block a user