feat(network): SSE optimization for local LLM streaming (#31 #32 #33) (#26)
CI / CI (push) Successful in 3m35s
CI / CI (push) Successful in 3m35s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly --------- Co-authored-by: poison <[email protected]> Reviewed-on: #26 Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #26.
This commit is contained in:
@@ -76,10 +76,56 @@ echo "▸ SQS service"
|
||||
assert "sqs/list-queues" 401 \
|
||||
-X GET -H "X-Service: sqs" -H "X-Resource: list-queues" "${GW}/"
|
||||
|
||||
# ── Workflow (gRPC needs content-type → 400) ──
|
||||
# ── Workflow visibility (namespace pass-down) ──
|
||||
echo "▸ Workflow service"
|
||||
assert "workflow/list (no grpc content-type → 400)" 400 \
|
||||
-X GET -H "X-Service: workflow" -H "X-Resource: list" "${GW}/"
|
||||
|
||||
# Test 1: List workflows in poimen-harness namespace (should see 4 terminated workflows)
|
||||
echo " Testing workflow visibility in poimen-harness namespace..."
|
||||
WF_LIST=$(curl -s -X POST \
|
||||
-H "X-Service: workflow" \
|
||||
-H "X-Resource: list" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"namespace": "poimen-harness"}' \
|
||||
"${GW}/" 2>/dev/null || echo '{}')
|
||||
|
||||
# Check if response contains workflows
|
||||
if echo "$WF_LIST" | grep -q '"executions"'; then
|
||||
echo " ✓ Workflow list returned (poimen-harness namespace)"
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo " ✗ Workflow list failed to return executions"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
TOTAL=$((TOTAL + 1))
|
||||
|
||||
# Test 2: Verify we can query terminated workflows
|
||||
echo " Testing terminated workflow visibility..."
|
||||
if echo "$WF_LIST" | grep -q '"Completed\|"status"'; then
|
||||
echo " ✓ Found completed/terminated workflows in response"
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo " ⚠ No terminated workflows found in response (may be empty namespace)"
|
||||
# Don't fail if namespace is empty - just note it
|
||||
fi
|
||||
TOTAL=$((TOTAL + 1))
|
||||
|
||||
# Test 3: Verify namespace is required (missing namespace → 400)
|
||||
echo " Testing namespace validation..."
|
||||
NO_NS=$(curl -s -w '%{http_code}' -X POST \
|
||||
-H "X-Service: workflow" \
|
||||
-H "X-Resource: list" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{}' \
|
||||
"${GW}/" 2>/dev/null || echo "000")
|
||||
|
||||
if [ "$NO_NS" = "400" ]; then
|
||||
echo " ✓ Correctly rejected list without namespace (400)"
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo " ✗ Expected 400 for missing namespace, got $NO_NS"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
TOTAL=$((TOTAL + 1))
|
||||
|
||||
echo ""
|
||||
echo "═══ Results: ${PASS}/${TOTAL} passed, ${FAIL} failed ═══"
|
||||
|
||||
Reference in New Issue
Block a user