Admin Bot
cc9a32f53a
CI / CI (pull_request) Successful in 3m11s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering - Paired with ResponseController.Flush() for unbuffered token delivery **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost from default (2) to 10 - ForceAttemptHTTP2 on outbound Transport for upstream connections - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load - Upstream Transport respects backpressure when clients read slowly **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly - Both pass at 0.11s and 0.06s respectively Fixes all three streaming performance issues in one coherent change.
100 lines
3.4 KiB
YAML
100 lines
3.4 KiB
YAML
apiVersion: tekton.dev/v1
|
|
kind: Pipeline
|
|
metadata:
|
|
name: sse-optimization-tests
|
|
namespace: api
|
|
labels:
|
|
app: api-gateway
|
|
component: ci-cd
|
|
spec:
|
|
description: >
|
|
Test pipeline for SSE optimization (issues #31, #32, #33).
|
|
Runs both functional integration tests and performance load tests.
|
|
|
|
params:
|
|
- name: image
|
|
type: string
|
|
description: "Container image to test (repo:tag)"
|
|
- name: gateway-port
|
|
type: string
|
|
default: "8080"
|
|
|
|
tasks:
|
|
# Functional integration tests first (quick smoke test)
|
|
- name: integration-tests
|
|
taskRef:
|
|
name: integration-test
|
|
params:
|
|
- name: image
|
|
value: $(params.image)
|
|
- name: gateway-port
|
|
value: $(params.gateway-port)
|
|
|
|
# Performance load tests (runs after integration tests pass)
|
|
- name: load-tests
|
|
runAfter:
|
|
- integration-tests
|
|
taskRef:
|
|
name: load-test-sse-streaming
|
|
params:
|
|
- name: image
|
|
value: $(params.image)
|
|
- name: gateway-port
|
|
value: $(params.gateway-port)
|
|
- name: concurrent-streams
|
|
value: "10"
|
|
- name: events-per-stream
|
|
value: "100"
|
|
- name: event-interval-ms
|
|
value: "50"
|
|
|
|
# Summary reporter
|
|
- name: report-results
|
|
runAfter:
|
|
- load-tests
|
|
taskSpec:
|
|
description: "Report combined test results"
|
|
params:
|
|
- name: integration-result
|
|
type: string
|
|
- name: integration-summary
|
|
type: string
|
|
- name: load-result
|
|
type: string
|
|
- name: load-summary
|
|
type: string
|
|
- name: load-metrics
|
|
type: string
|
|
steps:
|
|
- name: print-summary
|
|
image: busybox
|
|
script: |
|
|
#!/bin/sh
|
|
echo "╔════════════════════════════════════════════════════╗"
|
|
echo "║ SSE Optimization Test Results (PR #26) ║"
|
|
echo "╠════════════════════════════════════════════════════╣"
|
|
echo "║ ║"
|
|
echo "║ Integration Tests: ║"
|
|
echo "║ Status: $(params.integration-result)"
|
|
echo "║ Summary: $(params.integration-summary)"
|
|
echo "║ ║"
|
|
echo "║ Load Tests (Issues #31, #32, #33): ║"
|
|
echo "║ Status: $(params.load-result)"
|
|
echo "║ Summary: $(params.load-summary)"
|
|
echo "║ ║"
|
|
echo "║ Performance Metrics: ║"
|
|
echo "║ $(params.load-metrics)"
|
|
echo "║ ║"
|
|
echo "╚════════════════════════════════════════════════════╝"
|
|
params:
|
|
- name: integration-result
|
|
value: $(tasks.integration-tests.results.result)
|
|
- name: integration-summary
|
|
value: $(tasks.integration-tests.results.summary)
|
|
- name: load-result
|
|
value: $(tasks.load-tests.results.result)
|
|
- name: load-summary
|
|
value: $(tasks.load-tests.results.summary)
|
|
- name: load-metrics
|
|
value: $(tasks.load-tests.results.metrics)
|