feat(network): SSE optimization for local LLM streaming (#31 #32 #33) (#26)

Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly --------- Co-authored-by: poison <[email protected]> Reviewed-on: #26 Co-authored-by: poimen <[email protected]>
2026-09-13 23:37:33 +00:00
parent 7de71180b3
commit 30e0a83a50
23 changed files with 1606 additions and 797 deletions
@@ -0,0 +1,84 @@
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: workflow-visibility-test
+  namespace: api
+  labels:
+    app: api-gateway
+    component: testing
+spec:
+  description: >
+    Test workflow visibility via WorkflowAdapter.
+    Verifies that the gateway provides visibility into terminated workflows
+    in the poimen-harness namespace via X-Service: workflow routing.
+    This ensures namespace pass-down is working correctly.
+  
+  params:
+  - name: image
+    type: string
+    description: "Container image to test (repo:tag)"
+  - name: gateway-port
+    type: string
+    default: "8080"
+  
+  results:
+  - name: result
+    type: string
+    description: "pass or fail"
+  - name: summary
+    type: string
+    description: "Test summary"
+  - name: workflow-count
+    type: string
+    description: "Number of workflows found in poimen-harness"
+  
+  sidecars:
+  - name: gateway
+    image: $(params.image)
+    env:
+    - name: LISTEN_ADDR
+      value: "0.0.0.0:$(params.gateway-port)"
+    - name: CONFIG_PATH
+      value: /etc/gateway/config.yaml
+    - name: LOG_LEVEL
+      value: info
+    - name: AUTH_CLIENT_SECRET
+      valueFrom:
+        secretKeyRef:
+          name: api-gw-client-secret
+          key: client-secret
+          optional: true
+    volumeMounts:
+    - name: gateway-config
+      mountPath: /etc/gateway
+      readOnly: true
+  
+  steps:
+  - name: run-workflow-visibility-test
+    image: curlimages/curl:8.13.0
+    env:
+    - name: GW
+      value: "http://localhost:$(params.gateway-port)"
+    - name: RESULTS_DIR
+      value: /tekton/results
+    command: ["sh", "/scripts/workflow-visibility-test.sh"]
+    volumeMounts:
+    - name: test-script
+      mountPath: /scripts
+      readOnly: true
+    computeResources:
+      requests:
+        cpu: 100m
+        memory: 64Mi
+      limits:
+        cpu: 200m
+        memory: 128Mi
+  
+  volumes:
+  - name: gateway-config
+    secret:
+      secretName: api-gateway-config
+  - name: test-script
+    configMap:
+      name: workflow-visibility-test-script
+      defaultMode: 0755