feat(network): SSE optimization for local LLM streaming (#31 #32 #33)
CI / CI (pull_request) Successful in 3m4s

Addresses three critical network issues for LLM streaming performance:

**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery

**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking

**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly

**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively

Fixes all three streaming performance issues in one coherent change.
This commit is contained in:
Admin Bot
2026-09-14 08:12:03 +09:00
parent 7de71180b3
commit be4d33eb87
10 changed files with 528 additions and 13 deletions
+27
View File
@@ -0,0 +1,27 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: api
resources:
- serviceaccount.yaml
- service.yaml
- deployment.yaml
- network-policy.yaml
- gateway-config-secret.enc.yaml
# The deployed image tag lives here and nowhere else. CI publishes
# forgejo.riotpiao.com/rock/api-gateway:<commit-sha> and tags it as :latest on main.
# ArgoCD auto-syncs when the latest image is available.
images:
- name: forgejo.riotpiao.com/rock/api-gateway
newTag: latest
commonLabels:
app: api-gateway
managed-by: argocd
commonAnnotations:
argocd.argoproj.io/sync-wave: "2"
# Wave 2 ensures the gateway is ready before anything that depends on it
# Kong remains on wave 7 unchanged
+29
View File
@@ -0,0 +1,29 @@
apiVersion: v1
kind: Secret
metadata:
name: smtp-credentials
namespace: api
labels:
app: api-gateway
component: notification
type: Opaque
data:
host: <base64-encoded SMTP hostname>
port: <base64-encoded SMTP port, e.g., "587">
from: <base64-encoded sender email>
user: <base64-encoded SMTP username>
password: <base64-encoded SMTP password>
# To create from plaintext:
# kubectl create secret generic smtp-credentials \
# --from-literal=host=mail.example.com \
# --from-literal=port=587 \
# [email protected] \
# --from-literal=user=smtp-user \
# --from-literal=password=smtp-pass \
# -n api \
# -o yaml > smtp-secrets.yaml
#
# Then encrypt with SOPS:
# sops -e smtp-secrets.yaml > smtp-secrets.enc.yaml
# rm smtp-secrets.yaml