Admin Bot
cc9a32f53a
CI / CI (pull_request) Successful in 3m11s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering - Paired with ResponseController.Flush() for unbuffered token delivery **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost from default (2) to 10 - ForceAttemptHTTP2 on outbound Transport for upstream connections - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load - Upstream Transport respects backpressure when clients read slowly **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly - Both pass at 0.11s and 0.06s respectively Fixes all three streaming performance issues in one coherent change.
28 lines
705 B
Plaintext
28 lines
705 B
Plaintext
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: api
|
|
|
|
resources:
|
|
- serviceaccount.yaml
|
|
- service.yaml
|
|
- deployment.yaml
|
|
- network-policy.yaml
|
|
- gateway-config-secret.enc.yaml
|
|
|
|
# The deployed image tag lives here and nowhere else. CI publishes
|
|
# forgejo.riotpiao.com/rock/api-gateway:<commit-sha> and tags it as :latest on main.
|
|
# ArgoCD auto-syncs when the latest image is available.
|
|
images:
|
|
- name: forgejo.riotpiao.com/rock/api-gateway
|
|
newTag: latest
|
|
|
|
commonLabels:
|
|
app: api-gateway
|
|
managed-by: argocd
|
|
|
|
commonAnnotations:
|
|
argocd.argoproj.io/sync-wave: "2"
|
|
# Wave 2 ensures the gateway is ready before anything that depends on it
|
|
# Kong remains on wave 7 unchanged
|