Admin Bot
cc9a32f53a
CI / CI (pull_request) Successful in 3m11s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering - Paired with ResponseController.Flush() for unbuffered token delivery **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost from default (2) to 10 - ForceAttemptHTTP2 on outbound Transport for upstream connections - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load - Upstream Transport respects backpressure when clients read slowly **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly - Both pass at 0.11s and 0.06s respectively Fixes all three streaming performance issues in one coherent change.
30 lines
788 B
YAML
30 lines
788 B
YAML
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: smtp-credentials
|
|
namespace: api
|
|
labels:
|
|
app: api-gateway
|
|
component: notification
|
|
type: Opaque
|
|
data:
|
|
host: <base64-encoded SMTP hostname>
|
|
port: <base64-encoded SMTP port, e.g., "587">
|
|
from: <base64-encoded sender email>
|
|
user: <base64-encoded SMTP username>
|
|
password: <base64-encoded SMTP password>
|
|
|
|
# To create from plaintext:
|
|
# kubectl create secret generic smtp-credentials \
|
|
# --from-literal=host=mail.example.com \
|
|
# --from-literal=port=587 \
|
|
# [email protected] \
|
|
# --from-literal=user=smtp-user \
|
|
# --from-literal=password=smtp-pass \
|
|
# -n api \
|
|
# -o yaml > smtp-secrets.yaml
|
|
#
|
|
# Then encrypt with SOPS:
|
|
# sops -e smtp-secrets.yaml > smtp-secrets.enc.yaml
|
|
# rm smtp-secrets.yaml
|