Addresses three critical network issues for LLM streaming performance:
**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery
**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking
**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly
**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively
Fixes all three streaming performance issues in one coherent change.
- Add TemporalConfig struct to internal/config
- Update gateway config with Temporal frontend service (port 7233)
- Update router with memory service adapter support
- Add config.local.yaml with memory service configuration
- Encrypt production config with SOPS (AES256_GCM)
- Support X-Service header routing pattern for service discovery
- Keep legacy path-based routes with deprecation warnings
- All 5 adapters preserved: workflow, memory, sqs, s3, iam
http.Server.WriteTimeout is an absolute deadline over the whole
response, not an inactivity timeout -- 15s was cutting off SSE streams
from the reasoning model mid-generation, surfacing to clients as a
"terminated" error well before the model finished. Switch to
ReadHeaderTimeout (protects against slow headers without capping body
duration) and raise WriteTimeout to match the edge nginx Ingress's
proxy-read/send-timeout of 3600s.
Baseline for the Kong replacement on api.riotpiao.com. Brings the working
tree under version control for the first time: gateway source, the task
board that drives the agent runs, test fixtures, and K8s manifests.
Anchor the gateway ignore rule to the repo root. Unanchored, "gateway"
also matched the cmd/gateway/ source directory, so the program entrypoint
was excluded from every commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Admin Bot