Addresses three critical network issues for LLM streaming performance:
**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery
**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking
**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly
**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively
Fixes all three streaming performance issues in one coherent change.
- Add internal/tracing package with OTel tracer initialization
- HTTP middleware for server-side tracing (request/response attributes)
- Transport wrapper for client-side upstream call tracing
- Update proxy to use tracing transport
- Add OTEL_* env vars to k8s deployment
Traces flow: api-gateway -> otel-collector -> tempo -> grafana
- 2.3: unknown model errors (400 + RFC 9457 problem+json with valid_models)
- 2.5: GET /v1/models endpoint (derived from config, not hardcoded)
- 2.6: POST /v1/embeddings passthrough (body-based dispatch, no rewrite)
- 2.7: POST /v1/rerank with path rewrite (/v1/rerank → /rerank)
- wire proxy.Handler in main.go (was using dummy handler)
- 140+ tests passing, race detector clean
- all requests: client → nginx → gateway → upstreams
- ready for config deployment to go live
- Dockerfile: multi-stage, distroless nonroot, CGO_ENABLED=0 static, commit
SHA stamped via VERSION build arg.
- .forgejo/workflows/ci.yaml: Forgejo reads .forgejo/, not .github/, and the
runner declares only the "docker" label. Verify job on every push; image
build and push gated to main.
- Drop .github/workflows/ci.yml — this remote is Forgejo, so it never ran.
- deployment.yaml: image from the Forgejo registry, forgejo-registry pull
secret, runAsUser 65532 to match distroless nonroot.
- kustomization.yaml: pin the tag in one place. Promoting a build is a
one-line newTag bump, never :latest.
Baseline for the Kong replacement on api.riotpiao.com. Brings the working
tree under version control for the first time: gateway source, the task
board that drives the agent runs, test fixtures, and K8s manifests.
Anchor the gateway ignore rule to the repo root. Unanchored, "gateway"
also matched the cmd/gateway/ source directory, so the program entrypoint
was excluded from every commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Admin Bot
rock
