riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

[Network] TCP backpressure for streaming LLM responses #31

New Issue
Open
opened 2026-09-11 01:26:14 +00:00 by poimen · 0 comments
poimen commented 2026-09-11 01:26:14 +00:00
Member
Copy Link
Copy Source

Problem

Fast GPU overwhelms slow client connections, floods gateway RAM.

Investigation

  • Gateway proxy in internal/proxy/ uses Flush() for SSE streaming
  • No backpressure — gateway buffers indefinitely if client reads slowly
  • Tests verify chunk arrival but not backpressure

Resolution

Apply TCP backpressure: bounded buffer in io.Copy, pause upstream on slow client. Add context.WithCancel on client disconnect.

Files

  • internal/proxy/bodydispatch.go
  • internal/proxy/streaming_test.go
## Problem Fast GPU overwhelms slow client connections, floods gateway RAM. ## Investigation - Gateway proxy in `internal/proxy/` uses `Flush()` for SSE streaming - No backpressure — gateway buffers indefinitely if client reads slowly - Tests verify chunk arrival but not backpressure ## Resolution Apply TCP backpressure: bounded buffer in `io.Copy`, pause upstream on slow client. Add `context.WithCancel` on client disconnect. ## Files - `internal/proxy/bodydispatch.go` - `internal/proxy/streaming_test.go`
poimen added this to the LLM Production Hardening milestone 2026-09-11 01:26:14 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/auth
Authentication & authorization
 area/authentik
Authentik OIDC provider config
 area/downstream
Downstream service changes
 area/event-bus
Event routing and notification pipeline
 area/gateway
API gateway code (homelab-frontend)
 area/notifications
Gotify, SMTP, push notifications
 area/observer
Poimen goal-driven observation and pattern detection
 priority/high
Must have
 priority/medium
Should have
 status/blocked
Blocked by dependency
 status/done
Completed
 status/in-progress
Currently being worked on
 status/review
Ready for review
 status/todo
Not started
 type/docs
Documentation
 type/feature
New feature
 type/infra
Infrastructure change
No labels
Milestone
No items
No Milestone LLM Production Hardening
Projects
Clear projects
No projects
Assignees
Clear assignees
poimen rock (Rock)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab#31