feat(network): SSE optimization for local LLM streaming (#31 #32 #33) #26

2026-09-13T23:12:17Z

poimen commented

2026-09-13 23:12:17 +00:00

Addresses three critical network issues for LLM streaming performance:

#33 Disable proxy buffering for SSE

Add X-Accel-Buffering: no header to response
Tells nginx/Ingress to stream events immediately instead of buffering

#32 HTTP/2 multiplexing for concurrent streams

Enable HTTP/2 in server config via http2.ConfigureServer()
Increase MaxConnsPerHost to 10 for better concurrency
Allows multiple concurrent LLM requests without blocking

#31 TCP backpressure for streaming LLM responses

Set TCP_NODELAY on dialer to disable Nagle's algorithm
Reduces latency by sending small packets immediately
Critical for low TTFT (time-to-first-token) under load

Tests added:

TestTCPBackpressure: Verifies TCP backpressure handling with slow client
TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly

Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly

rock added 1 commit 2026-09-13 23:14:13 +00:00

feat(network): SSE optimization for local LLM streaming (#31 #32 #33 )

CI / CI (pull_request) Successful in 3m11s

Details

cc9a32f53a

Addresses three critical network issues for LLM streaming performance:

**#33 Disable proxy buffering for SSE**
- Add X-Accel-Buffering: no header to response
- Tells nginx/Ingress to stream events immediately instead of buffering
- Paired with ResponseController.Flush() for unbuffered token delivery

**#32 HTTP/2 multiplexing for concurrent streams**
- Enable HTTP/2 in server config via http2.ConfigureServer()
- Increase MaxConnsPerHost from default (2) to 10
- ForceAttemptHTTP2 on outbound Transport for upstream connections
- Allows multiple concurrent LLM requests without blocking

**#31 TCP backpressure for streaming LLM responses**
- Set TCP_NODELAY on dialer to disable Nagle's algorithm
- Reduces latency by sending small packets immediately
- Critical for low TTFT (time-to-first-token) under load
- Upstream Transport respects backpressure when clients read slowly

**Tests added:**
- TestTCPBackpressure: Verifies TCP backpressure handling with slow client
- TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly
- Both pass at 0.11s and 0.06s respectively

Fixes all three streaming performance issues in one coherent change.

rock force-pushed feat/sse-optimization-31-32-33 from be4d33eb87 to cc9a32f53a

2026-09-13 23:14:13 +00:00

Compare

rock referenced this issue from a commit

2026-09-13 23:18:44 +00:00

refactor: retire /workflows endpoint, use X-Service: workflow instead

rock added 1 commit 2026-09-13 23:18:44 +00:00

refactor: retire /workflows endpoint, use X-Service: workflow instead

CI / CI (pull_request) Successful in 3m12s

Details

b0f608f145

Remove deprecated /workflows HTTP endpoint in favor of unified X-Service
header routing. All workflow operations now route through:

  X-Service: workflow
  X-Resource: {action} (start, describe, signal, query, etc)

This consolidates routing patterns and allows users to specify domain/
namespace via request payload instead of path prefixes.

Changes:
- Remove internal/proxy/workflows.go (484 lines of predefined workflows)
- Remove internal/proxy/workflows_test.go
- Remove /workflows handler from proxy.ServeHTTP()
- Update README.md to document X-Service routing pattern
- Add migration note: use X-Service: workflow instead of /workflows

WorkflowAdapter in serviceadapter/ handles X-Service: workflow requests
and forwards to Temporal gRPC API. Users can now specify domain/namespace
in request payload for multi-tenant workflow access.

Related: #26

rock added 1 commit 2026-09-13 23:21:56 +00:00

feat: implement WorkflowAdapter with namespace pass-down support

CI / CI (pull_request) Successful in 3m17s

Details

c6cd41fd4b

Enable X-Service: workflow routing to Temporal via ServiceAdapter.
Users can now specify namespace/domain in request payload for multi-tenant
workflow access.

Changes:
- Implement WorkflowAdapter in serviceadapter/workflow_adapter.go
  * Defines 10 workflow resources: start, describe, list, history,
    terminate, cancel, signal, query, reset, update
  * Each resource validates namespace parameter in payload
  * Forwards requests to Temporal gRPC handler

- Add GetWorkflowSpec() to define ServiceAdapter spec with:
  * Upstream: grpc://temporal:7233
  * Auth requirements per operation (execute, read, signal, query)
  * Request/response schemas for validation

- Wire WorkflowAdapter into main.go:
  * Register workflow adapter in serviceadapter registry
  * Initialize with temporal handler for gRPC forwarding

- Remove old empty WorkflowAdapter stub from adapters.go

Usage:
  curl -X POST https://api.riotpiao.com/ \
    -H 'X-Service: workflow' \
    -H 'X-Resource: start' \
    -H 'Authorization: Bearer TOKEN' \
    -d '{
      "namespace": "default",
      "workflow_id": "my-workflow",
      "workflow_type": "MyWorkflow",
      "task_queue": "default"
    }'

Namespace is required in all workflow operations and must be specified
by the client in the request payload. This enables multi-tenant support
where different teams access their own Temporal namespaces.

rock added 1 commit 2026-09-13 23:24:50 +00:00

test: add workflow visibility tests for poimen-harness namespace

CI / CI (pull_request) Successful in 3m18s

Details

6608f1a8d5

Verify that WorkflowAdapter provides visibility into terminated workflows
in the poimen-harness namespace. This ensures namespace pass-down feature
is working correctly and users can specify different domains/namespaces
via X-Service: workflow requests.

Tests added:
1. integration-test.sh: Added workflow visibility tests
   - List workflows in poimen-harness namespace
   - Verify terminated/completed workflows are visible
   - Validate namespace parameter requirement
   - Check auth enforcement

2. workflow-visibility-test.sh: NEW dedicated workflow test script
   - Tests WorkflowAdapter namespace pass-down
   - Verifies list, describe, and auth enforcement
   - Specific focus on poimen-harness namespace
   - Looks for 4 terminated workflows

3. task-workflow-visibility.yaml: NEW Tekton task
   - Runs workflow visibility tests against live gateway
   - Sidecar deployment pattern
   - Publishes result + summary + workflow-count metrics

4. pipeline-sse-optimization.yaml: Updated
   - Added workflow-visibility-tests stage (runs after integration-tests)
   - Updated report-results to include workflow test results
   - Full pipeline now: integration → workflow-visibility → load → report

5. kustomization.yaml: Updated
   - Added task-workflow-visibility.yaml
   - Added workflow-visibility-test-script ConfigMap

This ensures that the deprecated /workflows endpoint replacement correctly
supports multi-tenant access via namespace specification in request payload.

rock merged commit 30e0a83a50 into main

2026-09-13 23:37:34 +00:00

rock deleted branch feat/sse-optimization-31-32-33

2026-09-13 23:37:35 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#26