30e0a83a50
CI / CI (push) Successful in 3m35s
Addresses three critical network issues for LLM streaming performance: **#33 Disable proxy buffering for SSE** - Add X-Accel-Buffering: no header to response - Tells nginx/Ingress to stream events immediately instead of buffering **#32 HTTP/2 multiplexing for concurrent streams** - Enable HTTP/2 in server config via http2.ConfigureServer() - Increase MaxConnsPerHost to 10 for better concurrency - Allows multiple concurrent LLM requests without blocking **#31 TCP backpressure for streaming LLM responses** - Set TCP_NODELAY on dialer to disable Nagle's algorithm - Reduces latency by sending small packets immediately - Critical for low TTFT (time-to-first-token) under load **Tests added:** - TestTCPBackpressure: Verifies TCP backpressure handling with slow client - TestConcurrentSSEStreams: Confirms HTTP/2 multiplexing works correctly --------- Co-authored-by: poison <[email protected]> Reviewed-on: #26 Co-authored-by: poimen <[email protected]>
35 lines
788 B
Go
35 lines
788 B
Go
package serviceadapter
|
|
|
|
// SQSAdapter handles X-Service: sqs requests.
|
|
type SQSAdapter struct{}
|
|
|
|
// S3Adapter handles X-Service: s3 requests.
|
|
type S3Adapter struct{}
|
|
|
|
// IAMAdapter handles X-Service: iam requests.
|
|
type IAMAdapter struct{}
|
|
|
|
// MemoryAdapter handles X-Service: memory requests (core + extended).
|
|
type MemoryAdapter struct{
|
|
// Extended resources: notes, context, nodes (git-aware)
|
|
// Requires M3.5.6+, M3.7.7+, M3.7.8+, M3.5.9
|
|
}
|
|
|
|
// AdapterFactory creates adapters by type.
|
|
func AdapterFactory(serviceName string) interface{} {
|
|
switch serviceName {
|
|
case "workflow":
|
|
return &WorkflowAdapter{}
|
|
case "sqs":
|
|
return &SQSAdapter{}
|
|
case "s3":
|
|
return &S3Adapter{}
|
|
case "iam":
|
|
return &IAMAdapter{}
|
|
case "memory":
|
|
return &MemoryAdapter{}
|
|
default:
|
|
return nil
|
|
}
|
|
}
|