fix(proxy): address review findings on admission control (#64)
CI / CI (pull_request) Successful in 33m12s
CI / CI (pull_request) Successful in 33m12s
- tracing: responseWriter now forwards Flush and exposes Unwrap. Without it w.(http.Flusher) failed for every request passing the tracing middleware, so the parked-stream keep-alive never started: a parked streaming request wrote nothing for up to ADMISSION_MAX_PARK_SECONDS, which is exactly the client timeout the keep-alive was added to prevent. - proxy: once the SSE status line is spent, ModifyResponse frames an upstream response >= 400 as an event: error instead of letting a raw error body land inside the stream, where a streaming client reads it as an empty completion and reports success. net/http keeps the first status, so the upstream status cannot be forwarded after the park has written one. - engine_state: new Snapshot() returns the state and its tick channel under one lock. Get and Tick taken separately can miss the poll that lands between the two calls, parking a request for an extra interval. - engine_state: observer_degraded now tracks current parsability rather than being raised once on the first poll and never cleared, which pinned it at 1 after a single truncated response and never reported drift that appeared later. markDegraded only touches the exporter when the value changes. - deployment: ADMISSION_MAX_INFLIGHT is set explicitly to 4, documenting the coupling to the engine's --max-num-seqs, instead of silently defaulting to VLLM_MAX_NUM_SEQS.
This commit is contained in:
@@ -36,6 +36,21 @@ func (rw *responseWriter) Write(b []byte) (int, error) {
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Flush forwards to the wrapped writer so that streaming handlers can still
|
||||
// detect http.Flusher through this wrapper. Without it a bare
|
||||
// `w.(http.Flusher)` assertion fails for every request that passes through
|
||||
// this middleware, which silently disables SSE flushing and keep-alives.
|
||||
func (rw *responseWriter) Flush() {
|
||||
if f, ok := rw.ResponseWriter.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
// Unwrap exposes the wrapped writer to http.ResponseController.
|
||||
func (rw *responseWriter) Unwrap() http.ResponseWriter {
|
||||
return rw.ResponseWriter
|
||||
}
|
||||
|
||||
// Middleware returns an HTTP middleware that adds tracing to requests.
|
||||
func Middleware(next http.Handler) http.Handler {
|
||||
tracer := otel.Tracer(tracerName)
|
||||
|
||||
Reference in New Issue
Block a user