fix(proxy): address review findings on admission control (#64)

- tracing: responseWriter now forwards Flush and exposes Unwrap. Without it
  w.(http.Flusher) failed for every request passing the tracing middleware, so
  the parked-stream keep-alive never started: a parked streaming request wrote
  nothing for up to ADMISSION_MAX_PARK_SECONDS, which is exactly the client
  timeout the keep-alive was added to prevent.
- proxy: once the SSE status line is spent, ModifyResponse frames an upstream
  response >= 400 as an event: error instead of letting a raw error body land
  inside the stream, where a streaming client reads it as an empty completion
  and reports success. net/http keeps the first status, so the upstream status
  cannot be forwarded after the park has written one.
- engine_state: new Snapshot() returns the state and its tick channel under one
  lock. Get and Tick taken separately can miss the poll that lands between the
  two calls, parking a request for an extra interval.
- engine_state: observer_degraded now tracks current parsability rather than
  being raised once on the first poll and never cleared, which pinned it at 1
  after a single truncated response and never reported drift that appeared
  later. markDegraded only touches the exporter when the value changes.
- deployment: ADMISSION_MAX_INFLIGHT is set explicitly to 4, documenting the
  coupling to the engine's --max-num-seqs, instead of silently defaulting to
  VLLM_MAX_NUM_SEQS.
This commit is contained in:
Admin Bot
2026-09-30 19:32:10 +08:00
parent c5a91cf647
commit d71c88498b
5 changed files with 80 additions and 8 deletions
+4 -2
View File
@@ -80,7 +80,9 @@ func (b *Backend) acquire(ctx context.Context, onPark func()) error {
defer timer.Stop()
parked := false
for {
st := b.observer.Get()
// State and tick channel are read together under one lock: taking
// them separately can miss a poll that lands between the two calls.
st, tick := b.observer.Snapshot()
if !st.Healthy || (st.InFlight() < int64(b.maxInFlight) && st.KVUsage < b.maxKVUsage) {
break
}
@@ -92,7 +94,7 @@ func (b *Backend) acquire(ctx context.Context, onPark func()) error {
}
}
select {
case <-b.observer.Tick():
case <-tick:
case <-ctx.Done():
return ctx.Err()
case <-timer.C: