- tracing: responseWriter now forwards Flush and exposes Unwrap. Without it
w.(http.Flusher) failed for every request passing the tracing middleware, so
the parked-stream keep-alive never started: a parked streaming request wrote
nothing for up to ADMISSION_MAX_PARK_SECONDS, which is exactly the client
timeout the keep-alive was added to prevent.
- proxy: once the SSE status line is spent, ModifyResponse frames an upstream
response >= 400 as an event: error instead of letting a raw error body land
inside the stream, where a streaming client reads it as an empty completion
and reports success. net/http keeps the first status, so the upstream status
cannot be forwarded after the park has written one.
- engine_state: new Snapshot() returns the state and its tick channel under one
lock. Get and Tick taken separately can miss the poll that lands between the
two calls, parking a request for an extra interval.
- engine_state: observer_degraded now tracks current parsability rather than
being raised once on the first poll and never cleared, which pinned it at 1
after a single truncated response and never reported drift that appeared
later. markDegraded only touches the exporter when the value changes.
- deployment: ADMISSION_MAX_INFLIGHT is set explicitly to 4, documenting the
coupling to the engine's --max-num-seqs, instead of silently defaulting to
VLLM_MAX_NUM_SEQS.
The per-pod semaphore was not a cap: 3 replicas x 4 slots advertised 12
concurrent requests while the engine runs 4, and nothing coordinated the
replicas. Admission is now gated on the engine's own published gauges,
which is global by construction because every replica contends for the
same engine.
- new internal/proxy/engine_state.go: 2s scrape of the backend /metrics,
hand-rolled exposition parse (no new dependency), unhealthy after 3
consecutive failed polls, and a startup self-check that raises
observer_degraded rather than falling back to fail-open silently
- gate: park while kv_cache_usage_perc >= 0.90 or running + waiting >=
ADMISSION_MAX_INFLIGHT, bounded by ADMISSION_MAX_PARK_SECONDS, then
503 + Retry-After: 5. Unknown state fails open on purpose
- streaming requests receive SSE keep-alive heartbeats while parked
- router: extract only the top-level model field instead of unmarshalling
the whole body into a map; body cap from MAX_BODY_BYTES (1 MiB, the
model path previously inherited the 100 MB default)
- GOMAXPROCS from the cgroup CPU quota: the container is limited to 500m
while Go defaulted to the host core count and was CFS-throttled
- VLLM_MAX_QUEUE_DEPTH default 20 -> 8
- delete k8s/configmap.yaml: absent from kustomization.yaml, and the live
ConfigMap of that name carries an unrelated key
- correct the llm-client NetworkPolicy comment; llm-serving-ingress now
selects by namespace, not by that label
Observers run only for models named in ADMISSION_OBSERVE_MODELS.
reasoning is the only vLLM engine here; ornith and qwen-cpu are Ollama
and embeddings/rerank are TEI, none of which publish vllm:* series, so
observing them would hold observer_degraded at 1 permanently.
Companion change (engine flags + dashboard panels) is
riotpiao-poimen/homelab#51.
Admin Bot