homelab-frontend

riotpiao-poimen/homelab-frontend

Fork 0

Author	SHA1	Message	Date
Admin Bot	d71c88498b	fix(proxy): address review findings on admission control (#64 ) - tracing: responseWriter now forwards Flush and exposes Unwrap. Without it w.(http.Flusher) failed for every request passing the tracing middleware, so the parked-stream keep-alive never started: a parked streaming request wrote nothing for up to ADMISSION_MAX_PARK_SECONDS, which is exactly the client timeout the keep-alive was added to prevent. - proxy: once the SSE status line is spent, ModifyResponse frames an upstream response >= 400 as an event: error instead of letting a raw error body land inside the stream, where a streaming client reads it as an empty completion and reports success. net/http keeps the first status, so the upstream status cannot be forwarded after the park has written one. - engine_state: new Snapshot() returns the state and its tick channel under one lock. Get and Tick taken separately can miss the poll that lands between the two calls, parking a request for an extra interval. - engine_state: observer_degraded now tracks current parsability rather than being raised once on the first poll and never cleared, which pinned it at 1 after a single truncated response and never reported drift that appeared later. markDegraded only touches the exporter when the value changes. - deployment: ADMISSION_MAX_INFLIGHT is set explicitly to 4, documenting the coupling to the engine's --max-num-seqs, instead of silently defaulting to VLLM_MAX_NUM_SEQS.	2026-09-30 19:32:10 +08:00
Admin Bot	c5a91cf647	feat(proxy): engine-wide LLM admission control via vLLM gauges (#64 ) CI / CI (pull_request) Successful in 34m5s Details The per-pod semaphore was not a cap: 3 replicas x 4 slots advertised 12 concurrent requests while the engine runs 4, and nothing coordinated the replicas. Admission is now gated on the engine's own published gauges, which is global by construction because every replica contends for the same engine. - new internal/proxy/engine_state.go: 2s scrape of the backend /metrics, hand-rolled exposition parse (no new dependency), unhealthy after 3 consecutive failed polls, and a startup self-check that raises observer_degraded rather than falling back to fail-open silently - gate: park while kv_cache_usage_perc >= 0.90 or running + waiting >= ADMISSION_MAX_INFLIGHT, bounded by ADMISSION_MAX_PARK_SECONDS, then 503 + Retry-After: 5. Unknown state fails open on purpose - streaming requests receive SSE keep-alive heartbeats while parked - router: extract only the top-level model field instead of unmarshalling the whole body into a map; body cap from MAX_BODY_BYTES (1 MiB, the model path previously inherited the 100 MB default) - GOMAXPROCS from the cgroup CPU quota: the container is limited to 500m while Go defaulted to the host core count and was CFS-throttled - VLLM_MAX_QUEUE_DEPTH default 20 -> 8 - delete k8s/configmap.yaml: absent from kustomization.yaml, and the live ConfigMap of that name carries an unrelated key - correct the llm-client NetworkPolicy comment; llm-serving-ingress now selects by namespace, not by that label Observers run only for models named in ADMISSION_OBSERVE_MODELS. reasoning is the only vLLM engine here; ornith and qwen-cpu are Ollama and embeddings/rerank are TEI, none of which publish vllm:* series, so observing them would hold observer_degraded at 1 permanently. Companion change (engine flags + dashboard panels) is riotpiao-poimen/homelab#51.	2026-09-30 17:39:47 +08:00

Author

SHA1

Message

Date

Admin Bot

d71c88498b

fix(proxy): address review findings on admission control (#64 )

- tracing: responseWriter now forwards Flush and exposes Unwrap. Without it
  w.(http.Flusher) failed for every request passing the tracing middleware, so
  the parked-stream keep-alive never started: a parked streaming request wrote
  nothing for up to ADMISSION_MAX_PARK_SECONDS, which is exactly the client
  timeout the keep-alive was added to prevent.
- proxy: once the SSE status line is spent, ModifyResponse frames an upstream
  response >= 400 as an event: error instead of letting a raw error body land
  inside the stream, where a streaming client reads it as an empty completion
  and reports success. net/http keeps the first status, so the upstream status
  cannot be forwarded after the park has written one.
- engine_state: new Snapshot() returns the state and its tick channel under one
  lock. Get and Tick taken separately can miss the poll that lands between the
  two calls, parking a request for an extra interval.
- engine_state: observer_degraded now tracks current parsability rather than
  being raised once on the first poll and never cleared, which pinned it at 1
  after a single truncated response and never reported drift that appeared
  later. markDegraded only touches the exporter when the value changes.
- deployment: ADMISSION_MAX_INFLIGHT is set explicitly to 4, documenting the
  coupling to the engine's --max-num-seqs, instead of silently defaulting to
  VLLM_MAX_NUM_SEQS.

2026-09-30 19:32:10 +08:00

Admin Bot

c5a91cf647

feat(proxy): engine-wide LLM admission control via vLLM gauges (#64 )

CI / CI (pull_request) Successful in 34m5s

Details

The per-pod semaphore was not a cap: 3 replicas x 4 slots advertised 12
concurrent requests while the engine runs 4, and nothing coordinated the
replicas. Admission is now gated on the engine's own published gauges,
which is global by construction because every replica contends for the
same engine.

- new internal/proxy/engine_state.go: 2s scrape of the backend /metrics,
  hand-rolled exposition parse (no new dependency), unhealthy after 3
  consecutive failed polls, and a startup self-check that raises
  observer_degraded rather than falling back to fail-open silently
- gate: park while kv_cache_usage_perc >= 0.90 or running + waiting >=
  ADMISSION_MAX_INFLIGHT, bounded by ADMISSION_MAX_PARK_SECONDS, then
  503 + Retry-After: 5. Unknown state fails open on purpose
- streaming requests receive SSE keep-alive heartbeats while parked
- router: extract only the top-level model field instead of unmarshalling
  the whole body into a map; body cap from MAX_BODY_BYTES (1 MiB, the
  model path previously inherited the 100 MB default)
- GOMAXPROCS from the cgroup CPU quota: the container is limited to 500m
  while Go defaulted to the host core count and was CFS-throttled
- VLLM_MAX_QUEUE_DEPTH default 20 -> 8
- delete k8s/configmap.yaml: absent from kustomization.yaml, and the live
  ConfigMap of that name carries an unrelated key
- correct the llm-client NetworkPolicy comment; llm-serving-ingress now
  selects by namespace, not by that label

Observers run only for models named in ADMISSION_OBSERVE_MODELS.
reasoning is the only vLLM engine here; ornith and qwen-cpu are Ollama
and embeddings/rerank are TEI, none of which publish vllm:* series, so
observing them would hold observer_degraded at 1 permanently.

Companion change (engine flags + dashboard panels) is
riotpiao-poimen/homelab#51.

2026-09-30 17:39:47 +08:00

1 changed files with 1 additions and 7 deletions

									
										internal/proxy/proxy.go
									
		+1
		-7
	
												View File
												
					@@ -797,14 +797,8 @@ func (h *Handler) setupCAGProxy(ctx context.Context, w http.ResponseWriter, r *h

								if resp.StatusCode < http.StatusBadRequest {

								if resp.StatusCode < http.StatusBadRequest {

									return nil

									return nil

								}

								}

								body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))

								body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))

								resp.Body.Close()

								resp.Body.Close()

								if readErr != nil {

									logging.Warnf("reading upstream error body failed", map[string]string{

										"upstream": capturedRoute.Upstream.Address,

										"err":      readErr.Error(),

									})

								}

								logging.Warnf("upstream error after parked stream started", map[string]string{

								logging.Warnf("upstream error after parked stream started", map[string]string{

									"upstream": capturedRoute.Upstream.Address,

									"upstream": capturedRoute.Upstream.Address,

									"status":   strconv.Itoa(resp.StatusCode),

									"status":   strconv.Itoa(resp.StatusCode),

Compare commits