feat(proxy): engine-wide LLM admission control via vLLM gauges #65

2026-09-30T09:40:02Z

poimen commented

2026-09-30 09:40:02 +00:00

Closes #64

Replaces the per-pod concurrency semaphore as the effective cap with admission gated on the vLLM engine's own scheduler gauges.

Why

internal/proxy/proxy.go:119 built a semaphore of 4 slots inside each gateway process, and api-gateway runs 3 replicas, so the advertised cap was 12 while the engine's --max-num-seqs is 4. Nothing coordinated replicas, and the mismatch was invisible because the queue never rejected anything (queue_full_requests_total 0).

Measured on the engine while investigating a 10-minute TTFT:

vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"} = 152,432 tokens of KV capacity for the whole engine. PP=2 splits layers, so both ranks hold KV for the same tokens — not 2x that.
Uncached prefill runs at 262 tok/s contended, 525 tok/s over 7d (vllm:request_prefill_time_seconds_sum/count).
One request showed llm_ttft_seconds_sum 590.981 / count 5 at the gateway and a (480, 960] s entry in vllm:request_prefill_time_seconds_bucket, with engine-side generation throughput at 0.0-0.1 tok/s throughout: a 65k-token prompt costs ~250 s of prefill and starves every other request while it runs.
vllm:num_preemptions_total = 0; KV peaked at 0.5399.

Reading the engine's own gauges is global by construction, needs no lock and no new datastore, and it also accounts for callers that never pass the gateway — namespace poimen is admitted directly on 80/8080 by llm-serving-ingress, and memory/crates/mem-cli/src/kv_warmer.rs is such a caller. A distributed lock would have been blind to that traffic.

Changes

new internal/proxy/engine_state.go — 2s scrape of http://<addr>/metrics on its own client (the proxy transports carry socket options tuned for inference), hand-rolled exposition parse, Healthy=false after 3 consecutive failed polls with engine_state_stale_total counted once at the transition, and a startup self-check that raises observer_degraded and logs at error level instead of degrading to fail-open silently.
admission gate (internal/proxy/scheduler.go) — parks while kv_cache_usage_perc >= 0.90 or running + waiting >= ADMISSION_MAX_INFLIGHT, bounded by ADMISSION_MAX_PARK_SECONDS, then 503 + Retry-After: 5. Unknown or unhealthy state fails open on purpose: a metrics outage must not become an inference outage.
parked streaming heartbeat (internal/proxy/proxy.go) — stream: true requests get 200 + text/event-stream and a : keep-alive comment every 15s while parked, so a client idle timeout does not kill a request that is legitimately waiting. Non-streaming requests park silently.
internal/proxy/router.go — extract only the top-level model field with a json.Decoder token walk instead of unmarshalling the whole body into map[string]interface{} (that built the entire messages+tools tree per request, ~300 KB for a 65k-token prompt). MaxBodySize now comes from MAX_BODY_BYTES; the model path previously inherited the 100 MB default at proxy.go:93 against a 512Mi container limit.
cmd/gateway/main.go — GOMAXPROCS from the cgroup CPU quota (cgroup v2 cpu.max, v1 fallback). The container is limited to 500m while Go defaulted to the host core count and was being CFS-throttled.
k8s/deployment.yaml — ADMISSION_MAX_PARK_SECONDS=120, ADMISSION_OBSERVE_MODELS=reasoning, VLLM_MAX_QUEUE_DEPTH=8, MAX_BODY_BYTES=1048576; corrected the llm-client comment (the NetworkPolicy selects by namespace now, not by that label).
deleted k8s/configmap.yaml — absent from k8s/kustomization.yaml, and the live ConfigMap of that name carries an unrelated key (gw-config-fixed.yaml) plus a stale routing table.
internal/observability/ — admission_parked_total, admission_rejected_total{reason}, engine_state_stale_total, and gauges vllm_requests_running / vllm_kv_usage_perc (the gateway's own reading, deliberately mirroring the vLLM series so a disagreement is visible) and observer_degraded. Names follow the existing convention, which carries no prefix.

Out of scope here, in riotpiao-poimen/homelab#51: the engine flags (--max-num-partial-prefills, --max-long-partial-prefills, --long-prefill-token-threshold, --max-num-batched-tokens, --max-num-seqs, --gpu-memory-utilization) and the four dashboard panels in generate-dashboards.py.

Verification

go build ./... clean, go vet ./... clean.
go test ./... — 368 tests, 0 failures (baseline on main is 368/0).
structural-check.sh on the changed files — 0 errors, 1 pre-existing warning (:latest image tag).
go.mod / go.sum byte-identical: no new dependency.

Not yet verified: the real-engine load test (2x65k + 15x8k against reasoning). That run also produces the number for ADMISSION_MAX_PARK_SECONDS, which today is the provisional 120; retuning it is a follow-up since it is env-only and needs no rebuild.

Accepted risks

No tests were added for the observer or the gate (deliberate: production is the test bed). The mitigation is the startup self-check, which is production self-diagnosis rather than a test, plus the observer_degraded alert. A parse failure that outlives the first poll logs and surfaces as a metric; it does not silently disable gating without a signal.
The cap is soft, not hard. Between polls each replica may admit more than one request, so there is no clean bound. Exposure is low (~11,500 requests over 7d ≈ 1.4/min) and the engine remains the hard limit — a race parks requests in vLLM's queue rather than over-committing KV.
ADMISSION_MAX_INFLIGHT is manually coupled to --max-num-seqs and can drift silently. It defaults to VLLM_MAX_NUM_SEQS, and startup logs the effective value.
The first scrape waits one interval, so the first ~2s of admission is ungated (fail-open). Scraping at Start() instead would make the gateway contact every observed backend before any request needs one, which is wasted work and a surprising side effect.

Rollback

Revert this commit and redeploy. The admission path is fail-open, so the code can also be neutralised without reverting by unsetting ADMISSION_OBSERVE_MODELS. Separately, the orphaned cluster object can be removed by hand: kubectl delete cm -n api api-gateway-config (not a code change, deliberately left to a human).

Closes #64 Replaces the per-pod concurrency semaphore as the effective cap with admission gated on the vLLM engine's own scheduler gauges. ## Why `internal/proxy/proxy.go:119` built a semaphore of 4 slots *inside each gateway process*, and `api-gateway` runs 3 replicas, so the advertised cap was 12 while the engine's `--max-num-seqs` is 4. Nothing coordinated replicas, and the mismatch was invisible because the queue never rejected anything (`queue_full_requests_total 0`). Measured on the engine while investigating a 10-minute TTFT: - `vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}` = **152,432 tokens** of KV capacity for the whole engine. PP=2 splits layers, so both ranks hold KV for the same tokens — not 2x that. - Uncached prefill runs at **262 tok/s** contended, 525 tok/s over 7d (`vllm:request_prefill_time_seconds_sum/count`). - One request showed `llm_ttft_seconds_sum 590.981 / count 5` at the gateway and a `(480, 960]` s entry in `vllm:request_prefill_time_seconds_bucket`, with engine-side generation throughput at 0.0-0.1 tok/s throughout: a 65k-token prompt costs ~250 s of prefill and starves every other request while it runs. - `vllm:num_preemptions_total = 0`; KV peaked at `0.5399`. Reading the engine's own gauges is global by construction, needs no lock and no new datastore, and it also accounts for callers that never pass the gateway — namespace `poimen` is admitted directly on 80/8080 by `llm-serving-ingress`, and `memory/crates/mem-cli/src/kv_warmer.rs` is such a caller. A distributed lock would have been blind to that traffic. ## Changes - **new `internal/proxy/engine_state.go`** — 2s scrape of `http://<addr>/metrics` on its own client (the proxy transports carry socket options tuned for inference), hand-rolled exposition parse, `Healthy=false` after 3 consecutive failed polls with `engine_state_stale_total` counted once at the transition, and a startup self-check that raises `observer_degraded` and logs at error level instead of degrading to fail-open silently. - **admission gate** (`internal/proxy/scheduler.go`) — parks while `kv_cache_usage_perc >= 0.90` or `running + waiting >= ADMISSION_MAX_INFLIGHT`, bounded by `ADMISSION_MAX_PARK_SECONDS`, then `503 + Retry-After: 5`. Unknown or unhealthy state fails **open** on purpose: a metrics outage must not become an inference outage. - **parked streaming heartbeat** (`internal/proxy/proxy.go`) — `stream: true` requests get 200 + `text/event-stream` and a `: keep-alive` comment every 15s while parked, so a client idle timeout does not kill a request that is legitimately waiting. Non-streaming requests park silently. - **`internal/proxy/router.go`** — extract only the top-level `model` field with a `json.Decoder` token walk instead of unmarshalling the whole body into `map[string]interface{}` (that built the entire `messages`+`tools` tree per request, ~300 KB for a 65k-token prompt). `MaxBodySize` now comes from `MAX_BODY_BYTES`; the model path previously inherited the 100 MB default at `proxy.go:93` against a 512Mi container limit. - **`cmd/gateway/main.go`** — `GOMAXPROCS` from the cgroup CPU quota (cgroup v2 `cpu.max`, v1 fallback). The container is limited to `500m` while Go defaulted to the host core count and was being CFS-throttled. - **`k8s/deployment.yaml`** — `ADMISSION_MAX_PARK_SECONDS=120`, `ADMISSION_OBSERVE_MODELS=reasoning`, `VLLM_MAX_QUEUE_DEPTH=8`, `MAX_BODY_BYTES=1048576`; corrected the `llm-client` comment (the NetworkPolicy selects by namespace now, not by that label). - **deleted `k8s/configmap.yaml`** — absent from `k8s/kustomization.yaml`, and the live ConfigMap of that name carries an unrelated key (`gw-config-fixed.yaml`) plus a stale routing table. - **`internal/observability/`** — `admission_parked_total`, `admission_rejected_total{reason}`, `engine_state_stale_total`, and gauges `vllm_requests_running` / `vllm_kv_usage_perc` (the gateway's own reading, deliberately mirroring the vLLM series so a disagreement is visible) and `observer_degraded`. Names follow the existing convention, which carries no prefix. Out of scope here, in `riotpiao-poimen/homelab#51`: the engine flags (`--max-num-partial-prefills`, `--max-long-partial-prefills`, `--long-prefill-token-threshold`, `--max-num-batched-tokens`, `--max-num-seqs`, `--gpu-memory-utilization`) and the four dashboard panels in `generate-dashboards.py`. ## Verification - `go build ./...` clean, `go vet ./...` clean. - `go test ./...` — 368 tests, 0 failures (baseline on `main` is 368/0). - `structural-check.sh` on the changed files — 0 errors, 1 pre-existing warning (`:latest` image tag). - `go.mod` / `go.sum` byte-identical: no new dependency. Not yet verified: the real-engine load test (2x65k + 15x8k against `reasoning`). That run also produces the number for `ADMISSION_MAX_PARK_SECONDS`, which today is the provisional 120; retuning it is a follow-up since it is env-only and needs no rebuild. ## Accepted risks - **No tests were added** for the observer or the gate (deliberate: production is the test bed). The mitigation is the startup self-check, which is production self-diagnosis rather than a test, plus the `observer_degraded` alert. A parse failure that outlives the first poll logs and surfaces as a metric; it does not silently disable gating without a signal. - **The cap is soft, not hard.** Between polls each replica may admit more than one request, so there is no clean bound. Exposure is low (~11,500 requests over 7d ≈ 1.4/min) and the engine remains the hard limit — a race parks requests in vLLM's queue rather than over-committing KV. - **`ADMISSION_MAX_INFLIGHT` is manually coupled to `--max-num-seqs`** and can drift silently. It defaults to `VLLM_MAX_NUM_SEQS`, and startup logs the effective value. - The first scrape waits one interval, so the first ~2s of admission is ungated (fail-open). Scraping at `Start()` instead would make the gateway contact every observed backend before any request needs one, which is wasted work and a surprising side effect. ## Rollback Revert this commit and redeploy. The admission path is fail-open, so the code can also be neutralised without reverting by unsetting `ADMISSION_OBSERVE_MODELS`. Separately, the orphaned cluster object can be removed by hand: `kubectl delete cm -n api api-gateway-config` (not a code change, deliberately left to a human).

poimen added 1 commit 2026-09-30 09:40:03 +00:00

feat(proxy): engine-wide LLM admission control via vLLM gauges (#64 )

CI / CI (pull_request) Successful in 34m5s

Details

c5a91cf647

The per-pod semaphore was not a cap: 3 replicas x 4 slots advertised 12
concurrent requests while the engine runs 4, and nothing coordinated the
replicas. Admission is now gated on the engine's own published gauges,
which is global by construction because every replica contends for the
same engine.

- new internal/proxy/engine_state.go: 2s scrape of the backend /metrics,
  hand-rolled exposition parse (no new dependency), unhealthy after 3
  consecutive failed polls, and a startup self-check that raises
  observer_degraded rather than falling back to fail-open silently
- gate: park while kv_cache_usage_perc >= 0.90 or running + waiting >=
  ADMISSION_MAX_INFLIGHT, bounded by ADMISSION_MAX_PARK_SECONDS, then
  503 + Retry-After: 5. Unknown state fails open on purpose
- streaming requests receive SSE keep-alive heartbeats while parked
- router: extract only the top-level model field instead of unmarshalling
  the whole body into a map; body cap from MAX_BODY_BYTES (1 MiB, the
  model path previously inherited the 100 MB default)
- GOMAXPROCS from the cgroup CPU quota: the container is limited to 500m
  while Go defaulted to the host core count and was CFS-throttled
- VLLM_MAX_QUEUE_DEPTH default 20 -> 8
- delete k8s/configmap.yaml: absent from kustomization.yaml, and the live
  ConfigMap of that name carries an unrelated key
- correct the llm-client NetworkPolicy comment; llm-serving-ingress now
  selects by namespace, not by that label

Observers run only for models named in ADMISSION_OBSERVE_MODELS.
reasoning is the only vLLM engine here; ornith and qwen-cpu are Ollama
and embeddings/rerank are TEI, none of which publish vllm:* series, so
observing them would hold observer_degraded at 1 permanently.

Companion change (engine flags + dashboard panels) is
riotpiao-poimen/homelab#51.

rock merged commit 369d51a5c2 into main

2026-09-30 10:56:12 +00:00

rock deleted branch feat/cag4-cluster-admission-64

2026-09-30 10:56:13 +00:00

poimen referenced this pull request

2026-09-30 11:44:06 +00:00

fix(proxy): admission-control review fixes missing from the #65 merge #66

rock referenced this issue from a commit

2026-09-30 12:19:26 +00:00

fix(proxy): admission-control review fixes missing from the #65 merge (#66)

Sign in to join this conversation.