[Infra] reasoning: vLLM prefill scheduling flags + admission-control dashboard panels (homelab-frontend#64) #51

New Issue

2026-09-30T09:27:31Z

poimen commented

2026-09-30 09:27:31 +00:00

Companion to homelab-frontend#64 (CAG-4: cluster-wide LLM admission control). That issue adds gateway-side admission control; this one changes the engine and the dashboard. Both are needed for the same outcome and neither is sufficient alone.

Why

Measured on reasoning (Qwen3-32B-GPTQ-Int4, vLLM 0.11.0, 2x V100-32GB, PP=2):

Engine KV capacity is 152,432 tokens — vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}. PP=2 splits layers, so both ranks hold KV for the same tokens; the capacity is not 2x that number.
Uncached prefill runs at 262 tok/s contended (vllm:request_prefill_time_seconds_sum/count, 1h window), 525 tok/s over 7d.
A ~65k-token prompt therefore costs ~250 s of prefill, and during it the engine reported Avg generation throughput: 0.0-0.1 tokens/s with Running: 2-4 reqs — every other request is starved while one prompt prefills. vllm:request_prefill_time_seconds_bucket has one request in the (480, 960] s bucket, matching a gateway-observed llm_ttft_seconds_sum 590.981 / count 5 (one request ≈ 580 s TTFT).
vllm:num_preemptions_total = 0 today; the engine peaked at kv_cache_usage_perc = 0.5399.

Root cause is scheduling, not capacity: vLLM 0.11.0 defaults max_num_partial_prefills = 1 and max_long_partial_prefills = 1 (vllm/config/scheduler.py:49-59), so exactly one request may prefill at a time and a long one owns the prefill path end to end. --max-num-batched-tokens is unset, so it falls back to DEFAULT_MAX_NUM_BATCHED_TOKENS = 2048 (vllm/utils/__init__.py:88 via scheduler.py:172-173).

Change 1 — `k8s/apps/llm-serving/reasoning.yaml`

OLD:      - --max-num-seqs=4
          - --enable-chunked-prefill
          - --enable-prefix-caching
NEW:      - --max-num-seqs=5
          - --max-num-batched-tokens=4096
          - --max-num-partial-prefills=2
          - --max-long-partial-prefills=2
          - --long-prefill-token-threshold=8192
          - --enable-chunked-prefill
          - --enable-prefix-caching

plus --gpu-memory-utilization=0.90 -> 0.95.

Rationale, one line each: --max-long-partial-prefills=2 is the flag that makes two large prompts co-exist (default 1 serialises them); --max-num-partial-prefills=2 lifts the single-prefill-at-a-time default; --long-prefill-token-threshold=8192 classifies a 65k prompt as long (explicit beats the auto-derived int(131072*0.04)=5242 at scheduler.py:210-213); --max-num-batched-tokens=4096 makes chunking explicit and keeps decode scheduled in every step; --gpu-memory-utilization=0.95 buys the fifth sequence slot (2 large + 3 small = 161k = 97.4% of KV), since at 0.90 the KV-legal maximum is 4.

Do not raise --max-num-seqs to 16: the target mix needs 167k tokens against 152,432 available (109.6%), and preempting a 65k prompt means re-prefilling it from zero, ~250 s.

Change 2 — `k8s/infra/monitoring/dashboards/generate-dashboards.py`

Four panels in build_llm_observability(). The generated file llm-observability.yaml is written by generate-dashboards.py:464, so editing the YAML directly is overwritten on the next run.

Admission parked (5m) — increase(admission_parked_total[5m])
Admission rejected by reason — sum by (reason) (increase(admission_rejected_total[5m]))
Engine state stale — increase(engine_state_stale_total[5m])
Gateway view vs engine truth — overlay vllm_requests_running / vllm_kv_usage_perc (the gateway's own reading) against vllm:num_requests_running / vllm:kv_cache_usage_perc. A disagreement between those pairs is how a stale observer or a mistuned target shows itself, and nothing else will reveal it.

Series names come from homelab-frontend#64's Q14 decision: the existing gateway series carry no prefix (llm_ttft_seconds, queue_full_requests_total, warmup_requests_delayed_total), so the new ones follow that convention.

Change 3 — `k8s/infra/monitoring/alerts/api-gateway-alerts.yaml`

Alert on observer_degraded == 1 and on engine_state_stale_total rising. The gateway's admission gate fails open when the engine gauges go stale, which means a broken observer silently degrades to today's behaviour (admit everything). The metric is the only detection since there are no automated tests.

Acceptance

Two concurrent 65k-token prompts both start prefilling (observable as two entries in the prefill path, not one), and small requests keep generating during it.
vllm:num_preemptions_total stays 0 through a 2x65k + 15x8k run.
vllm:kv_cache_usage_perc stays below 0.95.
Gateway-side llm_ttft_seconds p95 for small prompts drops from the starvation case (~570 s observed) to tens of seconds.
The four panels render, and the gateway-view vs engine-truth pair agree.

Notes

Rollout is Recreate with a 20 min startupProbe budget, and higher --max-num-seqs grows CUDA-graph capture (scheduler.py:224, cuda_graph_sizes = [min(max_num_seqs*2, 512)]). Load-test after the rollout, not during.
Rollback is removing the five added flags and restoring --max-num-seqs=4 / --gpu-memory-utilization=0.90.
Prefix-cache discipline is the other half of the fix and is client-side: with a byte-identical system + tools prefix the ~45k static part is cached, cutting both prefill time and KV cost for the second and later calls. enable_prefix_caching=True is already set and the hit rate has been observed between 22% and 99.96%, so this is under-exploited rather than broken.

Companion to **homelab-frontend#64** (CAG-4: cluster-wide LLM admission control). That issue adds gateway-side admission control; this one changes the engine and the dashboard. Both are needed for the same outcome and neither is sufficient alone. ## Why Measured on `reasoning` (Qwen3-32B-GPTQ-Int4, vLLM 0.11.0, 2x V100-32GB, PP=2): - Engine KV capacity is **152,432 tokens** — `vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}`. PP=2 splits layers, so both ranks hold KV for the *same* tokens; the capacity is not 2x that number. - Uncached prefill runs at **262 tok/s** contended (`vllm:request_prefill_time_seconds_sum/count`, 1h window), 525 tok/s over 7d. - A ~65k-token prompt therefore costs **~250 s of prefill**, and during it the engine reported `Avg generation throughput: 0.0-0.1 tokens/s` with `Running: 2-4 reqs` — every other request is starved while one prompt prefills. `vllm:request_prefill_time_seconds_bucket` has one request in the `(480, 960]` s bucket, matching a gateway-observed `llm_ttft_seconds_sum 590.981 / count 5` (one request ≈ 580 s TTFT). - `vllm:num_preemptions_total = 0` today; the engine peaked at `kv_cache_usage_perc = 0.5399`. Root cause is scheduling, not capacity: vLLM 0.11.0 defaults `max_num_partial_prefills = 1` and `max_long_partial_prefills = 1` (`vllm/config/scheduler.py:49-59`), so exactly one request may prefill at a time and a long one owns the prefill path end to end. `--max-num-batched-tokens` is unset, so it falls back to `DEFAULT_MAX_NUM_BATCHED_TOKENS = 2048` (`vllm/utils/__init__.py:88` via `scheduler.py:172-173`). ## Change 1 — `k8s/apps/llm-serving/reasoning.yaml` ``` OLD: - --max-num-seqs=4 - --enable-chunked-prefill - --enable-prefix-caching NEW: - --max-num-seqs=5 - --max-num-batched-tokens=4096 - --max-num-partial-prefills=2 - --max-long-partial-prefills=2 - --long-prefill-token-threshold=8192 - --enable-chunked-prefill - --enable-prefix-caching ``` plus `--gpu-memory-utilization=0.90` -> `0.95`. Rationale, one line each: `--max-long-partial-prefills=2` is the flag that makes two large prompts co-exist (default 1 serialises them); `--max-num-partial-prefills=2` lifts the single-prefill-at-a-time default; `--long-prefill-token-threshold=8192` classifies a 65k prompt as long (explicit beats the auto-derived `int(131072*0.04)=5242` at `scheduler.py:210-213`); `--max-num-batched-tokens=4096` makes chunking explicit and keeps decode scheduled in every step; `--gpu-memory-utilization=0.95` buys the fifth sequence slot (2 large + 3 small = 161k = 97.4% of KV), since at 0.90 the KV-legal maximum is 4. Do **not** raise `--max-num-seqs` to 16: the target mix needs 167k tokens against 152,432 available (109.6%), and preempting a 65k prompt means re-prefilling it from zero, ~250 s. ## Change 2 — `k8s/infra/monitoring/dashboards/generate-dashboards.py` Four panels in `build_llm_observability()`. The generated file `llm-observability.yaml` is written by `generate-dashboards.py:464`, so editing the YAML directly is overwritten on the next run. - `Admission parked (5m)` — `increase(admission_parked_total[5m])` - `Admission rejected by reason` — `sum by (reason) (increase(admission_rejected_total[5m]))` - `Engine state stale` — `increase(engine_state_stale_total[5m])` - Gateway view vs engine truth — overlay `vllm_requests_running` / `vllm_kv_usage_perc` (the gateway's own reading) against `vllm:num_requests_running` / `vllm:kv_cache_usage_perc`. A disagreement between those pairs is how a stale observer or a mistuned target shows itself, and nothing else will reveal it. Series names come from homelab-frontend#64's Q14 decision: the existing gateway series carry no prefix (`llm_ttft_seconds`, `queue_full_requests_total`, `warmup_requests_delayed_total`), so the new ones follow that convention. ## Change 3 — `k8s/infra/monitoring/alerts/api-gateway-alerts.yaml` Alert on `observer_degraded == 1` and on `engine_state_stale_total` rising. The gateway's admission gate fails **open** when the engine gauges go stale, which means a broken observer silently degrades to today's behaviour (admit everything). The metric is the only detection since there are no automated tests. ## Acceptance - Two concurrent 65k-token prompts both start prefilling (observable as two entries in the prefill path, not one), and small requests keep generating during it. - `vllm:num_preemptions_total` stays 0 through a 2x65k + 15x8k run. - `vllm:kv_cache_usage_perc` stays below 0.95. - Gateway-side `llm_ttft_seconds` p95 for small prompts drops from the starvation case (~570 s observed) to tens of seconds. - The four panels render, and the gateway-view vs engine-truth pair agree. ## Notes - Rollout is `Recreate` with a 20 min startupProbe budget, and higher `--max-num-seqs` grows CUDA-graph capture (`scheduler.py:224`, `cuda_graph_sizes = [min(max_num_seqs*2, 512)]`). Load-test after the rollout, not during. - Rollback is removing the five added flags and restoring `--max-num-seqs=4` / `--gpu-memory-utilization=0.90`. - Prefix-cache discipline is the other half of the fix and is client-side: with a byte-identical system + tools prefix the ~45k static part is cached, cutting both prefill time and KV cost for the second and later calls. `enable_prefix_caching=True` is already set and the hit rate has been observed between 22% and 99.96%, so this is under-exploited rather than broken.

poimen added this to the LLM Production Hardening milestone 2026-09-30 09:27:31 +00:00