Companion to homelab-frontend#64 (admission control), which is already deployed (@sha256:64f2bfc1), so the new panels have live data.
The fix is not what the issue specified
The issue proposed --max-num-partial-prefills=2 / --max-long-partial-prefills=2. Both crash the pod. Verified in the live container:
vLLM 0.11.0 ships V1 only — vllm/core/ (V0) does not exist, and VLLM_USE_V1 is unset.
EngineArgs._is_v1_supported_oracle returns False for any non-default value of either flag, then _raise_or_fallback("Concurrent Partial Prefill") prints "Falling back to V0" — to an engine that isn't in the image.
Strategy is Recreate, so that is an outage, not a slow rollout.
V1 needs no such flag: it has no prefill/decode phase at all ("There's no decoding phase nor prefill phase in the scheduler"). max_num_partial_prefills appears 0 times in V1's scheduler.
The actual defect
V1 walks self.running and hands out tokens from one shared budget, applying the per-request cap before the budget clamp:
With the threshold at its default of 0 the cap is disabled, so one long prompt takes the entire budget, token_budget hits 0, the while ... and token_budget > 0 loop exits, and every other request is skipped — for all 32 steps. That is the measured 0.0–0.1 tok/s.
Because the cap precedes the clamp, threshold >= budget never binds. Replay of the loop on 2×65k + 3×8k + small:
budget
threshold
requests served per step
2048
0
1 ← today
8192
8192
1 ← still starved
8192
2048
4✅
16384
8192
2
Changes
reasoning.yaml: --max-num-batched-tokens=8192, --long-prefill-token-threshold=2048, --max-num-seqs=5, --gpu-memory-utilization=0.95. The two V0-only flags are not set, with a comment recording why so they are not reintroduced.
Dashboard: four admission panels, each with a description (an unexplained blank panel reads as a broken dashboard).
Alerts: observer_degraded, engine_state_stale_total (both with an absent() guard — the gate fails open, so a missing metric must alert), plus LLMPreemptions (the acceptance metric) and LLMPrefillStarvation (the regression direction this change introduces).
sum(vllm_requests_running) → max(...): one engine-global gauge is read by all three gateway replicas, so sum() read 3× and could never agree with the engine.
kubectl kustomize renders for both llm-serving and monitoring
All 9 new PromQL expressions and 15 alert expressions return status: success against live Prometheus
Both absent() guards confirmed silently non-firing today (series present, 3 each) — not false-firing, as one review initially claimed
Generator is idempotent: regenerating into a temp dir reproduces the committed YAML byte-for-byte
Accepted risk
--gpu-memory-utilization=0.95 leaves roughly 322 MiB of headroom per GPU (30808/32768 MiB in use at 0.90), and max_num_batched_tokens=8192 raises activation memory out of the same budget. The ~166k KV figure is marked in-comment as a projection; the authoritative number is vllm:cache_config_infonum_gpu_blocks read after boot. LLMPreemptions covers it — evicting a 65k prompt costs a full re-prefill (~250s).
Review
reviewer-sre: SIGN-OFF (0🔴) · reviewer-quality: SIGN-OFF (0🔴). The reviewers initially disagreed on whether the V0 flags crash; adjudicated against the live container, which confirmed they do.
Closes #51
Companion to homelab-frontend#64 (admission control), which is **already deployed** (`@sha256:64f2bfc1`), so the new panels have live data.
## The fix is not what the issue specified
The issue proposed `--max-num-partial-prefills=2` / `--max-long-partial-prefills=2`. **Both crash the pod.** Verified in the live container:
- vLLM 0.11.0 ships **V1 only** — `vllm/core/` (V0) does not exist, and `VLLM_USE_V1` is unset.
- `EngineArgs._is_v1_supported_oracle` returns `False` for any non-default value of either flag, then `_raise_or_fallback("Concurrent Partial Prefill")` prints *"Falling back to V0"* — to an engine that isn't in the image.
- Strategy is `Recreate`, so that is an **outage**, not a slow rollout.
V1 needs no such flag: it has **no prefill/decode phase at all** (`"There's no decoding phase nor prefill phase in the scheduler"`). `max_num_partial_prefills` appears **0 times** in V1's scheduler.
## The actual defect
V1 walks `self.running` and hands out tokens from one shared budget, applying the per-request cap *before* the budget clamp:
```python
if 0 < long_prefill_token_threshold < num_new_tokens: num_new_tokens = threshold
num_new_tokens = min(num_new_tokens, token_budget)
```
With the threshold at its **default of 0 the cap is disabled**, so one long prompt takes the entire budget, `token_budget` hits 0, the `while ... and token_budget > 0` loop exits, and every other request is skipped — for all 32 steps. That is the measured `0.0–0.1 tok/s`.
Because the cap precedes the clamp, **`threshold >= budget` never binds**. Replay of the loop on `2×65k + 3×8k + small`:
| budget | threshold | requests served per step |
|---|---|---|
| 2048 | 0 | 1 ← today |
| 8192 | 8192 | 1 ← still starved |
| **8192** | **2048** | **4** ✅ |
| 16384 | 8192 | 2 |
## Changes
`reasoning.yaml`: `--max-num-batched-tokens=8192`, `--long-prefill-token-threshold=2048`, `--max-num-seqs=5`, `--gpu-memory-utilization=0.95`. The two V0-only flags are **not set**, with a comment recording why so they are not reintroduced.
Dashboard: four admission panels, each with a `description` (an unexplained blank panel reads as a broken dashboard).
Alerts: `observer_degraded`, `engine_state_stale_total` (both with an `absent()` guard — the gate fails **open**, so a missing metric must alert), plus `LLMPreemptions` (the acceptance metric) and `LLMPrefillStarvation` (the regression direction this change introduces).
`sum(vllm_requests_running)` → `max(...)`: one engine-global gauge is read by all three gateway replicas, so `sum()` read 3× and could never agree with the engine.
## Verification
- Structural gate: `0 error, 0 warning` (notes: no rule covers python)
- `kubectl kustomize` renders for both `llm-serving` and `monitoring`
- All 9 new PromQL expressions and 15 alert expressions return `status: success` against live Prometheus
- Both `absent()` guards confirmed **silently non-firing** today (series present, 3 each) — not false-firing, as one review initially claimed
- Generator is idempotent: regenerating into a temp dir reproduces the committed YAML byte-for-byte
## Accepted risk
`--gpu-memory-utilization=0.95` leaves roughly 322 MiB of headroom per GPU (30808/32768 MiB in use at 0.90), and `max_num_batched_tokens=8192` raises activation memory out of the same budget. The `~166k` KV figure is marked in-comment as a **projection**; the authoritative number is `vllm:cache_config_info` `num_gpu_blocks` read **after** boot. `LLMPreemptions` covers it — evicting a 65k prompt costs a full re-prefill (~250s).
## Review
reviewer-sre: **SIGN-OFF** (0🔴) · reviewer-quality: **SIGN-OFF** (0🔴). The reviewers initially disagreed on whether the V0 flags crash; adjudicated against the live container, which confirmed they do.
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes#51
rock
merged commit d60b520968 into main2026-09-30 11:49:55 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #51
Companion to homelab-frontend#64 (admission control), which is already deployed (
@sha256:64f2bfc1), so the new panels have live data.The fix is not what the issue specified
The issue proposed
--max-num-partial-prefills=2/--max-long-partial-prefills=2. Both crash the pod. Verified in the live container:vllm/core/(V0) does not exist, andVLLM_USE_V1is unset.EngineArgs._is_v1_supported_oraclereturnsFalsefor any non-default value of either flag, then_raise_or_fallback("Concurrent Partial Prefill")prints "Falling back to V0" — to an engine that isn't in the image.Recreate, so that is an outage, not a slow rollout.V1 needs no such flag: it has no prefill/decode phase at all (
"There's no decoding phase nor prefill phase in the scheduler").max_num_partial_prefillsappears 0 times in V1's scheduler.The actual defect
V1 walks
self.runningand hands out tokens from one shared budget, applying the per-request cap before the budget clamp:With the threshold at its default of 0 the cap is disabled, so one long prompt takes the entire budget,
token_budgethits 0, thewhile ... and token_budget > 0loop exits, and every other request is skipped — for all 32 steps. That is the measured0.0–0.1 tok/s.Because the cap precedes the clamp,
threshold >= budgetnever binds. Replay of the loop on2×65k + 3×8k + small:Changes
reasoning.yaml:--max-num-batched-tokens=8192,--long-prefill-token-threshold=2048,--max-num-seqs=5,--gpu-memory-utilization=0.95. The two V0-only flags are not set, with a comment recording why so they are not reintroduced.Dashboard: four admission panels, each with a
description(an unexplained blank panel reads as a broken dashboard).Alerts:
observer_degraded,engine_state_stale_total(both with anabsent()guard — the gate fails open, so a missing metric must alert), plusLLMPreemptions(the acceptance metric) andLLMPrefillStarvation(the regression direction this change introduces).sum(vllm_requests_running)→max(...): one engine-global gauge is read by all three gateway replicas, sosum()read 3× and could never agree with the engine.Verification
0 error, 0 warning(notes: no rule covers python)kubectl kustomizerenders for bothllm-servingandmonitoringstatus: successagainst live Prometheusabsent()guards confirmed silently non-firing today (series present, 3 each) — not false-firing, as one review initially claimedAccepted risk
--gpu-memory-utilization=0.95leaves roughly 322 MiB of headroom per GPU (30808/32768 MiB in use at 0.90), andmax_num_batched_tokens=8192raises activation memory out of the same budget. The~166kKV figure is marked in-comment as a projection; the authoritative number isvllm:cache_config_infonum_gpu_blocksread after boot.LLMPreemptionscovers it — evicting a 65k prompt costs a full re-prefill (~250s).Review
reviewer-sre: SIGN-OFF (0🔴) · reviewer-quality: SIGN-OFF (0🔴). The reviewers initially disagreed on whether the V0 flags crash; adjudicated against the live container, which confirmed they do.
The reasoning engine starves every other request while one long prompt prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt, with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs. Root cause is NOT what the issue first diagnosed. The issue proposed --max-num-partial-prefills / --max-long-partial-prefills, which are V0-only knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/ is absent, so there is no V0 to fall back to), and EngineArgs. _is_v1_supported_oracle returns False whenever either flag differs from its default, calling _raise_or_fallback("Concurrent Partial Prefill") -> "Falling back to V0" -> AssertionError. With strategy Recreate that is an outage, not a slow rollout. Both flags are therefore NOT set, and a comment records why so they are not reintroduced. V1 needs no such flag: it has no prefill/decode phase at all. Every step it walks self.running and hands out tokens from one shared budget (vllm/v1/core/sched/scheduler.py). The real defect is that the per-request cap is disabled by default (long_prefill_token_threshold=0), so one long prompt takes the WHOLE budget, token_budget hits 0, the `while ... and token_budget > 0` loop exits, and every other request is skipped. The cap is applied before the budget clamp, so threshold >= budget never binds. Replaying the loop: 8192/8192 serves 1 request per step (still starved); 8192/2048 serves 4, and two long prompts advance together, which is the issue's actual acceptance criterion. --max-num-batched-tokens=8192 (was unset -> default 2048) --long-prefill-token-threshold=2048 (was unset -> 0, cap disabled) --max-num-seqs=5, --gpu-memory-utilization=0.95 The ~166k KV figure is marked as a projection; the live reading is num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative post-boot. A preemption alert covers the accept-and-watch risk. Observability for the companion admission gate (homelab-frontend#64, now deployed at 64f2bfc1): - four dashboard panels: parked, rejected by reason, engine state stale, and gateway-view vs engine-truth. Each carries a description, because an unexplained blank panel reads as a broken dashboard. - alerts on observer_degraded and engine_state_stale_total, both with an absent() guard: the gate fails OPEN, so a missing or renamed metric must alert rather than look healthy. Verified live that the guards do not false-fire (both series present, 3 each). - LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the regression direction this change introduces. - max(vllm_requests_running), not sum(): one engine-global gauge is read by all three gateway replicas, so sum() read 3x and could never agree. Also corrects comments that were factually wrong. An earlier revision of this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were dead code; homelab-frontend#63 wired them, verified live as 3 series each. request_duration_seconds is the genuinely dead one (0 samples, zero non-test call sites) and is now marked as such instead of being listed as working. Closes #51