rock
c7d52b5971
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes #51
63 lines
2.7 KiB
YAML
63 lines
2.7 KiB
YAML
# History, recorded so nobody re-derives it the hard way: on 2026-09-29 the
|
|
# three latency series below recorded nothing. internal/proxy/llm_metrics.go
|
|
# declared a capture wrapper (LLMMetricsCapture) that was never constructed
|
|
# anywhere, so the exporter emitted HELP-only stubs and the dashboard panels
|
|
# stayed empty.
|
|
#
|
|
# homelab-frontend#63 fixed exactly that. The wrapper is now
|
|
# ResponseWriterWrapper, constructed by NewResponseWriterWrapper
|
|
# (internal/proxy/llm_metrics.go:38) and installed on the streaming proxy
|
|
# request path. Verified live on 2026-09-29:
|
|
# llm_ttft_seconds count=7 sum=598.212s
|
|
# llm_itl_seconds count=16116
|
|
# llm_tokens_total 3 series
|
|
#
|
|
# These derived-from-SSE series cover EVERY model including the two Ollama ones
|
|
# (ornith:35b, qwen2.5:3b-instruct), which expose no metrics endpoint at all.
|
|
# They are proxy-side: TTFT is request-start to first byte and ITL is the gap
|
|
# between SSE writes, so both are granular to SSE write boundaries rather than
|
|
# to the engine's own token cadence. llm_tokens_total counts `data:` events,
|
|
# and the documented assumption in internal/proxy/llm_metrics.go is that these
|
|
# upstreams emit one such event per token — so it is a token count only as far
|
|
# as that assumption holds. vLLM's own histograms remain authoritative.
|
|
#
|
|
# Also recorded and working (3 series, one per replica):
|
|
# queue_full_requests_total counter backpressure rejects
|
|
# warmup_requests_delayed_total counter vLLM slot contention
|
|
# vllm_prefix_cache_hit_tokens_total counter prefix-cache hits
|
|
#
|
|
# NOT working, do not build on it: request_duration_seconds is declared and
|
|
# exported but has 0 samples, and its only writer Metrics.RecordRequest has
|
|
# zero call sites outside _test.go — verified by ast-grep, not assumed. It is
|
|
# the same dead-code shape this branch already corrected for the latency
|
|
# series, so it is listed here rather than left to be rediscovered.
|
|
#
|
|
# Also note the Deployment already carried prometheus.io/scrape annotations,
|
|
# but kube-prometheus-stack ignores pod annotations (no additionalScrapeConfigs,
|
|
# serviceMonitorSelectorNilUsesHelmValues: false) — so nothing at all was being
|
|
# collected before this ServiceMonitor existed.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: api-gateway
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/name: api-gateway
|
|
app.kubernetes.io/part-of: api
|
|
spec:
|
|
namespaceSelector:
|
|
matchNames:
|
|
- api
|
|
selector:
|
|
matchLabels:
|
|
app: api-gateway
|
|
endpoints:
|
|
# api-gateway Service port IS named `http` (unlike the KServe ones), so
|
|
# the name form works here.
|
|
- port: http
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
path: /metrics
|
|
scheme: http
|
|
|