# The admission gate fails OPEN when its view of the engine gauges goes
# stale, so a broken observer degrades silently to the pre-#64 behaviour
# of admitting everything. These two series are the only detection —
# the gate has no automated tests.
rules:
# The gate fails OPEN when it cannot read the engine, so "no data" must
# alert: otherwise a missing emitter, a renamed metric, or a broken
# scrape is indistinguishable from a healthy observer. Both expressions
# therefore OR in an absent() guard on the metric itself.
- alert:APIGatewayObserverDegraded
expr:observer_degraded == 1 or absent(observer_degraded)
for:2m
labels:
severity:warning
annotations:
summary:"Gateway admission observer degraded or not reporting — gate is failing open"
description:"observer_degraded is either 1 or absent. Absent means the emitter is missing, renamed, or unscraped, which is the fail-open case."
# Engine gauges went stale, or are not being reported at all
- alert:GatewayEngineStateStale
expr:increase(engine_state_stale_total[15m]) > 0 or absent(engine_state_stale_total)
for:2m
labels:
severity:warning
annotations:
summary:"Gateway engine state stale or not reporting — admission may be based on stale gauges"
description:"Non-zero means the observer lost the engine; absent means the metric is not being emitted at all."
# The acceptance metric for reasoning.yaml's prefill change. Preemption
# of a 65k prompt costs a full re-prefill (~250s at the measured
# 262 tok/s), so any sustained rate is a regression, not noise.
- alert:LLMPreemptions
# Guarded absent() is deliberately NOT used bare here: the engine
# restarts (strategy Recreate) and the series legitimately vanishes
# for the length of a boot, so a bare absent() would false-fire on
# every rollout. It only fires when the metric is missing WHILE the
# predictor is up, which is the real "renamed or unscraped" case.
expr:|
increase(vllm:num_preemptions_total[15m]) > 0
or (absent(vllm:num_preemptions_total) and on() up{job="reasoning-predictor"} == 1)
for:5m
labels:
severity:warning
annotations:
summary:"vLLM preempting requests ({{ $value }} in 15m) — KV cache over-committed, re-prefill cost is severe"
description:"Any sustained preemption rate is a regression for the reasoning prefill change: evicting a 65k-token prompt costs a full re-prefill (~250s at the measured 262 tok/s). Orchestrator verified `up{job=\"reasoning-predictor\"}` is the live job label — if that label is wrong the guard is inert but harmless."
# This change trades single-request latency for fairness: capping a long
# prompt at 2048 tokens/step means it takes more steps to finish, so a
# lone large prompt can now be slower even while preemptions stay 0.
# Without this alert the change has no way to show a regression.
- alert:LLMPrefillStarvation
expr:|
(sum(vllm:num_requests_waiting) > 0 and max(vllm:kv_cache_usage_perc) < 0.5)
or histogram_quantile(0.95, sum(rate(vllm:time_to_first_token_seconds_bucket[10m])) by (le)) > 60
for:10m
labels:
severity:warning
annotations:
summary:"reasoning queueing or TTFT p95 above 60s while KV is under 50% — scheduling, not capacity"
description:"Queue depth with spare KV means requests are being starved rather than blocked on memory; a high TTFT p95 with headroom points at the same. Both series verified live (vllm:num_requests_waiting and vllm:time_to_first_token_seconds exist)."
