CAG-4: cluster-wide LLM admission control + gateway CPU/GOMAXPROCS tuning #64

New Issue

2026-09-30T08:54:53Z

poimen commented

2026-09-30 08:54:53 +00:00

Problem

Two independent defects found while investigating a 10-minute TTFT on reasoning (Qwen3-32B-GPTQ-Int4, vLLM 0.11.0, 2x V100-32GB, --pipeline-parallel-size=2, --max-num-seqs=4).

Evidence for both came from vllm:* metrics via prometheus.riotpiao.com and the reasoning-predictor log; the engine's own /metrics is unreachable from the API server because llm-serving default-deny NetworkPolicy admits only pods labelled llm-client=true.

A. "Max concurrency" is a per-pod semaphore, so it is 3x too high

internal/proxy/proxy.go:119 — maxSlots := 4 builds a semaphore inside each gateway process; internal/proxy/scheduler.go:21-25 holds it per backend. api-gateway runs 3 replicas (k8s/deployment.yaml:9), so the effective cluster-wide cap is 12, not 4. No replica coordinates with the others.

Measured on the engine side: max_over_time(vllm:num_requests_running[7d]) = 4 (its own --max-num-seqs=4) while the gateway was willing to admit 12. The queue never rejected anything either — queue_full_requests_total 0, so the mismatch is invisible today.

The budget that matters is KV, and it is smaller than previously assumed:

vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"} = 152,432 tokens for the whole engine.
PP=2 splits layers, so both ranks hold KV for the same tokens. Do not multiply by 2 (an earlier estimate of 312k was wrong by 2x).
Two 65k-token prompts + generation = ~134k = 88% of it.
Over-admission causes preemption, and a preempted 65k prefill is recomputed from zero: ~250 s at the measured uncached prefill rate (262 tok/s contended, from vllm:request_prefill_time_seconds_sum/count).

So the admission cap must be one cluster-wide number sized against 152,432 tokens, not a per-pod constant multiplied by replica count.

B. Gateway CPU is the throughput limit; goroutine count is not the lever

Go does not bound throughput by goroutine count — net/http already runs one goroutine per in-flight request, and they are cheap. A fixed worker pool would cap concurrency and add queueing latency; it cannot speed anything up. The -race-free path to more throughput is:

GOMAXPROCS: the container is limited to 500m CPU (kubectl get deploy -n api api-gateway -o jsonpath -> {"limits":{"cpu":"500m","memory":"512Mi"}}) but GOMAXPROCS defaults to the host's visible core count, so the Go scheduler oversubscribes and gets CFS-throttled. It should equal the cgroup quota. No GOMAXPROCS/automaxprocs reference exists anywhere in the repo today.
Full-body buffering per chat request: internal/proxy/router.go:61 io.ReadAll(r.Body) then :74 json.Unmarshal then :128 json.Marshal — the entire prompt is buffered and round-tripped through JSON just to validate and optionally rewrite model. That is O(prompt bytes) of CPU and heap per request: ~300 KB for a 65k-token prompt.
Body cap vs memory limit: model routes are synthesized with MaxBodySize: h.defaultMaxBodySize (internal/proxy/router.go:154), i.e. 100 MB (internal/proxy/proxy.go:93), against a 512Mi container limit. A single large POST plus the unmarshalled JSON tree can OOMKill the pod. The configured per-route caps in testdata/config/valid.yaml (embeddings 5 MB, ornith 10 MB, reasoning 1 h read) never apply to the model path.
For scale: the LLM path is GPU-bound (measured aggregate ~30.9 tok/s generation over 1 day against 525 tok/s prefill). Gateway CPU only becomes the bottleneck with many concurrent large bodies, which is exactly the workload this issue is about.

Proposal to explore

A. Cluster-wide admission control

The requested primitive is Redlock (Redis distributed lock). Recommendation: do not use Redlock here, and explore the alternatives below instead.

Redlock is mutual exclusion. Admission control needs a counting lease/semaphore (N concurrent holders). A lock serialises requests; a semaphore does not.
Redlock's safety depends on bounded clock drift and bounded process pauses; it is documented as unsuitable for correctness-critical exclusion without fencing tokens. Here a lost lock means over-admission, which is a soft (not corrupting) failure, so Redlock would be tolerable — but it is still the wrong shape for the job.
There is no shared Redis in the cluster: argocd-redis, forgejo-redis, immich-redis, paperless-redis, poimen-redis and sqs/kmsvc-redis-master are all per-app. Adding Redis to api means a new stateful dependency and a new single point of failure for the LLM path.

Options, cheapest first:

Single authority for the reasoning route. Pin LLM routing to one replica (dedicated 1-replica Deployment or a coordination.k8s.io/Lease leader election so only the leader admits). No new dependency; the existing semaphore then is the cluster-wide cap. Cost: one 500m pod carries all LLM traffic.
Observe vLLM instead of locking. Read vllm:num_requests_running / vllm:kv_cache_usage_perc and shed with 503 above target. vLLM is already the true admission authority; a lock duplicates it. Latency is one scrape interval (15-30 s), which is fine for a KV budget measured in minutes.
Redis counting semaphore (if a shared Redis is accepted): INCR + per-holder EXPIRE lease, or a sorted set keyed by holder id with expiry-driven reaping. This is the shape Redlock should have been. Failure mode must be explicit: Redis unreachable -> fail-open (admit) or fail-closed (503)? Decide in the spike.
Postgres advisory lock / FOR UPDATE SKIP LOCKED against an existing database. No new runtime, but adds DB dependency to the LLM hot path.
Temporal, already deployed and already reachable from the gateway (TEMPORAL_HOST_PORT). A task-queue with a concurrency limit models admission naturally, but with substantially more moving parts than the problem needs.

Expected outcome of the spike: a decision between (1), (2) and (3), with (1)+(2) as the default recommendation.

B. Gateway CPU

Set GOMAXPROCS from the cgroup quota (uber-go/automaxprocs import, or an explicit GOMAXPROCS env in k8s/deployment.yaml) before touching anything else.
Raise the container CPU limit from 500m; measure whether SSE proxying of concurrent large bodies is throttled (container_cpu_cfs_throttled_seconds_total).
Stop round-tripping the whole body: stream/peek the model field, or only re-marshal when upstreamModel actually differs (internal/proxy/router.go:120-134).
Make the model-path body cap real and small (start at 1 MB), and confirm it is below the memory limit.

Acceptance criteria

One cluster-wide admission cap for the reasoning route, sized against 152,432 KV tokens, enforced regardless of gateway replica count.
Documented failure mode when the coordination backend is unavailable (explicit fail-open or fail-closed, plus a metric).
vllm:num_preemptions_total stays 0 under a load test of 2x 65k-token prompts plus concurrent 8k prompts.
GOMAXPROCS matches the container CPU quota; CFS throttling measured before/after.
Model-path body cap is a configured value below the container memory limit, and an oversized prompt returns a fast 4xx instead of entering prefill.
Load test reproduces against the gateway (not vLLM directly) and reports TTFT p50/p95 split by prompt size.

Out of scope

vLLM flag tuning (--max-num-batched-tokens, --max-num-partial-prefills, --max-num-seqs) — separate change in homelab/k8s/apps/llm-serving/reasoning.yaml.
Prefix-cache discipline in the client (stable system + tools prefix).
--scheduling-policy=priority and gateway priority injection.

SETTLED DESIGN — supersedes the Redlock proposal above

Design tree walked to completion 2026-09-30. Redlock is dropped, not deferred.

Core mechanism (replaces the per-pod semaphore's claim to be a cap)

The gateway reads the engine's own gauges before admitting. This is global by construction: every replica contends for the same engine, and the gauge already accounts for callers that never pass the gateway (namespace poimen is admitted directly on 80/8080 by the llm-serving-ingress NetworkPolicy; memory/crates/mem-cli/src/kv_warmer.rs is such a caller). No lock, no Redis, no k8s.io/client-go.

Measurements that size it:

vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"} = 152,432 tokens engine KV capacity. PP=2 splits layers, so both ranks hold KV for the same tokens; do not multiply by 2.
Uncached prefill 262 tok/s contended (1h), 525 tok/s (7d), from vllm:request_prefill_time_seconds_sum/count.
Three gauges available: vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc. No max_num_seqs gauge exists — the target must be configured and can drift silently.

Decisions

#	Decision
Q1	Engine observation only. No Lease, no redlock, no shared Redis (none exists in `api`; all Redis instances are per-app).
Q2	Both gates: `kv_cache_usage_perc >= 0.90` (primary) or `num_requests_running + num_requests_waiting >= N` (secondary), `N = max_num_seqs`. Asserted and logged at startup so drift is visible.
Q3	Bounded park, then `503 + Retry-After: 5`.
Q4	FIFO, no reservation, no priority. Fairness is delegated to the engine via concurrent partial prefill.
Q5	Fail-open when stale (>3 missed polls / 6s), plus `engine_state_stale_total`.
Q6	`VLLM_MAX_QUEUE_DEPTH=8` per pod; overflow returns 503 through the existing `ErrQueueFull` path at `internal/proxy/proxy.go:410-419`.
Q7	Env vars first, per-model config fields later. Delete the dead ConfigMap manifest and the orphaned live ConfigMap.
Q9	Park cap is env-configurable; provisional `ADMISSION_MAX_PARK_SECONDS=120`, final value from the load test.
Q10	Gate on `running + waiting`, keeping overflow in the gateway where it is visible and sheddable.
Q11	vLLM flags are a companion issue + PR in `riotpiao.com/homelab`. The gateway ships with `N = 4` and is correct against today's engine.
Q12	Delete repo `k8s/configmap.yaml` (dead: absent from `k8s/kustomization.yaml`) in this PR; the live ConfigMap `api-gateway-config` (key `gw-config-fixed.yaml`, declared in neither repo) is deleted by the human — it is a mutation outside the code change.
Q13	Verification is a manual real-engine run. No Tekton, no mock upstream.
Q14	Metrics named to match existing convention (these series carry no prefix today): `admission_parked_total`, `admission_rejected_total{reason}`, `engine_state_stale_total`, and shadow gauges `vllm_requests_running` / `vllm_kv_usage_perc`.
Q15	Four panels added to `build_llm_observability()` in `homelab/k8s/infra/monitoring/dashboards/generate-dashboards.py` — editing `llm-observability.yaml` directly is futile, it is generated at `generate-dashboards.py:464`.
Q16	One PR per repo.
Q17	No test infrastructure. Production is the test bed.
Q18	Multi-PR in `homelab-frontend`: PR-1 ships the mechanism with the provisional cap; a later PR sets the measured value.
Q19	The 65k load test is run from the api-gw session; evidence goes in the PR-1 description plus one comment here.
Q20	Body cap 1 MB global (`MAX_BODY_BYTES=1048576`).
Q21	Startup self-check: the first poll must parse all three series or the gateway logs ERROR, sets `observer_degraded 1`, and the alert in `homelab/k8s/infra/monitoring/alerts/api-gateway-alerts.yaml` fires.
Q22	Streaming requests (`stream: true`) receive 200 + `text/event-stream` headers and SSE comment heartbeats while parked, then hand off to the proxy. Precedent: `internal/temporal/sse.go:63` (15s ticker). Non-streaming requests park silently.

Corrections to the analysis above

N = max_num_seqs, not max_num_seqs - 1. --max-num-seqs is a hard engine limit, so a gateway poll race cannot over-commit KV; it only parks a few requests in vLLM's queue. Subtracting 1 wastes a slot.
The cap is soft, not hard. Between polls each replica may admit more than one request, so there is no clean "(replicas−1)" bound. Exposure is low (7d volume ~11,500 requests ≈ 1.4/min, measured), and the engine remains the hard limit. A guarantee would require the dropped Lease.

PR split

PR-1 homelab-frontend

new internal/proxy/engine_state.go — 2s poll of http://<addr>/metrics, hand-rolled text parse (no prometheus/client_golang in go.mod; internal/observability/prometheus.go already hand-rolls the exposition format), EngineState, degraded flag, startup assertion.
internal/proxy/scheduler.go:43 — gate before taking the local slot, park loop, cap, 503.
internal/proxy/proxy.go — start observers near :141; park in acquireScheduledBackend at :404; SSE heartbeat while parked inside setupCAGProxy (:606-636), which must not write its own status line afterwards.
internal/proxy/router.go:74 — extract only the top-level model field instead of json.Unmarshal into map[string]interface{} (avoids building the whole messages+tools tree per request); :154 — body cap from env.
internal/observability/prometheus.go — the five series.
cmd/gateway/main.go — GOMAXPROCS from the cgroup v2 quota (/sys/fs/cgroup/cpu.max); the container is limited to 500m while Go defaults to the host core count.
k8s/deployment.yaml — ADMISSION_MAX_PARK_SECONDS=120, VLLM_MAX_QUEUE_DEPTH=8, MAX_BODY_BYTES=1048576; also delete the now-false llm-client: "true" comment at :33-36 (llm-serving-ingress now selects by namespace, not label).
delete k8s/configmap.yaml.

PR-1b homelab-frontend — the measured ADMISSION_MAX_PARK_SECONDS.

PR-2 riotpiao.com/homelab — in k8s/apps/llm-serving/reasoning.yaml: --max-num-partial-prefills=2, --max-long-partial-prefills=2, --long-prefill-token-threshold=8192, --max-num-batched-tokens=4096, --max-num-seqs=5, --gpu-memory-utilization=0.95. In generate-dashboards.py: four panels in build_llm_observability(). In alerts/api-gateway-alerts.yaml: the observer_degraded alert.

Accepted risks

No automated tests. The mitigation is the Q21 startup self-check, which is production self-diagnosis, not a test.
Soft cap, not a guarantee.
The max_num_seqs to N coupling is manual; drift is silent unless the startup assertion is read.

## Problem Two independent defects found while investigating a 10-minute TTFT on `reasoning` (Qwen3-32B-GPTQ-Int4, vLLM 0.11.0, 2x V100-32GB, `--pipeline-parallel-size=2`, `--max-num-seqs=4`). Evidence for both came from `vllm:*` metrics via `prometheus.riotpiao.com` and the reasoning-predictor log; the engine's own `/metrics` is unreachable from the API server because `llm-serving` default-deny NetworkPolicy admits only pods labelled `llm-client=true`. ### A. "Max concurrency" is a per-pod semaphore, so it is 3x too high `internal/proxy/proxy.go:119` — `maxSlots := 4` builds a semaphore inside each gateway process; `internal/proxy/scheduler.go:21-25` holds it per backend. `api-gateway` runs 3 replicas (`k8s/deployment.yaml:9`), so the effective cluster-wide cap is **12**, not 4. No replica coordinates with the others. Measured on the engine side: `max_over_time(vllm:num_requests_running[7d]) = 4` (its own `--max-num-seqs=4`) while the gateway was willing to admit 12. The queue never rejected anything either — `queue_full_requests_total 0`, so the mismatch is invisible today. The budget that matters is KV, and it is smaller than previously assumed: - `vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}` = **152,432 tokens** for the whole engine. - PP=2 splits *layers*, so both ranks hold KV for the **same** tokens. Do not multiply by 2 (an earlier estimate of 312k was wrong by 2x). - Two 65k-token prompts + generation = ~134k = **88%** of it. - Over-admission causes preemption, and a preempted 65k prefill is recomputed from zero: **~250 s** at the measured uncached prefill rate (262 tok/s contended, from `vllm:request_prefill_time_seconds_sum/count`). So the admission cap must be one cluster-wide number sized against 152,432 tokens, not a per-pod constant multiplied by replica count. ### B. Gateway CPU is the throughput limit; goroutine count is not the lever Go does not bound throughput by goroutine count — `net/http` already runs one goroutine per in-flight request, and they are cheap. A fixed worker pool would *cap* concurrency and add queueing latency; it cannot speed anything up. The `-race`-free path to more throughput is: - **`GOMAXPROCS`**: the container is limited to `500m` CPU (`kubectl get deploy -n api api-gateway -o jsonpath` -> `{"limits":{"cpu":"500m","memory":"512Mi"}}`) but `GOMAXPROCS` defaults to the host's visible core count, so the Go scheduler oversubscribes and gets CFS-throttled. It should equal the cgroup quota. No `GOMAXPROCS`/`automaxprocs` reference exists anywhere in the repo today. - **Full-body buffering per chat request**: `internal/proxy/router.go:61` `io.ReadAll(r.Body)` then `:74` `json.Unmarshal` then `:128` `json.Marshal` — the entire prompt is buffered and round-tripped through JSON just to validate and optionally rewrite `model`. That is O(prompt bytes) of CPU and heap per request: ~300 KB for a 65k-token prompt. - **Body cap vs memory limit**: model routes are synthesized with `MaxBodySize: h.defaultMaxBodySize` (`internal/proxy/router.go:154`), i.e. **100 MB** (`internal/proxy/proxy.go:93`), against a **512Mi** container limit. A single large POST plus the unmarshalled JSON tree can OOMKill the pod. The configured per-route caps in `testdata/config/valid.yaml` (embeddings 5 MB, ornith 10 MB, reasoning 1 h read) never apply to the model path. - For scale: the LLM path is GPU-bound (measured aggregate ~30.9 tok/s generation over 1 day against 525 tok/s prefill). Gateway CPU only becomes the bottleneck with many concurrent large bodies, which is exactly the workload this issue is about. ## Proposal to explore ### A. Cluster-wide admission control The requested primitive is Redlock (Redis distributed lock). Recommendation: **do not use Redlock here**, and explore the alternatives below instead. - Redlock is *mutual exclusion*. Admission control needs a *counting* lease/semaphore (N concurrent holders). A lock serialises requests; a semaphore does not. - Redlock's safety depends on bounded clock drift and bounded process pauses; it is documented as unsuitable for correctness-critical exclusion without fencing tokens. Here a lost lock means over-admission, which is a soft (not corrupting) failure, so Redlock would be tolerable — but it is still the wrong shape for the job. - There is **no shared Redis in the cluster**: `argocd-redis`, `forgejo-redis`, `immich-redis`, `paperless-redis`, `poimen-redis` and `sqs/kmsvc-redis-master` are all per-app. Adding Redis to `api` means a new stateful dependency and a new single point of failure for the LLM path. Options, cheapest first: 1. **Single authority for the reasoning route.** Pin LLM routing to one replica (dedicated 1-replica Deployment or a `coordination.k8s.io/Lease` leader election so only the leader admits). No new dependency; the existing semaphore then *is* the cluster-wide cap. Cost: one 500m pod carries all LLM traffic. 2. **Observe vLLM instead of locking.** Read `vllm:num_requests_running` / `vllm:kv_cache_usage_perc` and shed with 503 above target. vLLM is already the true admission authority; a lock duplicates it. Latency is one scrape interval (15-30 s), which is fine for a KV budget measured in minutes. 3. **Redis counting semaphore** (if a shared Redis is accepted): `INCR` + per-holder `EXPIRE` lease, or a sorted set keyed by holder id with expiry-driven reaping. This is the shape Redlock should have been. Failure mode must be explicit: Redis unreachable -> fail-open (admit) or fail-closed (503)? Decide in the spike. 4. **Postgres advisory lock / `FOR UPDATE SKIP LOCKED`** against an existing database. No new runtime, but adds DB dependency to the LLM hot path. 5. **Temporal**, already deployed and already reachable from the gateway (`TEMPORAL_HOST_PORT`). A task-queue with a concurrency limit models admission naturally, but with substantially more moving parts than the problem needs. Expected outcome of the spike: a decision between (1), (2) and (3), with (1)+(2) as the default recommendation. ### B. Gateway CPU - Set `GOMAXPROCS` from the cgroup quota (`uber-go/automaxprocs` import, or an explicit `GOMAXPROCS` env in `k8s/deployment.yaml`) before touching anything else. - Raise the container CPU limit from `500m`; measure whether SSE proxying of concurrent large bodies is throttled (`container_cpu_cfs_throttled_seconds_total`). - Stop round-tripping the whole body: stream/peek the `model` field, or only re-marshal when `upstreamModel` actually differs (`internal/proxy/router.go:120-134`). - Make the model-path body cap real and small (start at 1 MB), and confirm it is below the memory limit. ## Acceptance criteria - [ ] One cluster-wide admission cap for the `reasoning` route, sized against 152,432 KV tokens, enforced regardless of gateway replica count. - [ ] Documented failure mode when the coordination backend is unavailable (explicit fail-open or fail-closed, plus a metric). - [ ] `vllm:num_preemptions_total` stays 0 under a load test of 2x 65k-token prompts plus concurrent 8k prompts. - [ ] `GOMAXPROCS` matches the container CPU quota; CFS throttling measured before/after. - [ ] Model-path body cap is a configured value below the container memory limit, and an oversized prompt returns a fast 4xx instead of entering prefill. - [ ] Load test reproduces against the gateway (not vLLM directly) and reports TTFT p50/p95 split by prompt size. ## Out of scope - vLLM flag tuning (`--max-num-batched-tokens`, `--max-num-partial-prefills`, `--max-num-seqs`) — separate change in `homelab/k8s/apps/llm-serving/reasoning.yaml`. - Prefix-cache discipline in the client (stable system + tools prefix). - `--scheduling-policy=priority` and gateway `priority` injection. --- # SETTLED DESIGN — supersedes the Redlock proposal above Design tree walked to completion 2026-09-30. Redlock is **dropped**, not deferred. ## Core mechanism (replaces the per-pod semaphore's claim to be a cap) The gateway reads the **engine's own gauges** before admitting. This is global by construction: every replica contends for the same engine, and the gauge already accounts for callers that never pass the gateway (namespace `poimen` is admitted directly on 80/8080 by the `llm-serving-ingress` NetworkPolicy; `memory/crates/mem-cli/src/kv_warmer.rs` is such a caller). No lock, no Redis, no `k8s.io/client-go`. Measurements that size it: - `vllm:cache_config_info{num_gpu_blocks="9527",block_size="16"}` = **152,432 tokens** engine KV capacity. PP=2 splits layers, so both ranks hold KV for the same tokens; do not multiply by 2. - Uncached prefill **262 tok/s** contended (1h), 525 tok/s (7d), from `vllm:request_prefill_time_seconds_sum/count`. - Three gauges available: `vllm:num_requests_running`, `vllm:num_requests_waiting`, `vllm:kv_cache_usage_perc`. No `max_num_seqs` gauge exists — the target must be configured and can drift silently. ## Decisions | # | Decision | |---|---| | Q1 | Engine observation only. No Lease, no redlock, no shared Redis (none exists in `api`; all Redis instances are per-app). | | Q2 | Both gates: `kv_cache_usage_perc >= 0.90` (primary) or `num_requests_running + num_requests_waiting >= N` (secondary), `N = max_num_seqs`. Asserted and logged at startup so drift is visible. | | Q3 | Bounded park, then `503 + Retry-After: 5`. | | Q4 | FIFO, no reservation, no priority. Fairness is delegated to the engine via concurrent partial prefill. | | Q5 | Fail-open when stale (>3 missed polls / 6s), plus `engine_state_stale_total`. | | Q6 | `VLLM_MAX_QUEUE_DEPTH=8` per pod; overflow returns 503 through the existing `ErrQueueFull` path at `internal/proxy/proxy.go:410-419`. | | Q7 | Env vars first, per-model config fields later. Delete the dead ConfigMap manifest and the orphaned live ConfigMap. | | Q9 | Park cap is env-configurable; provisional `ADMISSION_MAX_PARK_SECONDS=120`, final value from the load test. | | Q10 | Gate on `running + waiting`, keeping overflow in the gateway where it is visible and sheddable. | | Q11 | vLLM flags are a companion issue + PR in `riotpiao.com/homelab`. The gateway ships with `N = 4` and is correct against today's engine. | | Q12 | Delete repo `k8s/configmap.yaml` (dead: absent from `k8s/kustomization.yaml`) in this PR; the live ConfigMap `api-gateway-config` (key `gw-config-fixed.yaml`, declared in neither repo) is deleted by the human — it is a mutation outside the code change. | | Q13 | Verification is a manual real-engine run. No Tekton, no mock upstream. | | Q14 | Metrics named to match existing convention (these series carry no prefix today): `admission_parked_total`, `admission_rejected_total{reason}`, `engine_state_stale_total`, and shadow gauges `vllm_requests_running` / `vllm_kv_usage_perc`. | | Q15 | Four panels added to `build_llm_observability()` in `homelab/k8s/infra/monitoring/dashboards/generate-dashboards.py` — editing `llm-observability.yaml` directly is futile, it is generated at `generate-dashboards.py:464`. | | Q16 | One PR per repo. | | Q17 | No test infrastructure. Production is the test bed. | | Q18 | Multi-PR in `homelab-frontend`: PR-1 ships the mechanism with the provisional cap; a later PR sets the measured value. | | Q19 | The 65k load test is run from the api-gw session; evidence goes in the PR-1 description plus one comment here. | | Q20 | Body cap 1 MB global (`MAX_BODY_BYTES=1048576`). | | Q21 | Startup self-check: the first poll must parse all three series or the gateway logs ERROR, sets `observer_degraded 1`, and the alert in `homelab/k8s/infra/monitoring/alerts/api-gateway-alerts.yaml` fires. | | Q22 | Streaming requests (`stream: true`) receive 200 + `text/event-stream` headers and SSE comment heartbeats while parked, then hand off to the proxy. Precedent: `internal/temporal/sse.go:63` (15s ticker). Non-streaming requests park silently. | ## Corrections to the analysis above - `N = max_num_seqs`, **not** `max_num_seqs - 1`. `--max-num-seqs` is a hard engine limit, so a gateway poll race cannot over-commit KV; it only parks a few requests in vLLM's queue. Subtracting 1 wastes a slot. - **The cap is soft, not hard.** Between polls each replica may admit more than one request, so there is no clean "(replicas−1)" bound. Exposure is low (7d volume ~11,500 requests ≈ 1.4/min, measured), and the engine remains the hard limit. A guarantee would require the dropped Lease. ## PR split **PR-1 `homelab-frontend`** - new `internal/proxy/engine_state.go` — 2s poll of `http://<addr>/metrics`, hand-rolled text parse (no `prometheus/client_golang` in `go.mod`; `internal/observability/prometheus.go` already hand-rolls the exposition format), `EngineState`, degraded flag, startup assertion. - `internal/proxy/scheduler.go:43` — gate before taking the local slot, park loop, cap, 503. - `internal/proxy/proxy.go` — start observers near `:141`; park in `acquireScheduledBackend` at `:404`; SSE heartbeat while parked inside `setupCAGProxy` (`:606-636`), which must not write its own status line afterwards. - `internal/proxy/router.go:74` — extract only the top-level `model` field instead of `json.Unmarshal` into `map[string]interface{}` (avoids building the whole `messages`+`tools` tree per request); `:154` — body cap from env. - `internal/observability/prometheus.go` — the five series. - `cmd/gateway/main.go` — `GOMAXPROCS` from the cgroup v2 quota (`/sys/fs/cgroup/cpu.max`); the container is limited to `500m` while Go defaults to the host core count. - `k8s/deployment.yaml` — `ADMISSION_MAX_PARK_SECONDS=120`, `VLLM_MAX_QUEUE_DEPTH=8`, `MAX_BODY_BYTES=1048576`; also delete the now-false `llm-client: "true"` comment at `:33-36` (`llm-serving-ingress` now selects by namespace, not label). - delete `k8s/configmap.yaml`. **PR-1b `homelab-frontend`** — the measured `ADMISSION_MAX_PARK_SECONDS`. **PR-2 `riotpiao.com/homelab`** — in `k8s/apps/llm-serving/reasoning.yaml`: `--max-num-partial-prefills=2`, `--max-long-partial-prefills=2`, `--long-prefill-token-threshold=8192`, `--max-num-batched-tokens=4096`, `--max-num-seqs=5`, `--gpu-memory-utilization=0.95`. In `generate-dashboards.py`: four panels in `build_llm_observability()`. In `alerts/api-gateway-alerts.yaml`: the `observer_degraded` alert. ## Accepted risks - No automated tests. The mitigation is the Q21 startup self-check, which is production self-diagnosis, not a test. - Soft cap, not a guarantee. - The `max_num_seqs` to `N` coupling is manual; drift is silent unless the startup assertion is read.