CAG-2: vLLM-aware request routing — warmup delay + per-backend concurrency semaphore #45

New Issue

2026-09-27T09:57:59Z

rock commented

2026-09-27 09:57:59 +00:00

Context

vLLM runs with --max-num-seqs=4. Warmup requests (max_tokens=1) from poimen-memory must not starve real agent inference slots.

Requirement

Add in-memory bounded queue + per-backend semaphore for continuous batching support. Deprioritise warmup requests with artificial delay.

Acceptance Criteria

AC1: X-Poimen-Warmup: true on /v1/chat/completions → 500ms delay (configurable via WARMUP_DELAY_MS) before forwarding
AC2: Header stripped before upstream; X-Priority: low added
AC3: Per-backend in-memory queue: VLLM_MAX_QUEUE_DEPTH slots (default 20), VLLM_MAX_NUM_SEQS concurrent slots (default 4)
AC4: Queue full → 503 immediately (no goroutine pile-up)
AC5: warmup_requests_delayed_total counter exported at /metrics
AC6: Warmup sleeps BEFORE competing for queue slot (yields to agents arriving in that window)

Files

internal/proxy/scheduler.go (new)
internal/proxy/proxy.go
internal/observability/metrics.go
internal/observability/prometheus.go
internal/server/router.go
cmd/gateway/main.go

## Context vLLM runs with `--max-num-seqs=4`. Warmup requests (`max_tokens=1`) from poimen-memory must not starve real agent inference slots. ## Requirement Add in-memory bounded queue + per-backend semaphore for continuous batching support. Deprioritise warmup requests with artificial delay. ## Acceptance Criteria - AC1: `X-Poimen-Warmup: true` on `/v1/chat/completions` → 500ms delay (configurable via `WARMUP_DELAY_MS`) before forwarding - AC2: Header stripped before upstream; `X-Priority: low` added - AC3: Per-backend in-memory queue: `VLLM_MAX_QUEUE_DEPTH` slots (default 20), `VLLM_MAX_NUM_SEQS` concurrent slots (default 4) - AC4: Queue full → 503 immediately (no goroutine pile-up) - AC5: `warmup_requests_delayed_total` counter exported at `/metrics` - AC6: Warmup sleeps BEFORE competing for queue slot (yields to agents arriving in that window) ## Files - `internal/proxy/scheduler.go` (new) - `internal/proxy/proxy.go` - `internal/observability/metrics.go` - `internal/observability/prometheus.go` - `internal/server/router.go` - `cmd/gateway/main.go`

rock referenced this issue from a commit

2026-09-27 11:09:17 +00:00

feat(cag): memory standing-context adapter, warmup semaphore, cache-aware routing

rock referenced a pull request that will close this issue

2026-09-27 11:09:39 +00:00

feat(cag): CAG integration — memory adapter, warmup semaphore, cache-aware routing #47

rock referenced this issue from a commit

2026-09-27 11:16:42 +00:00

feat(cag): CAG integration — memory adapter, warmup semaphore, cache-aware routing (#47)

rock closed this issue

2026-09-27 11:16:42 +00:00

rock commented

2026-09-27 11:17:15 +00:00

Harness review — merged

RUN_ID: riotpiao-poimen/homelab-frontend#44-45-46-2026-09-27T09:35Z
PR: #47 (merged) Diff: 16 files +1603 / -168

Coders

coder-impl: DONE — scheduler.go (new), memory_adapter.go (new), proxy.go, router.go, config.go/loader.go, metrics.go/prometheus.go, logger.go, main.go, server/router.go, serviceadapter/router.go
coder-tests: DONE — scheduler_test.go (new), +JWT/queue/cache tests in proxy_test.go, +auth coverage in router_test.go, +addresses in models_test.go

Verification

go build ./... && go vet ./... && go test -race ./... → all pass, 0 races

reviewer-sre — SIGN-OFF

0🔴 3🟡 2🔵 — queue-full logged + counted, auth-bypass warned, all 3 CAG counters at /metrics

reviewer-quality — SIGN-OFF

CRAP: ServeHTTP 9.0 ✅ authenticateV1 8.5 ✅ — all 7 ACs covered by tests

Verdict: Merged ✅

## Harness review — merged **RUN_ID:** `riotpiao-poimen/homelab-frontend#44-45-46-2026-09-27T09:35Z` **PR:** #47 (merged) **Diff:** 16 files +1603 / -168 ### Coders - coder-impl: DONE — scheduler.go (new), memory_adapter.go (new), proxy.go, router.go, config.go/loader.go, metrics.go/prometheus.go, logger.go, main.go, server/router.go, serviceadapter/router.go - coder-tests: DONE — scheduler_test.go (new), +JWT/queue/cache tests in proxy_test.go, +auth coverage in router_test.go, +addresses in models_test.go ### Verification `go build ./... && go vet ./... && go test -race ./...` → all pass, 0 races ### reviewer-sre — SIGN-OFF 0🔴 3🟡 2🔵 — queue-full logged + counted, auth-bypass warned, all 3 CAG counters at /metrics ### reviewer-quality — SIGN-OFF CRAP: ServeHTTP 9.0 ✅ authenticateV1 8.5 ✅ — all 7 ACs covered by tests ### Verdict: Merged ✅

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#45