feat(cag): CAG integration — memory adapter, warmup semaphore, cache-aware routing #47

2026-09-27T11:09:39Z

rock commented

2026-09-27 11:09:39 +00:00

Summary

Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw.

#44 — Memory ServiceAdapter: standing-context resource

Code-registers memory adapter with standing-context resource (POST /memory/standing-context)
Fixes dispatcher bug: method-level Auth.Capability override now respected (was always using verb-derived capability)
Auth: memory:read required via method-level override; memory:write for ingest
MEMORY_UPSTREAM_URL env configures upstream

#45 — Continuous batching: warmup deprioritisation + per-backend semaphore

In-memory bounded queue per backend: VLLM_MAX_QUEUE_DEPTH slots (default 20)
Concurrency limiter: VLLM_MAX_NUM_SEQS slots (default 4)
X-Poimen-Warmup: true → sleep WARMUP_DELAY_MS (default 500ms) before queue entry, strip header, add X-Priority: low
Queue full → 503 immediately, no goroutine pile-up
Metrics: warmup_requests_delayed_total, queue_full_requests_total at /metrics

#46 — Cache-aware routing: rendezvous hash + prefix cache hit logging

addresses: [...] YAML field for multi-backend models (backward-compat with address)
X-Poimen-Cache-Key header → rendezvous-hash backend selection (same key = same GPU)
No cache key → round-robin
Failed backend excluded 30s; next-highest-score fallback
vllm-num-cached-tokens response header captured → vllm_prefix_cache_hit_tokens_total counter + structured log

Verification

go build ./...    ✅
go vet ./...      ✅
go test -race ./... ✅ (all packages, 0 races)

Reviewer verdicts

reviewer-sre: SIGN-OFF
reviewer-quality: SIGN-OFF

Non-blocking findings (🟡🔵 only)

getOrCreateTransport not goroutine-safe by design; pre-population invariant documented
time.Sleep(warmupDelay) holds goroutine; context-cancel alternative left as TODO
OTel trace_id not propagated into CAG log events (existing gap)

Closes #44 Closes #45 Closes #46

## Summary Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw. --- ### #44 — Memory ServiceAdapter: standing-context resource - Code-registers `memory` adapter with `standing-context` resource (`POST /memory/standing-context`) - Fixes dispatcher bug: method-level `Auth.Capability` override now respected (was always using verb-derived capability) - Auth: `memory:read` required via method-level override; `memory:write` for ingest - `MEMORY_UPSTREAM_URL` env configures upstream ### #45 — Continuous batching: warmup deprioritisation + per-backend semaphore - In-memory bounded queue per backend: `VLLM_MAX_QUEUE_DEPTH` slots (default 20) - Concurrency limiter: `VLLM_MAX_NUM_SEQS` slots (default 4) - `X-Poimen-Warmup: true` → sleep `WARMUP_DELAY_MS` (default 500ms) before queue entry, strip header, add `X-Priority: low` - Queue full → 503 immediately, no goroutine pile-up - Metrics: `warmup_requests_delayed_total`, `queue_full_requests_total` at `/metrics` ### #46 — Cache-aware routing: rendezvous hash + prefix cache hit logging - `addresses: [...]` YAML field for multi-backend models (backward-compat with `address`) - `X-Poimen-Cache-Key` header → rendezvous-hash backend selection (same key = same GPU) - No cache key → round-robin - Failed backend excluded 30s; next-highest-score fallback - `vllm-num-cached-tokens` response header captured → `vllm_prefix_cache_hit_tokens_total` counter + structured log --- ## Verification ``` go build ./... ✅ go vet ./... ✅ go test -race ./... ✅ (all packages, 0 races) ``` ## Reviewer verdicts - reviewer-sre: SIGN-OFF - reviewer-quality: SIGN-OFF ## Non-blocking findings (🟡🔵 only) - `getOrCreateTransport` not goroutine-safe by design; pre-population invariant documented - `time.Sleep(warmupDelay)` holds goroutine; context-cancel alternative left as TODO - OTel trace_id not propagated into CAG log events (existing gap) Closes #44 Closes #45 Closes #46

rock added 5 commits 2026-09-27 11:09:40 +00:00

feat(cag): memory standing-context adapter, warmup semaphore, cache-aware routing fc5c0e967d

Issue #44: code-register memory adapter with standing-context resource; fix
dispatcher method-level Auth.Capability override (was always using verb-derived
capability, ignoring per-method Auth field).

Issue #45: per-backend bounded in-memory queue (VLLM_MAX_QUEUE_DEPTH=20) +
concurrency semaphore (VLLM_MAX_NUM_SEQS=4); warmup requests (X-Poimen-Warmup:
true) sleep WARMUP_DELAY_MS before queue entry; /metrics endpoint wired.

Issue #46: rendezvous-hash backend selection on X-Poimen-Cache-Key; round-robin
fallback; MarkFailed excludes unhealthy backends 30s; ModifyResponse captures
vllm-num-cached-tokens + emits structured log + increments counter.

Closes #44 Closes #45 Closes #46

fix(cag): resolve data races, add queue observability, fix test coverage a731185588

Cycle 2 fixes from reviewer-sre + reviewer-quality:

- scheduler.go: remove delete() under RLock (data race); expired entries
  skipped on read, cleaned lazily by next MarkFailed write
- proxy.go: pre-populate transports for scheduler backends at startup to
  prevent concurrent map writes from ServeHTTP goroutines
- proxy.go: log + IncrQueueFull() on queue-full 503 path (was silent)
- proxy.go: remove stub total_tokens/hit_rate from cache-hit log event
- serviceadapter/router.go: Warnf when Auth.Required but jwtValidator nil
- observability: add queue_full_requests_total counter + Prometheus export
- scheduler_test.go: TestBackendAcquireCancelledContext (ctx cancel branch)
- proxy_test.go: implement TestWarmupHeaderHandling (remove t.Skip)
- router_test.go: TestAuthenticate_MethodCapabilityOverrideDirect (auth coverage)

fix(cag): extract applyWarmup/acquireScheduledBackend helpers, add race-free tests d703ace2c6

- proxy.go: extract applyWarmup + acquireScheduledBackend to reduce ServeHTTP complexity
- proxy_test.go: TestProxyQueueFull, TestProxyCacheHitLogging (AC5/AC6/AC7 HTTP-level coverage)
- scheduler_test.go: TestBackendAcquireCancelledContext + runtime.Gosched spin-wait (no time.Sleep)
- router_test.go: TestAuthenticate_MethodCapabilityOverrideDirect with real RS256 JWT + JWKS server

refactor(proxy): decompose ServeHTTP into 6 helpers (comp 38→9, CRAP<30) a9a151a8b4

Extract handleAuthEndpoint, routeOrError, writeModelValidationError, authenticateV1,
enforceBodyLimit, setupCAGProxy. No behavior change. All tests pass with -race.

fix(proxy): authenticateV1 Warnf + JWT test coverage (CRAP 42→9)

CI / CI (pull_request) Successful in 2m36s

Details

6257b8c201

- proxy.go: log Warnf when jwtValidator==nil so auth bypass is observable
- proxy_test.go: TestAuthenticateV1ProxyLevel covers 401/403/pass branches
  (authenticateV1 cov: 10.5% → 77.3%, CRAP: 42 → 8.5)

rock added 1 commit 2026-09-27 11:13:43 +00:00

fix(proxy): mutex on transports, ctx-aware warmup delay, Retry-After on 503

CI / CI (pull_request) Successful in 2m11s

Details

78a5d6baa2

- transportsMu (sync.RWMutex): guard getOrCreateTransport reads/writes;
  double-checked locking pattern; safe for dynamic backends
- applyWarmup: replace time.Sleep with select{time.After, ctx.Done}
  so cancelled warmup requests release the goroutine immediately
- Retry-After: 5 header on queue-full 503 (client backoff hint)
- OTel trace_id skipped: collector in tracing ns, not monitoring ns

rock merged commit ef5e736673 into main

2026-09-27 11:16:41 +00:00

rock deleted branch harness/44-45-46-cag-integration

2026-09-27 11:16:42 +00:00

rock referenced this pull request

2026-09-27 11:17:13 +00:00

CAG-1: Register memory service standing-context resource in ServiceAdapter #44

rock referenced this pull request

2026-09-27 11:17:15 +00:00

CAG-2: vLLM-aware request routing — warmup delay + per-backend concurrency semaphore #45

rock referenced this pull request

2026-09-27 11:17:16 +00:00