3 Commits

Include renames

Author	SHA1	Message	Date
Admin Bot	a731185588	fix(cag): resolve data races, add queue observability, fix test coverage ... Cycle 2 fixes from reviewer-sre + reviewer-quality: - scheduler.go: remove delete() under RLock (data race); expired entries skipped on read, cleaned lazily by next MarkFailed write - proxy.go: pre-populate transports for scheduler backends at startup to prevent concurrent map writes from ServeHTTP goroutines - proxy.go: log + IncrQueueFull() on queue-full 503 path (was silent) - proxy.go: remove stub total_tokens/hit_rate from cache-hit log event - serviceadapter/router.go: Warnf when Auth.Required but jwtValidator nil - observability: add queue_full_requests_total counter + Prometheus export - scheduler_test.go: TestBackendAcquireCancelledContext (ctx cancel branch) - proxy_test.go: implement TestWarmupHeaderHandling (remove t.Skip) - router_test.go: TestAuthenticate_MethodCapabilityOverrideDirect (auth coverage)	2026-09-27 18:39:45 +08:00
Admin Bot	fc5c0e967d	feat(cag): memory standing-context adapter, warmup semaphore, cache-aware routing ... Issue #44: code-register memory adapter with standing-context resource; fix dispatcher method-level Auth.Capability override (was always using verb-derived capability, ignoring per-method Auth field). Issue #45: per-backend bounded in-memory queue (VLLM_MAX_QUEUE_DEPTH=20) + concurrency semaphore (VLLM_MAX_NUM_SEQS=4); warmup requests (X-Poimen-Warmup: true) sleep WARMUP_DELAY_MS before queue entry; /metrics endpoint wired. Issue #46: rendezvous-hash backend selection on X-Poimen-Cache-Key; round-robin fallback; MarkFailed excludes unhealthy backends 30s; ModifyResponse captures vllm-num-cached-tokens + emits structured log + increments counter. Closes #44 Closes #45 Closes #46	2026-09-27 18:21:41 +08:00
rockandpoimen	d439536ca9	Add TTFT & ITL Metrics for LLM Inference (#28) ... Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total Closes #31 #32 #33 --------- Co-authored-by: poimen <[email protected]> Reviewed-on: #28	2026-09-15 08:53:58 +00:00