feat(cag): memory standing-context adapter, warmup semaphore, cache-aware routing
Issue #44: code-register memory adapter with standing-context resource; fix dispatcher method-level Auth.Capability override (was always using verb-derived capability, ignoring per-method Auth field). Issue #45: per-backend bounded in-memory queue (VLLM_MAX_QUEUE_DEPTH=20) + concurrency semaphore (VLLM_MAX_NUM_SEQS=4); warmup requests (X-Poimen-Warmup: true) sleep WARMUP_DELAY_MS before queue entry; /metrics endpoint wired. Issue #46: rendezvous-hash backend selection on X-Poimen-Cache-Key; round-robin fallback; MarkFailed excludes unhealthy backends 30s; ModifyResponse captures vllm-num-cached-tokens + emits structured log + increments counter. Closes #44 Closes #45 Closes #46
This commit is contained in:
@@ -16,6 +16,7 @@ type Router struct {
|
||||
temporalHandler http.Handler
|
||||
upstreamHandler http.Handler
|
||||
forgejoWebhook *webhook.ForgejoHandler
|
||||
metricsHandler http.Handler
|
||||
}
|
||||
|
||||
// NewRouter creates a new router with health endpoints.
|
||||
@@ -33,6 +34,11 @@ func NewRouter(healthChecker *HealthChecker, dispatcher *serviceadapter.Dispatch
|
||||
}
|
||||
}
|
||||
|
||||
// SetMetricsHandler registers the Prometheus /metrics endpoint handler.
|
||||
func (r *Router) SetMetricsHandler(h http.Handler) {
|
||||
r.metricsHandler = h
|
||||
}
|
||||
|
||||
// ServeHTTP implements http.Handler.
|
||||
// Priority order:
|
||||
// 1. /healthz and /readyz to health handlers
|
||||
@@ -48,6 +54,11 @@ func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||
case "/readyz":
|
||||
ReadinessHandler(r.healthChecker)(w, req)
|
||||
return
|
||||
case "/metrics":
|
||||
if r.metricsHandler != nil {
|
||||
r.metricsHandler.ServeHTTP(w, req)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// X-Service (ServiceAdapter) routing - checked before path-based routing
|
||||
|
||||
Reference in New Issue
Block a user