rockandAdmin Bot
ef5e736673
CI / CI (push) Successful in 3m20s
## Summary Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw. --- ### #44 — Memory ServiceAdapter: standing-context resource - Code-registers `memory` adapter with `standing-context` resource (`POST /memory/standing-context`) - Fixes dispatcher bug: method-level `Auth.Capability` override now respected (was always using verb-derived capability) - Auth: `memory:read` required via method-level override; `memory:write` for ingest - `MEMORY_UPSTREAM_URL` env configures upstream ### #45 — Continuous batching: warmup deprioritisation + per-backend semaphore - In-memory bounded queue per backend: `VLLM_MAX_QUEUE_DEPTH` slots (default 20) - Concurrency limiter: `VLLM_MAX_NUM_SEQS` slots (default 4) - `X-Poimen-Warmup: true` → sleep `WARMUP_DELAY_MS` (default 500ms) before queue entry, strip header, add `X-Priority: low` - Queue full → 503 immediately, no goroutine pile-up - Metrics: `warmup_requests_delayed_total`, `queue_full_requests_total` at `/metrics` ### #46 — Cache-aware routing: rendezvous hash + prefix cache hit logging - `addresses: [...]` YAML field for multi-backend models (backward-compat with `address`) - `X-Poimen-Cache-Key` header → rendezvous-hash backend selection (same key = same GPU) - No cache key → round-robin - Failed backend excluded 30s; next-highest-score fallback - `vllm-num-cached-tokens` response header captured → `vllm_prefix_cache_hit_tokens_total` counter + structured log --- ## Verification ``` go build ./... ✅ go vet ./... ✅ go test -race ./... ✅ (all packages, 0 races) ``` ## Reviewer verdicts - reviewer-sre: SIGN-OFF - reviewer-quality: SIGN-OFF ## Non-blocking findings (🟡🔵 only) - `getOrCreateTransport` not goroutine-safe by design; pre-population invariant documented - `time.Sleep(warmupDelay)` holds goroutine; context-cancel alternative left as TODO - OTel trace_id not propagated into CAG log events (existing gap) Closes #44 Closes #45 Closes #46 --------- Co-authored-by: Admin Bot <[email protected]> Reviewed-on: #47
92 lines
2.9 KiB
Go
92 lines
2.9 KiB
Go
package server
|
|
|
|
import (
|
|
"net/http"
|
|
|
|
"forgejo.riotpiao.com/rock/homelab-frontend/internal/serviceadapter"
|
|
"forgejo.riotpiao.com/rock/homelab-frontend/internal/webhook"
|
|
)
|
|
|
|
// Router implements an HTTP handler that routes health endpoints,
|
|
// ServiceAdapter X-Service requests, Temporal workflow endpoints,
|
|
// and other requests to upstream handlers.
|
|
type Router struct {
|
|
healthChecker *HealthChecker
|
|
dispatcher *serviceadapter.Dispatcher
|
|
temporalHandler http.Handler
|
|
upstreamHandler http.Handler
|
|
forgejoWebhook *webhook.ForgejoHandler
|
|
metricsHandler http.Handler
|
|
}
|
|
|
|
// NewRouter creates a new router with health endpoints.
|
|
// Health endpoints (/healthz and /readyz) are handled locally.
|
|
// X-Service requests are dispatched via ServiceAdapter CRD.
|
|
// Temporal endpoints (/workflow*) are routed to temporalHandler.
|
|
// All other paths are passed to the upstream handler.
|
|
func NewRouter(healthChecker *HealthChecker, dispatcher *serviceadapter.Dispatcher, temporalHandler http.Handler, upstreamHandler http.Handler) *Router {
|
|
return &Router{
|
|
healthChecker: healthChecker,
|
|
dispatcher: dispatcher,
|
|
temporalHandler: temporalHandler,
|
|
upstreamHandler: upstreamHandler,
|
|
forgejoWebhook: webhook.NewForgejoHandler(),
|
|
}
|
|
}
|
|
|
|
// SetMetricsHandler registers the Prometheus /metrics endpoint handler.
|
|
func (r *Router) SetMetricsHandler(h http.Handler) {
|
|
r.metricsHandler = h
|
|
}
|
|
|
|
// ServeHTTP implements http.Handler.
|
|
// Priority order:
|
|
// 1. /healthz and /readyz to health handlers
|
|
// 2. X-Service header to ServiceAdapter dispatcher (phase 8) - PREFERRED routing method
|
|
// 3. /workflow* to temporal handler - DEPRECATED: use X-Service: workflow instead
|
|
// 4. All other paths to upstream handler (phase 0-7)
|
|
func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
|
// Health endpoints first
|
|
switch req.URL.Path {
|
|
case "/healthz":
|
|
LivenessHandler(r.healthChecker)(w, req)
|
|
return
|
|
case "/readyz":
|
|
ReadinessHandler(r.healthChecker)(w, req)
|
|
return
|
|
case "/metrics":
|
|
if r.metricsHandler != nil {
|
|
r.metricsHandler.ServeHTTP(w, req)
|
|
return
|
|
}
|
|
}
|
|
|
|
// X-Service (ServiceAdapter) routing - checked before path-based routing
|
|
// PREFERRED: All service routing should use X-Service header pattern for consistency,
|
|
// auth enforcement, and resource-based access control.
|
|
if req.Header.Get("X-Service") != "" {
|
|
if r.dispatcher != nil {
|
|
r.dispatcher.Dispatch(w, req)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Forgejo webhook receiver — no auth, HMAC-verified by handler
|
|
if req.URL.Path == "/v1/webhooks/forgejo" {
|
|
r.forgejoWebhook.ServeHTTP(w, req)
|
|
return
|
|
}
|
|
|
|
// Workflow endpoints
|
|
// DEPRECATED: Path-based /workflow routing is legacy.
|
|
// New clients should use X-Service: workflow header instead for consistent auth.
|
|
switch req.URL.Path {
|
|
case "/workflow", "/workflow/health", "/workflow/metrics":
|
|
r.temporalHandler.ServeHTTP(w, req)
|
|
return
|
|
}
|
|
|
|
// Default: upstream handler (all other paths)
|
|
r.upstreamHandler.ServeHTTP(w, req)
|
|
}
|