rockandAdmin Bot
ef5e736673
CI / CI (push) Successful in 3m20s
## Summary Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw. --- ### #44 — Memory ServiceAdapter: standing-context resource - Code-registers `memory` adapter with `standing-context` resource (`POST /memory/standing-context`) - Fixes dispatcher bug: method-level `Auth.Capability` override now respected (was always using verb-derived capability) - Auth: `memory:read` required via method-level override; `memory:write` for ingest - `MEMORY_UPSTREAM_URL` env configures upstream ### #45 — Continuous batching: warmup deprioritisation + per-backend semaphore - In-memory bounded queue per backend: `VLLM_MAX_QUEUE_DEPTH` slots (default 20) - Concurrency limiter: `VLLM_MAX_NUM_SEQS` slots (default 4) - `X-Poimen-Warmup: true` → sleep `WARMUP_DELAY_MS` (default 500ms) before queue entry, strip header, add `X-Priority: low` - Queue full → 503 immediately, no goroutine pile-up - Metrics: `warmup_requests_delayed_total`, `queue_full_requests_total` at `/metrics` ### #46 — Cache-aware routing: rendezvous hash + prefix cache hit logging - `addresses: [...]` YAML field for multi-backend models (backward-compat with `address`) - `X-Poimen-Cache-Key` header → rendezvous-hash backend selection (same key = same GPU) - No cache key → round-robin - Failed backend excluded 30s; next-highest-score fallback - `vllm-num-cached-tokens` response header captured → `vllm_prefix_cache_hit_tokens_total` counter + structured log --- ## Verification ``` go build ./... ✅ go vet ./... ✅ go test -race ./... ✅ (all packages, 0 races) ``` ## Reviewer verdicts - reviewer-sre: SIGN-OFF - reviewer-quality: SIGN-OFF ## Non-blocking findings (🟡🔵 only) - `getOrCreateTransport` not goroutine-safe by design; pre-population invariant documented - `time.Sleep(warmupDelay)` holds goroutine; context-cancel alternative left as TODO - OTel trace_id not propagated into CAG log events (existing gap) Closes #44 Closes #45 Closes #46 --------- Co-authored-by: Admin Bot <[email protected]> Reviewed-on: #47
68 lines
1.7 KiB
Go
68 lines
1.7 KiB
Go
package serviceadapter
|
|
|
|
import "os"
|
|
|
|
// NewMemoryAdapter creates a code-registered ServiceAdapter for the memory service.
|
|
// Takes precedence over any config-file loaded memory adapter (main.go skips config
|
|
// adapters when a code-registered one with the same name exists).
|
|
// Upstream URL configured via MEMORY_UPSTREAM_URL env var.
|
|
func NewMemoryAdapter() *ServiceAdapter {
|
|
upstreamURL := os.Getenv("MEMORY_UPSTREAM_URL")
|
|
if upstreamURL == "" {
|
|
upstreamURL = "http://poimen-memory.poimen.svc.cluster.local:8080"
|
|
}
|
|
return &ServiceAdapter{
|
|
Namespace: "poimen",
|
|
ServiceName: "memory",
|
|
Spec: *GetMemorySpec(upstreamURL),
|
|
}
|
|
}
|
|
|
|
// GetMemorySpec returns the ServiceAdapter spec for the memory service.
|
|
func GetMemorySpec(upstreamURL string) *Spec {
|
|
return &Spec{
|
|
ServiceName: "memory",
|
|
Upstream: Upstream{
|
|
URL: upstreamURL,
|
|
TimeoutSeconds: 30,
|
|
},
|
|
Auth: Auth{
|
|
Required: true,
|
|
Capability: "memory:read",
|
|
},
|
|
Resources: []Resource{
|
|
{
|
|
Name: "ingest",
|
|
Methods: []Method{{
|
|
Verb: "POST",
|
|
UpstreamPath: "/memory/ingest",
|
|
Auth: &Auth{Required: true, Capability: "memory:write"},
|
|
}},
|
|
},
|
|
{
|
|
Name: "query",
|
|
Methods: []Method{{
|
|
Verb: "POST",
|
|
UpstreamPath: "/memory/query",
|
|
Auth: &Auth{Required: true, Capability: "memory:read"},
|
|
}},
|
|
},
|
|
{
|
|
Name: "skills",
|
|
Methods: []Method{{
|
|
Verb: "GET",
|
|
UpstreamPath: "/memory/skills",
|
|
}},
|
|
},
|
|
{
|
|
Name: "standing-context",
|
|
Methods: []Method{{
|
|
Verb: "POST",
|
|
UpstreamPath: "/memory/standing-context",
|
|
Auth: &Auth{Required: true, Capability: "memory:read"},
|
|
}},
|
|
},
|
|
},
|
|
}
|
|
}
|