feat(cag): CAG integration — memory adapter, warmup semaphore, cache-aware routing (#47)
CI / CI (push) Successful in 3m20s

## Summary

Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw.

---

### #44 — Memory ServiceAdapter: standing-context resource

- Code-registers `memory` adapter with `standing-context` resource (`POST /memory/standing-context`)
- Fixes dispatcher bug: method-level `Auth.Capability` override now respected (was always using verb-derived capability)
- Auth: `memory:read` required via method-level override; `memory:write` for ingest
- `MEMORY_UPSTREAM_URL` env configures upstream

### #45 — Continuous batching: warmup deprioritisation + per-backend semaphore

- In-memory bounded queue per backend: `VLLM_MAX_QUEUE_DEPTH` slots (default 20)
- Concurrency limiter: `VLLM_MAX_NUM_SEQS` slots (default 4)
- `X-Poimen-Warmup: true` → sleep `WARMUP_DELAY_MS` (default 500ms) before queue entry, strip header, add `X-Priority: low`
- Queue full → 503 immediately, no goroutine pile-up
- Metrics: `warmup_requests_delayed_total`, `queue_full_requests_total` at `/metrics`

### #46 — Cache-aware routing: rendezvous hash + prefix cache hit logging

- `addresses: [...]` YAML field for multi-backend models (backward-compat with `address`)
- `X-Poimen-Cache-Key` header → rendezvous-hash backend selection (same key = same GPU)
- No cache key → round-robin
- Failed backend excluded 30s; next-highest-score fallback
- `vllm-num-cached-tokens` response header captured → `vllm_prefix_cache_hit_tokens_total` counter + structured log

---

## Verification

```
go build ./...    ✅
go vet ./...      ✅
go test -race ./... ✅ (all packages, 0 races)
```

## Reviewer verdicts

- reviewer-sre: SIGN-OFF
- reviewer-quality: SIGN-OFF

## Non-blocking findings (🟡🔵 only)

- `getOrCreateTransport` not goroutine-safe by design; pre-population invariant documented
- `time.Sleep(warmupDelay)` holds goroutine; context-cancel alternative left as TODO
- OTel trace_id not propagated into CAG log events (existing gap)

Closes #44 Closes #45 Closes #46

---------

Co-authored-by: Admin Bot <[email protected]>
Reviewed-on: #47
This commit was merged in pull request #47.
This commit is contained in:
2026-09-27 11:16:40 +00:00
co-authored by Admin Bot
parent aaa9a5d1cb
commit ef5e736673
16 changed files with 1597 additions and 162 deletions
+17 -8
View File
@@ -37,9 +37,10 @@ type rawRoute struct {
// rawModel represents a single model entry in the YAML configuration.
type rawModel struct {
Name string `yaml:"name"`
Address string `yaml:"address"`
Path string `yaml:"path"`
Name string `yaml:"name"`
Address string `yaml:"address"`
Addresses []string `yaml:"addresses"`
Path string `yaml:"path"`
UpstreamModel string `yaml:"upstreamModel"`
AuthRequired *bool `yaml:"authRequired"`
}
@@ -121,11 +122,18 @@ func LoadRoutesAndModelsFromFile(path string) (map[string]*Route, map[string]*Mo
if _, exists := models[rawModel.Name]; exists {
return nil, nil, nil, AuthConfig{}, fmt.Errorf("duplicate model: \"%s\"", rawModel.Name)
}
if rawModel.Address == "" {
return nil, nil, nil, AuthConfig{}, fmt.Errorf("model \"%s\": field 'address' is required", rawModel.Name)
// Support both 'address' (single) and 'addresses' (multi-backend) YAML fields.
addresses := rawModel.Addresses
if len(addresses) == 0 && rawModel.Address != "" {
addresses = []string{rawModel.Address}
}
if _, _, err := net.SplitHostPort(rawModel.Address); err != nil {
return nil, nil, nil, AuthConfig{}, fmt.Errorf("model \"%s\": invalid address \"%s\": %w", rawModel.Name, rawModel.Address, err)
if len(addresses) == 0 {
return nil, nil, nil, AuthConfig{}, fmt.Errorf("model \"%s\": 'address' or 'addresses' is required", rawModel.Name)
}
for _, addr := range addresses {
if _, _, err := net.SplitHostPort(addr); err != nil {
return nil, nil, nil, AuthConfig{}, fmt.Errorf("model \"%s\": invalid address \"%s\": %w", rawModel.Name, addr, err)
}
}
// Default authRequired to global auth.enabled if not specified per-model
authRequired := false
@@ -134,7 +142,8 @@ func LoadRoutesAndModelsFromFile(path string) (map[string]*Route, map[string]*Mo
}
models[rawModel.Name] = &ModelUpstream{
Name: rawModel.Name,
Address: rawModel.Address,
Address: addresses[0],
Addresses: addresses,
Path: rawModel.Path,
UpstreamModel: rawModel.UpstreamModel,
AuthRequired: authRequired,