riotpiao-poimen/homelab-frontend
2
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

CAG-3: Cache-aware routing (rendezvous hash) + prefix cache hit logging #46

New Issue
Closed
opened 2026-09-27 09:58:00 +00:00 by rock · 1 comment
rock commented 2026-09-27 09:58:00 +00:00
Owner
Copy Link
Copy Source

Context

To maximise KV cache reuse, requests sharing the same context prefix must hit the same GPU. vLLM 0.11.0 returns vllm-num-cached-tokens in response headers.

Requirement

Implement cache-aware backend selection using rendezvous hashing on X-Poimen-Cache-Key. Log prefix cache hit stats.

Acceptance Criteria

  • AC1: Multiple backend addresses per model via addresses: [...] in config (backward compat with single address)
  • AC2: Requests with X-Poimen-Cache-Key → rendezvous-hash backend selection (same key always → same backend)
  • AC3: Requests without key → round-robin across backends
  • AC4: Failed backend excluded from routing for 30s; next-highest-score backend used (next-in-ring fallback)
  • AC5: X-Poimen-Cache-Key forwarded to vLLM upstream automatically
  • AC6: vllm-num-cached-tokens extracted from response; structured log emitted: {event,cache_key,cached_tokens,total_tokens,hit_rate}
  • AC7: vllm_prefix_cache_hit_tokens_total counter exported at /metrics

Files

  • internal/config/config.go
  • internal/config/loader.go
  • internal/proxy/proxy.go (wires scheduler from CAG-2)
  • internal/observability/metrics.go
  • internal/observability/prometheus.go
  • internal/logging/logger.go
## Context To maximise KV cache reuse, requests sharing the same context prefix must hit the same GPU. vLLM 0.11.0 returns `vllm-num-cached-tokens` in response headers. ## Requirement Implement cache-aware backend selection using rendezvous hashing on `X-Poimen-Cache-Key`. Log prefix cache hit stats. ## Acceptance Criteria - AC1: Multiple backend addresses per model via `addresses: [...]` in config (backward compat with single `address`) - AC2: Requests with `X-Poimen-Cache-Key` → rendezvous-hash backend selection (same key always → same backend) - AC3: Requests without key → round-robin across backends - AC4: Failed backend excluded from routing for 30s; next-highest-score backend used (next-in-ring fallback) - AC5: `X-Poimen-Cache-Key` forwarded to vLLM upstream automatically - AC6: `vllm-num-cached-tokens` extracted from response; structured log emitted: `{event,cache_key,cached_tokens,total_tokens,hit_rate}` - AC7: `vllm_prefix_cache_hit_tokens_total` counter exported at `/metrics` ## Files - `internal/config/config.go` - `internal/config/loader.go` - `internal/proxy/proxy.go` (wires scheduler from CAG-2) - `internal/observability/metrics.go` - `internal/observability/prometheus.go` - `internal/logging/logger.go`
rock closed this issue 2026-09-27 11:16:42 +00:00
rock commented 2026-09-27 11:17:16 +00:00
Author
Owner
Copy Link
Copy Source

Harness review — merged

RUN_ID: riotpiao-poimen/homelab-frontend#44-45-46-2026-09-27T09:35Z
PR: #47 (merged) Diff: 16 files +1603 / -168

Coders

  • coder-impl: DONE — scheduler.go (new), memory_adapter.go (new), proxy.go, router.go, config.go/loader.go, metrics.go/prometheus.go, logger.go, main.go, server/router.go, serviceadapter/router.go
  • coder-tests: DONE — scheduler_test.go (new), +JWT/queue/cache tests in proxy_test.go, +auth coverage in router_test.go, +addresses in models_test.go

Verification

go build ./... && go vet ./... && go test -race ./... → all pass, 0 races

reviewer-sre — SIGN-OFF

0🔴 3🟡 2🔵 — queue-full logged + counted, auth-bypass warned, all 3 CAG counters at /metrics

reviewer-quality — SIGN-OFF

CRAP: ServeHTTP 9.0 ✅ authenticateV1 8.5 ✅ — all 7 ACs covered by tests

Verdict: Merged ✅

## Harness review — merged **RUN_ID:** `riotpiao-poimen/homelab-frontend#44-45-46-2026-09-27T09:35Z` **PR:** #47 (merged) **Diff:** 16 files +1603 / -168 ### Coders - coder-impl: DONE — scheduler.go (new), memory_adapter.go (new), proxy.go, router.go, config.go/loader.go, metrics.go/prometheus.go, logger.go, main.go, server/router.go, serviceadapter/router.go - coder-tests: DONE — scheduler_test.go (new), +JWT/queue/cache tests in proxy_test.go, +auth coverage in router_test.go, +addresses in models_test.go ### Verification `go build ./... && go vet ./... && go test -race ./...` → all pass, 0 races ### reviewer-sre — SIGN-OFF 0🔴 3🟡 2🔵 — queue-full logged + counted, auth-bypass warned, all 3 CAG counters at /metrics ### reviewer-quality — SIGN-OFF CRAP: ServeHTTP 9.0 ✅ authenticateV1 8.5 ✅ — all 7 ACs covered by tests ### Verdict: Merged ✅
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/auth
Authentication & authorization
 area/authentik
Authentik OIDC provider config
 area/downstream
Downstream service changes
 area/gateway
API gateway code
 priority/high
Must have
 priority/low
Nice to have
 priority/medium
Should have
 status/blocked
Blocked by dependency
 status/done
Completed
 status/in-progress
Currently being worked on
 status/review
Ready for review
 status/todo
Not started
 type/docs
Documentation
 type/feature
New feature
 type/infra
Infrastructure change
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
poimen rock (Rock)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#46