11 Commits

Include renames

Author	SHA1	Message	Date
rockandAdmin Bot	ef5e736673	feat(cag): CAG integration — memory adapter, warmup semaphore, cache-aware routing (#47) ... ## Summary Three issues in one PR. Implements Cache-Augmented Generation (CAG) support in the api-gw. --- ### #44 — Memory ServiceAdapter: standing-context resource - Code-registers `memory` adapter with `standing-context` resource (`POST /memory/standing-context`) - Fixes dispatcher bug: method-level `Auth.Capability` override now respected (was always using verb-derived capability) - Auth: `memory:read` required via method-level override; `memory:write` for ingest - `MEMORY_UPSTREAM_URL` env configures upstream ### #45 — Continuous batching: warmup deprioritisation + per-backend semaphore - In-memory bounded queue per backend: `VLLM_MAX_QUEUE_DEPTH` slots (default 20) - Concurrency limiter: `VLLM_MAX_NUM_SEQS` slots (default 4) - `X-Poimen-Warmup: true` → sleep `WARMUP_DELAY_MS` (default 500ms) before queue entry, strip header, add `X-Priority: low` - Queue full → 503 immediately, no goroutine pile-up - Metrics: `warmup_requests_delayed_total`, `queue_full_requests_total` at `/metrics` ### #46 — Cache-aware routing: rendezvous hash + prefix cache hit logging - `addresses: [...]` YAML field for multi-backend models (backward-compat with `address`) - `X-Poimen-Cache-Key` header → rendezvous-hash backend selection (same key = same GPU) - No cache key → round-robin - Failed backend excluded 30s; next-highest-score fallback - `vllm-num-cached-tokens` response header captured → `vllm_prefix_cache_hit_tokens_total` counter + structured log --- ## Verification ``` go build ./... ✅ go vet ./... ✅ go test -race ./... ✅ (all packages, 0 races) ``` ## Reviewer verdicts - reviewer-sre: SIGN-OFF - reviewer-quality: SIGN-OFF ## Non-blocking findings (🟡🔵 only) - `getOrCreateTransport` not goroutine-safe by design; pre-population invariant documented - `time.Sleep(warmupDelay)` holds goroutine; context-cancel alternative left as TODO - OTel trace_id not propagated into CAG log events (existing gap) Closes #44 Closes #45 Closes #46 --------- Co-authored-by: Admin Bot <[email protected]> Reviewed-on: #47	2026-09-27 11:16:40 +00:00
rockandpoimen	d439536ca9	Add TTFT & ITL Metrics for LLM Inference (#28) ... Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) metrics for LLM inference observability Metrics: llm_ttft_seconds, llm_itl_seconds, llm_tokens_total Closes #31 #32 #33 --------- Co-authored-by: poimen <[email protected]> Reviewed-on: #28	2026-09-15 08:53:58 +00:00
Admin Bot	d7e1cbc62b	feat: implement gRPC forwarding for workflow adapter ... - Add HTTP/2 transport support for gRPC calls - Implement dispatchGRPC to forward requests to Temporal gRPC server - Replace 501 Not Implemented with actual gRPC proxy - Use golang.org/x/net/http2 for HTTP/2 protocol support - Supports ListWorkflowExecutions and other gRPC methods	2026-09-13 11:39:23 +09:00
Admin Bot	b8f95506ca	feat: add Temporal config and update routing with memory service integration ... - Add TemporalConfig struct to internal/config - Update gateway config with Temporal frontend service (port 7233) - Update router with memory service adapter support - Add config.local.yaml with memory service configuration - Encrypt production config with SOPS (AES256_GCM) - Support X-Service header routing pattern for service discovery - Keep legacy path-based routes with deprecation warnings - All 5 adapters preserved: workflow, memory, sqs, s3, iam	2026-09-13 10:56:18 +09:00
poimenandrock	74ecfe7107	feat(serviceadapter): enforce JWT auth on X-Service dispatch (#16) ... SQS dispatcher hardcoded a JWT validator pointing at authentik.riotpiao.com/application/o/sqs/jwks/ — provider doesn't exist. Every SQS request got 403 regardless of token. Co-authored-by: poimen <[email protected]>	2026-09-08 23:20:31 +00:00
Admin Bot	9d9395d938	feat: Phase 3.1 - SQS JWT validation against Authentik JWKS ... Implements gateway-level JWT validation for SQS requests: - Validates JWT signature against Authentik JWKS - Verifies claims: iss, aud, exp, nbf (with 60s skew) - Checks 'permissions' claim for sqs:read/sqs:write/wildcard - Returns 403 with error details on validation failure - JWKS caching with 15min TTL and auto-refresh on key rotation Architecture: - SQS: Gateway validates JWT (kmsvc code unverified) - MinIO, Temporal: Native JWT support (pass-through) - Memory, IAM: Service-owned JWT validation Integration tests added: - Reject requests without Authorization header (403) - Accept requests with valid JWT from Authentik - Pass through Authorization header unchanged for other services Uses github.com/MicahParks/keyfunc/v2 for JWKS handling: - Automatic refresh every 15 minutes - On-demand refresh if kid not found - Handles RS256 signatures	2026-08-27 11:40:35 -07:00
Admin Bot	95045e80f6	fix: SQS gateway JWT validation, MinIO/Temporal native JWT support ... Auth strategy clarified per service: - SQS: Gateway validates JWT (kmsvc code unverified, needs Phase 3) - MinIO: Native JWT/OIDC support (validates itself, Phase 3: load-test) - Temporal: Native JWT via jwtKeyProvider (Phase 3: configure) - Memory, IAM: Services validate JWTs (dumb pipe) SQS now requires Authorization header at gateway. Phase 3 will implement actual JWT signature validation against Authentik.	2026-08-27 11:33:04 -07:00
Admin Bot	139bc80529	refactor: dispatcher as dumb pipe + add gRPC detection for Temporal ... BREAKING CHANGE: Gateway no longer validates JWTs at dispatcher level. Each upstream service (MinIO, Authentik, Temporal) validates bearer tokens independently. Gateway passes Authorization header through unchanged. Changes: - Removed JWT validation from Dispatcher - Removed internal/auth JWT validator usage - Added gRPC URL scheme detection (grpc://) - Added temporal-frontend with gRPC config (returns 501 not-implemented) - All adapters now auth: required: false (services validate own tokens) - Gateway is now a transparent routing layer, not auth gateway gRPC forwarding requires grpcproxy middleware (future Phase 9). For now, gRPC clients should connect directly to temporal-frontend:7233.	2026-08-27 11:16:15 -07:00
Admin Bot	df33203a72	feat: add JWT validation against Authentik JWKS for protected adapters ... Replaces stub 'check Authorization header' auth with real JWT validation: - Extracts Bearer token from Authorization header - Validates signature against Authentik JWKS endpoint - Verifies iss, aud, exp claims - Checks permissions claim for required capability - Handles key rotation with 15min cache TTL - Returns 403 with detailed error on auth failure Protected adapters (memory, iam) now require valid Authentik JWT tokens.	2026-08-27 11:07:20 -07:00
Admin Bot	bd4cbbd0a3	feat: proxy requests to service adapter upstreams instead of echoing	2026-08-27 08:55:41 -07:00
Admin Bot	425611ec42	feat: phase 8 serviceadapter crd rollout (32/33 tasks)	2026-08-26 13:47:36 -07:00