Compare commits

...
Author SHA1 Message Date
Admin Bot ba8800d565 fix(gateway): set jwksUrl so the dispatcher builds a validator
CI / CI (pull_request) Successful in 2m9s
CI / CI (push) Successful in 3m29s
cmd/gateway/main.go:144 constructs the adapter validator only when both
cfg.Auth.Enabled and cfg.Auth.JWKSURL are set. The config left jwksUrl unset on
purpose, so jwtValidator stayed nil and every adapter request passed with one
warning line:

  auth required but jwtValidator nil; request unauthenticated

That made auth.required cosmetic: sqs, workflow, memory, s3 and iam all answered
unauthenticated calls. Set the JWKS URL to the ZITADEL key endpoint.
2026-10-27 01:17:09 -07:00
Admin Bot 29f5c64fab feat(gateway): require a token on every adapter
CI / CI (pull_request) Successful in 3m0s
CI / CI (push) Successful in 3m26s
workflow, memory, s3 and iam served requests without any Authorization header.
Only sqs and the LLM models checked a token, so four upstreams were reachable
from inside the cluster with no authentication at all.

Set auth.required true on all five adapters. The dispatcher then enforces one
capability per method, derived from the verb
(internal/serviceadapter/router.go:142):

  GET/HEAD/OPTIONS -> "<service>:read"
  anything else    -> "<service>:write"

Those strings must exist as ZITADEL role keys with the colon. The project now
holds memory:read, memory:write, workflow:read, workflow:execute, s3:read,
s3:write, iam:read, iam:write, sqs:read and sqs:write, and the riotpiao user
grant carries them, so a browser session keeps working.

Correct the old comment: ZITADEL does store colon role keys. Verified by listing
the project roles, which returned all eleven colon keys.
2026-10-27 01:11:18 -07:00
Admin Bot 8875f36dde feat(auth): validate access tokens with the ZITADEL Go SDK
Replace the handwritten Authentik JWT checks with github.com/zitadel/zitadel-go/v3.

What the SDK now owns:
- OIDC discovery (client.Discover) to find issuer and JWKS URI
- remote, auto-refreshing JWKS (rp.NewRemoteKeySet)
- signature, expiry and issuer checks (op.NewAccessTokenVerifier,
  op.VerifyAccessToken)
- the verified claim set and role lookup (oauth.IntrospectionContext,
  IsGrantedRole)

What stays local:
- a multi-audience allow-list, because this gateway is a shared resource
  server and the SDK's oauth.WithJWT accepts exactly one client ID
- an adapter from IntrospectionContext to jwt.MapClaims so the rest of the
  gateway keeps one claim representation

Other changes:
- issuer is configuration-driven; the hardcoded authentik.riotpiao.com
  check is gone
- aud may be a string or an array (ZITADEL sends an array)
- CheckPermissions also reads urn:zitadel:iam:org:project:roles, the claim
  ZITADEL uses for project roles
- requiredCapability accepts a comma-separated list, so llm:inference and
  llm-inference both satisfy the LLM gate
- a five-segment token is reported as an encrypted JWE with the fix in the
  message, instead of failing as a bad signature

Config: auth.zitadelDomain turns the SDK path on; jwksUrl stays supported for
the legacy path and for tests.
2026-10-27 01:11:18 -07:00
727b9e1cb0 fix(proxy): admission-control review fixes missing from the #65 merge (#66)
CI / CI (push) Successful in 43m8s
Follow-up to #65, which is already merged. The review of that PR returned BLOCK with two high-severity defects; the fix commit was pushed **after** the merge landed, so none of it is in `main`.

## What happened

- `main` @ `369d51a` is the squash of the first commit only (`c5a91cf`).
- `d71c884` (review fixes) is on the old branch and was never merged — PR #65 is closed.
- Verified against `main` directly: `ADMISSION_MAX_INFLIGHT` 0 hits, `func (rw *responseWriter) Flush` 0 hits, `ModifyResponse` 0 hits, `Snapshot` 0 hits.
- The image in production (`forgejo.riotpiao.com/rock/api-gateway@sha256:64f2bfc1…`, 3/3 replicas) is built from that squash, so the defects below are **live right now**.

## Live state confirmed

`https://api.riotpiao.com/metrics` on the running pods:

```
observer_degraded 0
vllm_requests_running 1
vllm_kv_usage_perc 0.008293092588704587
admission_parked_total 0
engine_state_stale_total 0
```

The observer itself works: it is reading the real vLLM exposition body and the startup self-check passed (`observer_degraded 0`). No requests have parked yet, so the two defects below have not been observable in the wild — they are proven from code, not from a live failure.

## Defects fixed here

**1. The parked-stream keep-alive never runs in production (high).** `internal/tracing/middleware.go` wraps the `ResponseWriter` in a struct that embeds the interface and implements only `WriteHeader`/`Write`, so `w.(http.Flusher)` in `internal/proxy/proxy.go` is false for every real request (`cmd/gateway/main.go` → `tracing.Middleware` → `router`). `prewritten` therefore never becomes true, the heartbeat never starts, and a parked streaming request writes zero bytes for up to `ADMISSION_MAX_PARK_SECONDS` (120) — the client timeout the feature exists to prevent. Fixed by adding `Flush()` and `Unwrap()` to the wrapper.

**2. Upstream status masked once the SSE status is spent (high, latent behind 1).** `httputil.ReverseProxy` calls `rw.WriteHeader(res.StatusCode)` unconditionally; after the park wrote 200, net/http keeps 200 and the upstream error body is appended raw into the event stream, which a streaming client reads as an empty completion and reports as success. Fixed with a `ModifyResponse` that frames an upstream `>= 400` as an `event: error` when the status has already been written.

**3. Lost wakeup in the park loop.** `Get()` and `Tick()` were taken under two separate lock acquisitions, so a poll landing between them pushed the waiter onto the next generation's channel — up to one extra poll interval (2s) parked. `Snapshot()` now returns state and channel under one lock.

**4. `observer_degraded` pinned at 1 forever.** It was raised only on the *first* poll's parse failure and `SetObserverDegraded(false)` had no call site, so one truncated response pinned the gauge for the pod's life, while genuine metric-name drift appearing later was never reported at all. `markDegraded` now tracks current parsability in both directions and only touches the exporter on a change.

**5. `ADMISSION_MAX_INFLIGHT` silently defaulted.** Now set explicitly to `4` in `k8s/deployment.yaml` with the coupling to the engine's `--max-num-seqs` documented, instead of quietly inheriting `VLLM_MAX_NUM_SEQS`.

## Verification

`go build ./...` clean · `go vet ./...` clean · `go test ./...` 368 passed, 0 failed · `structural-check.sh` 0 errors (1 pre-existing `:latest` warning) · `go.mod` unchanged.

Cherry-picked onto `main` rather than merged from the old branch, so this diff is exactly the five fixes and nothing else: 5 files, +80/−8.

## Still open

The reviewer's BLOCK also rested on nine functions over CRAP>30 purely from 0% coverage — the deliberate "no new tests" decision. Worth recording that defect 1 is a one-line type assertion that a `httptest` case through the real middleware chain would have caught immediately. Raising it here so the decision is visible rather than implied.

---------

Co-authored-by: Poimen <[email protected]>
Reviewed-on: #66
Co-authored-by: poimen <[email protected]>
2026-09-30 12:19:24 +00:00
15 changed files with 875 additions and 72 deletions
+15 -3
View File
@@ -5,39 +5,51 @@ go 1.26.0
require (
github.com/MicahParks/keyfunc/v2 v2.1.0
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/zitadel/oidc/v3 v3.49.2
github.com/zitadel/zitadel-go/v3 v3.30.0
go.opentelemetry.io/otel v1.46.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0
go.opentelemetry.io/otel/sdk v1.46.0
go.opentelemetry.io/otel/trace v1.46.0
go.temporal.io/api v1.63.5
go.temporal.io/sdk v1.48.0
golang.org/x/net v0.58.0
google.golang.org/grpc v1.83.2
google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1
)
require (
github.com/bmatcuk/doublestar/v4 v4.10.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a // indirect
github.com/go-chi/chi/v5 v5.3.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.3 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
github.com/muhlemmer/gu v0.3.1 // indirect
github.com/muhlemmer/httpforwarded v0.1.0 // indirect
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 // indirect
github.com/nexus-rpc/sdk-go v0.7.0 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/rs/cors v1.11.1 // indirect
github.com/stretchr/objx v0.5.3 // indirect
github.com/stretchr/testify v1.12.1 // indirect
github.com/zitadel/schema v1.3.2 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
go.opentelemetry.io/otel/metric v1.46.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.41.0 // indirect
+26
View File
@@ -1,11 +1,19 @@
github.com/MicahParks/keyfunc/v2 v2.1.0 h1:6ZXKb9Rp6qp1bDbJefnG7cTH8yMN1IC/4nf+GVjO99k=
github.com/MicahParks/keyfunc/v2 v2.1.0/go.mod h1:rW42fi+xgLJ2FRRXAfNx9ZA8WpD4OeE/yHVMteCkw9k=
github.com/bmatcuk/doublestar/v4 v4.10.0 h1:zU9WiOla1YA122oLM6i4EXvGW62DvKZVxIe6TYWexEs=
github.com/bmatcuk/doublestar/v4 v4.10.0/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/envoyproxy/protoc-gen-validate v1.3.3 h1:MVQghNeW+LZcmXe7SY1V36Z+WFMDjpqGAGacLe2T0ds=
github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0=
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a h1:yDWHCSQ40h88yih2JAcL6Ls/kVkSE8GFACTGVnMPruw=
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a/go.mod h1:7Ga40egUymuWXxAe151lTNnCv97MddSOVsjpPPkityA=
github.com/go-chi/chi/v5 v5.3.1 h1:3j4HZLGZQ3JpMCrPJF/Jl3mYJfWLKBfNJ6quurUGCf8=
github.com/go-chi/chi/v5 v5.3.1/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -23,8 +31,12 @@ github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/securecookie v1.1.2 h1:YCIWL56dvtr73r6715mJs5ZvhtnY73hBvEF8kXD8ePA=
github.com/gorilla/securecookie v1.1.2/go.mod h1:NfCASbcHqRSY+3a8tlWJwsQap2VX5pwzwo4h3eOamfo=
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2 h1:sGm2vDRFUrQJO/Veii4h4zG2vvqG6uWNkBHSTqXOZk0=
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2/go.mod h1:wd1YpapPLivG6nQgbf7ZkG1hhSOXDhhn4MLTknx2aAc=
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.3 h1:B+8ClL/kCQkRiU82d9xajRPKYMrB7E0MbtzWVi1K4ns=
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.3/go.mod h1:NbCUVmiS4foBGBHOYlCT25+YmGpJ32dZPi75pGEUpj4=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@@ -33,6 +45,10 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/muhlemmer/gu v0.3.1 h1:7EAqmFrW7n3hETvuAdmFmn4hS8W+z3LgKtrnow+YzNM=
github.com/muhlemmer/gu v0.3.1/go.mod h1:YHtHR+gxM+bKEIIs7Hmi9sPT3ZDUvTN/i88wQpZkrdM=
github.com/muhlemmer/httpforwarded v0.1.0 h1:x4DLrzXdliq8mprgUMR0olDvHGkou5BJsK/vWUetyzY=
github.com/muhlemmer/httpforwarded v0.1.0/go.mod h1:yo9czKedo2pdZhoXe+yDkGVbU0TJ0q9oQ90BVoDEtw0=
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 h1:2fELd+9sqUtNu6Fg//pw8YFsxOvp8vZ8hfP0nHhNI80=
github.com/nexus-rpc/nexus-proto-annotations v0.1.0/go.mod h1:n3UjF1bPCW8llR8tHvbxJ+27yPWrhpo8w/Yg1IOuY0Y=
github.com/nexus-rpc/sdk-go v0.7.0 h1:38NrfY5rLnZAiMMs2ZfCKI/CSDzdfJG+27iAgfA8bUI=
@@ -41,6 +57,8 @@ github.com/robfig/cron v1.2.0 h1:ZjScXvvxeQ63Dbyxy76Fj3AT3Ut0aKsyd2/tl3DTMuQ=
github.com/robfig/cron v1.2.0/go.mod h1:JGuDeoQd7Z6yL4zQhZ3OPEVHB7fL6Ka6skscFHfmt2k=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/cors v1.11.1 h1:eU3gRzXLRK57F5rKMGMZURNdIG4EoAmX8k94r9wXWHA=
github.com/rs/cors v1.11.1/go.mod h1:XyqrcTp5zjWr1wsJ8PIRZssZ8b/WMcMf71DJnit4EMU=
github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4=
github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
@@ -48,6 +66,12 @@ github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUm
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/zitadel/oidc/v3 v3.49.2 h1:yKvB2Hx6rVWp0vOTk1pEyXAPxJusgIHNmr20AxOohmQ=
github.com/zitadel/oidc/v3 v3.49.2/go.mod h1:HwoguOGo0eem0RK5Gb+P6Q4aQLVinJ9LhomlVEA57ck=
github.com/zitadel/schema v1.3.2 h1:gfJvt7dOMfTmxzhscZ9KkapKo3Nei3B6cAxjav+lyjI=
github.com/zitadel/schema v1.3.2/go.mod h1:IZmdfF9Wu62Zu6tJJTH3UsArevs3Y4smfJIj3L8fzxw=
github.com/zitadel/zitadel-go/v3 v3.30.0 h1:yk/surQ+atLS/ocGaaJdHD/R6VbveFScihxJhIkGZ3Q=
github.com/zitadel/zitadel-go/v3 v3.30.0/go.mod h1:4R+aki0JFMwIPsr4CIVLYq5DRTVEAfLbehJpFNhFKQw=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
@@ -87,6 +111,8 @@ golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+21
View File
@@ -0,0 +1,21 @@
package auth
import (
"strings"
"testing"
)
func TestValidateBearerToken_JWEGetsClearError(t *testing.T) {
ti := newTestIssuer(t)
host, port, _ := strings.Cut(strings.TrimPrefix(ti.server.URL, "http://"), ":")
v, err := NewZitadelValidator(t.Context(), ti.server.URL, nil, ZitadelConfig{
Domain: host, InsecurePort: port, HTTPClient: ti.server.Client(),
})
if err != nil {
t.Fatalf("NewZitadelValidator: %v", err)
}
_, err = v.ValidateBearerToken("Bearer a.b.c.d.e")
if err == nil || !strings.Contains(err.Error(), "encrypted JWE") {
t.Fatalf("expected a JWE-specific error, got %v", err)
}
}
+188 -30
View File
@@ -3,37 +3,141 @@ package auth
import (
"context"
"fmt"
gohttp "net/http"
"strings"
"sync"
"time"
"github.com/MicahParks/keyfunc/v2"
"github.com/golang-jwt/jwt/v5"
"github.com/zitadel/zitadel-go/v3/pkg/authorization/oauth"
)
// isValidIssuer checks if issuer is from Authentik (any provider/app).
// Accepts: https://authentik.riotpiao.com/application/o/{provider}/
func isValidIssuer(iss string) bool {
return strings.Contains(iss, "authentik.riotpiao.com/application/o/") &&
strings.HasSuffix(iss, "/")
// issuerMatches reports whether iss is the configured issuer or a sub-path of
// it. Zitadel emits exactly "https://zitadel.riotpiao.com"; providers that
// mount an application path append a suffix (Authentik used
// "/application/o/<app>/"). A trailing slash on either side is ignored.
//
// An empty configured issuer disables the domain check; the JWKS signature
// check still applies. Production configuration always sets an issuer.
func issuerMatches(iss, configured string) bool {
if configured == "" {
return iss != ""
}
iss = strings.TrimSuffix(iss, "/")
configured = strings.TrimSuffix(configured, "/")
return iss == configured || strings.HasPrefix(iss, configured+"/")
}
// Validator validates JWTs against Authentik JWKS.
// Supports multi-issuer: any Authentik service account provider is accepted
// (portfolio-agent, memory-agent, api-gw, etc.) because all share the same
// JWKS signing key.
// audienceValues normalises the aud claim. Zitadel access tokens carry an
// array, older providers carry a single string.
func audienceValues(claims jwt.MapClaims) ([]string, bool) {
switch v := claims["aud"].(type) {
case string:
return []string{v}, v != ""
case []string:
return v, len(v) > 0
case []interface{}:
out := make([]string, 0, len(v))
for _, item := range v {
if s, ok := item.(string); ok {
out = append(out, s)
}
}
return out, len(out) > 0
case jwt.ClaimStrings:
return []string(v), len(v) > 0
}
return nil, false
}
// Validator validates JWTs against the JWKS of an OIDC provider (Zitadel).
// The issuer is configuration-driven: tokens whose iss claim equals the
// configured issuer, or lives under it as a sub-path, are accepted.
//
// When a ZitadelVerifier is attached (see NewZitadelValidator), validation
// delegates to the official ZITADEL Go SDK and this type only adapts the
// result to jwt.MapClaims for the rest of the gateway.
type Validator struct {
issuer string // Not used for validation (kept for logging); issuer regex check is sufficient
audience string // Not used for validation; any audience from valid Authentik issuer is accepted
issuer string // Expected issuer (iss claim); empty disables the domain check
audience string // Expected audience; currently logged only (any non-empty aud is accepted)
jwksURL string
jwks *keyfunc.JWKS
mu sync.Mutex
zitadel *ZitadelVerifier // non-nil when backed by the ZITADEL SDK
}
// NewZitadelValidator builds a Validator backed by the ZITADEL Go SDK.
// issuer is the expected iss claim, used for logging and for the legacy
// claim checks if the SDK verifier is ever bypassed. clientIDs is the
// audience allow-list. Discovery runs here, so a wrong domain fails at
// startup.
func NewZitadelValidator(ctx context.Context, issuer string, clientIDs []string, cfg ZitadelConfig) (*Validator, error) {
return NewZitadelValidatorWithClient(ctx, issuer, clientIDs, cfg, nil)
}
// NewZitadelValidatorWithClient is NewZitadelValidator with an injected HTTP
// client, for tests that serve discovery and JWKS from httptest.
func NewZitadelValidatorWithClient(ctx context.Context, issuer string, clientIDs []string, cfg ZitadelConfig, httpClient *gohttp.Client) (*Validator, error) {
cfg.ClientIDs = clientIDs
if cfg.HTTPClient == nil {
cfg.HTTPClient = httpClient
}
sdk, err := NewZitadelVerifier(ctx, cfg)
if err != nil {
return nil, err
}
return &Validator{
issuer: issuer,
jwksURL: "", // unused on the SDK path
zitadel: sdk,
}, nil
}
// claimsFromIntrospection converts the SDK's verified claim set into
// jwt.MapClaims so the rest of the gateway keeps one claim representation.
// The SDK has already checked the signature, exp, nbf and iss; this only
// copies values across.
func claimsFromIntrospection(authCtx *oauth.IntrospectionContext) jwt.MapClaims {
claims := jwt.MapClaims{}
if authCtx == nil {
return claims
}
for k, v := range authCtx.Claims {
claims[k] = v
}
if authCtx.Issuer != "" {
claims["iss"] = authCtx.Issuer
}
if authCtx.Subject != "" {
claims["sub"] = authCtx.Subject
}
if len(authCtx.Audience) > 0 {
aud := make([]interface{}, 0, len(authCtx.Audience))
for _, a := range authCtx.Audience {
aud = append(aud, a)
}
claims["aud"] = aud
}
if authCtx.Expiration != 0 {
claims["exp"] = float64(authCtx.Expiration)
}
if authCtx.NotBefore != 0 {
claims["nbf"] = float64(authCtx.NotBefore)
}
if authCtx.IssuedAt != 0 {
claims["iat"] = float64(authCtx.IssuedAt)
}
if authCtx.ClientID != "" {
claims["client_id"] = authCtx.ClientID
}
return claims
}
// NewValidator creates a new JWT validator for a service.
// issuer and audience params are deprecated (ignored for validation) but kept
// for backward compatibility. Multi-issuer validation via isValidIssuer() is used instead.
// JWKS fetching is lazy (deferred until first validation).
// issuer is the expected iss claim (exact value or prefix). audience is kept
// for compatibility; only non-emptiness is enforced. JWKS fetching is lazy
// (deferred until first validation).
func NewValidator(issuer, audience, jwksURL string) *Validator {
return &Validator{
issuer: issuer, // deprecated param, kept for compat
@@ -87,9 +191,32 @@ func (v *Validator) ValidateBearerToken(authHeader string) (jwt.MapClaims, error
}
// Ensure JWKS is loaded (lazy)
if v.jwks != nil || v.zitadel == nil {
if err := v.ensureJWKS(); err != nil {
return nil, err
}
}
// SDK path: ZITADEL Go SDK performs signature, expiry and issuer checks.
if v.zitadel != nil {
// ZITADEL issues two access-token shapes. A plain JWS has three
// dot-separated segments and can be verified locally. A JWE has five,
// is encrypted with the instance key, and only ZITADEL can read it.
// Report that case explicitly so the operator changes the client to
// OIDC_TOKEN_TYPE_BEARER instead of debugging a signature failure.
if strings.Count(tokenString, ".") == 4 {
return nil, fmt.Errorf("token is an encrypted JWE (5 segments); " +
"set accessTokenType=OIDC_TOKEN_TYPE_BEARER on the ZITADEL client so the gateway can verify it locally")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
authCtx, err := v.zitadel.CheckAuthorization(ctx, authHeader)
if err != nil {
return nil, fmt.Errorf("token validation failed: %v", err)
}
return claimsFromIntrospection(authCtx), nil
}
// Parse and validate
claims := jwt.MapClaims{}
@@ -120,34 +247,35 @@ func (v *Validator) ValidateBearerToken(authHeader string) (jwt.MapClaims, error
}
}
// Check iss (issuer) - accept any Authentik provider issuer
// (portfolio-agent, memory-agent, api-gw, etc.)
// All use same signing key so JWKS validation is sufficient
// Check iss against the configured issuer (exact match or sub-path).
if iss, ok := claims["iss"].(string); !ok {
return nil, fmt.Errorf("missing issuer claim")
} else if !isValidIssuer(iss) {
return nil, fmt.Errorf("invalid issuer: %s", iss)
} else if !issuerMatches(iss, v.issuer) {
return nil, fmt.Errorf("invalid issuer: %s (want %s)", iss, v.issuer)
}
// Check aud (audience) - accept any Authentik-provided audience
// since all Authentik service accounts use the same signing key.
// The issuer check above is sufficient to ensure JWT came from Authentik.
if aud, ok := claims["aud"].(string); !ok {
// Check aud - Zitadel sends an array; accept a string or array.
if auds, ok := audienceValues(claims); !ok {
return nil, fmt.Errorf("missing audience claim")
} else if aud == "" {
} else if len(auds) == 0 {
return nil, fmt.Errorf("empty audience claim")
}
// Note: Not hardcoding expected audience. Any audience from a valid Authentik
// issuer is accepted, since all service accounts are under the same trust boundary.
// The signature check plus the issuer check establish the trust boundary;
// the audience value itself is not hardcoded.
return claims, nil
}
// CheckPermissions checks if claims contain required permission(s).
// Checks both "permissions" claim (for users) and "roles" claim (for service accounts).
// Understands three token shapes:
// - "permissions": []string (generic / legacy tokens)
// - "roles": []string or map[string]… (legacy service-account tokens)
// - "urn:zitadel:iam:org:project:roles": map[string]… (Zitadel access tokens,
// where the map keys are role keys)
//
// Returns true if any required permission is found or wildcard "*" exists.
func (v *Validator) CheckPermissions(claims jwt.MapClaims, required ...string) bool {
// Try permissions claim first (for user tokens)
// 1. permissions claim (array of strings)
if permsIface, ok := claims["permissions"]; ok {
if perms, ok := permsIface.([]interface{}); ok {
if v.checkPermList(perms, required...) {
@@ -156,15 +284,45 @@ func (v *Validator) CheckPermissions(claims jwt.MapClaims, required ...string) b
}
}
// Fall back to roles claim (for service account tokens)
// 2. roles claim: array (legacy) or map keyed by role name
if rolesIface, ok := claims["roles"]; ok {
if roles, ok := rolesIface.([]interface{}); ok {
switch roles := rolesIface.(type) {
case []interface{}:
if v.checkPermList(roles, required...) {
return true
}
case map[string]interface{}:
if v.checkPermMap(roles, required...) {
return true
}
}
}
// 3. Zitadel access-token role assertion: map keyed by role key
if rolesIface, ok := claims["urn:zitadel:iam:org:project:roles"]; ok {
if roles, ok := rolesIface.(map[string]interface{}); ok {
if v.checkPermMap(roles, required...) {
return true
}
}
}
return false
}
// checkPermMap reports whether any key in a role map matches a required
// permission, or the map carries a wildcard key.
func (v *Validator) checkPermMap(roles map[string]interface{}, required ...string) bool {
for roleKey := range roles {
if roleKey == "*" {
return true
}
for _, req := range required {
if roleKey == req {
return true
}
}
}
return false
}
+157
View File
@@ -0,0 +1,157 @@
package auth
import (
"context"
"errors"
"fmt"
gohttp "net/http"
"slices"
"strings"
"github.com/zitadel/oidc/v3/pkg/client"
"github.com/zitadel/oidc/v3/pkg/client/rp"
"github.com/zitadel/oidc/v3/pkg/oidc"
"github.com/zitadel/oidc/v3/pkg/op"
"github.com/zitadel/zitadel-go/v3/pkg/authorization/oauth"
"github.com/zitadel/zitadel-go/v3/pkg/zitadel"
)
// ErrAudienceNotAllowed reports a token whose aud claim names none of the
// configured resource-server client IDs.
var ErrAudienceNotAllowed = errors.New("audience not allowed")
// ZitadelConfig configures the ZITADEL-backed verifier.
type ZitadelConfig struct {
// Domain is the ZITADEL instance host, for example "zitadel.riotpiao.com".
Domain string
// InsecurePort, when non-empty, speaks plain HTTP to that port instead of
// HTTPS on 443. Use only for a local port-forward.
InsecurePort string
// ClientIDs is the audience allow-list: a token is accepted when its aud
// claim contains at least one of these client IDs. Empty accepts any
// non-empty audience, which trusts every client of the same instance.
ClientIDs []string
// HTTPClient overrides the client used for discovery and JWKS fetches.
HTTPClient *gohttp.Client
}
// ZitadelVerifier validates ZITADEL access tokens with the official
// github.com/zitadel/zitadel-go SDK.
//
// It reuses the SDK's building blocks rather than reimplementing JWT parsing:
// - zitadel.New + Origin to address the instance
// - client.Discover for OIDC discovery (issuer and JWKS URI)
// - rp.NewRemoteKeySet for cached, auto-refreshing signing keys
// - op.NewAccessTokenVerifier / op.VerifyAccessToken for signature, expiry
// and issuer checks
// - oauth.IntrospectionContext for the verified claim set and its
// IsGrantedRole helper, which already understands the
// urn:zitadel:iam:org:project:roles claim
//
// The only behaviour added on top of the SDK is a multi-value audience
// allow-list, because this gateway is a shared resource server: callers hold
// their own client IDs (paperless, poimen, api-gw, ...) and the SDK's
// oauth.WithJWT helper accepts exactly one.
type ZitadelVerifier struct {
instance *zitadel.Zitadel
verifier *op.AccessTokenVerifier
clientIDs []string
}
// NewZitadelVerifier performs OIDC discovery and returns a ready verifier.
// Discovery happens once at startup so a misconfigured domain fails fast
// instead of on the first request.
func NewZitadelVerifier(ctx context.Context, cfg ZitadelConfig) (*ZitadelVerifier, error) {
if cfg.Domain == "" {
return nil, errors.New("zitadel: domain is required")
}
httpClient := cfg.HTTPClient
if httpClient == nil {
httpClient = gohttp.DefaultClient
}
var opts []zitadel.Option
if cfg.InsecurePort != "" {
opts = append(opts, zitadel.WithInsecure(cfg.InsecurePort))
}
instance := zitadel.New(cfg.Domain, opts...)
discovery, err := client.Discover(ctx, instance.Origin(), httpClient)
if err != nil {
return nil, fmt.Errorf("zitadel: OIDC discovery at %s: %w", instance.Origin(), err)
}
keySet := rp.NewRemoteKeySet(httpClient, discovery.JwksURI)
verifier := op.NewAccessTokenVerifier(discovery.Issuer, keySet)
return &ZitadelVerifier{
instance: instance,
verifier: verifier,
clientIDs: append([]string(nil), cfg.ClientIDs...),
}, nil
}
// Origin returns the instance origin used for discovery, for logging.
func (z *ZitadelVerifier) Origin() string {
if z == nil || z.instance == nil {
return ""
}
return z.instance.Origin()
}
// CheckAuthorization validates an "Authorization: Bearer <token>" header and
// returns the SDK's verified introspection context.
func (z *ZitadelVerifier) CheckAuthorization(ctx context.Context, authHeader string) (*oauth.IntrospectionContext, error) {
accessToken, ok := strings.CutPrefix(authHeader, oidc.BearerToken)
if !ok {
return nil, fmt.Errorf("%w: expected %q prefix", oauth.ErrInvalidAuthorizationHeader, oidc.BearerToken)
}
accessToken = strings.TrimSpace(accessToken)
if accessToken == "" {
return nil, fmt.Errorf("%w: empty bearer token", oauth.ErrInvalidAuthorizationHeader)
}
claims, err := op.VerifyAccessToken[*oidc.AccessTokenClaims](ctx, accessToken, z.verifier)
if err != nil {
return nil, fmt.Errorf("%w: %v", oauth.ErrInvalidToken, err)
}
if len(claims.Audience) == 0 {
return nil, fmt.Errorf("%w: empty aud", oauth.ErrInvalidToken)
}
if !z.audienceAllowed(claims.Audience) {
return nil, fmt.Errorf("%w: %v", ErrAudienceNotAllowed, claims.Audience)
}
authCtx := &oauth.IntrospectionContext{
IntrospectionResponse: oidc.IntrospectionResponse{
Active: true,
Issuer: claims.Issuer,
Subject: claims.Subject,
Audience: claims.Audience,
Expiration: claims.Expiration,
IssuedAt: claims.IssuedAt,
NotBefore: claims.NotBefore,
ClientID: claims.ClientID,
JWTID: claims.JWTID,
Claims: claims.Claims,
},
}
authCtx.SetToken(accessToken)
return authCtx, nil
}
// audienceAllowed reports whether any aud entry is in the allow-list.
// An empty allow-list accepts every non-empty audience.
func (z *ZitadelVerifier) audienceAllowed(audience []string) bool {
if len(z.clientIDs) == 0 {
return true
}
for _, aud := range audience {
if slices.Contains(z.clientIDs, aud) {
return true
}
}
return false
}
+294
View File
@@ -0,0 +1,294 @@
package auth
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/base64"
"encoding/json"
"fmt"
"math/big"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/golang-jwt/jwt/v5"
)
// testIssuer mimics one ZITADEL instance: OIDC discovery on
// /.well-known/openid-configuration and the JWKS on /oauth/v2/keys.
type testIssuer struct {
server *httptest.Server
key *rsa.PrivateKey
kid string
jwksHit int
}
func newTestIssuer(t *testing.T) *testIssuer {
t.Helper()
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatalf("generate key: %v", err)
}
ti := &testIssuer{key: key, kid: "test-kid-1"}
mux := http.NewServeMux()
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"issuer": ti.server.URL,
"jwks_uri": ti.server.URL + "/oauth/v2/keys",
})
})
mux.HandleFunc("/oauth/v2/keys", func(w http.ResponseWriter, r *http.Request) {
ti.jwksHit++
pub := key.PublicKey
_ = json.NewEncoder(w).Encode(map[string]any{
"keys": []map[string]any{{
"kty": "RSA",
"kid": ti.kid,
"use": "sig",
"alg": "RS256",
"n": base64.RawURLEncoding.EncodeToString(pub.N.Bytes()),
"e": base64.RawURLEncoding.EncodeToString(big.NewInt(int64(pub.E)).Bytes()),
}},
})
})
ti.server = httptest.NewServer(mux)
t.Cleanup(ti.server.Close)
return ti
}
// sign mints an access token with the given audience and project roles.
func (ti *testIssuer) sign(t *testing.T, aud []string, roles map[string]any) string {
t.Helper()
claims := jwt.MapClaims{
"iss": ti.server.URL,
"sub": "396757800000000001",
"exp": time.Now().Add(time.Hour).Unix(),
"iat": time.Now().Unix(),
}
if len(aud) > 0 {
audClaim := make([]interface{}, 0, len(aud))
for _, a := range aud {
audClaim = append(audClaim, a)
}
claims["aud"] = audClaim
}
if roles != nil {
claims["urn:zitadel:iam:org:project:roles"] = roles
}
tok := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)
tok.Header["kid"] = ti.kid
signed, err := tok.SignedString(ti.key)
if err != nil {
t.Fatalf("sign token: %v", err)
}
return signed
}
// verifierFor builds a ZitadelVerifier pointed at the httptest instance.
// The SDK is given the loopback host and its port, so discovery goes to the
// test server rather than a real domain.
func verifierFor(t *testing.T, ti *testIssuer, clientIDs []string) *ZitadelVerifier {
t.Helper()
host, port, ok := strings.Cut(strings.TrimPrefix(ti.server.URL, "http://"), ":")
if !ok {
t.Fatalf("unexpected test server URL %q", ti.server.URL)
}
v, err := NewZitadelVerifier(context.Background(), ZitadelConfig{
Domain: host,
InsecurePort: port,
ClientIDs: clientIDs,
HTTPClient: ti.server.Client(),
})
if err != nil {
t.Fatalf("NewZitadelVerifier: %v", err)
}
return v
}
func TestZitadelVerifier_AcceptsTokenWithAllowedAudience(t *testing.T) {
ti := newTestIssuer(t)
v := verifierFor(t, ti, []string{"396757803559027071", "396757800874672511"})
token := ti.sign(t, []string{"396757803559027071"}, map[string]any{
"llm:inference": map[string]any{"396757796898471937": "homelab"},
})
authCtx, err := v.CheckAuthorization(context.Background(), "Bearer "+token)
if err != nil {
t.Fatalf("expected token to pass, got %v", err)
}
if !authCtx.IsAuthorized() {
t.Error("expected IsAuthorized() true")
}
if want := "396757800000000001"; authCtx.UserID() != want {
t.Errorf("UserID() = %q, want %q", authCtx.UserID(), want)
}
if !authCtx.IsGrantedRole("llm:inference") {
t.Error("expected SDK IsGrantedRole to see llm:inference from the project roles claim")
}
if ti.jwksHit == 0 {
t.Error("expected the SDK to fetch the JWKS")
}
}
func TestZitadelVerifier_RejectsForeignAudience(t *testing.T) {
ti := newTestIssuer(t)
v := verifierFor(t, ti, []string{"396757803559027071"})
token := ti.sign(t, []string{"999999999999999999"}, nil)
_, err := v.CheckAuthorization(context.Background(), "Bearer "+token)
if err == nil {
t.Fatal("expected rejection for an audience outside the allow-list")
}
if !strings.Contains(err.Error(), ErrAudienceNotAllowed.Error()) {
t.Errorf("expected ErrAudienceNotAllowed, got %v", err)
}
}
func TestZitadelVerifier_AcceptsAnyAudienceWhenAllowListEmpty(t *testing.T) {
ti := newTestIssuer(t)
v := verifierFor(t, ti, nil)
token := ti.sign(t, []string{"396757803877794175"}, nil)
if _, err := v.CheckAuthorization(context.Background(), "Bearer "+token); err != nil {
t.Fatalf("empty allow-list should accept any audience, got %v", err)
}
}
func TestZitadelVerifier_RejectsMissingBearerPrefix(t *testing.T) {
ti := newTestIssuer(t)
v := verifierFor(t, ti, nil)
token := ti.sign(t, []string{"396757803877794175"}, nil)
if _, err := v.CheckAuthorization(context.Background(), token); err == nil {
t.Fatal("expected rejection when the Bearer prefix is missing")
}
}
func TestZitadelVerifier_RejectsExpiredToken(t *testing.T) {
ti := newTestIssuer(t)
v := verifierFor(t, ti, nil)
claims := jwt.MapClaims{
"iss": ti.server.URL,
"sub": "u1",
"aud": []interface{}{"396757803877794175"},
"exp": time.Now().Add(-2 * time.Hour).Unix(),
"iat": time.Now().Add(-3 * time.Hour).Unix(),
}
tok := jwt.NewWithClaims(jwt.SigningMethodRS256, claims)
tok.Header["kid"] = ti.kid
signed, err := tok.SignedString(ti.key)
if err != nil {
t.Fatalf("sign token: %v", err)
}
if _, err := v.CheckAuthorization(context.Background(), "Bearer "+signed); err == nil {
t.Fatal("expected rejection for an expired token")
}
}
// TestValidator_SDKPathFeedsCheckPermissions proves the adapter from the SDK
// introspection context to jwt.MapClaims keeps the capability gate working.
func TestValidator_SDKPathFeedsCheckPermissions(t *testing.T) {
ti := newTestIssuer(t)
host, port, _ := strings.Cut(strings.TrimPrefix(ti.server.URL, "http://"), ":")
validator, err := NewZitadelValidator(context.Background(), ti.server.URL, []string{"396757803559027071"}, ZitadelConfig{
Domain: host,
InsecurePort: port,
HTTPClient: ti.server.Client(),
})
if err != nil {
t.Fatalf("NewZitadelValidator: %v", err)
}
good := ti.sign(t, []string{"396757803559027071"}, map[string]any{
"llm:inference": map[string]any{"396757796898471937": "homelab"},
})
claims, err := validator.ValidateBearerToken("Bearer " + good)
if err != nil {
t.Fatalf("ValidateBearerToken: %v", err)
}
if !validator.CheckPermissions(claims, "llm:inference") {
t.Error("expected CheckPermissions to accept llm:inference via the SDK path")
}
if validator.CheckPermissions(claims, "sqs:write") {
t.Error("expected CheckPermissions to reject a role the token does not carry")
}
noRole := ti.sign(t, []string{"396757803559027071"}, nil)
claims, err = validator.ValidateBearerToken("Bearer " + noRole)
if err != nil {
t.Fatalf("ValidateBearerToken without roles: %v", err)
}
if validator.CheckPermissions(claims, "llm:inference") {
t.Error("expected CheckPermissions to reject a token with no project roles")
}
}
func TestIssuerMatches(t *testing.T) {
cases := []struct {
iss string
configured string
want bool
}{
{"https://zitadel.riotpiao.com", "https://zitadel.riotpiao.com", true},
{"https://zitadel.riotpiao.com/", "https://zitadel.riotpiao.com", true},
{"https://zitadel.riotpiao.com/oauth/v2", "https://zitadel.riotpiao.com", true},
{"https://evil.example.com", "https://zitadel.riotpiao.com", false},
{"https://zitadel.riotpiao.com.evil.com", "https://zitadel.riotpiao.com", false},
{"anything", "", true},
}
for _, c := range cases {
if got := issuerMatches(c.iss, c.configured); got != c.want {
t.Errorf("issuerMatches(%q, %q) = %v, want %v", c.iss, c.configured, got, c.want)
}
}
}
func TestClaimsFromIntrospection_Nil(t *testing.T) {
if got := claimsFromIntrospection(nil); got == nil || len(got) != 0 {
t.Errorf("expected empty non-nil claims, got %v", got)
}
}
func TestAudienceValues(t *testing.T) {
cases := []struct {
name string
claim any
want int
ok bool
}{
{"string", "api-gw", 1, true},
{"empty string", "", 1, false},
{"string slice", []string{"a", "b"}, 2, true},
{"iface slice", []interface{}{"a", 5}, 1, true},
{"missing", nil, 0, false},
}
for _, c := range cases {
claims := jwt.MapClaims{}
if c.claim != nil {
claims["aud"] = c.claim
}
got, ok := audienceValues(claims)
if ok != c.ok || len(got) != c.want {
t.Errorf("%s: audienceValues = (%v, %v), want len %d ok %v", c.name, got, ok, c.want, c.ok)
}
}
_ = fmt.Sprint()
}
+12 -1
View File
@@ -66,12 +66,23 @@ type AuthConfig struct {
JWKSURL string
// RequiredCapability is the permission required for LLM inference (e.g., "llm:inference").
RequiredCapability string
// TokenURL is the Authentik token endpoint for password/refresh grants.
// TokenURL is the OIDC token endpoint used for token exchange
// (client_credentials / password grants).
TokenURL string
// ClientID is the OAuth2 client ID for token exchange.
ClientID string
// ClientSecret is the OAuth2 client secret (loaded from env, never from config file).
ClientSecret string
// ZitadelDomain is the ZITADEL instance host (for example
// "zitadel.riotpiao.com"). When set, token validation uses the official
// ZITADEL Go SDK instead of the raw JWKS path.
ZitadelDomain string
// ZitadelInsecurePort, when non-empty, makes the SDK speak plain HTTP to
// that port. Only for a local port-forward.
ZitadelInsecurePort string
// ZitadelClientIDs is the audience allow-list for the SDK verifier. Empty
// accepts any audience issued by the same instance.
ZitadelClientIDs []string
}
// Route represents a single route and its upstream configuration.
+6
View File
@@ -27,6 +27,9 @@ type rawAuth struct {
RequiredCapability string `yaml:"requiredCapability"`
TokenURL string `yaml:"tokenUrl"`
ClientID string `yaml:"clientId"`
ZitadelDomain string `yaml:"zitadelDomain"`
ZitadelInsecurePort string `yaml:"zitadelInsecurePort"`
ZitadelClientIDs []string `yaml:"zitadelClientIds"`
}
// rawRoute represents a single route in the YAML configuration.
@@ -192,6 +195,9 @@ func LoadRoutesAndModelsFromFile(path string) (map[string]*Route, map[string]*Mo
TokenURL: raw.Auth.TokenURL,
ClientID: raw.Auth.ClientID,
ClientSecret: os.Getenv("AUTH_CLIENT_SECRET"),
ZitadelDomain: raw.Auth.ZitadelDomain,
ZitadelInsecurePort: raw.Auth.ZitadelInsecurePort,
ZitadelClientIDs: raw.Auth.ZitadelClientIDs,
}
return routes, models, adapters, authConfig, nil
+29 -6
View File
@@ -127,6 +127,30 @@ func (o *EngineObserver) Degraded() bool {
return o.degraded
}
// Snapshot returns the latest state together with the tick channel that the
// next successful poll will close. Both are read under one lock on purpose:
// calling Get and Tick separately can miss the wakeup that lands between the
// two calls, parking a request for an extra poll interval.
func (o *EngineObserver) Snapshot() (EngineState, <-chan struct{}) {
o.mu.Lock()
defer o.mu.Unlock()
return o.state, o.tick
}
// markDegraded records whether the metrics feed is currently unusable and
// mirrors that into observer_degraded. It is cleared on the next parseable
// scrape, so one truncated response cannot pin the gauge for the process
// lifetime while metric-name drift after the first poll is still reported.
func (o *EngineObserver) markDegraded(v bool) {
o.mu.Lock()
changed := o.degraded != v
o.degraded = v
o.mu.Unlock()
if changed {
observability.Global().SetObserverDegraded(v)
}
}
// poll scrapes /metrics once. startup marks the first poll, whose failure is a
// self-diagnosis signal rather than a transient blip.
func (o *EngineObserver) poll(ctx context.Context, startup bool) {
@@ -161,13 +185,11 @@ func (o *EngineObserver) poll(ctx context.Context, startup bool) {
running, waiting, kvUsage, err := ParseEngineMetrics(resp.Body)
if err != nil {
// Metric-name drift must be loud, never a silent fall-back to fail-open
// admission (decision Q21). The gauge tracks current state, so it clears
// on the next parseable scrape instead of pinning for the pod's life.
o.markDegraded(true)
if startup {
// Startup self-check (decision Q21): metric-name drift must be
// loud, never a silent fall-back to fail-open admission.
o.mu.Lock()
o.degraded = true
o.mu.Unlock()
observability.Global().SetObserverDegraded(true)
logging.Errorf("engine metrics unusable at startup", err, map[string]string{
"address": o.addr,
})
@@ -181,6 +203,7 @@ func (o *EngineObserver) poll(ctx context.Context, startup bool) {
return
}
o.markDegraded(false)
o.recordSuccess(running, waiting, kvUsage)
}
+76 -6
View File
@@ -98,8 +98,29 @@ func New(cfg *config.Config) *Handler {
defaultMaxBodySize: 100 * 1024 * 1024,
}
// Initialize JWT validator if auth is enabled
if cfg.Auth.Enabled && cfg.Auth.JWKSURL != "" {
// Initialize JWT validator if auth is enabled.
//
// Two paths, in order of preference:
// 1. ZitadelDomain set -> official ZITADEL Go SDK (discovery + remote JWKS
// + op.AccessTokenVerifier). This is the production path.
// 2. JWKSURL set -> legacy raw-JWKS validation, kept for tests and for
// any deployment that has not moved to ZITADEL yet.
if cfg.Auth.Enabled && cfg.Auth.ZitadelDomain != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
v, err := auth.NewZitadelValidator(ctx, cfg.Auth.Issuer, cfg.Auth.ZitadelClientIDs, auth.ZitadelConfig{
Domain: cfg.Auth.ZitadelDomain,
InsecurePort: cfg.Auth.ZitadelInsecurePort,
ClientIDs: cfg.Auth.ZitadelClientIDs,
})
if err != nil {
logging.Errorf("zitadel: verifier init failed; auth disabled until restart", err, map[string]string{
"domain": cfg.Auth.ZitadelDomain,
})
} else {
h.jwtValidator = v
}
} else if cfg.Auth.Enabled && cfg.Auth.JWKSURL != "" {
h.jwtValidator = auth.NewValidator(
cfg.Auth.Issuer,
cfg.Auth.Audience,
@@ -722,16 +743,17 @@ func (h *Handler) authenticateV1(w http.ResponseWriter, r *http.Request) bool {
return false
}
identity.Inject(r, claims)
if h.config.Auth.RequiredCapability != "" {
if !h.jwtValidator.CheckPermissions(claims, h.config.Auth.RequiredCapability, "*") {
if caps := requiredCapabilities(h.config.Auth.RequiredCapability); len(caps) > 0 {
args := append(caps, "*")
if !h.jwtValidator.CheckPermissions(claims, args...) {
writeProblemDetail(w, http.StatusForbidden,
"https://api.example.com/problems/insufficient-permissions",
"Insufficient Permissions",
fmt.Sprintf("Required capability: %s", h.config.Auth.RequiredCapability),
fmt.Sprintf("Required capability: %s", strings.Join(caps, " or ")),
nil)
logging.Errorf("auth failed", fmt.Errorf("insufficient permissions"), map[string]string{
"path": r.URL.Path,
"required": h.config.Auth.RequiredCapability,
"required": strings.Join(caps, ","),
})
return false
}
@@ -739,6 +761,22 @@ func (h *Handler) authenticateV1(w http.ResponseWriter, r *http.Request) bool {
return true
}
// requiredCapabilities splits the comma-separated requiredCapability setting
// into a list. Several spellings are accepted because the same capability is
// named differently by each identity provider: Authentik used "llm:inference"
// as a permission string, while ZITADEL uses role keys such as
// "llm-inference". A token satisfies the gate when it carries any one of them.
func requiredCapabilities(raw string) []string {
parts := strings.Split(raw, ",")
out := make([]string, 0, len(parts))
for _, p := range parts {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
// enforceBodyLimit caps request body size for non-chat routes.
// Returns false when the request is rejected (response already written).
func (h *Handler) enforceBodyLimit(w http.ResponseWriter, r *http.Request, route *Route) bool {
@@ -786,6 +824,38 @@ func (h *Handler) setupCAGProxy(ctx context.Context, w http.ResponseWriter, r *h
capturedBackend := selectedBackend
capturedRoute := route
if headersPrewritten {
// The 200 + text/event-stream status is already on the wire, so an
// upstream status can no longer be forwarded: httputil.ReverseProxy
// calls WriteHeader again and net/http keeps the first one. Frame a
// failure as an SSE event rather than letting a raw error body land
// inside the stream, which a streaming client reads as an empty
// completion and reports as success.
proxy.ModifyResponse = func(resp *http.Response) error {
if resp.StatusCode < http.StatusBadRequest {
return nil
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
logging.Warnf("reading upstream error body failed", map[string]string{
"upstream": capturedRoute.Upstream.Address,
"err": readErr.Error(),
})
}
logging.Warnf("upstream error after parked stream started", map[string]string{
"upstream": capturedRoute.Upstream.Address,
"status": strconv.Itoa(resp.StatusCode),
})
resp.Body = io.NopCloser(strings.NewReader(fmt.Sprintf(
"event: error\ndata: {\"upstream_status\":%d,\"error\":%q}\n\n",
resp.StatusCode, strings.TrimSpace(string(body)))))
resp.ContentLength = -1
resp.Header.Del("Content-Length")
resp.Header.Set("Content-Type", "text/event-stream")
return nil
}
}
proxy.ErrorHandler = func(w http.ResponseWriter, r *http.Request, err error) {
if capturedBackend != nil {
if s := h.schedulers[capturedRoute.ModelName]; s != nil {
+4 -2
View File
@@ -80,7 +80,9 @@ func (b *Backend) acquire(ctx context.Context, onPark func()) error {
defer timer.Stop()
parked := false
for {
st := b.observer.Get()
// State and tick channel are read together under one lock: taking
// them separately can miss a poll that lands between the two calls.
st, tick := b.observer.Snapshot()
if !st.Healthy || (st.InFlight() < int64(b.maxInFlight) && st.KVUsage < b.maxKVUsage) {
break
}
@@ -92,7 +94,7 @@ func (b *Backend) acquire(ctx context.Context, onPark func()) error {
}
}
select {
case <-b.observer.Tick():
case <-tick:
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
+15
View File
@@ -36,6 +36,21 @@ func (rw *responseWriter) Write(b []byte) (int, error) {
return n, err
}
// Flush forwards to the wrapped writer so that streaming handlers can still
// detect http.Flusher through this wrapper. Without it a bare
// `w.(http.Flusher)` assertion fails for every request that passes through
// this middleware, which silently disables SSE flushing and keep-alives.
func (rw *responseWriter) Flush() {
if f, ok := rw.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// Unwrap exposes the wrapped writer to http.ResponseController.
func (rw *responseWriter) Unwrap() http.ResponseWriter {
return rw.ResponseWriter
}
// Middleware returns an HTTP middleware that adds tracing to requests.
func Middleware(next http.Handler) http.Handler {
tracer := otel.Tracer(tracerName)
+6
View File
@@ -70,6 +70,12 @@ spec:
value: "info"
- name: ADMISSION_MAX_PARK_SECONDS
value: "120"
# Must track the engine's --max-num-seqs (homelab: reasoning.yaml).
# The gauge this compares against is engine-wide, so this number is a
# ceiling on concurrent requests across all gateway replicas, not a
# per-pod slot count. Drift here is silent; startup logs the value.
- name: ADMISSION_MAX_INFLIGHT
value: "4"
# Only backends that publish the vLLM scheduler gauges may be observed.
# `reasoning` is the only vLLM engine; ornith/qwen-cpu are Ollama and
# embeddings/rerank are TEI, so observing them would hold
File diff suppressed because one or more lines are too long
+8 -6
View File
@@ -99,19 +99,21 @@ spec:
ports:
- protocol: TCP
port: 8080
# Allow to IAM (Authentik) for JWKS fetch + API
# Authentik pod listens on 9000 (http) and 9443 (https)
# Service translates 80→9000, 443→9443
# NetworkPolicy matches destination pod port, not service port
# Allow to IAM (Zitadel) for JWKS fetch + API
# Zitadel API listens on 8080 (h2c/plaintext in-cluster); the login UI
# listens on 3000. Public access goes through ingress-nginx on 443 (already
# covered by the 10.96.0.0/12 CIDR below), but 8080 keeps direct in-cluster
# calls working if CoreDNS rewrite is bypassed.
# NetworkPolicy matches destination pod port, not service port.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: iam
ports:
- protocol: TCP
port: 9000
port: 8080
- protocol: TCP
port: 9443
port: 3000
# Allow to SQS (queue management service)
- to:
- namespaceSelector: