homelab/project-usage/jwt-auth-rollout.md

Unified Authentik JWT/OIDC Auth — Rollout Status & Plan

Goal: every service's API (not just browser logins) authenticates against Authentik as the single OIDC provider, with RBAC driven by Authentik group membership (groups claim) and/or the synthesized permissions claim (GROUP_PERMISSIONS dict in k8s/infra/iam/scripts/authentik-provision.py). Same shape as what's already working for Vault: register an Authentik OAuth2 provider/application, the service validates JWTs against Authentik's JWKS, claims map to policy/role.

Verified live 2026-08-27.

JWT validation & authorization contract (what a service must implement)

This is the checklist any service in any repo needs to satisfy to actually consume Authentik JWTs, not just have a placeholder issuer URL. Confirmed live against the vault app's discovery doc (2026-08-27); same shape for every other Authentik app/slug.

1. Discovery & key fetch

• Per-app issuer: https://authentik.riotpiao.com/application/o/<app-slug>/
• Discovery doc: <issuer>.well-known/openid-configuration
• JWKS: <issuer>jwks/ (also given as jwks_uri in the discovery doc)
• Gotcha: the token endpoint is shared across every app, not per-slug: https://authentik.riotpiao.com/application/o/token/. Don't construct it from the issuer the way jwks_uri is constructed.
• Fetch the JWKS through the discovery doc (or jwks_uri directly) and cache it with a TTL + refresh-on-kid-miss, not a hardcoded key blob — Authentik's signing key (SIGNING_KEY_PK in authentik-provision.py) can rotate, and a hardcoded key silently breaks every token validation the moment it does.

2. Token validation checklist

• Verify the signature against the fetched JWKS.
• Pin the algorithm allow-list to RS256 only (confirmed id_token_signing_alg_values_supported: ["RS256"] — nothing else is issued). Reject any token whose header claims a different alg, including "none" — this is the standard alg-confusion defense; don't trust the token's own header to pick the verification algorithm.
• Verify iss equals the exact expected issuer string for that app's slug (not just "some Authentik issuer" — a token minted for a different app should not validate here).
• Verify aud (or azp) equals the service's own client_id.
• Verify exp/nbf/iat with a small clock-skew tolerance (30-60s).

3. Claims to request and how to use them

• groups and permissions are non-default scopes — Authentik only includes them in the token if explicitly requested, both at provider registration (property_mappings — every app in authentik-provision.py's SERVICES loop already gets both via SCOPE_PKS) and at token-request time (e.g. Vault's OIDC role sets oidc_scopes: ["permissions"]; a service doing its own token requests needs scope=openid permissions or scope=openid groups in the auth/token request).
• Permissions-claim pattern (what kmsvc/temporal/etc. should do): read the permissions claim (list of strings like "kmsvc:read", "kmsvc:write"), check it contains the string the attempted action needs. homelab-admins members get the literal string "*" in that list — treat that as wildcard-allow, not as a literal permission string to match against.
• Groups-claim pattern (what Vault does, what Temporal's claimMapper or MinIO's policy claim expect): read the groups claim directly (raw Authentik group names) and map group name -> internal role/policy inside the service's own config, the way Vault's Identity Group aliases do. Prefer this pattern when the service already has its own native role/policy system to map into, instead of parsing the synthesized permissions strings.

4. Gap: machine-to-machine (no human/browser step) calls aren't wired anywhere yet

Authentik's server supports client_credentials — confirmed live in grant_types_supported on the discovery doc. But every provider currently registered in authentik-provision.py only declares grant_types: ["authorization_code", "refresh_token"] (see the SERVICES loop and the kubernetes public-client block) — none of them can issue a token via client_credentials today. If a service needs to call another service's API with no human/browser step at all (true service-to-service, not "a human logged in via browser, then the resulting token gets reused"), it needs its own confidential-client Application with client_credentials added to grant_types, and requests a token via POST /application/o/token/ with grant_type=client_credentials. Decide per-service whether this is actually needed before assuming a JWT is always available to attach to an outbound call — right now, none are set up for it.

5. Device code flow (CLI / headless API callers with no local browser)

For a caller that can't do a browser redirect itself (a CLI tool, a script on a headless box, an API client embedded in another service) but still needs a human to approve the login somewhere. Confirmed live 2026-08-27 against the vault provider (device-authorization leg only — the token-polling leg below is standard RFC 8628 + Authentik's declared support, not independently re-verified since completing it needs a real human approval step).

Gap, same shape as client_credentials above: every provider currently registered only has grant_types: ["authorization_code", "refresh_token"]. Requesting a device code against one of them fails with a generic invalid_client error — misleading, since it looks like a bad secret but is actually just the missing grant type. Confirmed by adding urn:ietf:params:oauth:grant-type:device_code to the vault provider's grant_types (temporarily, via the API, reverted after testing) — the exact same request then succeeded. Any service that wants this flow needs that grant type added to its own provider's grant_types list in authentik-provision.py.

Flow, what the customer/caller must specify:

1. POST https://authentik.riotpiao.com/application/o/device/, form-encoded, with client_id=<app's client id> and scope=openid ... (same non-default-scope rule as above — add permissions/groups if the resulting token needs to drive RBAC). Live response shape, confirmed:

   {
     "device_code": "<opaque, client polls with this, never shown to the human>",
     "user_code": "022228491",
     "verification_uri": "https://authentik.riotpiao.com/device",
     "verification_uri_complete": "https://authentik.riotpiao.com/device?code=022228491",
     "expires_in": 60,
     "interval": 5
   }
   

   No client_secret was required on this leg even for a confidential client — Authentik didn't enforce it here in testing. Don't rely on that as a security boundary; verify token-endpoint behavior below before assuming secrets are optional throughout the flow.
2. The caller shows user_code + verification_uri (or just the verification_uri_complete link) to the human — this can be on a completely different device. The human opens it, logs into Authentik, approves.
3. The caller polls POST https://authentik.riotpiao.com/application/o/token/ with grant_type=urn:ietf:params:oauth:grant-type:device_code, device_code=<from step 1>, client_id=<same client id> (+ client_secret if confidential — not independently confirmed on this leg), no faster than every interval seconds. Per RFC 8628/Authentik's declared support: authorization_pending while waiting, slow_down if polling too fast, expired_token past expires_in, access_denied if the human rejects it, or the normal token bundle on approval.

Done

Service	Authentik app	Validates JWT itself	Notes
grafana, forgejo, argocd, homarr, paperless, immich	yes	yes (native OIDC login)	browser session auth
vault	yes	yes, tested	vault login -method=oidc; policies attached via Vault Identity Groups aliased to Authentik group names (groups claim) — adding a new Vault policy needs zero Authentik-side change, just a new identity/group+identity/group-alias pair in Vault
minio	yes	likely yes, not yet load-tested	MINIO_IDENTITY_OPENID_SCOPES includes non-default scopes, which also enables AssumeRoleWithWebIdentity (real S3 API access via STS), not just console login. Need to actually mint a token and call AssumeRoleWithWebIdentity + an S3 op to confirm before calling this done.
kubectl	yes (public PKCE client kubernetes)	yes	kube-apiserver --oidc-* flags confirmed live (2026-08-27); per-service Role/RoleBinding in k8s/infra/rbac/*.yaml bind oidc:<service>-admins groups — should now actually be enforced, not inert as an earlier pass of this doc set assumed. Worth a real login test to confirm group->Role resolution end-to-end.

Not done — this repo can finish it (config only, no new app code)

• portainer — has a built-in OAuth login feature. Register an Authentik app (confidential client, redirect URI to Portainer's OAuth callback) the same way vault/immich/etc. are registered in authentik-provision.py, then flip on OAuth in Portainer's own settings (or via its API). No application code involved.

• kmsvc / management-service — KMSVC_AUTHENTIK_ISSUER_URL and KMSVC_AUTHENTIK_AUDIENCE env vars already exist in k8s/apps/messaging/management-service/values.yaml but are empty strings — placeholders, never wired. Registering the Authentik app and populating them is real progress, BUT: whether the deployed image (ghcr.io/riotpiaole/kmsvc-management-service) actually validates a JWT against those values is unverified — that's a claim about code in a separate repo this checkout doesn't have. Confirm there before assuming this is load-bearing (this repo already has one precedent of a README claiming JWT support that the code didn't actually have — the LLM gateway, see "Not done" below).

• temporal — Temporal's own Helm chart (k8s/apps/temporal/temporal-values.yaml, no authorization/jwtKeyProvider block currently set) has native JWT authorization support (server.config.authorization.jwtKeyProvider + claimMapper) — this is Helm values, not custom plugin code. The real target is the temporal-frontend gRPC service (port 7233 — what SDKs and workers actually connect to), not the Web UI. Verified live 2026-08-27: only temporal.riotpiao.com is ingressed, and it points at temporal-web (the UI); temporal-frontend/temporal-frontend-headless are ClusterIP-only, no public hostname, no temporal-frontend.riotpiao.com exists yet. Temporal's jwtKeyProvider/claimMapper auth gates the frontend service itself at the RPC level regardless of exposure — it would apply the same to in-cluster workers as to any external caller, so this is worth doing even with no public ingress. Needs:

  1. Point jwtKeyProvider.keySourceURIs at Authentik's JWKS endpoint for a dedicated temporal Authentik app (none exists yet — only the temporal-admins permissions-claim group, confirmed live in Authentik, no OAuth2 provider/application).
  2. Confirm whether Temporal's default claimMapper expects a claim shape compatible with this repo's permissions claim (["temporal:read","temporal:write"]) or needs a custom claim-mapper config to translate it into Temporal's own namespace-permission format.
  3. Open question, unresolved: is external (outside-cluster) worker/client access to temporal-frontend even needed? If yes, that's a separate decision on top of the JWT work — a gRPC-capable ingress would need to be added (same shape as kmsvc's grpcPathPrefix pattern in k8s/apps/messaging/management-service/values.yaml), since nothing currently exposes 7233 outside the cluster.

Not done — needs application code in a different repo

• poimen-memory (forgejo.riotpiao.com/rock/poimen.git) — confirmed current auth is a static API key (poimen-memory-api-key k8s Secret, 1 key, Opaque), not JWT/OIDC at all. No Authentik app exists. Two paths:

  1. Add real JWT validation to poimen-memory's own code (that repo).
  2. Bridge without touching poimen-memory: front it behind the Go API gateway, gateway validates the Authentik JWT once and forwards with the existing static API key internally. Doesn't fix poimen-memory itself, but unblocks unified auth for callers without waiting on that repo.

• LLM gateway (rock/homelab-frontend, deployed as api-gateway) — internal/auth is an empty directory per that repo's own task board (phase 3, tasks 2.9-2.15 also unbuilt — Anthropic dialect, streaming tool calls). LLM_TOOL_CALLS.md in that repo overclaims what's implemented; don't trust it without checking internal/auth directly.

Open design question, unresolved

Should each service validate JWTs itself (JWKS-fetch + claim-check logic duplicated per service), or should the Go gateway become the one chokepoint doing JWT validation for everything behind it (kmsvc, llm-serving, eventually poimen-memory), so that logic exists once instead of N times? This changes where phase-3 code actually goes. Not decided as of this doc.