rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Add JWT auth rollout status

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-27 12:29:39 -07:00
parent d055483aa2
commit 33e36f1d4d
2 changed files with 304 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+80
View File
@@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging
- **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
- **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
## ArgoCD — GitOps Deployment Flow
**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically.
No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
```
Developer pushes to git
↓
ArgoCD detects change (every 3 min or webhook)
↓
Syncs manifests to cluster
↓
Workloads reconcile automatically
```
Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies —
storage deploys before databases, databases before applications.
### Tracked Git Repositories
ArgoCD monitors these repos for changes:
| Repository | Purpose |
|------------|----------|
| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
To deploy a new application: create a git repo, add an Application manifest to the homelab repo's
`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
## Management Planes — Talos vs Kubernetes
This cluster has **two separate management planes**, each with different workflows:
| Plane | What it manages | Workflow | Tool |
|-------|-----------------|----------|------|
| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
**Critical distinction:**
- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
```bash
cd terraform && terraform apply -var-file=terraform.tfvars.local
cd .. && make apply-cp # talosctl apply-config to all 3 control planes
```
But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
### CoreDNS ConfigMap Ownership — Critical
⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl.
It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's
client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile
(server-side-apply conflict, no error surfaced).
**To update CoreDNS (e.g., add a hostname rewrite):**
1. Edit `terraform/files/coredns/Corefile`
2. Commit + push
3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
4. Run `make apply-cp` to push config to all control planes
5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
**If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
```bash
kubectl delete configmap coredns -n kube-system
# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
kubectl get configmap coredns -n kube-system -w
```
Or as a stopgap, apply the correct content yourself:
```bash
kubectl apply --server-side -f <(terraform output coredns_config)
```
## Quick Start — Deploying the Cluster
### 1. Bootstrap Talos Nodes
project-usage/jwt-auth-rollout.md
+224
View File
@@ -0,0 +1,224 @@
# Unified Authentik JWT/OIDC Auth — Rollout Status & Plan
Goal: every service's API (not just browser logins) authenticates against
Authentik as the single OIDC provider, with RBAC driven by Authentik group
membership (`groups` claim) and/or the synthesized `permissions` claim
(`GROUP_PERMISSIONS` dict in [k8s/infra/iam/scripts/authentik-provision.py](../k8s/infra/iam/scripts/authentik-provision.py)).
Same shape as what's already working for Vault: register an Authentik OAuth2
provider/application, the service validates JWTs against Authentik's JWKS,
claims map to policy/role.
Verified live 2026-08-27.
## JWT validation & authorization contract (what a service must implement)
This is the checklist any service in any repo needs to satisfy to actually
consume Authentik JWTs, not just have a placeholder issuer URL. Confirmed
live against the `vault` app's discovery doc (2026-08-27); same shape for
every other Authentik app/slug.
### 1. Discovery & key fetch
- Per-app issuer: `https://authentik.riotpiao.com/application/o/<app-slug>/`
- Discovery doc: `<issuer>.well-known/openid-configuration`
- JWKS: `<issuer>jwks/` (also given as `jwks_uri` in the discovery doc)
- **Gotcha:** the token endpoint is *shared* across every app, not per-slug:
`https://authentik.riotpiao.com/application/o/token/`. Don't construct it
from the issuer the way jwks_uri is constructed.
- Fetch the JWKS through the discovery doc (or `jwks_uri` directly) and
**cache it with a TTL + refresh-on-`kid`-miss**, not a hardcoded key blob —
Authentik's signing key (`SIGNING_KEY_PK` in `authentik-provision.py`) can
rotate, and a hardcoded key silently breaks every token validation the
moment it does.
### 2. Token validation checklist
- Verify the signature against the fetched JWKS.
- **Pin the algorithm allow-list to `RS256` only** (confirmed
`id_token_signing_alg_values_supported: ["RS256"]` — nothing else is
issued). Reject any token whose header claims a different `alg`,
including `"none"` — this is the standard alg-confusion defense; don't
trust the token's own header to pick the verification algorithm.
- Verify `iss` equals the exact expected issuer string for that app's slug
(not just "some Authentik issuer" — a token minted for a different app
should not validate here).
- Verify `aud` (or `azp`) equals the service's own `client_id`.
- Verify `exp`/`nbf`/`iat` with a small clock-skew tolerance (30-60s).
### 3. Claims to request and how to use them
- `groups` and `permissions` are **non-default scopes** — Authentik only
includes them in the token if explicitly requested, both at provider
registration (`property_mappings` — every app in `authentik-provision.py`'s
`SERVICES` loop already gets both via `SCOPE_PKS`) and at token-request
time (e.g. Vault's OIDC role sets `oidc_scopes: ["permissions"]`; a service
doing its own token requests needs `scope=openid permissions` or
`scope=openid groups` in the auth/token request).
- **Permissions-claim pattern** (what kmsvc/temporal/etc. should do): read
the `permissions` claim (list of strings like `"kmsvc:read"`,
`"kmsvc:write"`), check it contains the string the attempted action needs.
`homelab-admins` members get the literal string `"*"` in that list —
treat that as wildcard-allow, not as a literal permission string to match
against.
- **Groups-claim pattern** (what Vault does, what Temporal's `claimMapper`
or MinIO's `policy` claim expect): read the `groups` claim directly (raw
Authentik group names) and map group name -> internal role/policy inside
the service's own config, the way Vault's Identity Group aliases do. Prefer
this pattern when the service already has its own native role/policy
system to map into, instead of parsing the synthesized `permissions`
strings.
### 4. Gap: machine-to-machine (no human/browser step) calls aren't wired anywhere yet
Authentik's server supports `client_credentials` — confirmed live in
`grant_types_supported` on the discovery doc. But **every provider currently
registered in `authentik-provision.py` only declares
`grant_types: ["authorization_code", "refresh_token"]`** (see the `SERVICES`
loop and the `kubernetes` public-client block) — none of them can issue a
token via `client_credentials` today. If a service needs to call another
service's API with no human/browser step at all (true service-to-service,
not "a human logged in via browser, then the resulting token gets reused"),
it needs its own confidential-client Application with `client_credentials`
added to `grant_types`, and requests a token via
`POST /application/o/token/` with `grant_type=client_credentials`. Decide
per-service whether this is actually needed before assuming a JWT is always
available to attach to an outbound call — right now, none are set up for it.
### 5. Device code flow (CLI / headless API callers with no local browser)
For a caller that can't do a browser redirect itself (a CLI tool, a script
on a headless box, an API client embedded in another service) but still
needs a *human* to approve the login somewhere. Confirmed live 2026-08-27
against the `vault` provider (device-authorization leg only — the
token-polling leg below is standard RFC 8628 + Authentik's declared support,
not independently re-verified since completing it needs a real human
approval step).
**Gap, same shape as client_credentials above:** every provider currently
registered only has `grant_types: ["authorization_code", "refresh_token"]`.
Requesting a device code against one of them fails with a generic
`invalid_client` error — misleading, since it looks like a bad secret but is
actually just the missing grant type. Confirmed by adding
`urn:ietf:params:oauth:grant-type:device_code` to the `vault` provider's
`grant_types` (temporarily, via the API, reverted after testing) — the exact
same request then succeeded. Any service that wants this flow needs that
grant type added to its own provider's `grant_types` list in
`authentik-provision.py`.
**Flow, what the customer/caller must specify:**
1. `POST https://authentik.riotpiao.com/application/o/device/`,
form-encoded, with `client_id=<app's client id>` and `scope=openid
...` (same non-default-scope rule as above — add `permissions`/`groups`
if the resulting token needs to drive RBAC). Live response shape,
confirmed:
```json
{
"device_code": "<opaque, client polls with this, never shown to the human>",
"user_code": "022228491",
"verification_uri": "https://authentik.riotpiao.com/device",
"verification_uri_complete": "https://authentik.riotpiao.com/device?code=022228491",
"expires_in": 60,
"interval": 5
}
```
No client_secret was required on this leg even for a confidential client
— Authentik didn't enforce it here in testing. Don't rely on that as a
security boundary; verify token-endpoint behavior below before assuming
secrets are optional throughout the flow.
2. The caller shows `user_code` + `verification_uri` (or just the
`verification_uri_complete` link) to the human — this can be on a
completely different device. The human opens it, logs into Authentik,
approves.
3. The caller polls `POST https://authentik.riotpiao.com/application/o/token/`
with `grant_type=urn:ietf:params:oauth:grant-type:device_code`,
`device_code=<from step 1>`, `client_id=<same client id>` (+
`client_secret` if confidential — not independently confirmed on this
leg), no faster than every `interval` seconds. Per RFC 8628/Authentik's
declared support: `authorization_pending` while waiting, `slow_down` if
polling too fast, `expired_token` past `expires_in`, `access_denied` if
the human rejects it, or the normal token bundle on approval.
## Done
| Service | Authentik app | Validates JWT itself | Notes |
|---|---|---|---|
| grafana, forgejo, argocd, homarr, paperless, immich | yes | yes (native OIDC login) | browser session auth |
| **vault** | yes | yes, tested | `vault login -method=oidc`; policies attached via Vault Identity Groups aliased to Authentik group names (`groups` claim) — adding a new Vault policy needs zero Authentik-side change, just a new `identity/group`+`identity/group-alias` pair in Vault |
| **minio** | yes | likely yes, **not yet load-tested** | `MINIO_IDENTITY_OPENID_SCOPES` includes non-default scopes, which also enables `AssumeRoleWithWebIdentity` (real S3 API access via STS), not just console login. Need to actually mint a token and call `AssumeRoleWithWebIdentity` + an S3 op to confirm before calling this done. |
| kubectl | yes (public PKCE client `kubernetes`) | yes | kube-apiserver `--oidc-*` flags confirmed live (2026-08-27); per-service `Role`/`RoleBinding` in `k8s/infra/rbac/*.yaml` bind `oidc:<service>-admins` groups — should now actually be enforced, not inert as an earlier pass of this doc set assumed. Worth a real login test to confirm group->Role resolution end-to-end. |
## Not done — this repo can finish it (config only, no new app code)
- **portainer** — has a *built-in* OAuth login feature. Register an Authentik
app (confidential client, redirect URI to Portainer's OAuth callback) the
same way `vault`/`immich`/etc. are registered in `authentik-provision.py`,
then flip on OAuth in Portainer's own settings (or via its API). No
application code involved.
- **kmsvc / management-service** — `KMSVC_AUTHENTIK_ISSUER_URL` and
`KMSVC_AUTHENTIK_AUDIENCE` env vars already exist in
[k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml)
but are empty strings — placeholders, never wired. Registering the
Authentik app and populating them is real progress, BUT: **whether the
deployed image (`ghcr.io/riotpiaole/kmsvc-management-service`) actually
validates a JWT against those values is unverified** — that's a claim
about code in a separate repo this checkout doesn't have. Confirm there
before assuming this is load-bearing (this repo already has one precedent
of a README claiming JWT support that the code didn't actually have — the
LLM gateway, see "Not done" below).
- **temporal** — Temporal's own Helm chart
([k8s/apps/temporal/temporal-values.yaml](../k8s/apps/temporal/temporal-values.yaml),
no `authorization`/`jwtKeyProvider` block currently set) has **native**
JWT authorization support (`server.config.authorization.jwtKeyProvider` +
`claimMapper`) — this is Helm values, not custom plugin code. The real
target is the **`temporal-frontend` gRPC service** (port 7233 — what SDKs
and workers actually connect to), not the Web UI. Verified live
2026-08-27: only `temporal.riotpiao.com` is ingressed, and it points at
`temporal-web` (the UI); `temporal-frontend`/`temporal-frontend-headless`
are ClusterIP-only, no public hostname, no `temporal-frontend.riotpiao.com`
exists yet. Temporal's `jwtKeyProvider`/`claimMapper` auth gates the
frontend service itself at the RPC level regardless of exposure — it would
apply the same to in-cluster workers as to any external caller, so this
is worth doing even with no public ingress. Needs:
1. Point `jwtKeyProvider.keySourceURIs` at Authentik's JWKS endpoint for a
dedicated `temporal` Authentik app (none exists yet — only the
`temporal-admins` permissions-claim group, confirmed live in Authentik,
no OAuth2 provider/application).
2. Confirm whether Temporal's default `claimMapper` expects a claim shape
compatible with this repo's `permissions` claim
(`["temporal:read","temporal:write"]`) or needs a custom claim-mapper
config to translate it into Temporal's own namespace-permission format.
3. **Open question, unresolved:** is external (outside-cluster) worker/client
access to `temporal-frontend` even needed? If yes, that's a separate
decision on top of the JWT work — a gRPC-capable ingress would need to
be added (same shape as kmsvc's `grpcPathPrefix` pattern in
[k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml)),
since nothing currently exposes 7233 outside the cluster.
## Not done — needs application code in a different repo
- **poimen-memory** (`forgejo.riotpiao.com/rock/poimen.git`) — confirmed
current auth is a static API key (`poimen-memory-api-key` k8s Secret, 1
key, Opaque), not JWT/OIDC at all. No Authentik app exists. Two paths:
1. Add real JWT validation to poimen-memory's own code (that repo).
2. Bridge without touching poimen-memory: front it behind the Go API
gateway, gateway validates the Authentik JWT once and forwards with the
existing static API key internally. Doesn't fix poimen-memory itself,
but unblocks unified auth for callers without waiting on that repo.
- **LLM gateway** (`rock/homelab-frontend`, deployed as `api-gateway`) —
`internal/auth` is an empty directory per that repo's own task board
(phase 3, tasks 2.9-2.15 also unbuilt — Anthropic dialect, streaming tool
calls). `LLM_TOOL_CALLS.md` in that repo overclaims what's implemented;
don't trust it without checking `internal/auth` directly.
## Open design question, unresolved
Should each service validate JWTs itself (JWKS-fetch + claim-check logic
duplicated per service), or should the Go gateway become the one
chokepoint doing JWT validation for everything behind it (kmsvc,
llm-serving, eventually poimen-memory), so that logic exists once instead
of N times? This changes where phase-3 code actually goes. Not decided as
of this doc.