docs: Add JWT auth rollout status
This commit is contained in:
Story Crater Bot
@@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging
|
|||||||
- **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
|
- **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps
|
||||||
- **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
|
- **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure
|
||||||
|
|
||||||
|
## ArgoCD — GitOps Deployment Flow
|
||||||
|
|
||||||
|
**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically.
|
||||||
|
No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes.
|
||||||
|
|
||||||
|
```
|
||||||
|
Developer pushes to git
|
||||||
|
↓
|
||||||
|
ArgoCD detects change (every 3 min or webhook)
|
||||||
|
↓
|
||||||
|
Syncs manifests to cluster
|
||||||
|
↓
|
||||||
|
Workloads reconcile automatically
|
||||||
|
```
|
||||||
|
|
||||||
|
Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies —
|
||||||
|
storage deploys before databases, databases before applications.
|
||||||
|
|
||||||
|
### Tracked Git Repositories
|
||||||
|
|
||||||
|
ArgoCD monitors these repos for changes:
|
||||||
|
|
||||||
|
| Repository | Purpose |
|
||||||
|
|------------|----------|
|
||||||
|
| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) |
|
||||||
|
| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) |
|
||||||
|
| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) |
|
||||||
|
|
||||||
|
To deploy a new application: create a git repo, add an Application manifest to the homelab repo's
|
||||||
|
`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes.
|
||||||
|
|
||||||
|
## Management Planes — Talos vs Kubernetes
|
||||||
|
|
||||||
|
This cluster has **two separate management planes**, each with different workflows:
|
||||||
|
|
||||||
|
| Plane | What it manages | Workflow | Tool |
|
||||||
|
|-------|-----------------|----------|------|
|
||||||
|
| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` |
|
||||||
|
| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD |
|
||||||
|
|
||||||
|
**Critical distinction:**
|
||||||
|
- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply`
|
||||||
|
- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials)
|
||||||
|
|
||||||
|
Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then:
|
||||||
|
```bash
|
||||||
|
cd terraform && terraform apply -var-file=terraform.tfvars.local
|
||||||
|
cd .. && make apply-cp # talosctl apply-config to all 3 control planes
|
||||||
|
```
|
||||||
|
|
||||||
|
But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**.
|
||||||
|
|
||||||
|
### CoreDNS ConfigMap Ownership — Critical
|
||||||
|
|
||||||
|
⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl.
|
||||||
|
It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time.
|
||||||
|
|
||||||
|
**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's
|
||||||
|
client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile
|
||||||
|
(server-side-apply conflict, no error surfaced).
|
||||||
|
|
||||||
|
**To update CoreDNS (e.g., add a hostname rewrite):**
|
||||||
|
1. Edit `terraform/files/coredns/Corefile`
|
||||||
|
2. Commit + push
|
||||||
|
3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local`
|
||||||
|
4. Run `make apply-cp` to push config to all control planes
|
||||||
|
5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed
|
||||||
|
|
||||||
|
**If you accidentally edited the ConfigMap directly and broke Talos's ownership:**
|
||||||
|
```bash
|
||||||
|
kubectl delete configmap coredns -n kube-system
|
||||||
|
# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it
|
||||||
|
kubectl get configmap coredns -n kube-system -w
|
||||||
|
```
|
||||||
|
|
||||||
|
Or as a stopgap, apply the correct content yourself:
|
||||||
|
```bash
|
||||||
|
kubectl apply --server-side -f <(terraform output coredns_config)
|
||||||
|
```
|
||||||
|
|
||||||
## Quick Start — Deploying the Cluster
|
## Quick Start — Deploying the Cluster
|
||||||
|
|
||||||
### 1. Bootstrap Talos Nodes
|
### 1. Bootstrap Talos Nodes
|
||||||
|
|||||||
@@ -0,0 +1,224 @@
|
|||||||
|
# Unified Authentik JWT/OIDC Auth — Rollout Status & Plan
|
||||||
|
|
||||||
|
Goal: every service's API (not just browser logins) authenticates against
|
||||||
|
Authentik as the single OIDC provider, with RBAC driven by Authentik group
|
||||||
|
membership (`groups` claim) and/or the synthesized `permissions` claim
|
||||||
|
(`GROUP_PERMISSIONS` dict in [k8s/infra/iam/scripts/authentik-provision.py](../k8s/infra/iam/scripts/authentik-provision.py)).
|
||||||
|
Same shape as what's already working for Vault: register an Authentik OAuth2
|
||||||
|
provider/application, the service validates JWTs against Authentik's JWKS,
|
||||||
|
claims map to policy/role.
|
||||||
|
|
||||||
|
Verified live 2026-08-27.
|
||||||
|
|
||||||
|
## JWT validation & authorization contract (what a service must implement)
|
||||||
|
|
||||||
|
This is the checklist any service in any repo needs to satisfy to actually
|
||||||
|
consume Authentik JWTs, not just have a placeholder issuer URL. Confirmed
|
||||||
|
live against the `vault` app's discovery doc (2026-08-27); same shape for
|
||||||
|
every other Authentik app/slug.
|
||||||
|
|
||||||
|
### 1. Discovery & key fetch
|
||||||
|
|
||||||
|
- Per-app issuer: `https://authentik.riotpiao.com/application/o/<app-slug>/`
|
||||||
|
- Discovery doc: `<issuer>.well-known/openid-configuration`
|
||||||
|
- JWKS: `<issuer>jwks/` (also given as `jwks_uri` in the discovery doc)
|
||||||
|
- **Gotcha:** the token endpoint is *shared* across every app, not per-slug:
|
||||||
|
`https://authentik.riotpiao.com/application/o/token/`. Don't construct it
|
||||||
|
from the issuer the way jwks_uri is constructed.
|
||||||
|
- Fetch the JWKS through the discovery doc (or `jwks_uri` directly) and
|
||||||
|
**cache it with a TTL + refresh-on-`kid`-miss**, not a hardcoded key blob —
|
||||||
|
Authentik's signing key (`SIGNING_KEY_PK` in `authentik-provision.py`) can
|
||||||
|
rotate, and a hardcoded key silently breaks every token validation the
|
||||||
|
moment it does.
|
||||||
|
|
||||||
|
### 2. Token validation checklist
|
||||||
|
|
||||||
|
- Verify the signature against the fetched JWKS.
|
||||||
|
- **Pin the algorithm allow-list to `RS256` only** (confirmed
|
||||||
|
`id_token_signing_alg_values_supported: ["RS256"]` — nothing else is
|
||||||
|
issued). Reject any token whose header claims a different `alg`,
|
||||||
|
including `"none"` — this is the standard alg-confusion defense; don't
|
||||||
|
trust the token's own header to pick the verification algorithm.
|
||||||
|
- Verify `iss` equals the exact expected issuer string for that app's slug
|
||||||
|
(not just "some Authentik issuer" — a token minted for a different app
|
||||||
|
should not validate here).
|
||||||
|
- Verify `aud` (or `azp`) equals the service's own `client_id`.
|
||||||
|
- Verify `exp`/`nbf`/`iat` with a small clock-skew tolerance (30-60s).
|
||||||
|
|
||||||
|
### 3. Claims to request and how to use them
|
||||||
|
|
||||||
|
- `groups` and `permissions` are **non-default scopes** — Authentik only
|
||||||
|
includes them in the token if explicitly requested, both at provider
|
||||||
|
registration (`property_mappings` — every app in `authentik-provision.py`'s
|
||||||
|
`SERVICES` loop already gets both via `SCOPE_PKS`) and at token-request
|
||||||
|
time (e.g. Vault's OIDC role sets `oidc_scopes: ["permissions"]`; a service
|
||||||
|
doing its own token requests needs `scope=openid permissions` or
|
||||||
|
`scope=openid groups` in the auth/token request).
|
||||||
|
- **Permissions-claim pattern** (what kmsvc/temporal/etc. should do): read
|
||||||
|
the `permissions` claim (list of strings like `"kmsvc:read"`,
|
||||||
|
`"kmsvc:write"`), check it contains the string the attempted action needs.
|
||||||
|
`homelab-admins` members get the literal string `"*"` in that list —
|
||||||
|
treat that as wildcard-allow, not as a literal permission string to match
|
||||||
|
against.
|
||||||
|
- **Groups-claim pattern** (what Vault does, what Temporal's `claimMapper`
|
||||||
|
or MinIO's `policy` claim expect): read the `groups` claim directly (raw
|
||||||
|
Authentik group names) and map group name -> internal role/policy inside
|
||||||
|
the service's own config, the way Vault's Identity Group aliases do. Prefer
|
||||||
|
this pattern when the service already has its own native role/policy
|
||||||
|
system to map into, instead of parsing the synthesized `permissions`
|
||||||
|
strings.
|
||||||
|
|
||||||
|
### 4. Gap: machine-to-machine (no human/browser step) calls aren't wired anywhere yet
|
||||||
|
|
||||||
|
Authentik's server supports `client_credentials` — confirmed live in
|
||||||
|
`grant_types_supported` on the discovery doc. But **every provider currently
|
||||||
|
registered in `authentik-provision.py` only declares
|
||||||
|
`grant_types: ["authorization_code", "refresh_token"]`** (see the `SERVICES`
|
||||||
|
loop and the `kubernetes` public-client block) — none of them can issue a
|
||||||
|
token via `client_credentials` today. If a service needs to call another
|
||||||
|
service's API with no human/browser step at all (true service-to-service,
|
||||||
|
not "a human logged in via browser, then the resulting token gets reused"),
|
||||||
|
it needs its own confidential-client Application with `client_credentials`
|
||||||
|
added to `grant_types`, and requests a token via
|
||||||
|
`POST /application/o/token/` with `grant_type=client_credentials`. Decide
|
||||||
|
per-service whether this is actually needed before assuming a JWT is always
|
||||||
|
available to attach to an outbound call — right now, none are set up for it.
|
||||||
|
|
||||||
|
### 5. Device code flow (CLI / headless API callers with no local browser)
|
||||||
|
|
||||||
|
For a caller that can't do a browser redirect itself (a CLI tool, a script
|
||||||
|
on a headless box, an API client embedded in another service) but still
|
||||||
|
needs a *human* to approve the login somewhere. Confirmed live 2026-08-27
|
||||||
|
against the `vault` provider (device-authorization leg only — the
|
||||||
|
token-polling leg below is standard RFC 8628 + Authentik's declared support,
|
||||||
|
not independently re-verified since completing it needs a real human
|
||||||
|
approval step).
|
||||||
|
|
||||||
|
**Gap, same shape as client_credentials above:** every provider currently
|
||||||
|
registered only has `grant_types: ["authorization_code", "refresh_token"]`.
|
||||||
|
Requesting a device code against one of them fails with a generic
|
||||||
|
`invalid_client` error — misleading, since it looks like a bad secret but is
|
||||||
|
actually just the missing grant type. Confirmed by adding
|
||||||
|
`urn:ietf:params:oauth:grant-type:device_code` to the `vault` provider's
|
||||||
|
`grant_types` (temporarily, via the API, reverted after testing) — the exact
|
||||||
|
same request then succeeded. Any service that wants this flow needs that
|
||||||
|
grant type added to its own provider's `grant_types` list in
|
||||||
|
`authentik-provision.py`.
|
||||||
|
|
||||||
|
**Flow, what the customer/caller must specify:**
|
||||||
|
|
||||||
|
1. `POST https://authentik.riotpiao.com/application/o/device/`,
|
||||||
|
form-encoded, with `client_id=<app's client id>` and `scope=openid
|
||||||
|
...` (same non-default-scope rule as above — add `permissions`/`groups`
|
||||||
|
if the resulting token needs to drive RBAC). Live response shape,
|
||||||
|
confirmed:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"device_code": "<opaque, client polls with this, never shown to the human>",
|
||||||
|
"user_code": "022228491",
|
||||||
|
"verification_uri": "https://authentik.riotpiao.com/device",
|
||||||
|
"verification_uri_complete": "https://authentik.riotpiao.com/device?code=022228491",
|
||||||
|
"expires_in": 60,
|
||||||
|
"interval": 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
No client_secret was required on this leg even for a confidential client
|
||||||
|
— Authentik didn't enforce it here in testing. Don't rely on that as a
|
||||||
|
security boundary; verify token-endpoint behavior below before assuming
|
||||||
|
secrets are optional throughout the flow.
|
||||||
|
2. The caller shows `user_code` + `verification_uri` (or just the
|
||||||
|
`verification_uri_complete` link) to the human — this can be on a
|
||||||
|
completely different device. The human opens it, logs into Authentik,
|
||||||
|
approves.
|
||||||
|
3. The caller polls `POST https://authentik.riotpiao.com/application/o/token/`
|
||||||
|
with `grant_type=urn:ietf:params:oauth:grant-type:device_code`,
|
||||||
|
`device_code=<from step 1>`, `client_id=<same client id>` (+
|
||||||
|
`client_secret` if confidential — not independently confirmed on this
|
||||||
|
leg), no faster than every `interval` seconds. Per RFC 8628/Authentik's
|
||||||
|
declared support: `authorization_pending` while waiting, `slow_down` if
|
||||||
|
polling too fast, `expired_token` past `expires_in`, `access_denied` if
|
||||||
|
the human rejects it, or the normal token bundle on approval.
|
||||||
|
|
||||||
|
## Done
|
||||||
|
|
||||||
|
| Service | Authentik app | Validates JWT itself | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| grafana, forgejo, argocd, homarr, paperless, immich | yes | yes (native OIDC login) | browser session auth |
|
||||||
|
| **vault** | yes | yes, tested | `vault login -method=oidc`; policies attached via Vault Identity Groups aliased to Authentik group names (`groups` claim) — adding a new Vault policy needs zero Authentik-side change, just a new `identity/group`+`identity/group-alias` pair in Vault |
|
||||||
|
| **minio** | yes | likely yes, **not yet load-tested** | `MINIO_IDENTITY_OPENID_SCOPES` includes non-default scopes, which also enables `AssumeRoleWithWebIdentity` (real S3 API access via STS), not just console login. Need to actually mint a token and call `AssumeRoleWithWebIdentity` + an S3 op to confirm before calling this done. |
|
||||||
|
| kubectl | yes (public PKCE client `kubernetes`) | yes | kube-apiserver `--oidc-*` flags confirmed live (2026-08-27); per-service `Role`/`RoleBinding` in `k8s/infra/rbac/*.yaml` bind `oidc:<service>-admins` groups — should now actually be enforced, not inert as an earlier pass of this doc set assumed. Worth a real login test to confirm group->Role resolution end-to-end. |
|
||||||
|
|
||||||
|
## Not done — this repo can finish it (config only, no new app code)
|
||||||
|
|
||||||
|
- **portainer** — has a *built-in* OAuth login feature. Register an Authentik
|
||||||
|
app (confidential client, redirect URI to Portainer's OAuth callback) the
|
||||||
|
same way `vault`/`immich`/etc. are registered in `authentik-provision.py`,
|
||||||
|
then flip on OAuth in Portainer's own settings (or via its API). No
|
||||||
|
application code involved.
|
||||||
|
|
||||||
|
- **kmsvc / management-service** — `KMSVC_AUTHENTIK_ISSUER_URL` and
|
||||||
|
`KMSVC_AUTHENTIK_AUDIENCE` env vars already exist in
|
||||||
|
[k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml)
|
||||||
|
but are empty strings — placeholders, never wired. Registering the
|
||||||
|
Authentik app and populating them is real progress, BUT: **whether the
|
||||||
|
deployed image (`ghcr.io/riotpiaole/kmsvc-management-service`) actually
|
||||||
|
validates a JWT against those values is unverified** — that's a claim
|
||||||
|
about code in a separate repo this checkout doesn't have. Confirm there
|
||||||
|
before assuming this is load-bearing (this repo already has one precedent
|
||||||
|
of a README claiming JWT support that the code didn't actually have — the
|
||||||
|
LLM gateway, see "Not done" below).
|
||||||
|
|
||||||
|
- **temporal** — Temporal's own Helm chart
|
||||||
|
([k8s/apps/temporal/temporal-values.yaml](../k8s/apps/temporal/temporal-values.yaml),
|
||||||
|
no `authorization`/`jwtKeyProvider` block currently set) has **native**
|
||||||
|
JWT authorization support (`server.config.authorization.jwtKeyProvider` +
|
||||||
|
`claimMapper`) — this is Helm values, not custom plugin code. The real
|
||||||
|
target is the **`temporal-frontend` gRPC service** (port 7233 — what SDKs
|
||||||
|
and workers actually connect to), not the Web UI. Verified live
|
||||||
|
2026-08-27: only `temporal.riotpiao.com` is ingressed, and it points at
|
||||||
|
`temporal-web` (the UI); `temporal-frontend`/`temporal-frontend-headless`
|
||||||
|
are ClusterIP-only, no public hostname, no `temporal-frontend.riotpiao.com`
|
||||||
|
exists yet. Temporal's `jwtKeyProvider`/`claimMapper` auth gates the
|
||||||
|
frontend service itself at the RPC level regardless of exposure — it would
|
||||||
|
apply the same to in-cluster workers as to any external caller, so this
|
||||||
|
is worth doing even with no public ingress. Needs:
|
||||||
|
1. Point `jwtKeyProvider.keySourceURIs` at Authentik's JWKS endpoint for a
|
||||||
|
dedicated `temporal` Authentik app (none exists yet — only the
|
||||||
|
`temporal-admins` permissions-claim group, confirmed live in Authentik,
|
||||||
|
no OAuth2 provider/application).
|
||||||
|
2. Confirm whether Temporal's default `claimMapper` expects a claim shape
|
||||||
|
compatible with this repo's `permissions` claim
|
||||||
|
(`["temporal:read","temporal:write"]`) or needs a custom claim-mapper
|
||||||
|
config to translate it into Temporal's own namespace-permission format.
|
||||||
|
3. **Open question, unresolved:** is external (outside-cluster) worker/client
|
||||||
|
access to `temporal-frontend` even needed? If yes, that's a separate
|
||||||
|
decision on top of the JWT work — a gRPC-capable ingress would need to
|
||||||
|
be added (same shape as kmsvc's `grpcPathPrefix` pattern in
|
||||||
|
[k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml)),
|
||||||
|
since nothing currently exposes 7233 outside the cluster.
|
||||||
|
|
||||||
|
## Not done — needs application code in a different repo
|
||||||
|
|
||||||
|
- **poimen-memory** (`forgejo.riotpiao.com/rock/poimen.git`) — confirmed
|
||||||
|
current auth is a static API key (`poimen-memory-api-key` k8s Secret, 1
|
||||||
|
key, Opaque), not JWT/OIDC at all. No Authentik app exists. Two paths:
|
||||||
|
1. Add real JWT validation to poimen-memory's own code (that repo).
|
||||||
|
2. Bridge without touching poimen-memory: front it behind the Go API
|
||||||
|
gateway, gateway validates the Authentik JWT once and forwards with the
|
||||||
|
existing static API key internally. Doesn't fix poimen-memory itself,
|
||||||
|
but unblocks unified auth for callers without waiting on that repo.
|
||||||
|
|
||||||
|
- **LLM gateway** (`rock/homelab-frontend`, deployed as `api-gateway`) —
|
||||||
|
`internal/auth` is an empty directory per that repo's own task board
|
||||||
|
(phase 3, tasks 2.9-2.15 also unbuilt — Anthropic dialect, streaming tool
|
||||||
|
calls). `LLM_TOOL_CALLS.md` in that repo overclaims what's implemented;
|
||||||
|
don't trust it without checking `internal/auth` directly.
|
||||||
|
|
||||||
|
## Open design question, unresolved
|
||||||
|
|
||||||
|
Should each service validate JWTs itself (JWKS-fetch + claim-check logic
|
||||||
|
duplicated per service), or should the Go gateway become the one
|
||||||
|
chokepoint doing JWT validation for everything behind it (kmsvc,
|
||||||
|
llm-serving, eventually poimen-memory), so that logic exists once instead
|
||||||
|
of N times? This changes where phase-3 code actually goes. Not decided as
|
||||||
|
of this doc.
|
||||||
Reference in New Issue
Block a user