From 33e36f1d4deeaa559f9473e62a50beecd92f9f88 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Thu, 27 Aug 2026 12:29:39 -0700 Subject: [PATCH] docs: Add JWT auth rollout status --- README.md | 80 +++++++++++ project-usage/jwt-auth-rollout.md | 224 ++++++++++++++++++++++++++++++ 2 files changed, 304 insertions(+) create mode 100644 project-usage/jwt-auth-rollout.md diff --git a/README.md b/README.md index 655d74c..d17ff9e 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,86 @@ All logs + metrics centralized in Grafana for debugging - **Secrets at rest** — Vault + encrypted etcd; credentials never in logs or ConfigMaps - **Infrastructure-as-code** — Every service deployed via Helmfile; one `helmfile apply` recovers from total failure +## ArgoCD — GitOps Deployment Flow + +**ArgoCD** pulls infrastructure changes from git and syncs the cluster automatically. +No manual `kubectl apply` — push to git, ArgoCD detects the change, and deploys within ~3 minutes. + +``` +Developer pushes to git + ↓ +ArgoCD detects change (every 3 min or webhook) + ↓ +Syncs manifests to cluster + ↓ +Workloads reconcile automatically +``` + +Applications are deployed in waves (numbered 00, 10, 20, 30, ...) to respect dependencies — +storage deploys before databases, databases before applications. + +### Tracked Git Repositories + +ArgoCD monitors these repos for changes: + +| Repository | Purpose | +|------------|----------| +| `https://github.com/Riotpiaole/riotpiao.homelab.com` | Main infrastructure repo (all manifests in `k8s/argocd/apps/`) | +| `https://forgejo.riotpiao.com/rock/*` | Any `rock/*` repo in in-cluster Forgejo (apps + configs) | +| `https://github.com/Riotpiaole/Poimen-*` | External Poimen services (memory, workflows) | + +To deploy a new application: create a git repo, add an Application manifest to the homelab repo's +`k8s/argocd/apps/`, commit + push, and ArgoCD syncs within 3 minutes. + +## Management Planes — Talos vs Kubernetes + +This cluster has **two separate management planes**, each with different workflows: + +| Plane | What it manages | Workflow | Tool | +|-------|-----------------|----------|------| +| **Talos (OS)** | Node configuration, kernel params, networking, CoreDNS, machine state | Edit `terraform/` → `terraform apply` → `make apply-cp` | `terraform` + `talosctl` | +| **Kubernetes (workloads)** | All pods, services, deployments, ingresses, databases | Edit `k8s/argocd/apps/` → `git push` → ArgoCD syncs | `git` + ArgoCD | + +**Critical distinction:** +- **Kubernetes resources** (`k8s/**`) flow through **git → ArgoCD** — never use `kubectl apply` +- **Talos machine config** (`terraform/**`) uses **local `terraform apply`** (sanctioned exception — CI can't hold node credentials) + +Example: To add a CoreDNS hostname rewrite, you edit `terraform/files/coredns/Corefile`, then: +```bash +cd terraform && terraform apply -var-file=terraform.tfvars.local +cd .. && make apply-cp # talosctl apply-config to all 3 control planes +``` + +But to add a new Kubernetes Deployment or update an Ingress, you only `git push` — **never `kubectl apply`**. + +### CoreDNS ConfigMap Ownership — Critical + +⚠️ **Warning:** The `coredns` ConfigMap in `kube-system` namespace is **owned by Talos**, not ArgoCD or kubectl. +It is rendered from `terraform/files/coredns/Corefile` into Talos's machine config at bootstrap time. + +**Do not `kubectl apply` or `kubectl edit` this ConfigMap directly.** Doing so transfers field ownership to kubectl's +client-side-apply mechanism, and Talos's inline-manifest controller will silently no-op on every future reconcile +(server-side-apply conflict, no error surfaced). + +**To update CoreDNS (e.g., add a hostname rewrite):** +1. Edit `terraform/files/coredns/Corefile` +2. Commit + push +3. Run `cd terraform && terraform apply -var-file=terraform.tfvars.local` +4. Run `make apply-cp` to push config to all control planes +5. CoreDNS picks up changes via its `reload` plugin — no pod restart needed + +**If you accidentally edited the ConfigMap directly and broke Talos's ownership:** +```bash +kubectl delete configmap coredns -n kube-system +# Wait ~30s for Talos's k8s.ManifestApplyController to recreate it +kubectl get configmap coredns -n kube-system -w +``` + +Or as a stopgap, apply the correct content yourself: +```bash +kubectl apply --server-side -f <(terraform output coredns_config) +``` + ## Quick Start — Deploying the Cluster ### 1. Bootstrap Talos Nodes diff --git a/project-usage/jwt-auth-rollout.md b/project-usage/jwt-auth-rollout.md new file mode 100644 index 0000000..eaf80f2 --- /dev/null +++ b/project-usage/jwt-auth-rollout.md @@ -0,0 +1,224 @@ +# Unified Authentik JWT/OIDC Auth — Rollout Status & Plan + +Goal: every service's API (not just browser logins) authenticates against +Authentik as the single OIDC provider, with RBAC driven by Authentik group +membership (`groups` claim) and/or the synthesized `permissions` claim +(`GROUP_PERMISSIONS` dict in [k8s/infra/iam/scripts/authentik-provision.py](../k8s/infra/iam/scripts/authentik-provision.py)). +Same shape as what's already working for Vault: register an Authentik OAuth2 +provider/application, the service validates JWTs against Authentik's JWKS, +claims map to policy/role. + +Verified live 2026-08-27. + +## JWT validation & authorization contract (what a service must implement) + +This is the checklist any service in any repo needs to satisfy to actually +consume Authentik JWTs, not just have a placeholder issuer URL. Confirmed +live against the `vault` app's discovery doc (2026-08-27); same shape for +every other Authentik app/slug. + +### 1. Discovery & key fetch + +- Per-app issuer: `https://authentik.riotpiao.com/application/o//` +- Discovery doc: `.well-known/openid-configuration` +- JWKS: `jwks/` (also given as `jwks_uri` in the discovery doc) +- **Gotcha:** the token endpoint is *shared* across every app, not per-slug: + `https://authentik.riotpiao.com/application/o/token/`. Don't construct it + from the issuer the way jwks_uri is constructed. +- Fetch the JWKS through the discovery doc (or `jwks_uri` directly) and + **cache it with a TTL + refresh-on-`kid`-miss**, not a hardcoded key blob — + Authentik's signing key (`SIGNING_KEY_PK` in `authentik-provision.py`) can + rotate, and a hardcoded key silently breaks every token validation the + moment it does. + +### 2. Token validation checklist + +- Verify the signature against the fetched JWKS. +- **Pin the algorithm allow-list to `RS256` only** (confirmed + `id_token_signing_alg_values_supported: ["RS256"]` — nothing else is + issued). Reject any token whose header claims a different `alg`, + including `"none"` — this is the standard alg-confusion defense; don't + trust the token's own header to pick the verification algorithm. +- Verify `iss` equals the exact expected issuer string for that app's slug + (not just "some Authentik issuer" — a token minted for a different app + should not validate here). +- Verify `aud` (or `azp`) equals the service's own `client_id`. +- Verify `exp`/`nbf`/`iat` with a small clock-skew tolerance (30-60s). + +### 3. Claims to request and how to use them + +- `groups` and `permissions` are **non-default scopes** — Authentik only + includes them in the token if explicitly requested, both at provider + registration (`property_mappings` — every app in `authentik-provision.py`'s + `SERVICES` loop already gets both via `SCOPE_PKS`) and at token-request + time (e.g. Vault's OIDC role sets `oidc_scopes: ["permissions"]`; a service + doing its own token requests needs `scope=openid permissions` or + `scope=openid groups` in the auth/token request). +- **Permissions-claim pattern** (what kmsvc/temporal/etc. should do): read + the `permissions` claim (list of strings like `"kmsvc:read"`, + `"kmsvc:write"`), check it contains the string the attempted action needs. + `homelab-admins` members get the literal string `"*"` in that list — + treat that as wildcard-allow, not as a literal permission string to match + against. +- **Groups-claim pattern** (what Vault does, what Temporal's `claimMapper` + or MinIO's `policy` claim expect): read the `groups` claim directly (raw + Authentik group names) and map group name -> internal role/policy inside + the service's own config, the way Vault's Identity Group aliases do. Prefer + this pattern when the service already has its own native role/policy + system to map into, instead of parsing the synthesized `permissions` + strings. + +### 4. Gap: machine-to-machine (no human/browser step) calls aren't wired anywhere yet + +Authentik's server supports `client_credentials` — confirmed live in +`grant_types_supported` on the discovery doc. But **every provider currently +registered in `authentik-provision.py` only declares +`grant_types: ["authorization_code", "refresh_token"]`** (see the `SERVICES` +loop and the `kubernetes` public-client block) — none of them can issue a +token via `client_credentials` today. If a service needs to call another +service's API with no human/browser step at all (true service-to-service, +not "a human logged in via browser, then the resulting token gets reused"), +it needs its own confidential-client Application with `client_credentials` +added to `grant_types`, and requests a token via +`POST /application/o/token/` with `grant_type=client_credentials`. Decide +per-service whether this is actually needed before assuming a JWT is always +available to attach to an outbound call — right now, none are set up for it. + +### 5. Device code flow (CLI / headless API callers with no local browser) + +For a caller that can't do a browser redirect itself (a CLI tool, a script +on a headless box, an API client embedded in another service) but still +needs a *human* to approve the login somewhere. Confirmed live 2026-08-27 +against the `vault` provider (device-authorization leg only — the +token-polling leg below is standard RFC 8628 + Authentik's declared support, +not independently re-verified since completing it needs a real human +approval step). + +**Gap, same shape as client_credentials above:** every provider currently +registered only has `grant_types: ["authorization_code", "refresh_token"]`. +Requesting a device code against one of them fails with a generic +`invalid_client` error — misleading, since it looks like a bad secret but is +actually just the missing grant type. Confirmed by adding +`urn:ietf:params:oauth:grant-type:device_code` to the `vault` provider's +`grant_types` (temporarily, via the API, reverted after testing) — the exact +same request then succeeded. Any service that wants this flow needs that +grant type added to its own provider's `grant_types` list in +`authentik-provision.py`. + +**Flow, what the customer/caller must specify:** + +1. `POST https://authentik.riotpiao.com/application/o/device/`, + form-encoded, with `client_id=` and `scope=openid + ...` (same non-default-scope rule as above — add `permissions`/`groups` + if the resulting token needs to drive RBAC). Live response shape, + confirmed: + ```json + { + "device_code": "", + "user_code": "022228491", + "verification_uri": "https://authentik.riotpiao.com/device", + "verification_uri_complete": "https://authentik.riotpiao.com/device?code=022228491", + "expires_in": 60, + "interval": 5 + } + ``` + No client_secret was required on this leg even for a confidential client + — Authentik didn't enforce it here in testing. Don't rely on that as a + security boundary; verify token-endpoint behavior below before assuming + secrets are optional throughout the flow. +2. The caller shows `user_code` + `verification_uri` (or just the + `verification_uri_complete` link) to the human — this can be on a + completely different device. The human opens it, logs into Authentik, + approves. +3. The caller polls `POST https://authentik.riotpiao.com/application/o/token/` + with `grant_type=urn:ietf:params:oauth:grant-type:device_code`, + `device_code=`, `client_id=` (+ + `client_secret` if confidential — not independently confirmed on this + leg), no faster than every `interval` seconds. Per RFC 8628/Authentik's + declared support: `authorization_pending` while waiting, `slow_down` if + polling too fast, `expired_token` past `expires_in`, `access_denied` if + the human rejects it, or the normal token bundle on approval. + +## Done + +| Service | Authentik app | Validates JWT itself | Notes | +|---|---|---|---| +| grafana, forgejo, argocd, homarr, paperless, immich | yes | yes (native OIDC login) | browser session auth | +| **vault** | yes | yes, tested | `vault login -method=oidc`; policies attached via Vault Identity Groups aliased to Authentik group names (`groups` claim) — adding a new Vault policy needs zero Authentik-side change, just a new `identity/group`+`identity/group-alias` pair in Vault | +| **minio** | yes | likely yes, **not yet load-tested** | `MINIO_IDENTITY_OPENID_SCOPES` includes non-default scopes, which also enables `AssumeRoleWithWebIdentity` (real S3 API access via STS), not just console login. Need to actually mint a token and call `AssumeRoleWithWebIdentity` + an S3 op to confirm before calling this done. | +| kubectl | yes (public PKCE client `kubernetes`) | yes | kube-apiserver `--oidc-*` flags confirmed live (2026-08-27); per-service `Role`/`RoleBinding` in `k8s/infra/rbac/*.yaml` bind `oidc:-admins` groups — should now actually be enforced, not inert as an earlier pass of this doc set assumed. Worth a real login test to confirm group->Role resolution end-to-end. | + +## Not done — this repo can finish it (config only, no new app code) + +- **portainer** — has a *built-in* OAuth login feature. Register an Authentik + app (confidential client, redirect URI to Portainer's OAuth callback) the + same way `vault`/`immich`/etc. are registered in `authentik-provision.py`, + then flip on OAuth in Portainer's own settings (or via its API). No + application code involved. + +- **kmsvc / management-service** — `KMSVC_AUTHENTIK_ISSUER_URL` and + `KMSVC_AUTHENTIK_AUDIENCE` env vars already exist in + [k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml) + but are empty strings — placeholders, never wired. Registering the + Authentik app and populating them is real progress, BUT: **whether the + deployed image (`ghcr.io/riotpiaole/kmsvc-management-service`) actually + validates a JWT against those values is unverified** — that's a claim + about code in a separate repo this checkout doesn't have. Confirm there + before assuming this is load-bearing (this repo already has one precedent + of a README claiming JWT support that the code didn't actually have — the + LLM gateway, see "Not done" below). + +- **temporal** — Temporal's own Helm chart + ([k8s/apps/temporal/temporal-values.yaml](../k8s/apps/temporal/temporal-values.yaml), + no `authorization`/`jwtKeyProvider` block currently set) has **native** + JWT authorization support (`server.config.authorization.jwtKeyProvider` + + `claimMapper`) — this is Helm values, not custom plugin code. The real + target is the **`temporal-frontend` gRPC service** (port 7233 — what SDKs + and workers actually connect to), not the Web UI. Verified live + 2026-08-27: only `temporal.riotpiao.com` is ingressed, and it points at + `temporal-web` (the UI); `temporal-frontend`/`temporal-frontend-headless` + are ClusterIP-only, no public hostname, no `temporal-frontend.riotpiao.com` + exists yet. Temporal's `jwtKeyProvider`/`claimMapper` auth gates the + frontend service itself at the RPC level regardless of exposure — it would + apply the same to in-cluster workers as to any external caller, so this + is worth doing even with no public ingress. Needs: + 1. Point `jwtKeyProvider.keySourceURIs` at Authentik's JWKS endpoint for a + dedicated `temporal` Authentik app (none exists yet — only the + `temporal-admins` permissions-claim group, confirmed live in Authentik, + no OAuth2 provider/application). + 2. Confirm whether Temporal's default `claimMapper` expects a claim shape + compatible with this repo's `permissions` claim + (`["temporal:read","temporal:write"]`) or needs a custom claim-mapper + config to translate it into Temporal's own namespace-permission format. + 3. **Open question, unresolved:** is external (outside-cluster) worker/client + access to `temporal-frontend` even needed? If yes, that's a separate + decision on top of the JWT work — a gRPC-capable ingress would need to + be added (same shape as kmsvc's `grpcPathPrefix` pattern in + [k8s/apps/messaging/management-service/values.yaml](../k8s/apps/messaging/management-service/values.yaml)), + since nothing currently exposes 7233 outside the cluster. + +## Not done — needs application code in a different repo + +- **poimen-memory** (`forgejo.riotpiao.com/rock/poimen.git`) — confirmed + current auth is a static API key (`poimen-memory-api-key` k8s Secret, 1 + key, Opaque), not JWT/OIDC at all. No Authentik app exists. Two paths: + 1. Add real JWT validation to poimen-memory's own code (that repo). + 2. Bridge without touching poimen-memory: front it behind the Go API + gateway, gateway validates the Authentik JWT once and forwards with the + existing static API key internally. Doesn't fix poimen-memory itself, + but unblocks unified auth for callers without waiting on that repo. + +- **LLM gateway** (`rock/homelab-frontend`, deployed as `api-gateway`) — + `internal/auth` is an empty directory per that repo's own task board + (phase 3, tasks 2.9-2.15 also unbuilt — Anthropic dialect, streaming tool + calls). `LLM_TOOL_CALLS.md` in that repo overclaims what's implemented; + don't trust it without checking `internal/auth` directly. + +## Open design question, unresolved + +Should each service validate JWTs itself (JWKS-fetch + claim-check logic +duplicated per service), or should the Go gateway become the one +chokepoint doing JWT validation for everything behind it (kmsvc, +llm-serving, eventually poimen-memory), so that logic exists once instead +of N times? This changes where phase-3 code actually goes. Not decided as +of this doc.