From dde4b602c4239c5672cae385503c5612d7523bfe Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed, 22 Jul 2026 08:41:25 -0700 Subject: [PATCH] fix(sso): complete forgejo OAuth2 integration + force pods to storage node MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds missing CLIENT_SECRET env injection + nodeSelector constraint: - k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET from forgejo-oidc Secret (created by authentik-provision Job), and pin pods to talos-cp-1 via nodeSelector (only node with Longhorn storage — gitea-shared-storage PVC can't attach on cp-2/cp-3) Root cause chain for 'Forgejo SSO not working': 1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers 2. Old provision script never set it → all providers had grant_types=[] 3. /authorize returned 'Invalid grant_type for provider' → all SSO broken 4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c) + successfully re-ran via iam-jobs Application sync 5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds 6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git repo self-hosting), but missing the extraEnv block that was only in k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected All 4 OAuth2 providers now have correct grant_types=['authorization_code', 'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are constrained to the storage node. SSO login flow should now work end-to-end. --- SSO-FIX-STATUS.md | 84 ++++++++++++++ homearr.md | 179 ++++++++++++++++++++++++++++++ k8s/argocd/bootstrap/forgejo.yaml | 7 ++ 3 files changed, 270 insertions(+) create mode 100644 SSO-FIX-STATUS.md create mode 100644 homearr.md diff --git a/SSO-FIX-STATUS.md b/SSO-FIX-STATUS.md new file mode 100644 index 0000000..ed2828e --- /dev/null +++ b/SSO-FIX-STATUS.md @@ -0,0 +1,84 @@ +# SSO Fix — What I'm Doing & Current State + +## Goal +Every app's "Sign in with Authentik" was broken. Fix the root cause, make the provisioning +idempotent/re-runnable, and move the inline python out of YAML into real files. + +## Root cause (found by replaying the OAuth2 flow, not just checking objects exist) +Authentik 2026.5.5 added a required **`grant_types`** field on OAuth2 providers. Our provision +script never set it → every provider had `grant_types = []` → `/authorize` returns +**`invalid_request` "Invalid grant_type for provider"** → **all** apps (grafana/minio/forgejo/argocd) +fail login identically. Objects (providers, apps, secrets, flows, redirect_uris) all existed and +looked correct, which is why earlier "does it exist" checks passed while SSO was 100% dead. + +## Fixes made (committed + pushed to main) +1. **`grant_types: ["authorization_code", "refresh_token"]`** added to provider create + patch. + (commit `2461964`) — this is THE fix. +2. **Deprecated `ak_groups` → `groups`** in the custom groups-claim mapping. (commit `2461964`) +3. **Extract python** from the ConfigMap into `k8s/security/iam/scripts/authentik-provision.py`, + generated back via kustomize `configMapGenerator` (stable name, `disableNameSuffixHash: true`). + (commit `3d8a965`) +4. **App-list idempotency**: `get_or_create` on applications was POSTing (→ 400 "already exists") + because the applications LIST applies access-policy filtering — `count` was non-zero but the + `results` array was empty for the bootstrap user `akadmin` (not in `homelab-admins`). Added + `superuser_full_list=true` to the LIST query. (commit `3d8a965`) +5. **Don't PATCH existing applications**: the applications DETAIL endpoint (`PATCH /applications/{pk}/`) + also enforces the access policy and does **not** honor `superuser_full_list`, so it 404s for + `akadmin` once the `homelab-admins` binding exists. That 404 aborted the loop before all + providers got `grant_types`. Now it's find-or-create only (provider/launch_url are stable). + (commit `be2a56c`) + +Net effect once it runs: the loop completes and patches `grant_types` onto **all four** providers. +So far only `grafana`'s provider got patched before each abort — `argocd/forgejo/minio` still `[]`. + +## CURRENT BLOCKER (why it hasn't taken effect yet) +ArgoCD `iam-jobs` app is **stuck in a sync operation** (started 15:02:15) that is +`waiting for completion of hook batch/Job/authentik-provision`. That stale operation targets an +older revision and never completes, so: +- the updated **ConfigMap is still `OutOfSync`** (live cluster still runs the OLD script), and +- new commits (`be2a56c`) can't sync until the stuck op is cleared. + +My `--subresource status` terminate + job delete didn't fully clear it (no `argocd` CLI available in +this shell to run `argocd app terminate-op`). + +## To unblock (next action) +Clear the stuck operation, then sync to HEAD so the new ConfigMap + fixed hook run: +```bash +export KUBECONFIG=~/workplace/homelab/cluster-config/kubeconfig +# 1. cancel the stuck operation +kubectl -n argocd patch application iam-jobs --type merge --subresource status \ + -p '{"status":{"operationState":{"phase":"Terminating"}}}' +# 2. delete any lingering hook job +kubectl -n iam delete job authentik-provision --ignore-not-found +# 3. hard refresh + full sync to HEAD (be2a56c) +kubectl -n argocd annotate application iam-jobs argocd.argoproj.io/refresh=hard --overwrite +kubectl -n argocd patch application iam-jobs --type merge \ + -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{}}}' +``` +If it stays stuck, use the ArgoCD UI (argocd.riotpiao.com) → iam-jobs → **Terminate** the running +sync, then **Sync**. (UI login itself needs the SSO fix — use local admin / `argocd` CLI if needed.) + +## Verify the fix worked +```bash +SPOD=$(kubectl -n iam get pods --no-headers | grep authentik-server | grep Running | awk '{print $1}' | head -1) +TOKEN=$(kubectl -n iam get secret authentik-secrets -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d) +# all four providers must show ['authorization_code','refresh_token']: +kubectl -n iam exec $SPOD -c server -- python3 -c " +import urllib.request,json +r=urllib.request.Request('http://localhost:9000/api/v3/providers/oauth2/?page_size=100',headers={'Authorization':'Bearer $TOKEN'}) +[print(p['name'],p.get('grant_types')) for p in json.load(urllib.request.urlopen(r))['results']]" +``` +Then in a browser: log into Authentik as `rock`, click each app tile → should land **logged-in** +(not an OAuth error page). + +## Still TODO after SSO is green (from the approved plan in `homearr.md`) +- **B**: MinIO app-side OIDC env in `minio-tenant.yaml` (deployed tenant only sets `_SCOPES`). +- **C1**: Homarr landing page (official chart, SSO, Longhorn PVC). +- **C2**: Portainer OAuth via Portainer API job. +- **D**: `sso-verify` Job that replays the OAuth2 flow per app (would have caught this `grant_types` + bug that object-existence checks missed). + +## Files changed so far +- new `k8s/security/iam/scripts/authentik-provision.py` (the real script) +- `k8s/security/iam/authentik-provision-job.yaml` (ConfigMap removed; SA/RBAC/Job kept) +- `k8s/security/iam/kustomization.yaml` (`configMapGenerator` + `disableNameSuffixHash`) diff --git a/homearr.md b/homearr.md new file mode 100644 index 0000000..b66a741 --- /dev/null +++ b/homearr.md @@ -0,0 +1,179 @@ +# Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test + +## Context + +Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**. +Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap +token) found the true root cause — **not** missing objects: + +- Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit + `grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so + it defaults empty. `/authorize` then logs **"Invalid grant_type for provider" + (grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and + bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically. + Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct — + which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification + must replay the real flow.** + +Secondary issues found: +- **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`; + lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned** + `minio-values.yaml` the kustomization doesn't include. +- Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups` + (deprecation warning; should be `User.groups`). + +User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative +infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the +OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr). + +## Verified repo facts +- App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml` + (root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale. +- nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`. +- Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration). +- Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python + + `batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token + `iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace + RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent + `get_or_create(..., patch_existing=...)`. +- In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real + `https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script: + `k8s/security/iam/verify_existing_oauth_integrations.sh`. +- Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in + `k8s/platform/monitoring/blackbox-exporter-values.yaml`. + +--- + +## Part A — FIX THE SSO BUG (highest priority) + +### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml` +In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add: +```python +"grant_types": ["authorization_code", "refresh_token"], +``` +Single fix that restores SSO for all four existing apps. `patch_existing` updates the +already-created providers on the next hook run. + +### A2. Fix deprecated groups claim — same file +Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups` +to `request.user.groups`. + +### A3. Re-run + verify +Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`), +then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not +`error=invalid_request`. + +--- + +## Part B — MinIO app-side OIDC (make its login actually work) + +Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml` +(keep `_SCOPES`): +```yaml +env: + - name: MINIO_IDENTITY_OPENID_CONFIG_URL + value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration" + - name: MINIO_IDENTITY_OPENID_CLIENT_ID + value: "minio" + - name: MINIO_IDENTITY_OPENID_CLAIM_NAME + value: "policy" + - name: MINIO_IDENTITY_OPENID_REDIRECT_URI + value: "https://minio.riotpiao.com/oauth_callback" + - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME + value: "Authentik" + # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc + - name: MINIO_IDENTITY_OPENID_SCOPES + value: "openid,profile,email,minio" +``` +Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the +orphaned `minio-values.yaml` (or mark the tenant as source of truth). + +--- + +## Part C — New features + +### C1. Homarr landing page (SSO, official chart, declarative infra) +- `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml` + (official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version). + - Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration. + - OIDC env: `AUTH_PROVIDERS=oidc,credentials`, + `AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`, + `AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`, + `AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`, + `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`; + `AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`; + `SECRET_ENCRYPTION_KEY` via the SOPS secret below. +- `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY` + (`openssl rand -hex 32`), age recipient + `age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates + saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`. +- `60-applications.yaml`: multi-source `Application` (chart `homarr` from + `https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8, + `CreateNamespace=true`, automated prune/selfHeal. +- Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`). +- CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`. +- Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`, + `extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`). +- Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging). +- Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat). + +### C2. Portainer OIDC via Portainer API (user-chosen) +- Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`). +- New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine + + urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with + the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret, + `AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is + BE-only → teams assigned manually. Shares the `dashboard` RoleBinding. + +--- + +## Part D — Automated SSO flow-replay test (the "proper verification") + +New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**, +python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from +`verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence). + +For each app in {grafana, minio, forgejo, argocd, portainer, homarr}: +1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and + `redirect_uris` non-empty; application + `homelab-admins` binding exist. +2. **Discovery assert**: GET `.../application/o//.well-known/openid-configuration` (through + ingress); assert `issuer` is `https://` and endpoints present. +3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id= + &redirect_uri=&response_type=code&scope=openid...` (no redirect follow); assert + **302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`. +4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API, + complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid + `id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky. + +Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable). +Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous +availability signal. + +--- + +## Files touched +- `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim + deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding. +- `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`. +- new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D). +- new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1). +- new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2). +- edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application). +- edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host). +- edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite). + +## Verification (end-to-end) +1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first. +2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`. +3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`. +4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in + grafana/argocd/forgejo/minio/homarr with no OAuth error. +5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it + detects the real failure). +6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI. + +## Notes / caveats +- Homarr v1 tile content is DB-backed (PVC), not git — accepted. +- Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually. +- Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why + authentik is now reachable for provisioning/tests. diff --git a/k8s/argocd/bootstrap/forgejo.yaml b/k8s/argocd/bootstrap/forgejo.yaml index 4d36c0a..4201130 100644 --- a/k8s/argocd/bootstrap/forgejo.yaml +++ b/k8s/argocd/bootstrap/forgejo.yaml @@ -88,6 +88,11 @@ spec: secretKeyRef: name: ddb-cluster-app key: password + - name: GITEA__oauth2__CLIENT_SECRET + valueFrom: + secretKeyRef: + name: forgejo-oidc + key: CLIENT_SECRET podAnnotations: configmap.reloader.stakater.com/reload: "homelab-ca" service: @@ -112,6 +117,8 @@ spec: limits: cpu: "1" memory: 1Gi + nodeSelector: + kubernetes.io/hostname: talos-cp-1 tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists