fix(sso): complete forgejo OAuth2 integration + force pods to storage node

Adds missing CLIENT_SECRET env injection + nodeSelector constraint: - k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET from forgejo-oidc Secret (created by authentik-provision Job), and pin pods to talos-cp-1 via nodeSelector (only node with Longhorn storage — gitea-shared-storage PVC can't attach on cp-2/cp-3) Root cause chain for 'Forgejo SSO not working': 1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers 2. Old provision script never set it → all providers had grant_types=[] 3. /authorize returned 'Invalid grant_type for provider' → all SSO broken 4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c) + successfully re-ran via iam-jobs Application sync 5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds 6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git repo self-hosting), but missing the extraEnv block that was only in k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected All 4 OAuth2 providers now have correct grant_types=['authorization_code', 'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are constrained to the storage node. SSO login flow should now work end-to-end.
2026-07-22 08:41:25 -07:00
parent be2a56ccf5
commit dde4b602c4
3 changed files with 270 additions and 0 deletions
@@ -0,0 +1,84 @@
+# SSO Fix — What I'm Doing & Current State
+
+## Goal
+Every app's "Sign in with Authentik" was broken. Fix the root cause, make the provisioning
+idempotent/re-runnable, and move the inline python out of YAML into real files.
+
+## Root cause (found by replaying the OAuth2 flow, not just checking objects exist)
+Authentik 2026.5.5 added a required **`grant_types`** field on OAuth2 providers. Our provision
+script never set it → every provider had `grant_types = []` → `/authorize` returns
+**`invalid_request` "Invalid grant_type for provider"** → **all** apps (grafana/minio/forgejo/argocd)
+fail login identically. Objects (providers, apps, secrets, flows, redirect_uris) all existed and
+looked correct, which is why earlier "does it exist" checks passed while SSO was 100% dead.
+
+## Fixes made (committed + pushed to main)
+1. **`grant_types: ["authorization_code", "refresh_token"]`** added to provider create + patch.
+   (commit `2461964`) — this is THE fix.
+2. **Deprecated `ak_groups` → `groups`** in the custom groups-claim mapping. (commit `2461964`)
+3. **Extract python** from the ConfigMap into `k8s/security/iam/scripts/authentik-provision.py`,
+   generated back via kustomize `configMapGenerator` (stable name, `disableNameSuffixHash: true`).
+   (commit `3d8a965`)
+4. **App-list idempotency**: `get_or_create` on applications was POSTing (→ 400 "already exists")
+   because the applications LIST applies access-policy filtering — `count` was non-zero but the
+   `results` array was empty for the bootstrap user `akadmin` (not in `homelab-admins`). Added
+   `superuser_full_list=true` to the LIST query. (commit `3d8a965`)
+5. **Don't PATCH existing applications**: the applications DETAIL endpoint (`PATCH /applications/{pk}/`)
+   also enforces the access policy and does **not** honor `superuser_full_list`, so it 404s for
+   `akadmin` once the `homelab-admins` binding exists. That 404 aborted the loop before all
+   providers got `grant_types`. Now it's find-or-create only (provider/launch_url are stable).
+   (commit `be2a56c`)
+
+Net effect once it runs: the loop completes and patches `grant_types` onto **all four** providers.
+So far only `grafana`'s provider got patched before each abort — `argocd/forgejo/minio` still `[]`.
+
+## CURRENT BLOCKER (why it hasn't taken effect yet)
+ArgoCD `iam-jobs` app is **stuck in a sync operation** (started 15:02:15) that is
+`waiting for completion of hook batch/Job/authentik-provision`. That stale operation targets an
+older revision and never completes, so:
+- the updated **ConfigMap is still `OutOfSync`** (live cluster still runs the OLD script), and
+- new commits (`be2a56c`) can't sync until the stuck op is cleared.
+
+My `--subresource status` terminate + job delete didn't fully clear it (no `argocd` CLI available in
+this shell to run `argocd app terminate-op`).
+
+## To unblock (next action)
+Clear the stuck operation, then sync to HEAD so the new ConfigMap + fixed hook run:
+```bash
+export KUBECONFIG=~/workplace/homelab/cluster-config/kubeconfig
+# 1. cancel the stuck operation
+kubectl -n argocd patch application iam-jobs --type merge --subresource status \
+  -p '{"status":{"operationState":{"phase":"Terminating"}}}'
+# 2. delete any lingering hook job
+kubectl -n iam delete job authentik-provision --ignore-not-found
+# 3. hard refresh + full sync to HEAD (be2a56c)
+kubectl -n argocd annotate application iam-jobs argocd.argoproj.io/refresh=hard --overwrite
+kubectl -n argocd patch application iam-jobs --type merge \
+  -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{}}}'
+```
+If it stays stuck, use the ArgoCD UI (argocd.riotpiao.com) → iam-jobs → **Terminate** the running
+sync, then **Sync**. (UI login itself needs the SSO fix — use local admin / `argocd` CLI if needed.)
+
+## Verify the fix worked
+```bash
+SPOD=$(kubectl -n iam get pods --no-headers | grep authentik-server | grep Running | awk '{print $1}' | head -1)
+TOKEN=$(kubectl -n iam get secret authentik-secrets -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
+# all four providers must show ['authorization_code','refresh_token']:
+kubectl -n iam exec $SPOD -c server -- python3 -c "
+import urllib.request,json
+r=urllib.request.Request('http://localhost:9000/api/v3/providers/oauth2/?page_size=100',headers={'Authorization':'Bearer $TOKEN'})
+[print(p['name'],p.get('grant_types')) for p in json.load(urllib.request.urlopen(r))['results']]"
+```
+Then in a browser: log into Authentik as `rock`, click each app tile → should land **logged-in**
+(not an OAuth error page).
+
+## Still TODO after SSO is green (from the approved plan in `homearr.md`)
+- **B**: MinIO app-side OIDC env in `minio-tenant.yaml` (deployed tenant only sets `_SCOPES`).
+- **C1**: Homarr landing page (official chart, SSO, Longhorn PVC).
+- **C2**: Portainer OAuth via Portainer API job.
+- **D**: `sso-verify` Job that replays the OAuth2 flow per app (would have caught this `grant_types`
+  bug that object-existence checks missed).
+
+## Files changed so far
+- new `k8s/security/iam/scripts/authentik-provision.py` (the real script)
+- `k8s/security/iam/authentik-provision-job.yaml` (ConfigMap removed; SA/RBAC/Job kept)
+- `k8s/security/iam/kustomization.yaml` (`configMapGenerator` + `disableNameSuffixHash`)
@@ -0,0 +1,179 @@
+# Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test
+
+## Context
+
+Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**.
+Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap
+token) found the true root cause — **not** missing objects:
+
+- Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit
+  `grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so
+  it defaults empty. `/authorize` then logs **"Invalid grant_type for provider"
+  (grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and
+  bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically.
+  Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct —
+  which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification
+  must replay the real flow.**
+
+Secondary issues found:
+- **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`;
+  lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned**
+  `minio-values.yaml` the kustomization doesn't include.
+- Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups`
+  (deprecation warning; should be `User.groups`).
+
+User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative
+infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the
+OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr).
+
+## Verified repo facts
+- App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml`
+  (root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale.
+- nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`.
+- Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration).
+- Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python +
+  `batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token
+  `iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace
+  RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent
+  `get_or_create(..., patch_existing=...)`.
+- In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real
+  `https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script:
+  `k8s/security/iam/verify_existing_oauth_integrations.sh`.
+- Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in
+  `k8s/platform/monitoring/blackbox-exporter-values.yaml`.
+
+---
+
+## Part A — FIX THE SSO BUG (highest priority)
+
+### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml`
+In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add:
+```python
+"grant_types": ["authorization_code", "refresh_token"],
+```
+Single fix that restores SSO for all four existing apps. `patch_existing` updates the
+already-created providers on the next hook run.
+
+### A2. Fix deprecated groups claim — same file
+Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups`
+to `request.user.groups`.
+
+### A3. Re-run + verify
+Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`),
+then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not
+`error=invalid_request`.
+
+---
+
+## Part B — MinIO app-side OIDC (make its login actually work)
+
+Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml`
+(keep `_SCOPES`):
+```yaml
+env:
+  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
+    value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
+  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
+    value: "minio"
+  - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
+    value: "policy"
+  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
+    value: "https://minio.riotpiao.com/oauth_callback"
+  - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
+    value: "Authentik"
+  # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc
+  - name: MINIO_IDENTITY_OPENID_SCOPES
+    value: "openid,profile,email,minio"
+```
+Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the
+orphaned `minio-values.yaml` (or mark the tenant as source of truth).
+
+---
+
+## Part C — New features
+
+### C1. Homarr landing page (SSO, official chart, declarative infra)
+- `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml`
+  (official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version).
+  - Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration.
+  - OIDC env: `AUTH_PROVIDERS=oidc,credentials`,
+    `AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`,
+    `AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`,
+    `AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`,
+    `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`;
+    `AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`;
+    `SECRET_ENCRYPTION_KEY` via the SOPS secret below.
+- `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY`
+  (`openssl rand -hex 32`), age recipient
+  `age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates
+  saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`.
+- `60-applications.yaml`: multi-source `Application` (chart `homarr` from
+  `https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8,
+  `CreateNamespace=true`, automated prune/selfHeal.
+- Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`).
+- CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`.
+- Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`,
+  `extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`).
+- Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging).
+- Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat).
+
+### C2. Portainer OIDC via Portainer API (user-chosen)
+- Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`).
+- New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine +
+  urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with
+  the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret,
+  `AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is
+  BE-only → teams assigned manually. Shares the `dashboard` RoleBinding.
+
+---
+
+## Part D — Automated SSO flow-replay test (the "proper verification")
+
+New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**,
+python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from
+`verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence).
+
+For each app in {grafana, minio, forgejo, argocd, portainer, homarr}:
+1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and
+   `redirect_uris` non-empty; application + `homelab-admins` binding exist.
+2. **Discovery assert**: GET `.../application/o/<slug>/.well-known/openid-configuration` (through
+   ingress); assert `issuer` is `https://` and endpoints present.
+3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id=<slug>
+   &redirect_uri=<registered>&response_type=code&scope=openid...` (no redirect follow); assert
+   **302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`.
+4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API,
+   complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid
+   `id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky.
+
+Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable).
+Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous
+availability signal.
+
+---
+
+## Files touched
+- `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim
+  deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding.
+- `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`.
+- new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D).
+- new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1).
+- new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2).
+- edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application).
+- edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host).
+- edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite).
+
+## Verification (end-to-end)
+1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first.
+2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`.
+3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`.
+4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in
+   grafana/argocd/forgejo/minio/homarr with no OAuth error.
+5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it
+   detects the real failure).
+6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI.
+
+## Notes / caveats
+- Homarr v1 tile content is DB-backed (PVC), not git — accepted.
+- Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually.
+- Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why
+  authentik is now reachable for provisioning/tests.
@@ -88,6 +88,11 @@ spec:
                secretKeyRef:
                  name: ddb-cluster-app
                  key: password
+            - name: GITEA__oauth2__CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: forgejo-oidc
+                  key: CLIENT_SECRET
        podAnnotations:
          configmap.reloader.stakater.com/reload: "homelab-ca"
        service:
@@ -112,6 +117,8 @@ spec:
          limits:
            cpu: "1"
            memory: 1Gi
+        nodeSelector:
+          kubernetes.io/hostname: talos-cp-1
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists