homelab/SSO-FIX-STATUS.md

SSO Fix — What I'm Doing & Current State

Goal

Every app's "Sign in with Authentik" was broken. Fix the root cause, make the provisioning idempotent/re-runnable, and move the inline python out of YAML into real files.

Root cause (found by replaying the OAuth2 flow, not just checking objects exist)

Authentik 2026.5.5 added a required grant_types field on OAuth2 providers. Our provision script never set it → every provider had grant_types = [] → /authorize returns invalid_request "Invalid grant_type for provider" → all apps (grafana/minio/forgejo/argocd) fail login identically. Objects (providers, apps, secrets, flows, redirect_uris) all existed and looked correct, which is why earlier "does it exist" checks passed while SSO was 100% dead.

Fixes made (committed + pushed to main)

1. grant_types: ["authorization_code", "refresh_token"] added to provider create + patch. (commit 2461964) — this is THE fix.
2. Deprecated ak_groups → groups in the custom groups-claim mapping. (commit 2461964)
3. Extract python from the ConfigMap into k8s/security/iam/scripts/authentik-provision.py, generated back via kustomize configMapGenerator (stable name, disableNameSuffixHash: true). (commit 3d8a965)
4. App-list idempotency: get_or_create on applications was POSTing (→ 400 "already exists") because the applications LIST applies access-policy filtering — count was non-zero but the results array was empty for the bootstrap user akadmin (not in homelab-admins). Added superuser_full_list=true to the LIST query. (commit 3d8a965)
5. Don't PATCH existing applications: the applications DETAIL endpoint (PATCH /applications/{pk}/) also enforces the access policy and does not honor superuser_full_list, so it 404s for akadmin once the homelab-admins binding exists. That 404 aborted the loop before all providers got grant_types. Now it's find-or-create only (provider/launch_url are stable). (commit be2a56c)

Net effect once it runs: the loop completes and patches grant_types onto all four providers. So far only grafana's provider got patched before each abort — argocd/forgejo/minio still [].

CURRENT BLOCKER (why it hasn't taken effect yet)

ArgoCD iam-jobs app is stuck in a sync operation (started 15:02:15) that is waiting for completion of hook batch/Job/authentik-provision. That stale operation targets an older revision and never completes, so:

• the updated ConfigMap is still OutOfSync (live cluster still runs the OLD script), and
• new commits (be2a56c) can't sync until the stuck op is cleared.

My --subresource status terminate + job delete didn't fully clear it (no argocd CLI available in this shell to run argocd app terminate-op).

To unblock (next action)

Clear the stuck operation, then sync to HEAD so the new ConfigMap + fixed hook run:

export KUBECONFIG=~/workplace/homelab/cluster-config/kubeconfig
# 1. cancel the stuck operation
kubectl -n argocd patch application iam-jobs --type merge --subresource status \
  -p '{"status":{"operationState":{"phase":"Terminating"}}}'
# 2. delete any lingering hook job
kubectl -n iam delete job authentik-provision --ignore-not-found
# 3. hard refresh + full sync to HEAD (be2a56c)
kubectl -n argocd annotate application iam-jobs argocd.argoproj.io/refresh=hard --overwrite
kubectl -n argocd patch application iam-jobs --type merge \
  -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{}}}'

If it stays stuck, use the ArgoCD UI (argocd.riotpiao.com) → iam-jobs → Terminate the running sync, then Sync. (UI login itself needs the SSO fix — use local admin / argocd CLI if needed.)

Verify the fix worked

SPOD=$(kubectl -n iam get pods --no-headers | grep authentik-server | grep Running | awk '{print $1}' | head -1)
TOKEN=$(kubectl -n iam get secret authentik-secrets -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d)
# all four providers must show ['authorization_code','refresh_token']:
kubectl -n iam exec $SPOD -c server -- python3 -c "
import urllib.request,json
r=urllib.request.Request('http://localhost:9000/api/v3/providers/oauth2/?page_size=100',headers={'Authorization':'Bearer $TOKEN'})
[print(p['name'],p.get('grant_types')) for p in json.load(urllib.request.urlopen(r))['results']]"

Then in a browser: log into Authentik as rock, click each app tile → should land logged-in (not an OAuth error page).

Still TODO after SSO is green (from the approved plan in homearr.md)

• B: MinIO app-side OIDC env in minio-tenant.yaml (deployed tenant only sets _SCOPES).
• C1: Homarr landing page (official chart, SSO, Longhorn PVC).
• C2: Portainer OAuth via Portainer API job.
• D: sso-verify Job that replays the OAuth2 flow per app (would have caught this grant_types bug that object-existence checks missed).

Files changed so far

• new k8s/security/iam/scripts/authentik-provision.py (the real script)
• k8s/security/iam/authentik-provision-job.yaml (ConfigMap removed; SA/RBAC/Job kept)
• k8s/security/iam/kustomization.yaml (configMapGenerator + disableNameSuffixHash)