homelab/homearr.md

Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test

Context

Reported symptom: clicking any app in the Authentik launcher, no application lets you sign in. Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap token) found the true root cause — not missing objects:

• Every OAuth2 provider has grant_types = []. Authentik 2026.5.5 added an explicit grant_types list field; the provision script (authentik-provision-job.yaml) never sets it, so it defaults empty. /authorize then logs "Invalid grant_type for provider" (grant_type=authorization_code) → invalid_request "The request is otherwise malformed" and bounces an error back to the app. Breaks all apps (grafana/minio/forgejo/argocd) identically. Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct — which is why "check the objects exist" checks passed while SSO was 100% broken. Verification must replay the real flow.

Secondary issues found:

• MinIO app side unwired: deployed minio-tenant.yaml sets only MINIO_IDENTITY_OPENID_SCOPES; lacks CONFIG_URL/CLIENT_ID/envFrom minio-oidc. Full config sits in an orphaned minio-values.yaml the kustomization doesn't include.
• Custom homelab: groups claim property mapping uses deprecated User.ak_groups (deprecation warning; should be User.groups).

User-requested additions: Homarr landing page (Authentik SSO, official Helm chart, declarative infra), Portainer OIDC wired via the Portainer API, and an automated SSO test that replays the OAuth2 flow across portainer/grafana/minio/argocd (+forgejo/homarr).

Verified repo facts

• App registration: individual Application CRs; user apps in k8s/argocd/apps/60-applications.yaml (root homelab-root → path: k8s/argocd/apps). layer-N files are stale.
• nginx default cert ingress-nginx/riotpiao-com-tls (*.riotpiao.com) → ingresses need no tls:.
• Default SC longhorn-wffc (WFFC, single-node → needs nodeSelector zone=az-a + CP toleration).
• Provision job pattern (k8s/security/iam/authentik-provision-job.yaml): ConfigMap python + batch/v1 Job, PostSync hook, python:3.12-alpine + stdlib urllib, bootstrap token iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN, SA authentik-provisioner with per-namespace RoleBindings (iam/cicd/argocd/logging/storage — no dashboard). SERVICES dict + idempotent get_or_create(..., patch_existing=...).
• In-cluster reach: CoreDNS rewrites *.riotpiao.com → nginx, so pods can curl real https://authentik.riotpiao.com/... (valid LE cert). Reference verify script: k8s/security/iam/verify_existing_oauth_integrations.sh.
• Blackbox exporter already probes app URLs (availability only) via serviceMonitor.targets in k8s/platform/monitoring/blackbox-exporter-values.yaml.

---

Part A — FIX THE SSO BUG (highest priority)

A1. Set grant_types on every provider — k8s/security/iam/authentik-provision-job.yaml

In the provider get_or_create (create payload and patch_existing, ~lines 281-308) add:

"grant_types": ["authorization_code", "refresh_token"],

Single fix that restores SSO for all four existing apps. patch_existing updates the already-created providers on the next hook run.

A2. Fix deprecated groups claim — same file

Update the homelab: groups claim property-mapping expression from request.user.ak_groups to request.user.groups.

A3. Re-run + verify

Trigger the PostSync hook (kubectl -n argocd patch application iam-jobs ... syncStrategy.hook), then re-run the flow-replay (Part D) — /authorize must now 302 to the authentication flow, not error=invalid_request.

---

Part B — MinIO app-side OIDC (make its login actually work)

Extend the env: block in the deployed Tenant k8s/infrastructure/minio/minio-tenant.yaml (keep _SCOPES):

env:
  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
    value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
    value: "minio"
  - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
    value: "policy"
  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
    value: "https://minio.riotpiao.com/oauth_callback"
  - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
    value: "Authentik"
  # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc
  - name: MINIO_IDENTITY_OPENID_SCOPES
    value: "openid,profile,email,minio"

Client secret from storage/minio-oidc key MINIO_IDENTITY_OPENID_CLIENT_SECRET. Retire the orphaned minio-values.yaml (or mark the tenant as source of truth).

---

Part C — New features

C1. Homarr landing page (SSO, official chart, declarative infra)

• k8s/applications/homarr/: kustomization.yaml (stub, ns dashboard) + homarr-values.yaml (official homarr-labs/homarr chart, image ghcr.io/homarr-labs/homarr, pinned version).
  • Persistence PVC longhorn-wffc (2–5Gi RWO) + nodeSelector zone=az-a + CP toleration.
  • OIDC env: AUTH_PROVIDERS=oidc,credentials, AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/, AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc, AUTH_OIDC_CLIENT_NAME=Authentik, AUTH_OIDC_GROUPS_ATTRIBUTE=groups, OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true, BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com; AUTH_OIDC_CLIENT_ID/SECRET via secretKeyRef → dashboard/homarr-oidc; SECRET_ENCRYPTION_KEY via the SOPS secret below.
• k8s/applications/homarr/homarr-secrets.enc.yaml: SOPS-encrypted SECRET_ENCRYPTION_KEY (openssl rand -hex 32), age recipient age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh (stable — a new key invalidates saved integrations). Follows k8s/applications/temporal/temporal-secrets.enc.yaml.
• 60-applications.yaml: multi-source Application (chart homarr from https://homarr-labs.github.io/charts + in-repo $values), ns dashboard, wave 8, CreateNamespace=true, automated prune/selfHeal.
• Ingress homarr.riotpiao.com → k8s/bootstrap/ingress/ingress.yaml (no tls:).
• CoreDNS rewrite for homarr.riotpiao.com → k8s/bootstrap/coredns/coredns-configmap.yaml.
• Add homarr to SERVICES (secret dashboard/homarr-oidc, generate_if_missing, extra_secret_literals {client-id: homarr}, redirect .../api/auth/callback/oidc).
• Add a dashboard RoleBinding for SA authentik-provisioner (mirror storage/logging).
• Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat).

C2. Portainer OIDC via Portainer API (user-chosen)

• Add portainer to SERVICES (authentik provider+app+secret, redirect https://portainer.riotpiao.com/).
• New k8s/applications/portainer/portainer-oauth-job.yaml (PostSync hook, python:3.12-alpine + urllib): authenticate to Portainer API (admin creds from SOPS secret), PUT /api/settings with the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret, AuthenticationMethod: 3). Handle first-run admin init. CE caveat: login works, team auto-map is BE-only → teams assigned manually. Shares the dashboard RoleBinding.

---

Part D — Automated SSO flow-replay test (the "proper verification")

New k8s/security/iam/sso-verify-job.yaml — ConfigMap python + batch/v1 Job, PostSync hook, python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from verify_existing_oauth_integrations.sh but replays real OAuth2 (not just object existence).

For each app in {grafana, minio, forgejo, argocd, portainer, homarr}:

1. Provider assert (catches THIS bug): assert authorization_code in grant_types and redirect_uris non-empty; application + homelab-admins binding exist.
2. Discovery assert: GET .../application/o/<slug>/.well-known/openid-configuration (through ingress); assert issuer is https:// and endpoints present.
3. Authorize-replay assert (key check): GET /application/o/authorize/?client_id=<slug> &redirect_uri=<registered>&response_type=code&scope=openid... (no redirect follow); assert 302 → Authentik authentication flow (/flows/), not error=invalid_request.
4. (Stretch) full code exchange: authenticate a dedicated test user via the flow executor API, complete /authorize → code, POST /application/o/token/ with client secret, assert valid id_token (iss match, groups claim present). Optional to keep the hook fast/non-flaky.

Job fails non-zero on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable). Optionally add per-app .well-known targets to blackbox-exporter-values.yaml for a continuous availability signal.

---

Files touched

• k8s/security/iam/authentik-provision-job.yaml — grant_types fix (A1), groups-claim deprecation (A2), homarr + portainer SERVICES entries, dashboard RoleBinding.
• k8s/infrastructure/minio/minio-tenant.yaml — MinIO OIDC env (B); retire minio-values.yaml.
• new k8s/security/iam/sso-verify-job.yaml — flow-replay SSO test (D).
• new k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml} (C1).
• new k8s/applications/portainer/portainer-oauth-job.yaml (C2).
• edit k8s/argocd/apps/60-applications.yaml (Homarr Application).
• edit k8s/bootstrap/ingress/ingress.yaml (homarr host).
• edit k8s/bootstrap/coredns/coredns-configmap.yaml (homarr rewrite).

Verification (end-to-end)

1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first.
2. Re-trigger iam-jobs; confirm provider grant_types now includes authorization_code.
3. Flow-replay: /authorize per app returns 302 → /flows/, not invalid_request.
4. Browser: log into Authentik as rock, click each tile → lands logged-in in grafana/argocd/forgejo/minio/homarr with no OAuth error.
5. sso-verify-job completes green; reverting grant_types in a scratch test turns it red (proves it detects the real failure).
6. Homarr reachable at https://homarr.riotpiao.com, SSO works; add tiles in UI.

Notes / caveats

• Homarr v1 tile content is DB-backed (PVC), not git — accepted.
• Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually.
• Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why authentik is now reachable for provisioning/tests.