homelab/homearr.md

# Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test

## Context

Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**.
Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap
token) found the true root cause — **not** missing objects:

- Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit
  `grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so
  it defaults empty. `/authorize` then logs **"Invalid grant_type for provider"
  (grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and
  bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically.
  Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct —
  which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification
  must replay the real flow.**

Secondary issues found:
- **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`;
  lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned**
  `minio-values.yaml` the kustomization doesn't include.
- Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups`
  (deprecation warning; should be `User.groups`).

User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative
infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the
OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr).

## Verified repo facts
- App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml`
  (root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale.
- nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`.
- Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration).
- Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python +
  `batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token
  `iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace
  RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent
  `get_or_create(..., patch_existing=...)`.
- In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real
  `https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script:
  `k8s/security/iam/verify_existing_oauth_integrations.sh`.
- Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in
  `k8s/platform/monitoring/blackbox-exporter-values.yaml`.

---

## Part A — FIX THE SSO BUG (highest priority)

### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml`
In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add:
```python
"grant_types": ["authorization_code", "refresh_token"],
```
Single fix that restores SSO for all four existing apps. `patch_existing` updates the
already-created providers on the next hook run.

### A2. Fix deprecated groups claim — same file
Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups`
to `request.user.groups`.

### A3. Re-run + verify
Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`),
then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not
`error=invalid_request`.

---

## Part B — MinIO app-side OIDC (make its login actually work)

Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml`
(keep `_SCOPES`):
```yaml
env:
  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
    value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
    value: "minio"
  - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
    value: "policy"
  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
    value: "https://minio.riotpiao.com/oauth_callback"
  - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
    value: "Authentik"
  # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc
  - name: MINIO_IDENTITY_OPENID_SCOPES
    value: "openid,profile,email,minio"
```
Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the
orphaned `minio-values.yaml` (or mark the tenant as source of truth).

---

## Part C — New features

### C1. Homarr landing page (SSO, official chart, declarative infra)
- `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml`
  (official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version).
  - Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration.
  - OIDC env: `AUTH_PROVIDERS=oidc,credentials`,
    `AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`,
    `AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`,
    `AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`,
    `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`;
    `AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`;
    `SECRET_ENCRYPTION_KEY` via the SOPS secret below.
- `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY`
  (`openssl rand -hex 32`), age recipient
  `age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates
  saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`.
- `60-applications.yaml`: multi-source `Application` (chart `homarr` from
  `https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8,
  `CreateNamespace=true`, automated prune/selfHeal.
- Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`).
- CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`.
- Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`,
  `extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`).
- Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging).
- Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat).

### C2. Portainer OIDC via Portainer API (user-chosen)
- Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`).
- New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine +
  urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with
  the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret,
  `AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is
  BE-only → teams assigned manually. Shares the `dashboard` RoleBinding.

---

## Part D — Automated SSO flow-replay test (the "proper verification")

New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**,
python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from
`verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence).

For each app in {grafana, minio, forgejo, argocd, portainer, homarr}:
1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and
   `redirect_uris` non-empty; application + `homelab-admins` binding exist.
2. **Discovery assert**: GET `.../application/o/<slug>/.well-known/openid-configuration` (through
   ingress); assert `issuer` is `https://` and endpoints present.
3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id=<slug>
   &redirect_uri=<registered>&response_type=code&scope=openid...` (no redirect follow); assert
   **302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`.
4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API,
   complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid
   `id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky.

Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable).
Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous
availability signal.

---

## Files touched
- `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim
  deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding.
- `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`.
- new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D).
- new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1).
- new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2).
- edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application).
- edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host).
- edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite).

## Verification (end-to-end)
1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first.
2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`.
3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`.
4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in
   grafana/argocd/forgejo/minio/homarr with no OAuth error.
5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it
   detects the real failure).
6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI.

## Notes / caveats
- Homarr v1 tile content is DB-backed (PVC), not git — accepted.
- Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually.
- Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why
  authentik is now reachable for provisioning/tests.