From 06978047a227d05181690fcc2f6b4094b27d7568 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed, 22 Jul 2026 09:03:21 -0700 Subject: [PATCH] chore: remove markdown docs (violates hard rule - only CLAUDE.example.md/README.md/ARCHITECTURE.md allowed) --- FINAL-STATUS.md | 93 ----------------- HOMARR-NEXT-STEPS.md | 137 ------------------------- SSO-AND-STORAGE-HA-COMPLETE.md | 40 -------- homearr.md | 179 --------------------------------- 4 files changed, 449 deletions(-) delete mode 100644 FINAL-STATUS.md delete mode 100644 HOMARR-NEXT-STEPS.md delete mode 100644 SSO-AND-STORAGE-HA-COMPLETE.md delete mode 100644 homearr.md diff --git a/FINAL-STATUS.md b/FINAL-STATUS.md deleted file mode 100644 index 09f7928..0000000 --- a/FINAL-STATUS.md +++ /dev/null @@ -1,93 +0,0 @@ -# Final Status: SSO + Storage HA Complete - -## ✅ SSO Working (All 4 Services) -- **Root cause fixed:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers -- **All providers updated:** grafana, minio, forgejo, argocd now have `grant_types: ['authorization_code', 'refresh_token']` -- **Forgejo CLIENT_SECRET:** Added missing env injection -- **Test login:** https://forgejo.riotpiao.com → "Sign in with OpenID Connect" - - User: `rock` - - Password: `UAKLX8lv0pRbMaeBh2LQDZfU` - -## ✅ Storage HA Working (3-Node Replication) -- **Longhorn DaemonSet:** 3/3 pods (one per node) -- **All volumes:** 19/19 volumes now have 3 replicas (verified) -- **Automatic expansion:** PostSync hook Job expands existing volumes on every sync -- **Workload distribution:** Pods can now schedule on any node (no more nodeSelector pinning) - -## What Was Fixed - -### 1. Longhorn Taint Toleration -**File:** `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml` -- Allows Longhorn to run on nodes with `node-role.kubernetes.io/control-plane:NoSchedule` taint -- Without this, only talos-cp-1 had storage - -### 2. Explicit Node CRDs -**File:** `k8s/infrastructure/longhorn/longhorn-nodes.yaml` -- Created Longhorn Node CRDs for talos-cp-2 and talos-cp-3 -- Auto-discovery doesn't work when nodes have taints - -### 3. StorageClass 3-Replica Default -**File:** `k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml` -- Changed `numberOfReplicas` from 1 → 3 -- New volumes automatically get 3 replicas across 3 nodes - -### 4. Automatic Volume Expansion -**File:** `k8s/infrastructure/longhorn/expand-replicas-job.yaml` -- PostSync hook Job that expands all existing volumes to 3 replicas -- Runs idempotently on every longhorn-config Application sync -- Verified working: all 19 volumes now have 3 replicas - -### 5. Forgejo OAuth + Distribution -**File:** `k8s/argocd/bootstrap/forgejo.yaml` -- Added `GITEA__oauth2__CLIENT_SECRET` env injection -- Removed `nodeSelector: talos-cp-1` constraint - -## Verification - -```bash -# Check Longhorn nodes -kubectl -n longhorn-system get nodes.longhorn.io -# All 3 nodes should show Ready=True - -# Check DaemonSet -kubectl -n longhorn-system get ds longhorn-manager -# Should show DESIRED=3, CURRENT=3, READY=3 - -# Check volume replicas -kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas' -# All should show REPLICAS=3 - -# Test SSO -# Visit https://forgejo.riotpiao.com/user/login -# Click "Sign in with OpenID Connect" -# Login with rock / UAKLX8lv0pRbMaeBh2LQDZfU -``` - -## Commits to Push (6 total) - -``` -1685bca fix(longhorn): use jq instead of jsonpath for node/volume queries -e76ad91 feat(longhorn): auto-expand all volumes to 3 replicas via PostSync hook -30c5197 docs: SSO + Storage HA completion summary -6d1c055 fix(forgejo): remove nodeSelector now that Longhorn runs on all nodes -be7881d feat(storage): enable Longhorn on all 3 control-plane nodes for true HA -dde4b60 fix(sso): complete forgejo OAuth2 integration + force pods to storage node -``` - -**Ready to push when git credentials are refreshed.** - -## Benefits Achieved - -### Before -- ❌ SSO broken for all services (grant_types=[] → OAuth flow failed) -- ❌ Longhorn only on talos-cp-1 (cp-2/cp-3 couldn't run storage pods) -- ❌ All workloads pinned to single node (no HA, cluster wasted) -- ❌ Volumes had 1 replica (lose cp-1 = lose all data) - -### After -- ✅ SSO working for all 4 services (grant_types fixed) -- ✅ Longhorn on all 3 nodes (true distributed storage) -- ✅ Workloads can schedule anywhere (real HA cluster) -- ✅ Every volume has 3 replicas (lose any 1 node, data still available) - -**Result:** True 3-node HA cluster with distributed storage and working SSO. diff --git a/HOMARR-NEXT-STEPS.md b/HOMARR-NEXT-STEPS.md deleted file mode 100644 index 13551a4..0000000 --- a/HOMARR-NEXT-STEPS.md +++ /dev/null @@ -1,137 +0,0 @@ -# Homarr Deployment - Next Steps - -## Completed -✅ MinIO OIDC fully wired (k8s/infrastructure/minio/minio-tenant.yaml) -✅ Homarr base config created: - - homarr-values.yaml (SSO env vars) - - homarr-secrets.enc.yaml (SECRET_ENCRYPTION_KEY) - - kustomization.yaml (namespace: dashboard) - -## TODO to Complete Homarr Deployment - -### 1. Add Homarr to Authentik Provision Script -**File:** `k8s/security/iam/scripts/authentik-provision.py` -**Location:** In the `SERVICES` dict (around line 205), add: - -```python -"homarr": { - "client_secret_source": ("dashboard", "homarr-oidc", "client-secret"), - "generate_if_missing": True, - "extra_secret_literals": {"client-id": "homarr"}, - "redirect_uris": [ - "https://homarr.riotpiao.com/api/auth/callback/oidc", - ], - "launch_url": "https://homarr.riotpiao.com", - "display_name": "Homarr", -}, -``` - -### 2. Add Dashboard RoleBinding for Authentik Provisioner -**File:** `k8s/security/iam/kustomization.yaml` -**Location:** In the `resources:` list, add: - -```yaml - - rbac-dashboard-rolebinding.yaml -``` - -**New file:** `k8s/security/iam/rbac-dashboard-rolebinding.yaml`: - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: authentik-provisioner - namespace: dashboard -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: authentik-provisioner -subjects: -- kind: ServiceAccount - name: authentik-provisioner - namespace: iam -``` - -### 3. Add Homarr Application to ArgoCD -**File:** `k8s/argocd/apps/60-applications.yaml` -**Location:** Add at the end (before the final `---`): - -```yaml ---- -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: homarr - namespace: argocd - annotations: - argocd.argoproj.io/sync-wave: "8" -spec: - project: homelab - sources: - - repoURL: https://homarr-labs.github.io/charts - chart: homarr - targetRevision: "*" - helm: - valueFiles: - - $values/k8s/applications/homarr/homarr-values.yaml - - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git - targetRevision: main - ref: values - destination: - server: https://kubernetes.default.svc - namespace: dashboard - syncPolicy: - automated: - prune: true - selfHeal: true - syncOptions: - - CreateNamespace=true -``` - -### 4. Add Ingress Rule -**File:** `k8s/bootstrap/ingress/ingress.yaml` -**Location:** In the `rules:` list, add: - -```yaml - - host: homarr.riotpiao.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: homarr - port: - number: 3000 -``` - -### 5. Add CoreDNS Rewrite -**File:** `k8s/bootstrap/coredns/coredns-configmap.yaml` -**Location:** In the `rewrite name` section, add: - -``` - rewrite name homarr.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local -``` - -## Verification Steps - -After committing and pushing all changes: - -1. Wait for ArgoCD to sync iam-jobs (authentik-provision runs) -2. Check Authentik: `kubectl -n iam port-forward svc/authentik-server 9000:9000` - - Visit http://localhost:9000, login as rock - - Should see "Homarr" tile in application launcher -3. Wait for homarr Application to sync -4. Check pod: `kubectl -n dashboard get pods` -5. Test SSO: https://homarr.riotpiao.com - - Should redirect to Authentik login - - Login as rock → should land logged-in to Homarr dashboard - -## Chart Info - -Homarr uses the official chart from https://homarr-labs.github.io/charts -- Chart name: `homarr` -- Pinned image: `ghcr.io/homarr-labs/homarr:1.0.0` -- Persistence: 5Gi RWO on longhorn-wffc (3-replica HA) -- Tile content is UI-managed (saved to PVC, not git) - diff --git a/SSO-AND-STORAGE-HA-COMPLETE.md b/SSO-AND-STORAGE-HA-COMPLETE.md deleted file mode 100644 index 61826e1..0000000 --- a/SSO-AND-STORAGE-HA-COMPLETE.md +++ /dev/null @@ -1,40 +0,0 @@ -# SSO + Storage HA — Complete - -## What Was Fixed - -### 1. SSO (Authentik OAuth2 for all services) -**Root cause:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers. The provision script never set it → all providers had `grant_types=[]` → `/authorize` returned "Invalid grant_type for provider" → all SSO broken. - -**Fixed:** -- Updated `k8s/security/iam/scripts/authentik-provision.py` to set `grant_types: ['authorization_code', 'refresh_token']` on create + patch -- Fixed app-list idempotency bug (added `superuser_full_list=true` to bypass access-policy filtering) -- Removed PATCH of existing applications (detail endpoint enforces policy, 404s for akadmin) -- Added missing `GITEA__oauth2__CLIENT_SECRET` env injection to Forgejo bootstrap Application -- All 4 providers (grafana, minio, forgejo, argocd) now have correct grant_types - -**Test:** Login to any service with user `rock` / password `UAKLX8lv0pRbMaeBh2LQDZfU` - -### 2. Storage HA (Longhorn on all 3 control-plane nodes) -**Root cause:** Longhorn DaemonSet had no toleration for `node-role.kubernetes.io/control-plane:NoSchedule` taint → only ran on talos-cp-1 → all workloads with PVCs forced to single node → no HA. - -**Fixed:** -- Created `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml` (Setting that adds toleration) -- Created `k8s/infrastructure/longhorn/longhorn-nodes.yaml` (explicit Node CRDs for cp-2/cp-3, auto-discovery doesn't work with taints) -- Updated StorageClass `numberOfReplicas: 1→3` (true HA: each volume gets 3 copies across 3 nodes) -- Removed Forgejo's `nodeSelector: talos-cp-1` workaround - -**Verified:** -- Longhorn DaemonSet: 3/3 pods (one per node) -- All 3 nodes show `Ready: True` with storage available -- Forgejo pods can now schedule on any node (PVC attachments work everywhere) - -## Current State -- ✓ SSO working for all 4 services (Forgejo, Grafana, MinIO, ArgoCD) -- ✓ 3-node HA storage (Longhorn replicas across all control-plane nodes) -- ✓ All changes committed via GitOps (no manual kubectl patches retained) - -## Commits -- `be2a56c`: fix provision script (grant_types + idempotency) -- `dde4b60`: add Forgejo CLIENT_SECRET + temp nodeSelector workaround -- `be7881d`: enable Longhorn on all 3 nodes (taint toleration + Node CRDs + 3 replicas) -- `6d1c055`: remove Forgejo nodeSelector (no longer needed) diff --git a/homearr.md b/homearr.md deleted file mode 100644 index b66a741..0000000 --- a/homearr.md +++ /dev/null @@ -1,179 +0,0 @@ -# Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test - -## Context - -Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**. -Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap -token) found the true root cause — **not** missing objects: - -- Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit - `grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so - it defaults empty. `/authorize` then logs **"Invalid grant_type for provider" - (grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and - bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically. - Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct — - which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification - must replay the real flow.** - -Secondary issues found: -- **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`; - lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned** - `minio-values.yaml` the kustomization doesn't include. -- Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups` - (deprecation warning; should be `User.groups`). - -User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative -infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the -OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr). - -## Verified repo facts -- App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml` - (root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale. -- nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`. -- Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration). -- Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python + - `batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token - `iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace - RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent - `get_or_create(..., patch_existing=...)`. -- In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real - `https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script: - `k8s/security/iam/verify_existing_oauth_integrations.sh`. -- Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in - `k8s/platform/monitoring/blackbox-exporter-values.yaml`. - ---- - -## Part A — FIX THE SSO BUG (highest priority) - -### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml` -In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add: -```python -"grant_types": ["authorization_code", "refresh_token"], -``` -Single fix that restores SSO for all four existing apps. `patch_existing` updates the -already-created providers on the next hook run. - -### A2. Fix deprecated groups claim — same file -Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups` -to `request.user.groups`. - -### A3. Re-run + verify -Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`), -then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not -`error=invalid_request`. - ---- - -## Part B — MinIO app-side OIDC (make its login actually work) - -Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml` -(keep `_SCOPES`): -```yaml -env: - - name: MINIO_IDENTITY_OPENID_CONFIG_URL - value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration" - - name: MINIO_IDENTITY_OPENID_CLIENT_ID - value: "minio" - - name: MINIO_IDENTITY_OPENID_CLAIM_NAME - value: "policy" - - name: MINIO_IDENTITY_OPENID_REDIRECT_URI - value: "https://minio.riotpiao.com/oauth_callback" - - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME - value: "Authentik" - # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc - - name: MINIO_IDENTITY_OPENID_SCOPES - value: "openid,profile,email,minio" -``` -Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the -orphaned `minio-values.yaml` (or mark the tenant as source of truth). - ---- - -## Part C — New features - -### C1. Homarr landing page (SSO, official chart, declarative infra) -- `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml` - (official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version). - - Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration. - - OIDC env: `AUTH_PROVIDERS=oidc,credentials`, - `AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`, - `AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`, - `AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`, - `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`; - `AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`; - `SECRET_ENCRYPTION_KEY` via the SOPS secret below. -- `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY` - (`openssl rand -hex 32`), age recipient - `age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates - saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`. -- `60-applications.yaml`: multi-source `Application` (chart `homarr` from - `https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8, - `CreateNamespace=true`, automated prune/selfHeal. -- Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`). -- CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`. -- Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`, - `extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`). -- Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging). -- Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat). - -### C2. Portainer OIDC via Portainer API (user-chosen) -- Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`). -- New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine + - urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with - the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret, - `AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is - BE-only → teams assigned manually. Shares the `dashboard` RoleBinding. - ---- - -## Part D — Automated SSO flow-replay test (the "proper verification") - -New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**, -python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from -`verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence). - -For each app in {grafana, minio, forgejo, argocd, portainer, homarr}: -1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and - `redirect_uris` non-empty; application + `homelab-admins` binding exist. -2. **Discovery assert**: GET `.../application/o//.well-known/openid-configuration` (through - ingress); assert `issuer` is `https://` and endpoints present. -3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id= - &redirect_uri=&response_type=code&scope=openid...` (no redirect follow); assert - **302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`. -4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API, - complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid - `id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky. - -Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable). -Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous -availability signal. - ---- - -## Files touched -- `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim - deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding. -- `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`. -- new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D). -- new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1). -- new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2). -- edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application). -- edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host). -- edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite). - -## Verification (end-to-end) -1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first. -2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`. -3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`. -4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in - grafana/argocd/forgejo/minio/homarr with no OAuth error. -5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it - detects the real failure). -6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI. - -## Notes / caveats -- Homarr v1 tile content is DB-backed (PVC), not git — accepted. -- Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually. -- Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why - authentik is now reachable for provisioning/tests.