rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: remove markdown docs (violates hard rule - only CLAUDE.example.md/README.md/ARCHITECTURE.md allowed)

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-18 15:08:03 -07:00
parent 7155ca38d1
commit 06978047a2
4 changed files with 0 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files
FINAL-STATUS.md
-93
View File
@@ -1,93 +0,0 @@
# Final Status: SSO + Storage HA Complete
## ✅ SSO Working (All 4 Services)
- **Root cause fixed:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers
- **All providers updated:** grafana, minio, forgejo, argocd now have `grant_types: ['authorization_code', 'refresh_token']`
- **Forgejo CLIENT_SECRET:** Added missing env injection
- **Test login:** https://forgejo.riotpiao.com → "Sign in with OpenID Connect"
- User: `rock`
- Password: `UAKLX8lv0pRbMaeBh2LQDZfU`
## ✅ Storage HA Working (3-Node Replication)
- **Longhorn DaemonSet:** 3/3 pods (one per node)
- **All volumes:** 19/19 volumes now have 3 replicas (verified)
- **Automatic expansion:** PostSync hook Job expands existing volumes on every sync
- **Workload distribution:** Pods can now schedule on any node (no more nodeSelector pinning)
## What Was Fixed
### 1. Longhorn Taint Toleration
**File:** `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml`
- Allows Longhorn to run on nodes with `node-role.kubernetes.io/control-plane:NoSchedule` taint
- Without this, only talos-cp-1 had storage
### 2. Explicit Node CRDs
**File:** `k8s/infrastructure/longhorn/longhorn-nodes.yaml`
- Created Longhorn Node CRDs for talos-cp-2 and talos-cp-3
- Auto-discovery doesn't work when nodes have taints
### 3. StorageClass 3-Replica Default
**File:** `k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml`
- Changed `numberOfReplicas` from 1 → 3
- New volumes automatically get 3 replicas across 3 nodes
### 4. Automatic Volume Expansion
**File:** `k8s/infrastructure/longhorn/expand-replicas-job.yaml`
- PostSync hook Job that expands all existing volumes to 3 replicas
- Runs idempotently on every longhorn-config Application sync
- Verified working: all 19 volumes now have 3 replicas
### 5. Forgejo OAuth + Distribution
**File:** `k8s/argocd/bootstrap/forgejo.yaml`
- Added `GITEA__oauth2__CLIENT_SECRET` env injection
- Removed `nodeSelector: talos-cp-1` constraint
## Verification
```bash
# Check Longhorn nodes
kubectl -n longhorn-system get nodes.longhorn.io
# All 3 nodes should show Ready=True
# Check DaemonSet
kubectl -n longhorn-system get ds longhorn-manager
# Should show DESIRED=3, CURRENT=3, READY=3
# Check volume replicas
kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas'
# All should show REPLICAS=3
# Test SSO
# Visit https://forgejo.riotpiao.com/user/login
# Click "Sign in with OpenID Connect"
# Login with rock / UAKLX8lv0pRbMaeBh2LQDZfU
```
## Commits to Push (6 total)
```
1685bca fix(longhorn): use jq instead of jsonpath for node/volume queries
e76ad91 feat(longhorn): auto-expand all volumes to 3 replicas via PostSync hook
30c5197 docs: SSO + Storage HA completion summary
6d1c055 fix(forgejo): remove nodeSelector now that Longhorn runs on all nodes
be7881d feat(storage): enable Longhorn on all 3 control-plane nodes for true HA
dde4b60 fix(sso): complete forgejo OAuth2 integration + force pods to storage node
```
**Ready to push when git credentials are refreshed.**
## Benefits Achieved
### Before
- ❌ SSO broken for all services (grant_types=[] → OAuth flow failed)
- ❌ Longhorn only on talos-cp-1 (cp-2/cp-3 couldn't run storage pods)
- ❌ All workloads pinned to single node (no HA, cluster wasted)
- ❌ Volumes had 1 replica (lose cp-1 = lose all data)
### After
- ✅ SSO working for all 4 services (grant_types fixed)
- ✅ Longhorn on all 3 nodes (true distributed storage)
- ✅ Workloads can schedule anywhere (real HA cluster)
- ✅ Every volume has 3 replicas (lose any 1 node, data still available)
**Result:** True 3-node HA cluster with distributed storage and working SSO.
HOMARR-NEXT-STEPS.md
-137
View File
@@ -1,137 +0,0 @@
# Homarr Deployment - Next Steps
## Completed
✅ MinIO OIDC fully wired (k8s/infrastructure/minio/minio-tenant.yaml)
✅ Homarr base config created:
- homarr-values.yaml (SSO env vars)
- homarr-secrets.enc.yaml (SECRET_ENCRYPTION_KEY)
- kustomization.yaml (namespace: dashboard)
## TODO to Complete Homarr Deployment
### 1. Add Homarr to Authentik Provision Script
**File:** `k8s/security/iam/scripts/authentik-provision.py`
**Location:** In the `SERVICES` dict (around line 205), add:
```python
"homarr": {
"client_secret_source": ("dashboard", "homarr-oidc", "client-secret"),
"generate_if_missing": True,
"extra_secret_literals": {"client-id": "homarr"},
"redirect_uris": [
"https://homarr.riotpiao.com/api/auth/callback/oidc",
],
"launch_url": "https://homarr.riotpiao.com",
"display_name": "Homarr",
},
```
### 2. Add Dashboard RoleBinding for Authentik Provisioner
**File:** `k8s/security/iam/kustomization.yaml`
**Location:** In the `resources:` list, add:
```yaml
- rbac-dashboard-rolebinding.yaml
```
**New file:** `k8s/security/iam/rbac-dashboard-rolebinding.yaml`:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
```
### 3. Add Homarr Application to ArgoCD
**File:** `k8s/argocd/apps/60-applications.yaml`
**Location:** Add at the end (before the final `---`):
```yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: homarr
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "8"
spec:
project: homelab
sources:
- repoURL: https://homarr-labs.github.io/charts
chart: homarr
targetRevision: "*"
helm:
valueFiles:
- $values/k8s/applications/homarr/homarr-values.yaml
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: dashboard
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
### 4. Add Ingress Rule
**File:** `k8s/bootstrap/ingress/ingress.yaml`
**Location:** In the `rules:` list, add:
```yaml
- host: homarr.riotpiao.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: homarr
port:
number: 3000
```
### 5. Add CoreDNS Rewrite
**File:** `k8s/bootstrap/coredns/coredns-configmap.yaml`
**Location:** In the `rewrite name` section, add:
```
rewrite name homarr.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
```
## Verification Steps
After committing and pushing all changes:
1. Wait for ArgoCD to sync iam-jobs (authentik-provision runs)
2. Check Authentik: `kubectl -n iam port-forward svc/authentik-server 9000:9000`
- Visit http://localhost:9000, login as rock
- Should see "Homarr" tile in application launcher
3. Wait for homarr Application to sync
4. Check pod: `kubectl -n dashboard get pods`
5. Test SSO: https://homarr.riotpiao.com
- Should redirect to Authentik login
- Login as rock → should land logged-in to Homarr dashboard
## Chart Info
Homarr uses the official chart from https://homarr-labs.github.io/charts
- Chart name: `homarr`
- Pinned image: `ghcr.io/homarr-labs/homarr:1.0.0`
- Persistence: 5Gi RWO on longhorn-wffc (3-replica HA)
- Tile content is UI-managed (saved to PVC, not git)
SSO-AND-STORAGE-HA-COMPLETE.md
-40
View File
@@ -1,40 +0,0 @@
# SSO + Storage HA — Complete
## What Was Fixed
### 1. SSO (Authentik OAuth2 for all services)
**Root cause:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers. The provision script never set it → all providers had `grant_types=[]` → `/authorize` returned "Invalid grant_type for provider" → all SSO broken.
**Fixed:**
- Updated `k8s/security/iam/scripts/authentik-provision.py` to set `grant_types: ['authorization_code', 'refresh_token']` on create + patch
- Fixed app-list idempotency bug (added `superuser_full_list=true` to bypass access-policy filtering)
- Removed PATCH of existing applications (detail endpoint enforces policy, 404s for akadmin)
- Added missing `GITEA__oauth2__CLIENT_SECRET` env injection to Forgejo bootstrap Application
- All 4 providers (grafana, minio, forgejo, argocd) now have correct grant_types
**Test:** Login to any service with user `rock` / password `UAKLX8lv0pRbMaeBh2LQDZfU`
### 2. Storage HA (Longhorn on all 3 control-plane nodes)
**Root cause:** Longhorn DaemonSet had no toleration for `node-role.kubernetes.io/control-plane:NoSchedule` taint → only ran on talos-cp-1 → all workloads with PVCs forced to single node → no HA.
**Fixed:**
- Created `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml` (Setting that adds toleration)
- Created `k8s/infrastructure/longhorn/longhorn-nodes.yaml` (explicit Node CRDs for cp-2/cp-3, auto-discovery doesn't work with taints)
- Updated StorageClass `numberOfReplicas: 1→3` (true HA: each volume gets 3 copies across 3 nodes)
- Removed Forgejo's `nodeSelector: talos-cp-1` workaround
**Verified:**
- Longhorn DaemonSet: 3/3 pods (one per node)
- All 3 nodes show `Ready: True` with storage available
- Forgejo pods can now schedule on any node (PVC attachments work everywhere)
## Current State
- ✓ SSO working for all 4 services (Forgejo, Grafana, MinIO, ArgoCD)
- ✓ 3-node HA storage (Longhorn replicas across all control-plane nodes)
- ✓ All changes committed via GitOps (no manual kubectl patches retained)
## Commits
- `be2a56c`: fix provision script (grant_types + idempotency)
- `dde4b60`: add Forgejo CLIENT_SECRET + temp nodeSelector workaround
- `be7881d`: enable Longhorn on all 3 nodes (taint toleration + Node CRDs + 3 replicas)
- `6d1c055`: remove Forgejo nodeSelector (no longer needed)
homearr.md
-179
View File
@@ -1,179 +0,0 @@
# Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test
## Context
Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**.
Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap
token) found the true root cause — **not** missing objects:
- Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit
`grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so
it defaults empty. `/authorize` then logs **"Invalid grant_type for provider"
(grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and
bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically.
Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct —
which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification
must replay the real flow.**
Secondary issues found:
- **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`;
lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned**
`minio-values.yaml` the kustomization doesn't include.
- Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups`
(deprecation warning; should be `User.groups`).
User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative
infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the
OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr).
## Verified repo facts
- App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml`
(root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale.
- nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`.
- Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration).
- Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python +
`batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token
`iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace
RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent
`get_or_create(..., patch_existing=...)`.
- In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real
`https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script:
`k8s/security/iam/verify_existing_oauth_integrations.sh`.
- Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in
`k8s/platform/monitoring/blackbox-exporter-values.yaml`.
---
## Part A — FIX THE SSO BUG (highest priority)
### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml`
In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add:
```python
"grant_types": ["authorization_code", "refresh_token"],
```
Single fix that restores SSO for all four existing apps. `patch_existing` updates the
already-created providers on the next hook run.
### A2. Fix deprecated groups claim — same file
Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups`
to `request.user.groups`.
### A3. Re-run + verify
Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`),
then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not
`error=invalid_request`.
---
## Part B — MinIO app-side OIDC (make its login actually work)
Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml`
(keep `_SCOPES`):
```yaml
env:
- name: MINIO_IDENTITY_OPENID_CONFIG_URL
value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
- name: MINIO_IDENTITY_OPENID_CLIENT_ID
value: "minio"
- name: MINIO_IDENTITY_OPENID_CLAIM_NAME
value: "policy"
- name: MINIO_IDENTITY_OPENID_REDIRECT_URI
value: "https://minio.riotpiao.com/oauth_callback"
- name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
value: "Authentik"
# + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc
- name: MINIO_IDENTITY_OPENID_SCOPES
value: "openid,profile,email,minio"
```
Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the
orphaned `minio-values.yaml` (or mark the tenant as source of truth).
---
## Part C — New features
### C1. Homarr landing page (SSO, official chart, declarative infra)
- `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml`
(official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version).
- Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration.
- OIDC env: `AUTH_PROVIDERS=oidc,credentials`,
`AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`,
`AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`,
`AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`,
`OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`;
`AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`;
`SECRET_ENCRYPTION_KEY` via the SOPS secret below.
- `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY`
(`openssl rand -hex 32`), age recipient
`age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates
saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`.
- `60-applications.yaml`: multi-source `Application` (chart `homarr` from
`https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8,
`CreateNamespace=true`, automated prune/selfHeal.
- Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`).
- CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`.
- Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`,
`extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`).
- Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging).
- Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat).
### C2. Portainer OIDC via Portainer API (user-chosen)
- Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`).
- New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine +
urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with
the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret,
`AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is
BE-only → teams assigned manually. Shares the `dashboard` RoleBinding.
---
## Part D — Automated SSO flow-replay test (the "proper verification")
New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**,
python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from
`verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence).
For each app in {grafana, minio, forgejo, argocd, portainer, homarr}:
1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and
`redirect_uris` non-empty; application + `homelab-admins` binding exist.
2. **Discovery assert**: GET `.../application/o/<slug>/.well-known/openid-configuration` (through
ingress); assert `issuer` is `https://` and endpoints present.
3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id=<slug>
&redirect_uri=<registered>&response_type=code&scope=openid...` (no redirect follow); assert
**302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`.
4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API,
complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid
`id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky.
Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable).
Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous
availability signal.
---
## Files touched
- `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim
deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding.
- `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`.
- new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D).
- new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1).
- new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2).
- edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application).
- edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host).
- edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite).
## Verification (end-to-end)
1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first.
2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`.
3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`.
4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in
grafana/argocd/forgejo/minio/homarr with no OAuth error.
5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it
detects the real failure).
6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI.
## Notes / caveats
- Homarr v1 tile content is DB-backed (PVC), not git — accepted.
- Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually.
- Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why
authentik is now reachable for provisioning/tests.