chore: remove markdown docs (violates hard rule - only CLAUDE.example.md/README.md/ARCHITECTURE.md allowed)

2026-07-22 09:03:21 -07:00
parent 4faf8115c3
commit 2b94114310
4 changed files with 0 additions and 449 deletions
@@ -1,93 +0,0 @@
 # Final Status: SSO + Storage HA Complete
 ## ✅ SSO Working (All 4 Services)
 - **Root cause fixed:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers
 - **All providers updated:** grafana, minio, forgejo, argocd now have `grant_types: ['authorization_code', 'refresh_token']`
 - **Forgejo CLIENT_SECRET:** Added missing env injection
 - **Test login:** https://forgejo.riotpiao.com → "Sign in with OpenID Connect"
  - User: `rock`
  - Password: `UAKLX8lv0pRbMaeBh2LQDZfU`
 ## ✅ Storage HA Working (3-Node Replication)
 - **Longhorn DaemonSet:** 3/3 pods (one per node)
 - **All volumes:** 19/19 volumes now have 3 replicas (verified)
 - **Automatic expansion:** PostSync hook Job expands existing volumes on every sync
 - **Workload distribution:** Pods can now schedule on any node (no more nodeSelector pinning)
 ## What Was Fixed
 ### 1. Longhorn Taint Toleration
 **File:** `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml`
 - Allows Longhorn to run on nodes with `node-role.kubernetes.io/control-plane:NoSchedule` taint
 - Without this, only talos-cp-1 had storage
 ### 2. Explicit Node CRDs
 **File:** `k8s/infrastructure/longhorn/longhorn-nodes.yaml`
 - Created Longhorn Node CRDs for talos-cp-2 and talos-cp-3
 - Auto-discovery doesn't work when nodes have taints
 ### 3. StorageClass 3-Replica Default
 **File:** `k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml`
 - Changed `numberOfReplicas` from 1 → 3
 - New volumes automatically get 3 replicas across 3 nodes
 ### 4. Automatic Volume Expansion
 **File:** `k8s/infrastructure/longhorn/expand-replicas-job.yaml`
 - PostSync hook Job that expands all existing volumes to 3 replicas
 - Runs idempotently on every longhorn-config Application sync
 - Verified working: all 19 volumes now have 3 replicas
 ### 5. Forgejo OAuth + Distribution
 **File:** `k8s/argocd/bootstrap/forgejo.yaml`
 - Added `GITEA__oauth2__CLIENT_SECRET` env injection
 - Removed `nodeSelector: talos-cp-1` constraint
 ## Verification
 ```bash
 # Check Longhorn nodes
 kubectl -n longhorn-system get nodes.longhorn.io
 # All 3 nodes should show Ready=True
 # Check DaemonSet
 kubectl -n longhorn-system get ds longhorn-manager
 # Should show DESIRED=3, CURRENT=3, READY=3
 # Check volume replicas
 kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.numberOfReplicas'
 # All should show REPLICAS=3
 # Test SSO
 # Visit https://forgejo.riotpiao.com/user/login
 # Click "Sign in with OpenID Connect"
 # Login with rock / UAKLX8lv0pRbMaeBh2LQDZfU
 ```
 ## Commits to Push (6 total)
 ```
 1685bca fix(longhorn): use jq instead of jsonpath for node/volume queries
 e76ad91 feat(longhorn): auto-expand all volumes to 3 replicas via PostSync hook
 30c5197 docs: SSO + Storage HA completion summary
 6d1c055 fix(forgejo): remove nodeSelector now that Longhorn runs on all nodes
 be7881d feat(storage): enable Longhorn on all 3 control-plane nodes for true HA
 dde4b60 fix(sso): complete forgejo OAuth2 integration + force pods to storage node
 ```
 **Ready to push when git credentials are refreshed.**
 ## Benefits Achieved
 ### Before
 - ❌ SSO broken for all services (grant_types=[] → OAuth flow failed)
 - ❌ Longhorn only on talos-cp-1 (cp-2/cp-3 couldn't run storage pods)
 - ❌ All workloads pinned to single node (no HA, cluster wasted)
 - ❌ Volumes had 1 replica (lose cp-1 = lose all data)
 ### After
 - ✅ SSO working for all 4 services (grant_types fixed)
 - ✅ Longhorn on all 3 nodes (true distributed storage)
 - ✅ Workloads can schedule anywhere (real HA cluster)
 - ✅ Every volume has 3 replicas (lose any 1 node, data still available)
 **Result:** True 3-node HA cluster with distributed storage and working SSO.
@@ -1,137 +0,0 @@
 # Homarr Deployment - Next Steps
 ## Completed
 ✅ MinIO OIDC fully wired (k8s/infrastructure/minio/minio-tenant.yaml)
 ✅ Homarr base config created:
   - homarr-values.yaml (SSO env vars)
   - homarr-secrets.enc.yaml (SECRET_ENCRYPTION_KEY)
   - kustomization.yaml (namespace: dashboard)
 ## TODO to Complete Homarr Deployment
 ### 1. Add Homarr to Authentik Provision Script
 **File:** `k8s/security/iam/scripts/authentik-provision.py`
 **Location:** In the `SERVICES` dict (around line 205), add:
 ```python
 "homarr": {
    "client_secret_source": ("dashboard", "homarr-oidc", "client-secret"),
    "generate_if_missing": True,
    "extra_secret_literals": {"client-id": "homarr"},
    "redirect_uris": [
        "https://homarr.riotpiao.com/api/auth/callback/oidc",
    ],
    "launch_url": "https://homarr.riotpiao.com",
    "display_name": "Homarr",
 },
 ```
 ### 2. Add Dashboard RoleBinding for Authentik Provisioner
 **File:** `k8s/security/iam/kustomization.yaml`
 **Location:** In the `resources:` list, add:
 ```yaml
  - rbac-dashboard-rolebinding.yaml
 ```
 **New file:** `k8s/security/iam/rbac-dashboard-rolebinding.yaml`:
 ```yaml
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: authentik-provisioner
  namespace: dashboard
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: authentik-provisioner
 subjects:
 - kind: ServiceAccount
  name: authentik-provisioner
  namespace: iam
 ```
 ### 3. Add Homarr Application to ArgoCD
 **File:** `k8s/argocd/apps/60-applications.yaml`
 **Location:** Add at the end (before the final `---`):
 ```yaml
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: homarr
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "8"
 spec:
  project: homelab
  sources:
    - repoURL: https://homarr-labs.github.io/charts
      chart: homarr
      targetRevision: "*"
      helm:
        valueFiles:
          - $values/k8s/applications/homarr/homarr-values.yaml
    - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: dashboard
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
 ```
 ### 4. Add Ingress Rule
 **File:** `k8s/bootstrap/ingress/ingress.yaml`
 **Location:** In the `rules:` list, add:
 ```yaml
  - host: homarr.riotpiao.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: homarr
            port:
              number: 3000
 ```
 ### 5. Add CoreDNS Rewrite
 **File:** `k8s/bootstrap/coredns/coredns-configmap.yaml`
 **Location:** In the `rewrite name` section, add:
 ```
        rewrite name homarr.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ```
 ## Verification Steps
 After committing and pushing all changes:
 1. Wait for ArgoCD to sync iam-jobs (authentik-provision runs)
 2. Check Authentik: `kubectl -n iam port-forward svc/authentik-server 9000:9000`
   - Visit http://localhost:9000, login as rock
   - Should see "Homarr" tile in application launcher
 3. Wait for homarr Application to sync
 4. Check pod: `kubectl -n dashboard get pods`
 5. Test SSO: https://homarr.riotpiao.com
   - Should redirect to Authentik login
   - Login as rock → should land logged-in to Homarr dashboard
 ## Chart Info
 Homarr uses the official chart from https://homarr-labs.github.io/charts
 - Chart name: `homarr`
 - Pinned image: `ghcr.io/homarr-labs/homarr:1.0.0`
 - Persistence: 5Gi RWO on longhorn-wffc (3-replica HA)
 - Tile content is UI-managed (saved to PVC, not git)
@@ -1,40 +0,0 @@
 # SSO + Storage HA — Complete
 ## What Was Fixed
 ### 1. SSO (Authentik OAuth2 for all services)
 **Root cause:** Authentik 2026.5.5 requires explicit `grant_types` on OAuth2 providers. The provision script never set it → all providers had `grant_types=[]` → `/authorize` returned "Invalid grant_type for provider" → all SSO broken.
 **Fixed:**
 - Updated `k8s/security/iam/scripts/authentik-provision.py` to set `grant_types: ['authorization_code', 'refresh_token']` on create + patch
 - Fixed app-list idempotency bug (added `superuser_full_list=true` to bypass access-policy filtering)
 - Removed PATCH of existing applications (detail endpoint enforces policy, 404s for akadmin)
 - Added missing `GITEA__oauth2__CLIENT_SECRET` env injection to Forgejo bootstrap Application
 - All 4 providers (grafana, minio, forgejo, argocd) now have correct grant_types
 **Test:** Login to any service with user `rock` / password `UAKLX8lv0pRbMaeBh2LQDZfU`
 ### 2. Storage HA (Longhorn on all 3 control-plane nodes)
 **Root cause:** Longhorn DaemonSet had no toleration for `node-role.kubernetes.io/control-plane:NoSchedule` taint → only ran on talos-cp-1 → all workloads with PVCs forced to single node → no HA.
 **Fixed:**
 - Created `k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml` (Setting that adds toleration)
 - Created `k8s/infrastructure/longhorn/longhorn-nodes.yaml` (explicit Node CRDs for cp-2/cp-3, auto-discovery doesn't work with taints)
 - Updated StorageClass `numberOfReplicas: 1→3` (true HA: each volume gets 3 copies across 3 nodes)
 - Removed Forgejo's `nodeSelector: talos-cp-1` workaround
 **Verified:**
 - Longhorn DaemonSet: 3/3 pods (one per node)
 - All 3 nodes show `Ready: True` with storage available
 - Forgejo pods can now schedule on any node (PVC attachments work everywhere)
 ## Current State
 - ✓ SSO working for all 4 services (Forgejo, Grafana, MinIO, ArgoCD)
 - ✓ 3-node HA storage (Longhorn replicas across all control-plane nodes)
 - ✓ All changes committed via GitOps (no manual kubectl patches retained)
 ## Commits
 - `be2a56c`: fix provision script (grant_types + idempotency)
 - `dde4b60`: add Forgejo CLIENT_SECRET + temp nodeSelector workaround
 - `be7881d`: enable Longhorn on all 3 nodes (taint toleration + Node CRDs + 3 replicas)
 - `6d1c055`: remove Forgejo nodeSelector (no longer needed)
@@ -1,179 +0,0 @@
 # Plan: Fix homelab SSO end-to-end, add Homarr landing page, add OAuth flow-replay test
 ## Context
 Reported symptom: clicking any app in the Authentik launcher, **no application lets you sign in**.
 Read-only diagnosis (replaying the OAuth2 authorize flow against Authentik with the bootstrap
 token) found the true root cause — **not** missing objects:
 - Every OAuth2 provider has **`grant_types = []`**. Authentik 2026.5.5 added an explicit
  `grant_types` list field; the provision script (`authentik-provision-job.yaml`) never sets it, so
  it defaults empty. `/authorize` then logs **"Invalid grant_type for provider"
  (grant_type=authorization_code) → invalid_request "The request is otherwise malformed"** and
  bounces an error back to the app. Breaks **all** apps (grafana/minio/forgejo/argocd) identically.
  Providers, apps, client secrets, flows, redirect_uris, signing key all exist and are correct —
  which is why "check the objects exist" checks passed while SSO was 100% broken. **Verification
  must replay the real flow.**
 Secondary issues found:
 - **MinIO app side unwired**: deployed `minio-tenant.yaml` sets only `MINIO_IDENTITY_OPENID_SCOPES`;
  lacks `CONFIG_URL`/`CLIENT_ID`/`envFrom minio-oidc`. Full config sits in an **orphaned**
  `minio-values.yaml` the kustomization doesn't include.
 - Custom **`homelab: groups claim`** property mapping uses deprecated `User.ak_groups`
  (deprecation warning; should be `User.groups`).
 User-requested additions: **Homarr** landing page (Authentik SSO, official Helm chart, declarative
 infra), **Portainer** OIDC wired via the Portainer API, and an **automated SSO test that replays the
 OAuth2 flow** across portainer/grafana/minio/argocd (+forgejo/homarr).
 ## Verified repo facts
 - App registration: individual `Application` CRs; user apps in `k8s/argocd/apps/60-applications.yaml`
  (root `homelab-root` → `path: k8s/argocd/apps`). `layer-N` files are stale.
 - nginx default cert `ingress-nginx/riotpiao-com-tls` (`*.riotpiao.com`) → ingresses need no `tls:`.
 - Default SC `longhorn-wffc` (WFFC, single-node → needs `nodeSelector zone=az-a` + CP toleration).
 - Provision job pattern (`k8s/security/iam/authentik-provision-job.yaml`): ConfigMap python +
  `batch/v1` Job, PostSync hook, `python:3.12-alpine` + stdlib urllib, bootstrap token
  `iam/authentik-secrets:AUTHENTIK_BOOTSTRAP_TOKEN`, SA `authentik-provisioner` with per-namespace
  RoleBindings (iam/cicd/argocd/logging/storage — **no dashboard**). `SERVICES` dict + idempotent
  `get_or_create(..., patch_existing=...)`.
 - In-cluster reach: CoreDNS rewrites `*.riotpiao.com` → nginx, so pods can curl real
  `https://authentik.riotpiao.com/...` (valid LE cert). Reference verify script:
  `k8s/security/iam/verify_existing_oauth_integrations.sh`.
 - Blackbox exporter already probes app URLs (availability only) via `serviceMonitor.targets` in
  `k8s/platform/monitoring/blackbox-exporter-values.yaml`.
 ---
 ## Part A — FIX THE SSO BUG (highest priority)
 ### A1. Set `grant_types` on every provider — `k8s/security/iam/authentik-provision-job.yaml`
 In the provider `get_or_create` (create payload **and** `patch_existing`, ~lines 281-308) add:
 ```python
 "grant_types": ["authorization_code", "refresh_token"],
 ```
 Single fix that restores SSO for all four existing apps. `patch_existing` updates the
 already-created providers on the next hook run.
 ### A2. Fix deprecated groups claim — same file
 Update the `homelab: groups claim` property-mapping expression from `request.user.ak_groups`
 to `request.user.groups`.
 ### A3. Re-run + verify
 Trigger the PostSync hook (`kubectl -n argocd patch application iam-jobs ... syncStrategy.hook`),
 then re-run the flow-replay (Part D) — `/authorize` must now 302 to the authentication flow, not
 `error=invalid_request`.
 ---
 ## Part B — MinIO app-side OIDC (make its login actually work)
 Extend the `env:` block in the **deployed** Tenant `k8s/infrastructure/minio/minio-tenant.yaml`
 (keep `_SCOPES`):
 ```yaml
 env:
  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
    value: "https://authentik.riotpiao.com/application/o/minio/.well-known/openid-configuration"
  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
    value: "minio"
  - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
    value: "policy"
  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
    value: "https://minio.riotpiao.com/oauth_callback"
  - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
    value: "Authentik"
  # + MINIO_IDENTITY_OPENID_CLIENT_SECRET from secret storage/minio-oidc
  - name: MINIO_IDENTITY_OPENID_SCOPES
    value: "openid,profile,email,minio"
 ```
 Client secret from `storage/minio-oidc` key `MINIO_IDENTITY_OPENID_CLIENT_SECRET`. Retire the
 orphaned `minio-values.yaml` (or mark the tenant as source of truth).
 ---
 ## Part C — New features
 ### C1. Homarr landing page (SSO, official chart, declarative infra)
 - `k8s/applications/homarr/`: `kustomization.yaml` (stub, ns `dashboard`) + `homarr-values.yaml`
  (official `homarr-labs/homarr` chart, image `ghcr.io/homarr-labs/homarr`, pinned version).
  - Persistence PVC `longhorn-wffc` (2–5Gi RWO) + `nodeSelector zone=az-a` + CP toleration.
  - OIDC env: `AUTH_PROVIDERS=oidc,credentials`,
    `AUTH_OIDC_ISSUER=https://authentik.riotpiao.com/application/o/homarr/`,
    `AUTH_OIDC_CALLBACK_URL=https://homarr.riotpiao.com/api/auth/callback/oidc`,
    `AUTH_OIDC_CLIENT_NAME=Authentik`, `AUTH_OIDC_GROUPS_ATTRIBUTE=groups`,
    `OAUTH_ALLOW_DANGEROUS_EMAIL_ACCOUNT_LINKING=true`, `BASE_URL/NEXTAUTH_URL=https://homarr.riotpiao.com`;
    `AUTH_OIDC_CLIENT_ID/SECRET` via secretKeyRef → `dashboard/homarr-oidc`;
    `SECRET_ENCRYPTION_KEY` via the SOPS secret below.
 - `k8s/applications/homarr/homarr-secrets.enc.yaml`: SOPS-encrypted `SECRET_ENCRYPTION_KEY`
  (`openssl rand -hex 32`), age recipient
  `age1smu533f803gmd0jq60s2zaj9zlznajy0ca6rtewd4r37mr2hs3uqsrldfh` (stable — a new key invalidates
  saved integrations). Follows `k8s/applications/temporal/temporal-secrets.enc.yaml`.
 - `60-applications.yaml`: multi-source `Application` (chart `homarr` from
  `https://homarr-labs.github.io/charts` + in-repo `$values`), ns `dashboard`, wave 8,
  `CreateNamespace=true`, automated prune/selfHeal.
 - Ingress `homarr.riotpiao.com` → `k8s/bootstrap/ingress/ingress.yaml` (no `tls:`).
 - CoreDNS rewrite for `homarr.riotpiao.com` → `k8s/bootstrap/coredns/coredns-configmap.yaml`.
 - Add `homarr` to `SERVICES` (secret `dashboard/homarr-oidc`, `generate_if_missing`,
  `extra_secret_literals {client-id: homarr}`, redirect `.../api/auth/callback/oidc`).
 - Add a **dashboard** RoleBinding for SA `authentik-provisioner` (mirror storage/logging).
 - Tile content is UI-managed on the PVC (Homarr v1 has no config-as-code — accepted caveat).
 ### C2. Portainer OIDC via Portainer API (user-chosen)
 - Add `portainer` to `SERVICES` (authentik provider+app+secret, redirect `https://portainer.riotpiao.com/`).
 - New `k8s/applications/portainer/portainer-oauth-job.yaml` (PostSync hook, python:3.12-alpine +
  urllib): authenticate to Portainer API (admin creds from SOPS secret), `PUT /api/settings` with
  the OAuth block (AuthorizationURL/AccessTokenURI/ResourceURI/RedirectURI/ClientID/ClientSecret,
  `AuthenticationMethod: 3`). Handle first-run admin init. CE caveat: login works, team auto-map is
  BE-only → teams assigned manually. Shares the `dashboard` RoleBinding.
 ---
 ## Part D — Automated SSO flow-replay test (the "proper verification")
 New `k8s/security/iam/sso-verify-job.yaml` — ConfigMap python + `batch/v1` Job, **PostSync hook**,
 python:3.12-alpine + stdlib urllib, bootstrap-token access. Logic ported from
 `verify_existing_oauth_integrations.sh` but **replays real OAuth2** (not just object existence).
 For each app in {grafana, minio, forgejo, argocd, portainer, homarr}:
 1. **Provider assert** (catches THIS bug): assert `authorization_code in grant_types` and
   `redirect_uris` non-empty; application + `homelab-admins` binding exist.
 2. **Discovery assert**: GET `.../application/o/<slug>/.well-known/openid-configuration` (through
   ingress); assert `issuer` is `https://` and endpoints present.
 3. **Authorize-replay assert** (key check): GET `/application/o/authorize/?client_id=<slug>
   &redirect_uri=<registered>&response_type=code&scope=openid...` (no redirect follow); assert
   **302 → Authentik authentication flow** (`/flows/`), **not** `error=invalid_request`.
 4. **(Stretch) full code exchange**: authenticate a dedicated test user via the flow executor API,
   complete `/authorize` → `code`, POST `/application/o/token/` with client secret, assert valid
   `id_token` (`iss` match, `groups` claim present). Optional to keep the hook fast/non-flaky.
 Job **fails non-zero** on any assertion failure → ArgoCD marks the hook Degraded (visible + alertable).
 Optionally add per-app `.well-known` targets to `blackbox-exporter-values.yaml` for a continuous
 availability signal.
 ---
 ## Files touched
 - `k8s/security/iam/authentik-provision-job.yaml` — **grant_types fix (A1)**, groups-claim
  deprecation (A2), homarr + portainer SERVICES entries, `dashboard` RoleBinding.
 - `k8s/infrastructure/minio/minio-tenant.yaml` — MinIO OIDC env (B); retire `minio-values.yaml`.
 - new `k8s/security/iam/sso-verify-job.yaml` — flow-replay SSO test (D).
 - new `k8s/applications/homarr/{kustomization.yaml,homarr-values.yaml,homarr-secrets.enc.yaml}` (C1).
 - new `k8s/applications/portainer/portainer-oauth-job.yaml` (C2).
 - edit `k8s/argocd/apps/60-applications.yaml` (Homarr Application).
 - edit `k8s/bootstrap/ingress/ingress.yaml` (homarr host).
 - edit `k8s/bootstrap/coredns/coredns-configmap.yaml` (homarr rewrite).
 ## Verification (end-to-end)
 1. Commit/push each logical change; ArgoCD auto-syncs. Order: A1/A2 (grant_types) first.
 2. Re-trigger `iam-jobs`; confirm provider `grant_types` now includes `authorization_code`.
 3. Flow-replay: `/authorize` per app returns **302 → /flows/**, not `invalid_request`.
 4. Browser: log into Authentik as `rock`, click each tile → lands **logged-in** in
   grafana/argocd/forgejo/minio/homarr with no OAuth error.
 5. `sso-verify-job` completes green; reverting grant_types in a scratch test turns it red (proves it
   detects the real failure).
 6. Homarr reachable at `https://homarr.riotpiao.com`, SSO works; add tiles in UI.
 ## Notes / caveats
 - Homarr v1 tile content is DB-backed (PVC), not git — accepted.
 - Portainer CE: login works but no group→team auto-map (BE-only); teams assigned manually.
 - Authentik liveness kill-loop already fixed earlier this session (probe 3s→15s), which is why
  authentik is now reachable for provisioning/tests.