homelab/k8s/talos-iam/README.md

# Talos IAM — Authentik

Standalone **SSO / Identity Provider** for the homelab.
Authentik gives every homelab app one login (OIDC / OAuth2 / SAML / forward-auth). It is an
**identity provider, not a secret vault** — its system of record is PostgreSQL (users, apps,
tokens, policies) with Redis for cache/queue. There is no MinIO/S3 involvement.

> Want a Vault-style secret store instead? That's a different tool — OpenBao / HashiCorp Vault /
> Infisical — and a separate setup. This folder is SSO only.

## Architecture

```
                        ┌─────────────────────────────────────┐
                        │           namespace: iam             │
                        │                                      │
  Browser / CLI ───────▶│  authentik-server  (UI + API :80)   │
                        │  authentik-worker  (tasks / flows)  │
                        │  postgresql        (Longhorn 8Gi)   │
                        │  redis             (ephemeral)       │
                        └───────────────┬─────────────────────┘
                                        │ OIDC / OAuth2
                      ┌─────────────────┼─────────────────┐
                      ▼                 ▼                  ▼
              ns: logging          ns: storage        ns: iam
              Grafana              MinIO              (CronJob)
              client_id=grafana    client_id=minio    key-rotation
```

**OIDC clients provisioned:**

| App | Namespace | Client ID | Redirect URI |
|---|---|---|---|
| Grafana | `logging` | `grafana` | `http://localhost:3000/login/generic_oauth` |
| MinIO | `storage` | `minio` | `http://localhost:9001/oauth_callback` |
| Portainer | `portainer` | — | placeholder (CE has no OIDC) |

**Groups:**

| Group | Maps to |
|---|---|
| `grafana-admins` | Grafana Admin role |
| `grafana-viewers` | Grafana Viewer role |
| `minio-admins` | MinIO `readwrite` policy |
| `minio-readonly` | MinIO `readonly` policy |

---

## Layout

```
talos-iam/
├── setup_talos_iam.sh                    # one-shot deploy (namespace → helm → verify → provision)
├── authentik-values.yaml                 # Helm values (Postgres on Longhorn, Redis ephemeral, tolerations)
├── provision_oidc.py                     # idempotent OIDC provisioner (providers, groups, K8s secrets)
├── register_oauth_app.py                 # 🆕 register new services with OAuth + Vault JWT (recommended)
├── example_register_dashboard.py         # 🆕 example: register 'dashboard-service' with full RBAC
├── verify_existing_oauth_integrations.sh # 🆕 verify Grafana/MinIO/Forgejo/Argo CD OIDC still work
├── OAUTH_APP_SETUP.md                    # 🆕 comprehensive guide: manual & automated OAuth setup
├── key_rotate.rs                         # stdlib-only Rust script — rotates OIDC signing key
├── key-rotation-cronjob.yaml             # K8s CronJob running key_rotate.rs quarterly
├── setup_vault.sh                        # Vault initialization & JWT auth wiring (run after Authentik)
├── go-example-oidc/                      # Go Authorization Code flow example against Authentik
│   ├── main.go
│   ├── go.mod
│   └── .env.example
├── .env.example                          # required secrets — copy to ~/.authentik/.env and fill in
└── README.md                              # you are here
```

---

## Quickstart

```bash
cp k8s/talos-iam/.env.example ~/.authentik/.env
# fill in the secrets (generators are in .env.example)

bash k8s/talos-iam/setup_talos_iam.sh
```

The script creates the `iam` namespace, installs the Authentik Helm chart (server, worker, bundled
PostgreSQL + Redis), waits for rollout, probes the readiness endpoint, then calls
`provision_oidc.py` to wire up Grafana and MinIO as OIDC clients.

---

## Access

```bash
kubectl port-forward svc/authentik-server -n iam 7000:80
# Admin UI:  http://localhost:7000/if/admin/
# Login:     akadmin / <AUTHENTIK_BOOTSTRAP_PASSWORD>
```

---

## OIDC Provisioning

`provision_oidc.py` idempotently creates all Authentik resources from the API — safe to re-run.

```bash
# port-forward must be active (localhost:7000)
source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py
```

What it provisions:
- RSA-4096 signing certificate `homelab-oidc`
- OAuth2 providers for Grafana and MinIO (with the signing cert attached)
- Property mapping that injects a `policy` JWT claim for MinIO access control
- Groups: `grafana-admins`, `grafana-viewers`, `minio-admins`, `minio-readonly`
- K8s secrets `grafana-oidc` (ns: `logging`) and `minio-oidc` (ns: `storage`)

---

## Key Rotation

OIDC signing keys should be rotated periodically. The `key_rotate.rs` script generates a new
RSA-4096 cert in Authentik and patches all providers to use it. The old cert stays in the JWKS
endpoint until you delete it — existing tokens remain valid through their TTL (default 5 min).

**Manual rotation** (port-forward must be active):
```bash
source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py --rotate
```

**Automated rotation** (quarterly CronJob in-cluster):
```bash
# One-time setup
kubectl create configmap key-rotation-script \
  --from-file=rotate_key.rs=k8s/talos-iam/key_rotate.rs \
  -n iam --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic authentik-key-rotation-token \
  --from-literal=AUTHENTIK_BOOTSTRAP_TOKEN="${AUTHENTIK_BOOTSTRAP_TOKEN}" \
  -n iam --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f k8s/talos-iam/key-rotation-cronjob.yaml

# Test the job immediately
kubectl create job --from=cronjob/authentik-key-rotation test-rotation -n iam
kubectl logs -n iam -l job-name=test-rotation -f
```

**Validate rotation:**
```bash
# Confirm signing_key is set and changed
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
  "http://localhost:7000/api/v3/providers/oauth2/?name=grafana" \
  | python3 -c "import json,sys; p=json.load(sys.stdin)['results'][0]; print('signing_key:', p['signing_key'])"

# Confirm JWKS shows both old and new key during transition
curl -s http://localhost:7000/application/o/grafana/.well-known/jwks.json \
  | python3 -c "import json,sys; [print('kid:', k['kid']) for k in json.load(sys.stdin)['keys']]"
```

> After rotation, nothing in `~/.authentik/.env` changes. Client secrets, the bootstrap token,
> and `AUTHENTIK_SECRET_KEY` are all separate from the OIDC signing keypair.

---

## Go OIDC Example

A minimal Authorization Code flow demo against Authentik — useful for verifying the IdP
end-to-end or as a starting point for a new OIDC client.

```bash
cp k8s/talos-iam/go-example-oidc/.env.example ~/.authentik/.env
# add OIDC_CLIENT_ID and OIDC_CLIENT_SECRET for an app you register in Authentik

cd k8s/talos-iam/go-example-oidc
go mod tidy && go run .
# open http://localhost:8080/login
```

The callback prints the verified ID token claims as JSON — `email`, `name`, `sub`, and any
custom claims (e.g. the MinIO `policy` claim).

---

## Node Resilience — Auto-Start on Reboot

All four components (server, worker, PostgreSQL, Redis) carry the control-plane toleration:

```yaml
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
```

- **Any node goes down** → Kubernetes reschedules all Authentik pods onto the surviving node.
- **PostgreSQL PVC** (8Gi Longhorn `ReadWriteOnce`) → Longhorn reattaches automatically (~2 min). All user/app/token data is preserved.
- **Redis** is ephemeral (no PVC) — restarts clean, which is correct (cache/queue only).
- **Full cluster reboot** → cp-1 comes up first; Kubernetes reconciles Deployments; Longhorn reattaches. Zero manual action needed.

**Startup order after reboot:**
1. `kubelet` starts on both nodes
2. `etcd` + API server on cp-1
3. Controllers reconcile Authentik Deployments and PostgreSQL StatefulSet
4. PostgreSQL starts (Authentik server/worker wait via init probes)
5. Redis starts
6. Authentik server + worker become ready

---

## Verify

```bash
kubectl get pods -n iam
# authentik-server, authentik-worker, authentik-postgresql-0, authentik-redis-master-0 → Running

curl -fsS -o /dev/null -w '%{http_code}\n' http://localhost:7000/-/health/ready/
# 204
```

---

## Registering New Services (OAuth App Workflow)

The homelab provides **automated OAuth registration** for new services via `register_oauth_app.py`:

```bash
# Register a new service with OAuth + Vault JWT auth (recommended)
python3 k8s/talos-iam/register_oauth_app.py \
  --service-name my-service \
  --namespace my-ns \
  --redirect-uri "https://my-service.riotpiao.homelab.com/oauth2/callback" \
  --service-name-in-vault MY_SERVICE \
  --vault-jwt-policy shell-secrets \
  --add-group my-service-admins
```

This automates:
1. ✅ Authentik OAuth2 provider creation (credentials from Vault)
2. ✅ Authentik application binding
3. ✅ Kubernetes secret provisioning (client ID/secret)
4. ✅ Vault JWT role creation (for service → Vault auth)
5. ✅ Group-based RBAC setup (optional)

**For manual control or step-by-step guidance,** see [`OAUTH_APP_SETUP.md`](OAUTH_APP_SETUP.md) which covers both automated and manual workflows.

### Example: Register "dashboard-service"

```bash
# 1. Generate client secret and store in Vault
DASHBOARD_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
talos put cluster/DASHBOARD_OIDC_CLIENT_SECRET DASHBOARD_OIDC_CLIENT_SECRET="$DASHBOARD_OIDC_CLIENT_SECRET"

# 2. Register with automation
export DASHBOARD_OIDC_CLIENT_SECRET
python3 k8s/talos-iam/register_oauth_app.py \
  --service-name dashboard-service \
  --namespace apps \
  --redirect-uri "https://dashboard.riotpiao.homelab.com/oauth2/callback" \
  --service-name-in-vault DASHBOARD \
  --vault-jwt-policy shell-secrets \
  --add-group dashboard-admins \
  --vault-jwt-bound-claims '{"groups":["dashboard-admins"]}'

# 3. Service now has:
#   - Authentik provider (dashboard-service)
#   - K8s secret (dashboard-service-oidc) in 'apps' namespace
#   - Vault JWT role (dashboard-service) with group-based access
```

### Verification

After registration, verify the integration:

```bash
# Check Authentik provider and app
curl -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
  http://localhost:7000/api/v3/core/applications/?slug=my-service | jq .

# Check K8s secret
kubectl get secret my-service-oidc -n my-ns -o yaml

# Check Vault JWT role
vault read auth/jwt/role/my-service

# Test OAuth login
# Browser: https://my-service.riotpiao.homelab.com/login
# Should redirect to Authentik → back to service with session
```

**Full verification script:**
```bash
bash k8s/talos-iam/verify_existing_oauth_integrations.sh
```

---

## Notes

- **`AUTHENTIK_SECRET_KEY` is set-once.** Rotating it invalidates all sessions, tokens, and encrypted fields in the database. Keep `~/.authentik/.env` safe and backed up.
- **PostgreSQL holds everything that matters** — users, providers, groups, tokens, certificates. It lives on an 8Gi Longhorn PVC. Redis is ephemeral by design.
- **No ingress.** Access is via port-forward, matching the rest of the homelab.
- **OIDC signing keys** (rotated by `provision_oidc.py --rotate`) are separate from all `.env` credentials. Rotation requires no client reconfiguration — Grafana and MinIO pick up the new public key from the JWKS endpoint automatically.
- **New app registration** is automated via `register_oauth_app.py` and fully documented in `OAUTH_APP_SETUP.md`. Both manual and automated workflows are supported.