homelab/k8s/security/iam/README.md

Talos IAM — Authentik

Standalone SSO / Identity Provider for the homelab. Authentik gives every homelab app one login (OIDC / OAuth2 / SAML / forward-auth). It is an identity provider, not a secret vault — its system of record is PostgreSQL (users, apps, tokens, policies) with Redis for cache/queue. There is no MinIO/S3 involvement.

Want a Vault-style secret store instead? That's a different tool — OpenBao / HashiCorp Vault / Infisical — and a separate setup. This folder is SSO only.

Architecture

                        ┌─────────────────────────────────────┐
                        │           namespace: iam             │
                        │                                      │
  Browser / CLI ───────▶│  authentik-server  (UI + API :80)   │
                        │  authentik-worker  (tasks / flows)  │
                        │  postgresql        (Longhorn 8Gi)   │
                        │  redis             (ephemeral)       │
                        └───────────────┬─────────────────────┘
                                        │ OIDC / OAuth2
                      ┌─────────────────┼─────────────────┐
                      ▼                 ▼                  ▼
              ns: logging          ns: storage        ns: iam
              Grafana              MinIO              (CronJob)
              client_id=grafana    client_id=minio    key-rotation

OIDC clients provisioned:

App	Namespace	Client ID	Redirect URI
Grafana	logging	grafana	http://localhost:3000/login/generic_oauth
MinIO	storage	minio	http://localhost:9001/oauth_callback
Portainer	portainer	—	placeholder (CE has no OIDC)

Groups:

Group	Maps to
grafana-admins	Grafana Admin role
grafana-viewers	Grafana Viewer role
minio-admins	MinIO readwrite policy
minio-readonly	MinIO readonly policy

---

Layout

talos-iam/
├── setup_talos_iam.sh                    # one-shot deploy (namespace → helm → verify → provision)
├── authentik-values.yaml                 # Helm values (Postgres on Longhorn, Redis ephemeral, tolerations)
├── provision_oidc.py                     # idempotent OIDC provisioner (providers, groups, K8s secrets)
├── register_oauth_app.py                 # 🆕 register new services with OAuth + Vault JWT (recommended)
├── example_register_dashboard.py         # 🆕 example: register 'dashboard-service' with full RBAC
├── verify_existing_oauth_integrations.sh # 🆕 verify Grafana/MinIO/Forgejo/Argo CD OIDC still work
├── OAUTH_APP_SETUP.md                    # 🆕 comprehensive guide: manual & automated OAuth setup
├── key_rotate.rs                         # stdlib-only Rust script — rotates OIDC signing key
├── key-rotation-cronjob.yaml             # K8s CronJob running key_rotate.rs quarterly
├── setup_vault.sh                        # Vault initialization & JWT auth wiring (run after Authentik)
├── go-example-oidc/                      # Go Authorization Code flow example against Authentik
│   ├── main.go
│   ├── go.mod
│   └── .env.example
├── .env.example                          # required secrets — copy to ~/.authentik/.env and fill in
└── README.md                              # you are here

---

Quickstart

cp k8s/talos-iam/.env.example ~/.authentik/.env
# fill in the secrets (generators are in .env.example)

bash k8s/talos-iam/setup_talos_iam.sh

The script creates the iam namespace, installs the Authentik Helm chart (server, worker, bundled PostgreSQL + Redis), waits for rollout, probes the readiness endpoint, then calls provision_oidc.py to wire up Grafana and MinIO as OIDC clients.

---

Access

kubectl port-forward svc/authentik-server -n iam 7000:80
# Admin UI:  http://localhost:7000/if/admin/
# Login:     akadmin / <AUTHENTIK_BOOTSTRAP_PASSWORD>

---

OIDC Provisioning

provision_oidc.py idempotently creates all Authentik resources from the API — safe to re-run.

# port-forward must be active (localhost:7000)
source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py

What it provisions:

• RSA-4096 signing certificate homelab-oidc
• OAuth2 providers for Grafana and MinIO (with the signing cert attached)
• Property mapping that injects a policy JWT claim for MinIO access control
• Groups: grafana-admins, grafana-viewers, minio-admins, minio-readonly
• K8s secrets grafana-oidc (ns: logging) and minio-oidc (ns: storage)

---

Key Rotation

OIDC signing keys should be rotated periodically. The key_rotate.rs script generates a new RSA-4096 cert in Authentik and patches all providers to use it. The old cert stays in the JWKS endpoint until you delete it — existing tokens remain valid through their TTL (default 5 min).

Manual rotation (port-forward must be active):

source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py --rotate

Automated rotation (quarterly CronJob in-cluster):

# One-time setup
kubectl create configmap key-rotation-script \
  --from-file=rotate_key.rs=k8s/talos-iam/key_rotate.rs \
  -n iam --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic authentik-key-rotation-token \
  --from-literal=AUTHENTIK_BOOTSTRAP_TOKEN="${AUTHENTIK_BOOTSTRAP_TOKEN}" \
  -n iam --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f k8s/talos-iam/key-rotation-cronjob.yaml

# Test the job immediately
kubectl create job --from=cronjob/authentik-key-rotation test-rotation -n iam
kubectl logs -n iam -l job-name=test-rotation -f

Validate rotation:

# Confirm signing_key is set and changed
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
  "http://localhost:7000/api/v3/providers/oauth2/?name=grafana" \
  | python3 -c "import json,sys; p=json.load(sys.stdin)['results'][0]; print('signing_key:', p['signing_key'])"

# Confirm JWKS shows both old and new key during transition
curl -s http://localhost:7000/application/o/grafana/.well-known/jwks.json \
  | python3 -c "import json,sys; [print('kid:', k['kid']) for k in json.load(sys.stdin)['keys']]"

After rotation, nothing in ~/.authentik/.env changes. Client secrets, the bootstrap token, and AUTHENTIK_SECRET_KEY are all separate from the OIDC signing keypair.

---

Go OIDC Example

A minimal Authorization Code flow demo against Authentik — useful for verifying the IdP end-to-end or as a starting point for a new OIDC client.

cp k8s/talos-iam/go-example-oidc/.env.example ~/.authentik/.env
# add OIDC_CLIENT_ID and OIDC_CLIENT_SECRET for an app you register in Authentik

cd k8s/talos-iam/go-example-oidc
go mod tidy && go run .
# open http://localhost:8080/login

The callback prints the verified ID token claims as JSON — email, name, sub, and any custom claims (e.g. the MinIO policy claim).

---

Node Resilience — Auto-Start on Reboot

All four components (server, worker, PostgreSQL, Redis) carry the control-plane toleration:

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

• Any node goes down → Kubernetes reschedules all Authentik pods onto the surviving node.
• PostgreSQL PVC (8Gi Longhorn ReadWriteOnce) → Longhorn reattaches automatically (~2 min). All user/app/token data is preserved.
• Redis is ephemeral (no PVC) — restarts clean, which is correct (cache/queue only).
• Full cluster reboot → cp-1 comes up first; Kubernetes reconciles Deployments; Longhorn reattaches. Zero manual action needed.

Startup order after reboot:

1. kubelet starts on both nodes
2. etcd + API server on cp-1
3. Controllers reconcile Authentik Deployments and PostgreSQL StatefulSet
4. PostgreSQL starts (Authentik server/worker wait via init probes)
5. Redis starts
6. Authentik server + worker become ready

---

Verify

kubectl get pods -n iam
# authentik-server, authentik-worker, authentik-postgresql-0, authentik-redis-master-0 → Running

curl -fsS -o /dev/null -w '%{http_code}\n' http://localhost:7000/-/health/ready/
# 204

---

Registering New Services (OAuth App Workflow)

The homelab provides automated OAuth registration for new services via register_oauth_app.py:

# Register a new service with OAuth + Vault JWT auth (recommended)
python3 k8s/talos-iam/register_oauth_app.py \
  --service-name my-service \
  --namespace my-ns \
  --redirect-uri "https://my-service.riotpiao.com/oauth2/callback" \
  --service-name-in-vault MY_SERVICE \
  --vault-jwt-policy shell-secrets \
  --add-group my-service-admins

This automates:

1. ✅ Authentik OAuth2 provider creation (credentials from Vault)
2. ✅ Authentik application binding
3. ✅ Kubernetes secret provisioning (client ID/secret)
4. ✅ Vault JWT role creation (for service → Vault auth)
5. ✅ Group-based RBAC setup (optional)

For manual control or step-by-step guidance, see OAUTH_APP_SETUP.md which covers both automated and manual workflows.

Example: Register "dashboard-service"

# 1. Generate client secret and store in Vault
DASHBOARD_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
talos put cluster/DASHBOARD_OIDC_CLIENT_SECRET DASHBOARD_OIDC_CLIENT_SECRET="$DASHBOARD_OIDC_CLIENT_SECRET"

# 2. Register with automation
export DASHBOARD_OIDC_CLIENT_SECRET
python3 k8s/talos-iam/register_oauth_app.py \
  --service-name dashboard-service \
  --namespace apps \
  --redirect-uri "https://dashboard.riotpiao.com/oauth2/callback" \
  --service-name-in-vault DASHBOARD \
  --vault-jwt-policy shell-secrets \
  --add-group dashboard-admins \
  --vault-jwt-bound-claims '{"groups":["dashboard-admins"]}'

# 3. Service now has:
#   - Authentik provider (dashboard-service)
#   - K8s secret (dashboard-service-oidc) in 'apps' namespace
#   - Vault JWT role (dashboard-service) with group-based access

Verification

After registration, verify the integration:

# Check Authentik provider and app
curl -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
  http://localhost:7000/api/v3/core/applications/?slug=my-service | jq .

# Check K8s secret
kubectl get secret my-service-oidc -n my-ns -o yaml

# Check Vault JWT role
vault read auth/jwt/role/my-service

# Test OAuth login
# Browser: https://my-service.riotpiao.com/login
# Should redirect to Authentik → back to service with session

Full verification script:

bash k8s/talos-iam/verify_existing_oauth_integrations.sh

---

Notes

• AUTHENTIK_SECRET_KEY is set-once. Rotating it invalidates all sessions, tokens, and encrypted fields in the database. Keep ~/.authentik/.env safe and backed up.
• PostgreSQL holds everything that matters — users, providers, groups, tokens, certificates. It lives on an 8Gi Longhorn PVC. Redis is ephemeral by design.
• No ingress. Access is via port-forward, matching the rest of the homelab.
• OIDC signing keys (rotated by provision_oidc.py --rotate) are separate from all .env credentials. Rotation requires no client reconfiguration — Grafana and MinIO pick up the new public key from the JWKS endpoint automatically.
• New app registration is automated via register_oauth_app.py and fully documented in OAUTH_APP_SETUP.md. Both manual and automated workflows are supported.