Story Crater Bot
f08fb2bb75
Root cause of the provisioning Job never appearing in-cluster despite being
committed and pushed: k8s/security/iam/kustomization.yaml has an explicit
resources: allowlist (not a plain directory scan) and the new file was never
added to it, so ArgoCD's Kustomize build silently omitted every object in it
- no error, no drift shown, iam-jobs just reported Synced/Healthy against a
manifest set that never included the new Job/ConfigMap/RBAC at all.
Also removed the top-level transformer. It would have
force-rewritten metadata.namespace to iam on every resource in this
kustomization, including authentik-provision-job.yaml's RoleBindings which
deliberately target cicd/argocd/logging/storage (least-privilege access for
the authentik-provisioner ServiceAccount to read/create Secrets in exactly
those namespaces and no others). Every manifest in this directory already
sets its own explicit namespace, so dropping the transformer is a no-op for
the existing key-rotation-cronjob.yaml.
Verified with apiVersion: v1
kind: ServiceAccount
metadata:
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: authentik-provisioner
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: argocd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: cicd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: iam
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: logging
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: storage
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: v1
data:
authentik-provision.py: |
#!/usr/bin/env python3
"""
Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook).
Creates/updates, in order:
1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile
by default but NOT groups - required for ArgoCD RBAC group mapping and
Grafana's role_attribute_path, both of which read a `groups` claim).
2. Groups: homelab-admins (is_superuser=true), grafana-admins.
3. User "rock": created if missing, always (re-)synced into both groups above.
Password is generated once and only written to the k8s Secret
rock-credentials (iam ns) the first time the user is created - re-runs
never rotate an existing password.
4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd.
Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc)
if present, or generated once and written out (forgejo-oidc, oidc-secret)
the first time.
5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and
anyone else in that group) has guaranteed access regardless of each app's
default visibility.
Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80),
authenticating with the bootstrap token. Everything is done with GET-then-
create-or-patch so this can be re-run on every ArgoCD sync without duplicating
or clobbering objects (PostSync hook, not a one-shot Job with hook-delete).
kubectl is used only to read/write the small set of Secrets this script
touches - it shells out rather than using the Python k8s client to keep the
container image to stdlib Python + the kubectl binary, no pip installs.
"""
import json
import os
import secrets
import string
import subprocess
import sys
import urllib.error
import urllib.request
AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local"
TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"]
def api(method, path, data=None):
url = f"{AUTHENTIK_URL}{path}"
body = json.dumps(data).encode() if data is not None else None
req = urllib.request.Request(
url,
data=body,
method=method,
headers={
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else {})
except urllib.error.HTTPError as e:
raw = e.read()
try:
parsed = json.loads(raw) if raw else {}
except json.JSONDecodeError:
parsed = {"raw": raw.decode(errors="replace")}
return e.code, parsed
def die(msg):
print(f"FATAL: {msg}", file=sys.stderr)
sys.exit(1)
def gen_secret(n=40):
alphabet = string.ascii_letters + string.digits
return "".join(secrets.choice(alphabet) for _ in range(n))
def kubectl_get_secret_key(namespace, name, key):
"""Returns decoded value, or None if the secret/key doesn't exist."""
p = subprocess.run(
["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"],
capture_output=True, text=True,
)
if p.returncode != 0 or not p.stdout.strip():
return None
import base64
return base64.b64decode(p.stdout).decode()
def kubectl_create_secret(namespace, name, literals: dict):
"""Idempotent: create-or-update via dry-run|apply, same pattern used
elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh)."""
args = ["kubectl", "-n", namespace, "create", "secret", "generic", name]
for k, v in literals.items():
args += [f"--from-literal={k}={v}"]
args += ["--dry-run=client", "-o", "yaml"]
render = subprocess.run(args, capture_output=True, text=True)
if render.returncode != 0:
die(f"rendering secret {namespace}/{name}: {render.stderr}")
apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout,
capture_output=True, text=True)
if apply.returncode != 0:
die(f"applying secret {namespace}/{name}: {apply.stderr}")
print(f" secret {namespace}/{name}: {apply.stdout.strip()}")
def get_or_create(list_path, create_path, query, payload, patch_existing=None):
status, res = api("GET", f"{list_path}?{query}")
if status != 200:
die(f"GET {list_path}?{query} -> {status} {res}")
results = res.get("results", [])
if results:
obj = results[0]
if patch_existing:
status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing)
if status not in (200, 201):
die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}")
return obj2
return obj
status, obj = api("POST", create_path, payload)
if status not in (200, 201):
die(f"POST {create_path} -> {status} {obj}")
return obj
# -----------------------------------------------------------------------------
print("[1/5] Ensuring custom 'groups' scope mapping exists...")
groups_mapping = get_or_create(
"/api/v3/propertymappings/provider/scope/",
"/api/v3/propertymappings/provider/scope/",
"scope_name=groups",
{
"name": "homelab: groups claim",
"scope_name": "groups",
"expression": (
"return {\"groups\": [group.name for group in request.user.ak_groups.all()]}"
),
},
)
GROUPS_MAPPING_PK = groups_mapping["pk"]
# Fetch the standard openid/email/profile mapping pks (shipped by default).
status, res = api("GET", "/api/v3/propertymappings/provider/scope/")
by_scope = {m["scope_name"]: m["pk"] for m in res["results"]}
SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent")
AUTHORIZATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow")
INVALIDATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true")
SIGNING_KEY_PK = res["results"][0]["pk"]
# -----------------------------------------------------------------------------
print("[2/5] Ensuring groups homelab-admins / grafana-admins exist...")
homelab_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=homelab-admins",
{"name": "homelab-admins", "is_superuser": True},
)
grafana_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=grafana-admins",
{"name": "grafana-admins", "is_superuser": False},
)
# -----------------------------------------------------------------------------
print("[3/5] Ensuring user 'rock' exists with admin group membership...")
status, res = api("GET", "/api/v3/core/users/?username=rock")
rock_password = None
if res.get("results"):
rock = res["results"][0]
status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", {
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"is_active": True,
})
if status not in (200, 201):
die(f"PATCH user rock -> {status} {rock}")
print(" rock already exists, group membership synced (password unchanged)")
else:
rock_password = gen_secret(24)
status, rock = api("POST", "/api/v3/core/users/", {
"username": "rock",
"name": "Rock",
"is_active": True,
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"path": "users",
"type": "internal",
})
if status not in (200, 201):
die(f"POST user rock -> {status} {rock}")
status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/",
{"password": rock_password})
if status not in (200, 204):
die(f"set_password for rock -> {status} {pw_res}")
kubectl_create_secret("iam", "rock-credentials", {
"username": "rock",
"password": rock_password,
})
print(" rock created, credentials stored in iam/rock-credentials")
# -----------------------------------------------------------------------------
print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...")
SERVICES = {
"grafana": {
"client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"),
"redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"],
"launch_url": "https://grafana.riotpiao.com",
"display_name": "Grafana",
},
"minio": {
"client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"),
"redirect_uris": ["https://minio.riotpiao.com/oauth_callback"],
"launch_url": "https://minio.riotpiao.com",
"display_name": "MinIO",
},
"forgejo": {
# No secret exists yet for forgejo - generate + store on first run.
"client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"),
"generate_if_missing": True,
"redirect_uris": [
"https://forgejo.riotpiao.com/user/oauth2/authentik/callback",
"https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback",
],
"launch_url": "https://forgejo.riotpiao.com",
"display_name": "Forgejo",
},
"argocd": {
# oidc-secret uses hyphenated keys (client-id/client-secret) per
# argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs.
"client_secret_source": ("argocd", "oidc-secret", "client-secret"),
"generate_if_missing": True,
"extra_secret_literals": {"client-id": "argocd"},
"redirect_uris": ["https://argocd.riotpiao.com/auth/callback"],
"launch_url": "https://argocd.riotpiao.com",
"display_name": "Argo CD",
},
}
app_pks_for_binding = []
for name, cfg in SERVICES.items():
ns, secret_name, key = cfg["client_secret_source"]
client_secret = kubectl_get_secret_key(ns, secret_name, key)
if client_secret is None:
if not cfg.get("generate_if_missing"):
print(f" WARNING: {ns}/{secret_name} key {key} not found and "
f"generate_if_missing not set for '{name}' - skipping provider/app")
continue
client_secret = gen_secret(40)
literals = {key: client_secret}
literals.update(cfg.get("extra_secret_literals", {}))
kubectl_create_secret(ns, secret_name, literals)
print(f" {name}: generated new client secret -> {ns}/{secret_name}")
else:
print(f" {name}: using existing client secret from {ns}/{secret_name}")
provider = get_or_create(
"/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
f"name={name}",
{
"name": name,
"client_id": name,
"client_secret": client_secret,
"client_type": "confidential",
"authorization_flow": AUTHORIZATION_FLOW_PK,
"invalidation_flow": INVALIDATION_FLOW_PK,
"signing_key": SIGNING_KEY_PK,
"property_mappings": SCOPE_PKS,
"sub_mode": "hashed_user_id",
"include_claims_in_id_token": True,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
# Keep the redirect_uris/mappings in sync on re-run, but never touch
# client_secret again once created (that's the source of truth in the
# k8s Secret, and re-sending it here is harmless/idempotent anyway).
patch_existing={
"property_mappings": SCOPE_PKS,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
)
application = get_or_create(
"/api/v3/core/applications/", "/api/v3/core/applications/",
f"slug={name}",
{
"name": cfg["display_name"],
"slug": name,
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
patch_existing={
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
)
app_pks_for_binding.append((name, application["pk"]))
print(f" {name}: provider pk={provider['pk']} application pk={application['pk']}")
# -----------------------------------------------------------------------------
print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...")
for name, app_pk in app_pks_for_binding:
get_or_create(
"/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
f"target={app_pk}&group={homelab_admins['pk']}",
{
"target": app_pk,
"group": homelab_admins["pk"],
"order": 0,
"enabled": True,
},
)
print(f" {name}: homelab-admins bound")
print("\nDone. Summary:")
print(" groups: homelab-admins (superuser), grafana-admins")
print(" user: rock -> homelab-admins + grafana-admins")
print(f" apps: {', '.join(n for n, _ in app_pks_for_binding)}")
if rock_password:
print(" NOTE: rock's password was generated this run - see")
print(" kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d")
kind: ConfigMap
metadata:
name: authentik-provision-script
namespace: iam
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: authentik-key-rotation
namespace: iam
spec:
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- command:
- sh
- -c
- rustc /scripts/rotate_key.rs -o /tmp/rotate_key && /tmp/rotate_key
env:
- name: AUTHENTIK_BASE_URL
value: http://authentik-server.iam.svc.cluster.local
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-key-rotation-token
image: rust:1.82-slim
name: rotate
volumeMounts:
- mountPath: /scripts
name: script
restartPolicy: OnFailure
volumes:
- configMap:
name: key-rotation-script
name: script
schedule: 0 0 1 */3 *
---
apiVersion: batch/v1
kind: Job
metadata:
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
name: authentik-provision
namespace: iam
spec:
backoffLimit: 3
template:
spec:
containers:
- command:
- /bin/sh
- -c
- |
set -e
echo "waiting for authentik-server..."
until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do
sleep 5
done
echo "installing kubectl..."
apk add --no-cache curl >/dev/null
KVER=$(curl -sL https://dl.k8s.io/release/stable.txt)
curl -sLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl"
chmod +x /usr/local/bin/kubectl
echo "running provisioning script..."
python3 /script/authentik-provision.py
env:
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-secrets
image: python:3.12-alpine
name: provision
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /script
name: script
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
serviceAccountName: authentik-provisioner
volumes:
- configMap:
name: authentik-provision-script
name: script
ttlSecondsAfterFinished: 600 locally before pushing -
confirms all 5 RoleBindings land in their correct distinct namespaces
(iam/cicd/argocd/logging/storage) and every resource renders as valid YAML.
Talos IAM — Authentik
Standalone SSO / Identity Provider for the homelab. Authentik gives every homelab app one login (OIDC / OAuth2 / SAML / forward-auth). It is an identity provider, not a secret vault — its system of record is PostgreSQL (users, apps, tokens, policies) with Redis for cache/queue. There is no MinIO/S3 involvement.
Want a Vault-style secret store instead? That's a different tool — OpenBao / HashiCorp Vault / Infisical — and a separate setup. This folder is SSO only.
Architecture
┌─────────────────────────────────────┐
│ namespace: iam │
│ │
Browser / CLI ───────▶│ authentik-server (UI + API :80) │
│ authentik-worker (tasks / flows) │
│ postgresql (Longhorn 8Gi) │
│ redis (ephemeral) │
└───────────────┬─────────────────────┘
│ OIDC / OAuth2
┌─────────────────┼─────────────────┐
▼ ▼ ▼
ns: logging ns: storage ns: iam
Grafana MinIO (CronJob)
client_id=grafana client_id=minio key-rotation
OIDC clients provisioned:
| App | Namespace | Client ID | Redirect URI |
|---|---|---|---|
| Grafana | logging |
grafana |
http://localhost:3000/login/generic_oauth |
| MinIO | storage |
minio |
http://localhost:9001/oauth_callback |
| Portainer | portainer |
— | placeholder (CE has no OIDC) |
Groups:
| Group | Maps to |
|---|---|
grafana-admins |
Grafana Admin role |
grafana-viewers |
Grafana Viewer role |
minio-admins |
MinIO readwrite policy |
minio-readonly |
MinIO readonly policy |
Layout
talos-iam/
├── setup_talos_iam.sh # one-shot deploy (namespace → helm → verify → provision)
├── authentik-values.yaml # Helm values (Postgres on Longhorn, Redis ephemeral, tolerations)
├── provision_oidc.py # idempotent OIDC provisioner (providers, groups, K8s secrets)
├── register_oauth_app.py # 🆕 register new services with OAuth + Vault JWT (recommended)
├── example_register_dashboard.py # 🆕 example: register 'dashboard-service' with full RBAC
├── verify_existing_oauth_integrations.sh # 🆕 verify Grafana/MinIO/Forgejo/Argo CD OIDC still work
├── OAUTH_APP_SETUP.md # 🆕 comprehensive guide: manual & automated OAuth setup
├── key_rotate.rs # stdlib-only Rust script — rotates OIDC signing key
├── key-rotation-cronjob.yaml # K8s CronJob running key_rotate.rs quarterly
├── setup_vault.sh # Vault initialization & JWT auth wiring (run after Authentik)
├── go-example-oidc/ # Go Authorization Code flow example against Authentik
│ ├── main.go
│ ├── go.mod
│ └── .env.example
├── .env.example # required secrets — copy to ~/.authentik/.env and fill in
└── README.md # you are here
Quickstart
cp k8s/talos-iam/.env.example ~/.authentik/.env
# fill in the secrets (generators are in .env.example)
bash k8s/talos-iam/setup_talos_iam.sh
The script creates the iam namespace, installs the Authentik Helm chart (server, worker, bundled
PostgreSQL + Redis), waits for rollout, probes the readiness endpoint, then calls
provision_oidc.py to wire up Grafana and MinIO as OIDC clients.
Access
kubectl port-forward svc/authentik-server -n iam 7000:80
# Admin UI: http://localhost:7000/if/admin/
# Login: akadmin / <AUTHENTIK_BOOTSTRAP_PASSWORD>
OIDC Provisioning
provision_oidc.py idempotently creates all Authentik resources from the API — safe to re-run.
# port-forward must be active (localhost:7000)
source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py
What it provisions:
- RSA-4096 signing certificate
homelab-oidc - OAuth2 providers for Grafana and MinIO (with the signing cert attached)
- Property mapping that injects a
policyJWT claim for MinIO access control - Groups:
grafana-admins,grafana-viewers,minio-admins,minio-readonly - K8s secrets
grafana-oidc(ns:logging) andminio-oidc(ns:storage)
Key Rotation
OIDC signing keys should be rotated periodically. The key_rotate.rs script generates a new
RSA-4096 cert in Authentik and patches all providers to use it. The old cert stays in the JWKS
endpoint until you delete it — existing tokens remain valid through their TTL (default 5 min).
Manual rotation (port-forward must be active):
source ~/.authentik/.env
python k8s/talos-iam/provision_oidc.py --rotate
Automated rotation (quarterly CronJob in-cluster):
# One-time setup
kubectl create configmap key-rotation-script \
--from-file=rotate_key.rs=k8s/talos-iam/key_rotate.rs \
-n iam --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic authentik-key-rotation-token \
--from-literal=AUTHENTIK_BOOTSTRAP_TOKEN="${AUTHENTIK_BOOTSTRAP_TOKEN}" \
-n iam --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/talos-iam/key-rotation-cronjob.yaml
# Test the job immediately
kubectl create job --from=cronjob/authentik-key-rotation test-rotation -n iam
kubectl logs -n iam -l job-name=test-rotation -f
Validate rotation:
# Confirm signing_key is set and changed
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
"http://localhost:7000/api/v3/providers/oauth2/?name=grafana" \
| python3 -c "import json,sys; p=json.load(sys.stdin)['results'][0]; print('signing_key:', p['signing_key'])"
# Confirm JWKS shows both old and new key during transition
curl -s http://localhost:7000/application/o/grafana/.well-known/jwks.json \
| python3 -c "import json,sys; [print('kid:', k['kid']) for k in json.load(sys.stdin)['keys']]"
After rotation, nothing in
~/.authentik/.envchanges. Client secrets, the bootstrap token, andAUTHENTIK_SECRET_KEYare all separate from the OIDC signing keypair.
Go OIDC Example
A minimal Authorization Code flow demo against Authentik — useful for verifying the IdP end-to-end or as a starting point for a new OIDC client.
cp k8s/talos-iam/go-example-oidc/.env.example ~/.authentik/.env
# add OIDC_CLIENT_ID and OIDC_CLIENT_SECRET for an app you register in Authentik
cd k8s/talos-iam/go-example-oidc
go mod tidy && go run .
# open http://localhost:8080/login
The callback prints the verified ID token claims as JSON — email, name, sub, and any
custom claims (e.g. the MinIO policy claim).
Node Resilience — Auto-Start on Reboot
All four components (server, worker, PostgreSQL, Redis) carry the control-plane toleration:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- Any node goes down → Kubernetes reschedules all Authentik pods onto the surviving node.
- PostgreSQL PVC (8Gi Longhorn
ReadWriteOnce) → Longhorn reattaches automatically (~2 min). All user/app/token data is preserved. - Redis is ephemeral (no PVC) — restarts clean, which is correct (cache/queue only).
- Full cluster reboot → cp-1 comes up first; Kubernetes reconciles Deployments; Longhorn reattaches. Zero manual action needed.
Startup order after reboot:
kubeletstarts on both nodesetcd+ API server on cp-1- Controllers reconcile Authentik Deployments and PostgreSQL StatefulSet
- PostgreSQL starts (Authentik server/worker wait via init probes)
- Redis starts
- Authentik server + worker become ready
Verify
kubectl get pods -n iam
# authentik-server, authentik-worker, authentik-postgresql-0, authentik-redis-master-0 → Running
curl -fsS -o /dev/null -w '%{http_code}\n' http://localhost:7000/-/health/ready/
# 204
Registering New Services (OAuth App Workflow)
The homelab provides automated OAuth registration for new services via register_oauth_app.py:
# Register a new service with OAuth + Vault JWT auth (recommended)
python3 k8s/talos-iam/register_oauth_app.py \
--service-name my-service \
--namespace my-ns \
--redirect-uri "https://my-service.riotpiao.com/oauth2/callback" \
--service-name-in-vault MY_SERVICE \
--vault-jwt-policy shell-secrets \
--add-group my-service-admins
This automates:
- ✅ Authentik OAuth2 provider creation (credentials from Vault)
- ✅ Authentik application binding
- ✅ Kubernetes secret provisioning (client ID/secret)
- ✅ Vault JWT role creation (for service → Vault auth)
- ✅ Group-based RBAC setup (optional)
For manual control or step-by-step guidance, see OAUTH_APP_SETUP.md which covers both automated and manual workflows.
Example: Register "dashboard-service"
# 1. Generate client secret and store in Vault
DASHBOARD_OIDC_CLIENT_SECRET=$(openssl rand -hex 32)
talos put cluster/DASHBOARD_OIDC_CLIENT_SECRET DASHBOARD_OIDC_CLIENT_SECRET="$DASHBOARD_OIDC_CLIENT_SECRET"
# 2. Register with automation
export DASHBOARD_OIDC_CLIENT_SECRET
python3 k8s/talos-iam/register_oauth_app.py \
--service-name dashboard-service \
--namespace apps \
--redirect-uri "https://dashboard.riotpiao.com/oauth2/callback" \
--service-name-in-vault DASHBOARD \
--vault-jwt-policy shell-secrets \
--add-group dashboard-admins \
--vault-jwt-bound-claims '{"groups":["dashboard-admins"]}'
# 3. Service now has:
# - Authentik provider (dashboard-service)
# - K8s secret (dashboard-service-oidc) in 'apps' namespace
# - Vault JWT role (dashboard-service) with group-based access
Verification
After registration, verify the integration:
# Check Authentik provider and app
curl -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
http://localhost:7000/api/v3/core/applications/?slug=my-service | jq .
# Check K8s secret
kubectl get secret my-service-oidc -n my-ns -o yaml
# Check Vault JWT role
vault read auth/jwt/role/my-service
# Test OAuth login
# Browser: https://my-service.riotpiao.com/login
# Should redirect to Authentik → back to service with session
Full verification script:
bash k8s/talos-iam/verify_existing_oauth_integrations.sh
Notes
AUTHENTIK_SECRET_KEYis set-once. Rotating it invalidates all sessions, tokens, and encrypted fields in the database. Keep~/.authentik/.envsafe and backed up.- PostgreSQL holds everything that matters — users, providers, groups, tokens, certificates. It lives on an 8Gi Longhorn PVC. Redis is ephemeral by design.
- No ingress. Access is via port-forward, matching the rest of the homelab.
- OIDC signing keys (rotated by
provision_oidc.py --rotate) are separate from all.envcredentials. Rotation requires no client reconfiguration — Grafana and MinIO pick up the new public key from the JWKS endpoint automatically. - New app registration is automated via
register_oauth_app.pyand fully documented inOAUTH_APP_SETUP.md. Both manual and automated workflows are supported.