From f08fb2bb75272837264df4f412a55361a271890a Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Tue, 21 Jul 2026 16:33:53 -0700 Subject: [PATCH] fix(iam): register authentik-provision-job.yaml in kustomization + remove unsafe namespace transformer Root cause of the provisioning Job never appearing in-cluster despite being committed and pushed: k8s/security/iam/kustomization.yaml has an explicit resources: allowlist (not a plain directory scan) and the new file was never added to it, so ArgoCD's Kustomize build silently omitted every object in it - no error, no drift shown, iam-jobs just reported Synced/Healthy against a manifest set that never included the new Job/ConfigMap/RBAC at all. Also removed the top-level transformer. It would have force-rewritten metadata.namespace to iam on every resource in this kustomization, including authentik-provision-job.yaml's RoleBindings which deliberately target cicd/argocd/logging/storage (least-privilege access for the authentik-provisioner ServiceAccount to read/create Secrets in exactly those namespaces and no others). Every manifest in this directory already sets its own explicit namespace, so dropping the transformer is a no-op for the existing key-rotation-cronjob.yaml. Verified with apiVersion: v1 kind: ServiceAccount metadata: name: authentik-provisioner namespace: iam --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: authentik-provisioner rules: - apiGroups: - "" resources: - secrets verbs: - get - list - create - update - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: authentik-provisioner namespace: argocd roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: authentik-provisioner subjects: - kind: ServiceAccount name: authentik-provisioner namespace: iam --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: authentik-provisioner namespace: cicd roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: authentik-provisioner subjects: - kind: ServiceAccount name: authentik-provisioner namespace: iam --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: authentik-provisioner namespace: iam roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: authentik-provisioner subjects: - kind: ServiceAccount name: authentik-provisioner namespace: iam --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: authentik-provisioner namespace: logging roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: authentik-provisioner subjects: - kind: ServiceAccount name: authentik-provisioner namespace: iam --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: authentik-provisioner namespace: storage roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: authentik-provisioner subjects: - kind: ServiceAccount name: authentik-provisioner namespace: iam --- apiVersion: v1 data: authentik-provision.py: | #!/usr/bin/env python3 """ Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook). Creates/updates, in order: 1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile by default but NOT groups - required for ArgoCD RBAC group mapping and Grafana's role_attribute_path, both of which read a `groups` claim). 2. Groups: homelab-admins (is_superuser=true), grafana-admins. 3. User "rock": created if missing, always (re-)synced into both groups above. Password is generated once and only written to the k8s Secret rock-credentials (iam ns) the first time the user is created - re-runs never rotate an existing password. 4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd. Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc) if present, or generated once and written out (forgejo-oidc, oidc-secret) the first time. 5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and anyone else in that group) has guaranteed access regardless of each app's default visibility. Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80), authenticating with the bootstrap token. Everything is done with GET-then- create-or-patch so this can be re-run on every ArgoCD sync without duplicating or clobbering objects (PostSync hook, not a one-shot Job with hook-delete). kubectl is used only to read/write the small set of Secrets this script touches - it shells out rather than using the Python k8s client to keep the container image to stdlib Python + the kubectl binary, no pip installs. """ import json import os import secrets import string import subprocess import sys import urllib.error import urllib.request AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local" TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"] def api(method, path, data=None): url = f"{AUTHENTIK_URL}{path}" body = json.dumps(data).encode() if data is not None else None req = urllib.request.Request( url, data=body, method=method, headers={ "Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json", }, ) try: with urllib.request.urlopen(req, timeout=30) as resp: raw = resp.read() return resp.status, (json.loads(raw) if raw else {}) except urllib.error.HTTPError as e: raw = e.read() try: parsed = json.loads(raw) if raw else {} except json.JSONDecodeError: parsed = {"raw": raw.decode(errors="replace")} return e.code, parsed def die(msg): print(f"FATAL: {msg}", file=sys.stderr) sys.exit(1) def gen_secret(n=40): alphabet = string.ascii_letters + string.digits return "".join(secrets.choice(alphabet) for _ in range(n)) def kubectl_get_secret_key(namespace, name, key): """Returns decoded value, or None if the secret/key doesn't exist.""" p = subprocess.run( ["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"], capture_output=True, text=True, ) if p.returncode != 0 or not p.stdout.strip(): return None import base64 return base64.b64decode(p.stdout).decode() def kubectl_create_secret(namespace, name, literals: dict): """Idempotent: create-or-update via dry-run|apply, same pattern used elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh).""" args = ["kubectl", "-n", namespace, "create", "secret", "generic", name] for k, v in literals.items(): args += [f"--from-literal={k}={v}"] args += ["--dry-run=client", "-o", "yaml"] render = subprocess.run(args, capture_output=True, text=True) if render.returncode != 0: die(f"rendering secret {namespace}/{name}: {render.stderr}") apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout, capture_output=True, text=True) if apply.returncode != 0: die(f"applying secret {namespace}/{name}: {apply.stderr}") print(f" secret {namespace}/{name}: {apply.stdout.strip()}") def get_or_create(list_path, create_path, query, payload, patch_existing=None): status, res = api("GET", f"{list_path}?{query}") if status != 200: die(f"GET {list_path}?{query} -> {status} {res}") results = res.get("results", []) if results: obj = results[0] if patch_existing: status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing) if status not in (200, 201): die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}") return obj2 return obj status, obj = api("POST", create_path, payload) if status not in (200, 201): die(f"POST {create_path} -> {status} {obj}") return obj # ----------------------------------------------------------------------------- print("[1/5] Ensuring custom 'groups' scope mapping exists...") groups_mapping = get_or_create( "/api/v3/propertymappings/provider/scope/", "/api/v3/propertymappings/provider/scope/", "scope_name=groups", { "name": "homelab: groups claim", "scope_name": "groups", "expression": ( "return {\"groups\": [group.name for group in request.user.ak_groups.all()]}" ), }, ) GROUPS_MAPPING_PK = groups_mapping["pk"] # Fetch the standard openid/email/profile mapping pks (shipped by default). status, res = api("GET", "/api/v3/propertymappings/provider/scope/") by_scope = {m["scope_name"]: m["pk"] for m in res["results"]} SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK] status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent") AUTHORIZATION_FLOW_PK = res["results"][0]["pk"] status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow") INVALIDATION_FLOW_PK = res["results"][0]["pk"] status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true") SIGNING_KEY_PK = res["results"][0]["pk"] # ----------------------------------------------------------------------------- print("[2/5] Ensuring groups homelab-admins / grafana-admins exist...") homelab_admins = get_or_create( "/api/v3/core/groups/", "/api/v3/core/groups/", "name=homelab-admins", {"name": "homelab-admins", "is_superuser": True}, ) grafana_admins = get_or_create( "/api/v3/core/groups/", "/api/v3/core/groups/", "name=grafana-admins", {"name": "grafana-admins", "is_superuser": False}, ) # ----------------------------------------------------------------------------- print("[3/5] Ensuring user 'rock' exists with admin group membership...") status, res = api("GET", "/api/v3/core/users/?username=rock") rock_password = None if res.get("results"): rock = res["results"][0] status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", { "groups": [homelab_admins["pk"], grafana_admins["pk"]], "is_active": True, }) if status not in (200, 201): die(f"PATCH user rock -> {status} {rock}") print(" rock already exists, group membership synced (password unchanged)") else: rock_password = gen_secret(24) status, rock = api("POST", "/api/v3/core/users/", { "username": "rock", "name": "Rock", "is_active": True, "groups": [homelab_admins["pk"], grafana_admins["pk"]], "path": "users", "type": "internal", }) if status not in (200, 201): die(f"POST user rock -> {status} {rock}") status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/", {"password": rock_password}) if status not in (200, 204): die(f"set_password for rock -> {status} {pw_res}") kubectl_create_secret("iam", "rock-credentials", { "username": "rock", "password": rock_password, }) print(" rock created, credentials stored in iam/rock-credentials") # ----------------------------------------------------------------------------- print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...") SERVICES = { "grafana": { "client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"), "redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"], "launch_url": "https://grafana.riotpiao.com", "display_name": "Grafana", }, "minio": { "client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"), "redirect_uris": ["https://minio.riotpiao.com/oauth_callback"], "launch_url": "https://minio.riotpiao.com", "display_name": "MinIO", }, "forgejo": { # No secret exists yet for forgejo - generate + store on first run. "client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"), "generate_if_missing": True, "redirect_uris": [ "https://forgejo.riotpiao.com/user/oauth2/authentik/callback", "https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback", ], "launch_url": "https://forgejo.riotpiao.com", "display_name": "Forgejo", }, "argocd": { # oidc-secret uses hyphenated keys (client-id/client-secret) per # argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs. "client_secret_source": ("argocd", "oidc-secret", "client-secret"), "generate_if_missing": True, "extra_secret_literals": {"client-id": "argocd"}, "redirect_uris": ["https://argocd.riotpiao.com/auth/callback"], "launch_url": "https://argocd.riotpiao.com", "display_name": "Argo CD", }, } app_pks_for_binding = [] for name, cfg in SERVICES.items(): ns, secret_name, key = cfg["client_secret_source"] client_secret = kubectl_get_secret_key(ns, secret_name, key) if client_secret is None: if not cfg.get("generate_if_missing"): print(f" WARNING: {ns}/{secret_name} key {key} not found and " f"generate_if_missing not set for '{name}' - skipping provider/app") continue client_secret = gen_secret(40) literals = {key: client_secret} literals.update(cfg.get("extra_secret_literals", {})) kubectl_create_secret(ns, secret_name, literals) print(f" {name}: generated new client secret -> {ns}/{secret_name}") else: print(f" {name}: using existing client secret from {ns}/{secret_name}") provider = get_or_create( "/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/", f"name={name}", { "name": name, "client_id": name, "client_secret": client_secret, "client_type": "confidential", "authorization_flow": AUTHORIZATION_FLOW_PK, "invalidation_flow": INVALIDATION_FLOW_PK, "signing_key": SIGNING_KEY_PK, "property_mappings": SCOPE_PKS, "sub_mode": "hashed_user_id", "include_claims_in_id_token": True, "redirect_uris": [ {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"] ], }, # Keep the redirect_uris/mappings in sync on re-run, but never touch # client_secret again once created (that's the source of truth in the # k8s Secret, and re-sending it here is harmless/idempotent anyway). patch_existing={ "property_mappings": SCOPE_PKS, "redirect_uris": [ {"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"] ], }, ) application = get_or_create( "/api/v3/core/applications/", "/api/v3/core/applications/", f"slug={name}", { "name": cfg["display_name"], "slug": name, "provider": provider["pk"], "meta_launch_url": cfg["launch_url"], }, patch_existing={ "provider": provider["pk"], "meta_launch_url": cfg["launch_url"], }, ) app_pks_for_binding.append((name, application["pk"])) print(f" {name}: provider pk={provider['pk']} application pk={application['pk']}") # ----------------------------------------------------------------------------- print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...") for name, app_pk in app_pks_for_binding: get_or_create( "/api/v3/policies/bindings/", "/api/v3/policies/bindings/", f"target={app_pk}&group={homelab_admins['pk']}", { "target": app_pk, "group": homelab_admins["pk"], "order": 0, "enabled": True, }, ) print(f" {name}: homelab-admins bound") print("\nDone. Summary:") print(" groups: homelab-admins (superuser), grafana-admins") print(" user: rock -> homelab-admins + grafana-admins") print(f" apps: {', '.join(n for n, _ in app_pks_for_binding)}") if rock_password: print(" NOTE: rock's password was generated this run - see") print(" kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d") kind: ConfigMap metadata: name: authentik-provision-script namespace: iam --- apiVersion: batch/v1 kind: CronJob metadata: name: authentik-key-rotation namespace: iam spec: concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - command: - sh - -c - rustc /scripts/rotate_key.rs -o /tmp/rotate_key && /tmp/rotate_key env: - name: AUTHENTIK_BASE_URL value: http://authentik-server.iam.svc.cluster.local - name: AUTHENTIK_BOOTSTRAP_TOKEN valueFrom: secretKeyRef: key: AUTHENTIK_BOOTSTRAP_TOKEN name: authentik-key-rotation-token image: rust:1.82-slim name: rotate volumeMounts: - mountPath: /scripts name: script restartPolicy: OnFailure volumes: - configMap: name: key-rotation-script name: script schedule: 0 0 1 */3 * --- apiVersion: batch/v1 kind: Job metadata: annotations: argocd.argoproj.io/hook: PostSync argocd.argoproj.io/hook-delete-policy: BeforeHookCreation name: authentik-provision namespace: iam spec: backoffLimit: 3 template: spec: containers: - command: - /bin/sh - -c - | set -e echo "waiting for authentik-server..." until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do sleep 5 done echo "installing kubectl..." apk add --no-cache curl >/dev/null KVER=$(curl -sL https://dl.k8s.io/release/stable.txt) curl -sLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" chmod +x /usr/local/bin/kubectl echo "running provisioning script..." python3 /script/authentik-provision.py env: - name: AUTHENTIK_BOOTSTRAP_TOKEN valueFrom: secretKeyRef: key: AUTHENTIK_BOOTSTRAP_TOKEN name: authentik-secrets image: python:3.12-alpine name: provision securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL volumeMounts: - mountPath: /script name: script restartPolicy: Never securityContext: runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault serviceAccountName: authentik-provisioner volumes: - configMap: name: authentik-provision-script name: script ttlSecondsAfterFinished: 600 locally before pushing - confirms all 5 RoleBindings land in their correct distinct namespaces (iam/cicd/argocd/logging/storage) and every resource renders as valid YAML. --- k8s/security/iam/kustomization.yaml | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/k8s/security/iam/kustomization.yaml b/k8s/security/iam/kustomization.yaml index c97aed2..833d917 100644 --- a/k8s/security/iam/kustomization.yaml +++ b/k8s/security/iam/kustomization.yaml @@ -1,8 +1,19 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization -namespace: iam +# NOTE: no top-level `namespace:` transformer here (removed) - it used to +# force-rewrite metadata.namespace to "iam" on every resource in this +# kustomization, which was harmless while every manifest here only ever +# targeted the iam namespace itself. authentik-provision-job.yaml's +# RoleBindings deliberately target cicd/argocd/logging/storage (least- +# privilege access for the authentik-provisioner ServiceAccount to touch +# Secrets in those namespaces) - the namespace transformer would have +# silently rewritten all of them back to iam, breaking the RBAC. Every +# manifest in this directory already sets its own explicit +# metadata.namespace, so dropping the transformer changes nothing for the +# existing resources/. resources: - key-rotation-cronjob.yaml + - authentik-provision-job.yaml # authentik-migrations-job.yaml removed — redundant + broken. The authentik # `server` entrypoint runs migrations itself; this standalone job lacked the # authentik-secrets envFrom (Secret key missing) and always failed.