- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
TOPOLOGY CHANGE:
- All 3 control-plane nodes now schedulable (no NoSchedule taints)
- Pod distribution: ~59 on cp-1, ~21 on cp-2, ~23 on cp-3
- Better resource utilization across cluster
ADDED HARD RULE:
- Control-plane scheduling controlled via Terraform
- terraform.tfvars → allow_scheduling = true/false
- Never manual kubectl taint (Talos will revert)
- Workflow: terraform apply → talosctl apply-config
IMPLEMENTATION:
- Terraform: Set allow_scheduling=true for cp-2, cp-3
- Applied via talosctl --mode no-reboot (no disruption)
- Verified: kubectl get nodes shows no taints
ADDED:
- CloudNativePG (CNPG) Database Pattern section
- Explains shared 'app' user model (not per-app roles)
- Documents bootstrap.sh credential distribution pattern
- Working examples (Forgejo, Authentik)
- Prescriptive DO/DON'T guidance for new apps
FIXED:
- Storage topology: 3-node HA (not "sole Longhorn node")
- Verified: all 17 PVCs have replicas across all 3 nodes
- Updated last-modified date
This documents the architectural pattern established during CNPG refactor.
UNIFIED PATTERN: All apps follow same credential distribution
FORGEJO PATTERN (now universal):
1. CNPG creates ddb-cluster-app in ddb namespace (source)
2. bootstrap.sh copies to app namespaces (cicd, iam)
3. Apps reference local copy via secretKeyRef
4. No PostSync Jobs needed
CHANGES:
- bootstrap.sh: Copy ddb-cluster-app to iam namespace (like cicd)
- authentik-values.yaml: Reference local ddb-cluster-app via env vars
- Removed: sync-db-credentials PostSync Job (not needed)
- kustomization.yaml: Removed PostSync Job reference
BENEFITS:
✅ Same pattern as working Forgejo
✅ No complex PostSync Jobs
✅ bootstrap.sh handles setup for future clusters
✅ Simple secretKeyRef, no cross-namespace issues
✅ ArgoCD manages applications, not secrets
Database recreated with app owner (fresh migrations needed).
ARCHITECTURAL CHANGE: Align with CNPG design intent
BEFORE (Complex, broken):
- Per-app roles (authentik, temporal) with Database CR owner field
- Database CR doesn't transfer ownership properly
- Needed manual permission grants (PostSync Job)
- Apps couldn't create tables without grants from 'app' role
AFTER (Simple, works):
- All apps use shared 'app' bootstrap user
- Database CRs: owner: app (matches actual ownership)
- No permission grants needed (owner has full rights)
- Isolation via separate database names only
CHANGES:
- Database CRs: owner changed from app-specific to 'app'
- ddb-cluster.yaml: removed managed.roles section
- Deleted grant-schema-permissions PostSync Job
- Follows Forgejo pattern (already working this way)
MANUAL STEPS REQUIRED:
1. Update authentik-secrets: AUTHENTIK_POSTGRESQL__USER=app
2. Update temporal secrets: similar change
3. Recreate databases with app as owner
4. Restart applications
Benefits:
- Simpler architecture
- No permission grant complexity
- Aligns with CNPG single-cluster design
- Matches working Forgejo implementation
Fresh authentik deployment runs ~100 database migrations which takes 15-20
minutes. Previous startup probe failureThreshold of 60 (10 minutes) killed
the pod before migrations could complete, causing infinite restart loop.
Increased to 120 failures (20 minutes) to allow migrations to finish.
Fixes: nginx 503 due to pod never becoming Ready.
Authentik migrations need to CREATE SCHEMA (not just tables in public schema).
This requires GRANT CREATE ON DATABASE, not just schema-level permissions.
Added to PostSync Job:
- GRANT CREATE ON DATABASE authentik TO authentik
- GRANT CREATE ON DATABASE temporal TO temporal
- GRANT CREATE ON DATABASE temporal_visibility TO temporal
App user can grant these (it owns the databases).
Removes duplicate longhorn-kafka StorageClass managed by Kafka chart.
All applications now use single 'longhorn' StorageClass (3 replicas, Immediate binding).
Changes:
- Kafka chart: use 'longhorn' instead of 'longhorn-kafka'
- Delete Kafka StorageClass template (no longer needed)
- Update longhorn-storageclass.yaml to match deployed config (Immediate, not WaitForFirstConsumer)
Existing Kafka PVCs remain bound to old longhorn-kafka StorageClass (safe - no data loss).
New PVCs will use unified 'longhorn' StorageClass.
ROOT CAUSE: CNPG Database CR creates databases but doesn't grant schema
permissions to the owner role. Bootstrap DB owner 'app' retains CREATE
privilege on public schema, blocking authentik/temporal from creating tables.
SECURITY FIX: Removed insecure 'GRANT TO PUBLIC' from postInitApplicationSQL.
SOLUTION: PostSync Job connects as 'app' (DB owner) and grants schema
permissions to named roles (authentik, temporal) in their respective databases.
Runs after Database CRs reconcile, survives CNPG database recreation.
Pattern: Per-database grants via PostSync, not cluster-wide PUBLIC grants.
Adds SQL to postInitApplicationSQL granting schema permissions to PUBLIC.
Allows any role (authentik, temporal, etc) to create tables in databases.
For existing cluster: run SQL manually (done).
For future bootstrap: automatic via initdb.
Pattern for apps: Database CR + app-specific init Job optional (co-located).
CNPG Database CR creates DBs but doesn't grant schema permissions properly.
Database owner is 'app' instead of specified role (authentik, temporal).
PostSync Job grants ALL on schema public to both app and named roles,
ensuring applications can create tables. Runs after Database CRs reconcile.
Fixes: authentik InsufficientPrivilege error on migration.
Minio operator requires privileged securityContext. Without these labels,
StatefulSet stuck at 0/0 replicas (PodSecurity admission blocks pod creation).
CoreDNS rewrites *.riotpiao.com → ingress-nginx-controller but bootstrap
deployed as ingress-nginx-bootstrap-controller. Service alias makes both work.
- Add explicit tls block with riotpiao-com-tls secret
- Enables HTTPS access to https://forgejo.riotpiao.com
- Matches wildcard certificate (*.riotpiao.com)
The file comment mentioned TLS should be handled via default-ssl-certificate,
but explicit TLS blocks are needed for proper HTTPS routing.
Root cause: Longhorn refuses to schedule replicas on nodes without spec.name
field. talos-cp-1 was auto-discovered (has spec.name), but cp-2/cp-3 were
manually created CRDs without it.
Error: 'no node name provided to check node down or deleted'
Fix: Add spec.name matching metadata.name for both nodes.
Configure Homarr to use Authentik for OIDC authentication:
- AUTH_PROVIDERS: oidc,credentials (both SSO and local auth)
- AUTH_OIDC_ISSUER: Authentik endpoint
- CLIENT_ID/SECRET: from homarr-oidc secret
- Groups attribute for authorization
Allows users to sign in via Authentik SSO.
homarr-patches Application doesn't have SOPS support.
Secret is managed by sops-secrets Application instead.
Kustomization now only contains:
- fix-probes-job.yaml (PostSync hook)
Two fixes:
1. Added https://homarr-labs.github.io/charts to homelab AppProject sourceRepos
(ArgoCD rejected: "application repo is not permitted in project")
2. Removed env array from homarr-values.yaml
(Chart template error: "can't evaluate field AUTH_PROVIDERS in type interface {}")
Chart expects env as key-value object or doesn't support custom env at all.
Will configure env via post-deployment kubectl patch or Kustomize envFrom.
Allows Homarr Application to sync successfully.
ArgoCD was failing to query Authentik OIDC discovery endpoint with:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Root cause: ArgoCD's HTTP client doesn't properly trust the rootCA cert
even when specified in oidc.config.
Fixed by adding insecureSkipVerify: true to OIDC config. This is acceptable
for internal homelab with self-signed certificates.
Tested: ArgoCD SSO login via Authentik now works
Adds expand-replicas-job.yaml: PostSync hook Job that:
- Waits for all 3 Longhorn nodes to be Ready
- Patches every volume with numberOfReplicas < 3 to 3
- Runs idempotently on every longhorn-config sync (BeforeHookCreation
deletes previous job, so re-runs are safe)
This ensures existing 1-replica volumes (created before the HA setup) get
expanded automatically via GitOps, not via manual kubectl patch.
Why PostSync: needs to run AFTER the taint-toleration setting and Node CRDs
are applied, otherwise there aren't 3 nodes available yet and the expansion
would fail (Longhorn can't create replicas on nodes that don't exist).
All fixes applied and tested:
- SSO: Authentik OAuth2 grant_types fixed, all 4 services working
- Storage: Longhorn distributed across 3 nodes, 3-replica HA enabled
- Documented in SSO-AND-STORAGE-HA-COMPLETE.md
With Longhorn now running on all 3 control-plane nodes (commit be7881d),
Forgejo pods no longer need to be pinned to talos-cp-1. The gitea-shared-storage
PVC can attach on any node, and the scheduler will properly co-locate pod + volume
via WaitForFirstConsumer + 3-replica Longhorn volumes.
Removes the kubernetes.io/hostname: talos-cp-1 nodeSelector added in commit
dde4b60 (which was a workaround for single-node storage).
Changes:
- k8s/infrastructure/longhorn/longhorn-taint-toleration.yaml: new Setting
to tolerate node-role.kubernetes.io/control-plane:NoSchedule taint,
allowing Longhorn DaemonSet to run on cp-2/cp-3 (not just cp-1)
- k8s/infrastructure/longhorn/longhorn-nodes.yaml: explicit Node CRDs for
talos-cp-2 and talos-cp-3 (auto-discovery doesn't work when nodes have
taints; these define /var/lib/longhorn as the storage path)
- k8s/infrastructure/longhorn/longhorn-wffc-storageclass.yaml: bump
numberOfReplicas from 1→3 (true HA: each volume gets 3 copies across
3 nodes; if one node fails, 2 others still have the data)
- k8s/infrastructure/longhorn/kustomization.yaml: add new resources
Root cause: Longhorn was only running on talos-cp-1 (.213) because cp-2/cp-3
have the control-plane taint and Longhorn DaemonSet had no matching toleration.
Every workload with a PVC was forced to schedule on cp-1 (via nodeSelector or
implicit co-location with the storage), defeating the entire purpose of a 3-node
HA cluster.
With this fix:
- Longhorn manager runs on all 3 nodes
- Storage is replicated 3x (erasure-coded across nodes)
- Pods can schedule on any node without PVC attachment failures
- True HA: lose 1 node, cluster still serves all volumes
Adds missing CLIENT_SECRET env injection + nodeSelector constraint:
- k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET
from forgejo-oidc Secret (created by authentik-provision Job), and pin
pods to talos-cp-1 via nodeSelector (only node with Longhorn storage —
gitea-shared-storage PVC can't attach on cp-2/cp-3)
Root cause chain for 'Forgejo SSO not working':
1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers
2. Old provision script never set it → all providers had grant_types=[]
3. /authorize returned 'Invalid grant_type for provider' → all SSO broken
4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c)
+ successfully re-ran via iam-jobs Application sync
5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds
6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git
repo self-hosting), but missing the extraEnv block that was only in
k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected
All 4 OAuth2 providers now have correct grant_types=['authorization_code',
'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are
constrained to the storage node. SSO login flow should now work end-to-end.
CLAUDE.md was previously excluded from version control entirely (treated as
private local notes, with CLAUDE.example.md as the only git-tracked
counterpart). No longer justified - the file contains no secrets, just
architecture notes, private RFC1918 IPs, and operational lessons (same
sensitivity level as README.md, which is already tracked). Removing the
CLAUDE.md gitignore rule and committing it for the first time.
CLAUDE.md and the entire project-usage/ tree were written for a helmfile +
'core iam'/'core secrets' CLI workflow that has been fully retired - actual
practice is 100% ArgoCD app-of-apps GitOps (git commit -> push -> ArgoCD
sync), confirmed by an extended live debugging session that touched
Vault, MinIO, Temporal, Authentik provisioning, ingress-nginx, and
multiple ArgoCD Applications, none of which involved helmfile or core at
any point.
CLAUDE.md: replaced the helmfile-era assumptions with the actual GitOps
loop, and added a new 'GitOps / ArgoCD Gotchas' section capturing every
hard-won lesson from this session with live evidence for each:
- kustomization.yaml resources: allowlists silently dropping new files
- kustomization.yaml namespace: transformers clobbering cross-namespace
RBAC
- PreSync hooks deadlocking on same-Application RBAC dependencies
- ArgoCD hooks not being reconciled by selfHeal, requiring a genuinely
new sync operation to pick up fixes
- repo-server manifest caching
- repoURL port mismatches breaking every Application's sync
simultaneously when routed through an ingress-rewriting CoreDNS rule
- Bitnami's 2025 versioned-tag retirement
- apk-as-non-root permission failures
- Helm's lack of values.yaml schema validation (root cause of the
Temporal/PostgreSQL 'chart doesn't support this' misdiagnosis - it was
a schema mismatch between the pinned chart version and a newer
chart's values.yaml example, silently a no-op)
CLAUDE.example.md: fully rewritten as a sanitized, hardware-generic
template (explicit notice at top) - same lessons, genericized away from
this specific homelab's IPs/hostnames/secrets, intended to be reusable by
anyone running a similar bare-metal Talos + ArgoCD topology.
project-usage/gitops-workflow.md: new file - the accurate replacement for
'how do I actually deploy something' until the older helmfile-era docs in
this directory get a full rewrite (flagged as stale in CLAUDE.md's new
Documentation Map section rather than rewritten wholesale in this pass -
that's ~12 files, out of scope for this change).
Bitnami stopped publishing versioned image tags in 2025 - only 'latest' and
sha256-pinned digests remain for their free-tier images. Confirmed via
Docker Hub API before writing this fix: no '1.30' tag exists for
bitnami/kubectl, which caused an indefinite ImagePullBackOff (job stuck
'Running' with 0 pods able to start).
Switched to python:3.12-alpine + a stdlib urllib kubectl download, matching
the exact pattern already proven working in
k8s/security/iam/authentik-provision-job.yaml (which hit its own apk
permission problem on this same base image, now fixed the same way in
both places) - avoids depending on any third party's tagging policy.
PreSync hooks run BEFORE an Application's own normal (non-hook) resources
are synced. This Job's ServiceAccount/ClusterRole/RoleBindings are plain
resources in the same Application, so marking the Job PreSync created a
chicken-and-egg deadlock: confirmed live, the Job sat 'Running' for 14
minutes producing zero pods, with job-controller repeatedly logging
'serviceaccount temporal/temporal-db-secret-sync not found' - because that
ServiceAccount hadn't been created yet (it's created during the normal Sync
phase, which comes after PreSync).
Fixed to PostSync. This app (sync-wave 7) still fully completes - including
this hook - before the temporal Application (sync-wave 8) begins, so the
ordering guarantee we need (secret exists before Temporal's pods try to
mount it) is unaffected; only the intra-app hook-vs-normal-resource
ordering was wrong.
Job was crash-looping: 'apk add --no-cache curl' failed with Permission
denied - the container runs as non-root UID 1000 (securityContext.
runAsNonRoot: true), and both apk's working directories and /usr/local/bin
(where curl-downloaded kubectl was being written) are root-owned in the
python:3.12-alpine base image.
Replaced with a pure-Python download via urllib (stdlib, already a
dependency of this Job) writing to /tmp (world-writable) instead - no apk
install needed at all. PATH is extended to include /tmp before invoking the
provisioning script so authentik-provision.py's existing
subprocess.run(['kubectl', ...]) calls resolve it via normal PATH lookup,
no changes needed to the script itself.
Root cause: pinned to temporalio/helm-charts @ 0.74.0, which uses the OLD
flat persistence schema (server.config.persistence.<store>.driver/.sql),
NOT the datastores:-wrapped schema shown in the CURRENT chart's
values/values.postgresql.yaml example (that key was introduced in a later
major version). Our old values.yaml used the datastores: key, which doesn't
exist in 0.74.0 - Helm doesn't validate unknown keys, so it was silently a
no-op. persistence.default.driver / persistence.visibility.driver stayed at
their chart default ("cassandra", with empty hosts: []) the entire time,
regardless of anything nested under datastores:.
Verified before writing this fix: cloned temporalio/helm-charts, checked out
tag temporal-0.74.0 (exact pin), ran +
against our actual values.yaml - confirmed the rendered
schema-setup Job used CASSANDRA_HOST/temporal-cassandra-tool the whole time.
Re-rendered with the corrected flat schema - zero Cassandra references,
correct postgres12 pluginName/connectAddr wired to ddb-cluster-rw.
Also fixed two compounding no-ops found the same way:
- -> real keys are schema.setup.enabled /
schema.update.enabled / schema.createDatabase.enabled (jobs.autoSetup
doesn't exist anywhere in this chart's templates or values.yaml).
- cassandra.enabled was never actually set to false (stayed at chart
default true) - now explicitly false, along with mysql/elasticsearch/
prometheus/grafana (none of which we want).
Password wiring: existingSecret: temporal-db-role + secretKey: password,
pointing at the CNPG-generated Secret - avoids storing the DB password as
plaintext in this values file. Added a new temporal-db-secret-sync
Application (sync-wave 7, one before temporal's wave 8) with a PreSync hook
Job that copies that Secret from the ddb namespace into temporal (Secrets
are namespace-scoped; CNPG creates it in ddb, but Temporal's pods run in
temporal). Deliberately a standalone directory/Application rather than
folded into temporal/'s own kustomization.yaml, which has a The Temporal CLI manages, monitors, and debugs Temporal apps. It lets you run
a local Temporal Service, start Workflow Executions, pass messages to running
Workflows, inspect state, and more.
* Start a local development service:
`temporal server start-dev`
* View help: pass `--help` to any command:
`temporal activity complete --help`
Usage:
temporal [command]
Available Commands:
activity Operate on Activity Executions
batch Manage running batch jobs
completion Generate the autocompletion script for the specified shell
config Manage config files (EXPERIMENTAL)
env Manage environments
help Help about any command
operator Manage Temporal deployments
schedule Perform operations on Schedules
server Run Temporal Server
task-queue Manage Task Queues
worker Read or update Worker state
workflow Start, list, and operate on Workflows
Flags:
--client-connect-timeout duration
The client connection timeout. 0s means no timeout.
(default 0s)
--color string
Output coloring. Accepted values: always, never, auto.
(default "auto")
--command-timeout duration
The command execution timeout. 0s means no timeout.
(default 0s)
--config-file $CONFIG_PATH/temporalio/temporal.toml
File path to read TOML config from, defaults to
$CONFIG_PATH/temporalio/temporal.toml where
`$CONFIG_PATH` is defined as `$HOME/.config` on Unix,
`$HOME/Library/Application Support` on macOS, and
`%AppData%` on Windows.
--disable-config-env
If set, disables loading environment config from
environment variables.
--disable-config-file
If set, disables loading environment config from config file.
--env ENV
Active environment name (ENV). (default "default")
--env-file $HOME/.config/temporalio/temporal.yaml
Path to environment settings file. Defaults to
$HOME/.config/temporalio/temporal.yaml.
-h, --help
help for temporal
--log-format string
Log format. Accepted values: text, json. (default "text")
--log-level string
Log level. Default is "never" for most commands and
"warn" for "server start-dev". Accepted values: debug,
info, warn, error, never. (default "never")
--no-json-shorthand-payloads
Raw payload output, even if the JSON option was used.
-o, --output string
Non-logging data output format. Accepted values: text,
json, jsonl, none. (default "text")
--profile string
Profile to use for config file.
--time-format string
Time format. Accepted values: relative, iso, raw.
(default "relative")
-v, --version
version for temporal
Use "temporal [command] --help" for more information about a command. transformer that would silently rewrite the copy-job's ddb-scoped
RoleBinding back to temporal (same class of bug just fixed in
k8s/security/iam/kustomization.yaml).
Root cause of the provisioning Job never appearing in-cluster despite being
committed and pushed: k8s/security/iam/kustomization.yaml has an explicit
resources: allowlist (not a plain directory scan) and the new file was never
added to it, so ArgoCD's Kustomize build silently omitted every object in it
- no error, no drift shown, iam-jobs just reported Synced/Healthy against a
manifest set that never included the new Job/ConfigMap/RBAC at all.
Also removed the top-level transformer. It would have
force-rewritten metadata.namespace to iam on every resource in this
kustomization, including authentik-provision-job.yaml's RoleBindings which
deliberately target cicd/argocd/logging/storage (least-privilege access for
the authentik-provisioner ServiceAccount to read/create Secrets in exactly
those namespaces and no others). Every manifest in this directory already
sets its own explicit namespace, so dropping the transformer is a no-op for
the existing key-rotation-cronjob.yaml.
Verified with apiVersion: v1
kind: ServiceAccount
metadata:
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: authentik-provisioner
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: argocd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: cicd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: iam
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: logging
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: storage
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: v1
data:
authentik-provision.py: |
#!/usr/bin/env python3
"""
Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook).
Creates/updates, in order:
1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile
by default but NOT groups - required for ArgoCD RBAC group mapping and
Grafana's role_attribute_path, both of which read a `groups` claim).
2. Groups: homelab-admins (is_superuser=true), grafana-admins.
3. User "rock": created if missing, always (re-)synced into both groups above.
Password is generated once and only written to the k8s Secret
rock-credentials (iam ns) the first time the user is created - re-runs
never rotate an existing password.
4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd.
Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc)
if present, or generated once and written out (forgejo-oidc, oidc-secret)
the first time.
5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and
anyone else in that group) has guaranteed access regardless of each app's
default visibility.
Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80),
authenticating with the bootstrap token. Everything is done with GET-then-
create-or-patch so this can be re-run on every ArgoCD sync without duplicating
or clobbering objects (PostSync hook, not a one-shot Job with hook-delete).
kubectl is used only to read/write the small set of Secrets this script
touches - it shells out rather than using the Python k8s client to keep the
container image to stdlib Python + the kubectl binary, no pip installs.
"""
import json
import os
import secrets
import string
import subprocess
import sys
import urllib.error
import urllib.request
AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local"
TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"]
def api(method, path, data=None):
url = f"{AUTHENTIK_URL}{path}"
body = json.dumps(data).encode() if data is not None else None
req = urllib.request.Request(
url,
data=body,
method=method,
headers={
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else {})
except urllib.error.HTTPError as e:
raw = e.read()
try:
parsed = json.loads(raw) if raw else {}
except json.JSONDecodeError:
parsed = {"raw": raw.decode(errors="replace")}
return e.code, parsed
def die(msg):
print(f"FATAL: {msg}", file=sys.stderr)
sys.exit(1)
def gen_secret(n=40):
alphabet = string.ascii_letters + string.digits
return "".join(secrets.choice(alphabet) for _ in range(n))
def kubectl_get_secret_key(namespace, name, key):
"""Returns decoded value, or None if the secret/key doesn't exist."""
p = subprocess.run(
["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"],
capture_output=True, text=True,
)
if p.returncode != 0 or not p.stdout.strip():
return None
import base64
return base64.b64decode(p.stdout).decode()
def kubectl_create_secret(namespace, name, literals: dict):
"""Idempotent: create-or-update via dry-run|apply, same pattern used
elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh)."""
args = ["kubectl", "-n", namespace, "create", "secret", "generic", name]
for k, v in literals.items():
args += [f"--from-literal={k}={v}"]
args += ["--dry-run=client", "-o", "yaml"]
render = subprocess.run(args, capture_output=True, text=True)
if render.returncode != 0:
die(f"rendering secret {namespace}/{name}: {render.stderr}")
apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout,
capture_output=True, text=True)
if apply.returncode != 0:
die(f"applying secret {namespace}/{name}: {apply.stderr}")
print(f" secret {namespace}/{name}: {apply.stdout.strip()}")
def get_or_create(list_path, create_path, query, payload, patch_existing=None):
status, res = api("GET", f"{list_path}?{query}")
if status != 200:
die(f"GET {list_path}?{query} -> {status} {res}")
results = res.get("results", [])
if results:
obj = results[0]
if patch_existing:
status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing)
if status not in (200, 201):
die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}")
return obj2
return obj
status, obj = api("POST", create_path, payload)
if status not in (200, 201):
die(f"POST {create_path} -> {status} {obj}")
return obj
# -----------------------------------------------------------------------------
print("[1/5] Ensuring custom 'groups' scope mapping exists...")
groups_mapping = get_or_create(
"/api/v3/propertymappings/provider/scope/",
"/api/v3/propertymappings/provider/scope/",
"scope_name=groups",
{
"name": "homelab: groups claim",
"scope_name": "groups",
"expression": (
"return {\"groups\": [group.name for group in request.user.ak_groups.all()]}"
),
},
)
GROUPS_MAPPING_PK = groups_mapping["pk"]
# Fetch the standard openid/email/profile mapping pks (shipped by default).
status, res = api("GET", "/api/v3/propertymappings/provider/scope/")
by_scope = {m["scope_name"]: m["pk"] for m in res["results"]}
SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent")
AUTHORIZATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow")
INVALIDATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true")
SIGNING_KEY_PK = res["results"][0]["pk"]
# -----------------------------------------------------------------------------
print("[2/5] Ensuring groups homelab-admins / grafana-admins exist...")
homelab_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=homelab-admins",
{"name": "homelab-admins", "is_superuser": True},
)
grafana_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=grafana-admins",
{"name": "grafana-admins", "is_superuser": False},
)
# -----------------------------------------------------------------------------
print("[3/5] Ensuring user 'rock' exists with admin group membership...")
status, res = api("GET", "/api/v3/core/users/?username=rock")
rock_password = None
if res.get("results"):
rock = res["results"][0]
status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", {
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"is_active": True,
})
if status not in (200, 201):
die(f"PATCH user rock -> {status} {rock}")
print(" rock already exists, group membership synced (password unchanged)")
else:
rock_password = gen_secret(24)
status, rock = api("POST", "/api/v3/core/users/", {
"username": "rock",
"name": "Rock",
"is_active": True,
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"path": "users",
"type": "internal",
})
if status not in (200, 201):
die(f"POST user rock -> {status} {rock}")
status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/",
{"password": rock_password})
if status not in (200, 204):
die(f"set_password for rock -> {status} {pw_res}")
kubectl_create_secret("iam", "rock-credentials", {
"username": "rock",
"password": rock_password,
})
print(" rock created, credentials stored in iam/rock-credentials")
# -----------------------------------------------------------------------------
print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...")
SERVICES = {
"grafana": {
"client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"),
"redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"],
"launch_url": "https://grafana.riotpiao.com",
"display_name": "Grafana",
},
"minio": {
"client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"),
"redirect_uris": ["https://minio.riotpiao.com/oauth_callback"],
"launch_url": "https://minio.riotpiao.com",
"display_name": "MinIO",
},
"forgejo": {
# No secret exists yet for forgejo - generate + store on first run.
"client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"),
"generate_if_missing": True,
"redirect_uris": [
"https://forgejo.riotpiao.com/user/oauth2/authentik/callback",
"https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback",
],
"launch_url": "https://forgejo.riotpiao.com",
"display_name": "Forgejo",
},
"argocd": {
# oidc-secret uses hyphenated keys (client-id/client-secret) per
# argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs.
"client_secret_source": ("argocd", "oidc-secret", "client-secret"),
"generate_if_missing": True,
"extra_secret_literals": {"client-id": "argocd"},
"redirect_uris": ["https://argocd.riotpiao.com/auth/callback"],
"launch_url": "https://argocd.riotpiao.com",
"display_name": "Argo CD",
},
}
app_pks_for_binding = []
for name, cfg in SERVICES.items():
ns, secret_name, key = cfg["client_secret_source"]
client_secret = kubectl_get_secret_key(ns, secret_name, key)
if client_secret is None:
if not cfg.get("generate_if_missing"):
print(f" WARNING: {ns}/{secret_name} key {key} not found and "
f"generate_if_missing not set for '{name}' - skipping provider/app")
continue
client_secret = gen_secret(40)
literals = {key: client_secret}
literals.update(cfg.get("extra_secret_literals", {}))
kubectl_create_secret(ns, secret_name, literals)
print(f" {name}: generated new client secret -> {ns}/{secret_name}")
else:
print(f" {name}: using existing client secret from {ns}/{secret_name}")
provider = get_or_create(
"/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
f"name={name}",
{
"name": name,
"client_id": name,
"client_secret": client_secret,
"client_type": "confidential",
"authorization_flow": AUTHORIZATION_FLOW_PK,
"invalidation_flow": INVALIDATION_FLOW_PK,
"signing_key": SIGNING_KEY_PK,
"property_mappings": SCOPE_PKS,
"sub_mode": "hashed_user_id",
"include_claims_in_id_token": True,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
# Keep the redirect_uris/mappings in sync on re-run, but never touch
# client_secret again once created (that's the source of truth in the
# k8s Secret, and re-sending it here is harmless/idempotent anyway).
patch_existing={
"property_mappings": SCOPE_PKS,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
)
application = get_or_create(
"/api/v3/core/applications/", "/api/v3/core/applications/",
f"slug={name}",
{
"name": cfg["display_name"],
"slug": name,
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
patch_existing={
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
)
app_pks_for_binding.append((name, application["pk"]))
print(f" {name}: provider pk={provider['pk']} application pk={application['pk']}")
# -----------------------------------------------------------------------------
print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...")
for name, app_pk in app_pks_for_binding:
get_or_create(
"/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
f"target={app_pk}&group={homelab_admins['pk']}",
{
"target": app_pk,
"group": homelab_admins["pk"],
"order": 0,
"enabled": True,
},
)
print(f" {name}: homelab-admins bound")
print("\nDone. Summary:")
print(" groups: homelab-admins (superuser), grafana-admins")
print(" user: rock -> homelab-admins + grafana-admins")
print(f" apps: {', '.join(n for n, _ in app_pks_for_binding)}")
if rock_password:
print(" NOTE: rock's password was generated this run - see")
print(" kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d")
kind: ConfigMap
metadata:
name: authentik-provision-script
namespace: iam
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: authentik-key-rotation
namespace: iam
spec:
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- command:
- sh
- -c
- rustc /scripts/rotate_key.rs -o /tmp/rotate_key && /tmp/rotate_key
env:
- name: AUTHENTIK_BASE_URL
value: http://authentik-server.iam.svc.cluster.local
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-key-rotation-token
image: rust:1.82-slim
name: rotate
volumeMounts:
- mountPath: /scripts
name: script
restartPolicy: OnFailure
volumes:
- configMap:
name: key-rotation-script
name: script
schedule: 0 0 1 */3 *
---
apiVersion: batch/v1
kind: Job
metadata:
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
name: authentik-provision
namespace: iam
spec:
backoffLimit: 3
template:
spec:
containers:
- command:
- /bin/sh
- -c
- |
set -e
echo "waiting for authentik-server..."
until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do
sleep 5
done
echo "installing kubectl..."
apk add --no-cache curl >/dev/null
KVER=$(curl -sL https://dl.k8s.io/release/stable.txt)
curl -sLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl"
chmod +x /usr/local/bin/kubectl
echo "running provisioning script..."
python3 /script/authentik-provision.py
env:
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-secrets
image: python:3.12-alpine
name: provision
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /script
name: script
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
serviceAccountName: authentik-provisioner
volumes:
- configMap:
name: authentik-provision-script
name: script
ttlSecondsAfterFinished: 600 locally before pushing -
confirms all 5 RoleBindings land in their correct distinct namespaces
(iam/cicd/argocd/logging/storage) and every resource renders as valid YAML.
Adds k8s/security/iam/authentik-provision-job.yaml - a PostSync hook Job
(reruns every ArgoCD sync via hook-delete-policy: BeforeHookCreation) that
replaces the never-migrated setup_talos_iam.sh / provision_oidc.py workflow
(both referenced helmfile + a Python script that no longer exists in this
repo - OAuth was never actually provisioned since the ArgoCD migration).
Idempotently creates:
- Custom 'groups' OAuth2 scope mapping (Authentik doesn't ship one by
default; required for ArgoCD's RBAC groups claim and Grafana's
role_attribute_path, both of which read a groups claim from the token).
- Groups: homelab-admins (is_superuser), grafana-admins.
- User 'rock', member of both groups above - gets full Authentik superuser
access, ArgoCD role:admin via the existing
RBAC policy in argocd-values.yaml, and
Grafana Admin role via role_attribute_path. Password generated once,
stored in iam/rock-credentials (never rotated on re-run).
- OAuth2 providers + Applications for grafana, minio, forgejo, argocd.
Client secrets read from existing Secrets (grafana-oidc, minio-oidc) or
generated once and written out (forgejo-oidc, argocd's oidc-secret).
- PolicyBinding of homelab-admins -> every Application, guaranteeing rock
access regardless of each app's default visibility.
Also fixes forgejo-values.yaml: oauth2.CLIENT_ID was set but CLIENT_SECRET
was missing entirely (oauth2 login could never have worked). Added via
extraEnv -> GITEA__oauth2__CLIENT_SECRET sourced from the new forgejo-oidc
Secret, since the oauth2: values map can't reference a Secret inline.
RBAC: dedicated ServiceAccount + ClusterRole (secrets get/list/create/update/
patch only) bound via namespace-scoped RoleBindings in iam/cicd/argocd/
logging/storage - the only 5 namespaces this job ever touches, and the only
resource type it ever touches.
NOTE: MinIO's OIDC env vars were removed from minio-tenant.yaml earlier
(blocked IAM init because the provider/app didn't exist yet -> 404 on
discovery). Now that this job creates them, re-adding MinIO's OIDC config is
a safe follow-up in a separate change.
1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
Our Ingress objects intentionally omit per-rule tls: blocks (single wildcard
cert served via --default-ssl-certificate). nginx-ingress's normal ssl-redirect
setting only forces HTTPS 301 for hosts with an explicit TLS block on their own
Ingress resource - since none of ours do, plain http://*.riotpiao.com requests
were served directly instead of redirected, exposing every client-facing
console (forgejo, authentik, argocd, grafana, vault, etc.) over plaintext HTTP.
force-ssl-redirect forces the redirect globally regardless of per-ingress TLS
block presence. Verified fix works (tested via manual patch then reverted -
confirmed 308 redirects to https:// on forgejo/authentik/argocd) before
committing via GitOps.
Root cause of widespread 'Unknown' sync status / Skipping auto-sync across
almost every Application: CoreDNS rewrites forgejo.riotpiao.com to the nginx
ingress controller service (rewrite name forgejo.riotpiao.com -> ingress-nginx-
controller...), which only listens on 80/443, not 3000. Every git fetch from
argocd-repo-server to the :3000 repoURL was timing out (context deadline
exceeded), so ArgoCD couldn't compare desired vs live state for any app.
Fix: use https://forgejo.riotpiao.com (no port, TLS via nginx + wildcard cert)
consistent with the 'all external endpoints HTTPS' requirement. Verified git
smart-http response 200 on the new URL before committing.
- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator)
- minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled)
- minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking
plain-HTTP clients like Vault's S3 backend - this was the real cause of the
Vault S3 hang)
- argocd ingress: moved from namespace cicd -> argocd (service lives in argocd
namespace; ingress in wrong namespace can never route, was returning 503)
- removed duplicate kmsvc ingress (sqs namespace already has management-service
ingress with proper TLS block for same host/backend)
Audit method: cross-checked every ingress backend.service.{name,port} against
actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
Story Crater Bot