Story Crater Bot
f08fb2bb75
Root cause of the provisioning Job never appearing in-cluster despite being
committed and pushed: k8s/security/iam/kustomization.yaml has an explicit
resources: allowlist (not a plain directory scan) and the new file was never
added to it, so ArgoCD's Kustomize build silently omitted every object in it
- no error, no drift shown, iam-jobs just reported Synced/Healthy against a
manifest set that never included the new Job/ConfigMap/RBAC at all.
Also removed the top-level transformer. It would have
force-rewritten metadata.namespace to iam on every resource in this
kustomization, including authentik-provision-job.yaml's RoleBindings which
deliberately target cicd/argocd/logging/storage (least-privilege access for
the authentik-provisioner ServiceAccount to read/create Secrets in exactly
those namespaces and no others). Every manifest in this directory already
sets its own explicit namespace, so dropping the transformer is a no-op for
the existing key-rotation-cronjob.yaml.
Verified with apiVersion: v1
kind: ServiceAccount
metadata:
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: authentik-provisioner
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- create
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: argocd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: cicd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: iam
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: logging
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: authentik-provisioner
namespace: storage
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: authentik-provisioner
subjects:
- kind: ServiceAccount
name: authentik-provisioner
namespace: iam
---
apiVersion: v1
data:
authentik-provision.py: |
#!/usr/bin/env python3
"""
Authentik OAuth provisioning - idempotent, safe to re-run (ArgoCD PostSync hook).
Creates/updates, in order:
1. A custom "groups" OAuth2 scope mapping (Authentik ships openid/email/profile
by default but NOT groups - required for ArgoCD RBAC group mapping and
Grafana's role_attribute_path, both of which read a `groups` claim).
2. Groups: homelab-admins (is_superuser=true), grafana-admins.
3. User "rock": created if missing, always (re-)synced into both groups above.
Password is generated once and only written to the k8s Secret
rock-credentials (iam ns) the first time the user is created - re-runs
never rotate an existing password.
4. OAuth2/OIDC providers + Applications for: grafana, minio, forgejo, argocd.
Client secrets are read from existing k8s Secrets (grafana-oidc, minio-oidc)
if present, or generated once and written out (forgejo-oidc, oidc-secret)
the first time.
5. PolicyBinding of homelab-admins -> every Application above, so "rock" (and
anyone else in that group) has guaranteed access regardless of each app's
default visibility.
Talks to Authentik over the in-cluster Service (authentik-server.iam.svc:80),
authenticating with the bootstrap token. Everything is done with GET-then-
create-or-patch so this can be re-run on every ArgoCD sync without duplicating
or clobbering objects (PostSync hook, not a one-shot Job with hook-delete).
kubectl is used only to read/write the small set of Secrets this script
touches - it shells out rather than using the Python k8s client to keep the
container image to stdlib Python + the kubectl binary, no pip installs.
"""
import json
import os
import secrets
import string
import subprocess
import sys
import urllib.error
import urllib.request
AUTHENTIK_URL = "http://authentik-server.iam.svc.cluster.local"
TOKEN = os.environ["AUTHENTIK_BOOTSTRAP_TOKEN"]
def api(method, path, data=None):
url = f"{AUTHENTIK_URL}{path}"
body = json.dumps(data).encode() if data is not None else None
req = urllib.request.Request(
url,
data=body,
method=method,
headers={
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else {})
except urllib.error.HTTPError as e:
raw = e.read()
try:
parsed = json.loads(raw) if raw else {}
except json.JSONDecodeError:
parsed = {"raw": raw.decode(errors="replace")}
return e.code, parsed
def die(msg):
print(f"FATAL: {msg}", file=sys.stderr)
sys.exit(1)
def gen_secret(n=40):
alphabet = string.ascii_letters + string.digits
return "".join(secrets.choice(alphabet) for _ in range(n))
def kubectl_get_secret_key(namespace, name, key):
"""Returns decoded value, or None if the secret/key doesn't exist."""
p = subprocess.run(
["kubectl", "-n", namespace, "get", "secret", name, "-o", f"jsonpath={{.data.{key}}}"],
capture_output=True, text=True,
)
if p.returncode != 0 or not p.stdout.strip():
return None
import base64
return base64.b64decode(p.stdout).decode()
def kubectl_create_secret(namespace, name, literals: dict):
"""Idempotent: create-or-update via dry-run|apply, same pattern used
elsewhere in this repo (setup_vault.sh, apply-vault-secrets.sh)."""
args = ["kubectl", "-n", namespace, "create", "secret", "generic", name]
for k, v in literals.items():
args += [f"--from-literal={k}={v}"]
args += ["--dry-run=client", "-o", "yaml"]
render = subprocess.run(args, capture_output=True, text=True)
if render.returncode != 0:
die(f"rendering secret {namespace}/{name}: {render.stderr}")
apply = subprocess.run(["kubectl", "apply", "-f", "-"], input=render.stdout,
capture_output=True, text=True)
if apply.returncode != 0:
die(f"applying secret {namespace}/{name}: {apply.stderr}")
print(f" secret {namespace}/{name}: {apply.stdout.strip()}")
def get_or_create(list_path, create_path, query, payload, patch_existing=None):
status, res = api("GET", f"{list_path}?{query}")
if status != 200:
die(f"GET {list_path}?{query} -> {status} {res}")
results = res.get("results", [])
if results:
obj = results[0]
if patch_existing:
status, obj2 = api("PATCH", f"{create_path}{obj['pk']}/", patch_existing)
if status not in (200, 201):
die(f"PATCH {create_path}{obj['pk']}/ -> {status} {obj2}")
return obj2
return obj
status, obj = api("POST", create_path, payload)
if status not in (200, 201):
die(f"POST {create_path} -> {status} {obj}")
return obj
# -----------------------------------------------------------------------------
print("[1/5] Ensuring custom 'groups' scope mapping exists...")
groups_mapping = get_or_create(
"/api/v3/propertymappings/provider/scope/",
"/api/v3/propertymappings/provider/scope/",
"scope_name=groups",
{
"name": "homelab: groups claim",
"scope_name": "groups",
"expression": (
"return {\"groups\": [group.name for group in request.user.ak_groups.all()]}"
),
},
)
GROUPS_MAPPING_PK = groups_mapping["pk"]
# Fetch the standard openid/email/profile mapping pks (shipped by default).
status, res = api("GET", "/api/v3/propertymappings/provider/scope/")
by_scope = {m["scope_name"]: m["pk"] for m in res["results"]}
SCOPE_PKS = [by_scope["openid"], by_scope["email"], by_scope["profile"], GROUPS_MAPPING_PK]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent")
AUTHORIZATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/flows/instances/?slug=default-provider-invalidation-flow")
INVALIDATION_FLOW_PK = res["results"][0]["pk"]
status, res = api("GET", "/api/v3/crypto/certificatekeypairs/?has_key=true")
SIGNING_KEY_PK = res["results"][0]["pk"]
# -----------------------------------------------------------------------------
print("[2/5] Ensuring groups homelab-admins / grafana-admins exist...")
homelab_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=homelab-admins",
{"name": "homelab-admins", "is_superuser": True},
)
grafana_admins = get_or_create(
"/api/v3/core/groups/", "/api/v3/core/groups/",
"name=grafana-admins",
{"name": "grafana-admins", "is_superuser": False},
)
# -----------------------------------------------------------------------------
print("[3/5] Ensuring user 'rock' exists with admin group membership...")
status, res = api("GET", "/api/v3/core/users/?username=rock")
rock_password = None
if res.get("results"):
rock = res["results"][0]
status, rock = api("PATCH", f"/api/v3/core/users/{rock['pk']}/", {
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"is_active": True,
})
if status not in (200, 201):
die(f"PATCH user rock -> {status} {rock}")
print(" rock already exists, group membership synced (password unchanged)")
else:
rock_password = gen_secret(24)
status, rock = api("POST", "/api/v3/core/users/", {
"username": "rock",
"name": "Rock",
"is_active": True,
"groups": [homelab_admins["pk"], grafana_admins["pk"]],
"path": "users",
"type": "internal",
})
if status not in (200, 201):
die(f"POST user rock -> {status} {rock}")
status, pw_res = api("POST", f"/api/v3/core/users/{rock['pk']}/set_password/",
{"password": rock_password})
if status not in (200, 204):
die(f"set_password for rock -> {status} {pw_res}")
kubectl_create_secret("iam", "rock-credentials", {
"username": "rock",
"password": rock_password,
})
print(" rock created, credentials stored in iam/rock-credentials")
# -----------------------------------------------------------------------------
print("[4/5] Ensuring OAuth2 providers + applications for grafana/minio/forgejo/argocd...")
SERVICES = {
"grafana": {
"client_secret_source": ("logging", "grafana-oidc", "GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET"),
"redirect_uris": ["https://grafana.riotpiao.com/login/generic_oauth"],
"launch_url": "https://grafana.riotpiao.com",
"display_name": "Grafana",
},
"minio": {
"client_secret_source": ("storage", "minio-oidc", "MINIO_IDENTITY_OPENID_CLIENT_SECRET"),
"redirect_uris": ["https://minio.riotpiao.com/oauth_callback"],
"launch_url": "https://minio.riotpiao.com",
"display_name": "MinIO",
},
"forgejo": {
# No secret exists yet for forgejo - generate + store on first run.
"client_secret_source": ("cicd", "forgejo-oidc", "CLIENT_SECRET"),
"generate_if_missing": True,
"redirect_uris": [
"https://forgejo.riotpiao.com/user/oauth2/authentik/callback",
"https://forgejo.riotpiao.com/user/oauth2/openidconnect/callback",
],
"launch_url": "https://forgejo.riotpiao.com",
"display_name": "Forgejo",
},
"argocd": {
# oidc-secret uses hyphenated keys (client-id/client-secret) per
# argocd-values.yaml's `$oidc-secret:client-id` / `:client-secret` refs.
"client_secret_source": ("argocd", "oidc-secret", "client-secret"),
"generate_if_missing": True,
"extra_secret_literals": {"client-id": "argocd"},
"redirect_uris": ["https://argocd.riotpiao.com/auth/callback"],
"launch_url": "https://argocd.riotpiao.com",
"display_name": "Argo CD",
},
}
app_pks_for_binding = []
for name, cfg in SERVICES.items():
ns, secret_name, key = cfg["client_secret_source"]
client_secret = kubectl_get_secret_key(ns, secret_name, key)
if client_secret is None:
if not cfg.get("generate_if_missing"):
print(f" WARNING: {ns}/{secret_name} key {key} not found and "
f"generate_if_missing not set for '{name}' - skipping provider/app")
continue
client_secret = gen_secret(40)
literals = {key: client_secret}
literals.update(cfg.get("extra_secret_literals", {}))
kubectl_create_secret(ns, secret_name, literals)
print(f" {name}: generated new client secret -> {ns}/{secret_name}")
else:
print(f" {name}: using existing client secret from {ns}/{secret_name}")
provider = get_or_create(
"/api/v3/providers/oauth2/", "/api/v3/providers/oauth2/",
f"name={name}",
{
"name": name,
"client_id": name,
"client_secret": client_secret,
"client_type": "confidential",
"authorization_flow": AUTHORIZATION_FLOW_PK,
"invalidation_flow": INVALIDATION_FLOW_PK,
"signing_key": SIGNING_KEY_PK,
"property_mappings": SCOPE_PKS,
"sub_mode": "hashed_user_id",
"include_claims_in_id_token": True,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
# Keep the redirect_uris/mappings in sync on re-run, but never touch
# client_secret again once created (that's the source of truth in the
# k8s Secret, and re-sending it here is harmless/idempotent anyway).
patch_existing={
"property_mappings": SCOPE_PKS,
"redirect_uris": [
{"matching_mode": "strict", "url": u} for u in cfg["redirect_uris"]
],
},
)
application = get_or_create(
"/api/v3/core/applications/", "/api/v3/core/applications/",
f"slug={name}",
{
"name": cfg["display_name"],
"slug": name,
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
patch_existing={
"provider": provider["pk"],
"meta_launch_url": cfg["launch_url"],
},
)
app_pks_for_binding.append((name, application["pk"]))
print(f" {name}: provider pk={provider['pk']} application pk={application['pk']}")
# -----------------------------------------------------------------------------
print("[5/5] Binding homelab-admins to every application (guaranteed access for rock)...")
for name, app_pk in app_pks_for_binding:
get_or_create(
"/api/v3/policies/bindings/", "/api/v3/policies/bindings/",
f"target={app_pk}&group={homelab_admins['pk']}",
{
"target": app_pk,
"group": homelab_admins["pk"],
"order": 0,
"enabled": True,
},
)
print(f" {name}: homelab-admins bound")
print("\nDone. Summary:")
print(" groups: homelab-admins (superuser), grafana-admins")
print(" user: rock -> homelab-admins + grafana-admins")
print(f" apps: {', '.join(n for n, _ in app_pks_for_binding)}")
if rock_password:
print(" NOTE: rock's password was generated this run - see")
print(" kubectl -n iam get secret rock-credentials -o jsonpath='{.data.password}' | base64 -d")
kind: ConfigMap
metadata:
name: authentik-provision-script
namespace: iam
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: authentik-key-rotation
namespace: iam
spec:
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- command:
- sh
- -c
- rustc /scripts/rotate_key.rs -o /tmp/rotate_key && /tmp/rotate_key
env:
- name: AUTHENTIK_BASE_URL
value: http://authentik-server.iam.svc.cluster.local
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-key-rotation-token
image: rust:1.82-slim
name: rotate
volumeMounts:
- mountPath: /scripts
name: script
restartPolicy: OnFailure
volumes:
- configMap:
name: key-rotation-script
name: script
schedule: 0 0 1 */3 *
---
apiVersion: batch/v1
kind: Job
metadata:
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
name: authentik-provision
namespace: iam
spec:
backoffLimit: 3
template:
spec:
containers:
- command:
- /bin/sh
- -c
- |
set -e
echo "waiting for authentik-server..."
until wget -q -O /dev/null http://authentik-server.iam.svc.cluster.local/-/health/ready/ 2>/dev/null; do
sleep 5
done
echo "installing kubectl..."
apk add --no-cache curl >/dev/null
KVER=$(curl -sL https://dl.k8s.io/release/stable.txt)
curl -sLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl"
chmod +x /usr/local/bin/kubectl
echo "running provisioning script..."
python3 /script/authentik-provision.py
env:
- name: AUTHENTIK_BOOTSTRAP_TOKEN
valueFrom:
secretKeyRef:
key: AUTHENTIK_BOOTSTRAP_TOKEN
name: authentik-secrets
image: python:3.12-alpine
name: provision
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /script
name: script
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
serviceAccountName: authentik-provisioner
volumes:
- configMap:
name: authentik-provision-script
name: script
ttlSecondsAfterFinished: 600 locally before pushing -
confirms all 5 RoleBindings land in their correct distinct namespaces
(iam/cicd/argocd/logging/storage) and every resource renders as valid YAML.
Homelab Kubernetes Cluster
A bare-metal three-node Kubernetes cluster running Talos Linux with 18 Helm releases across 22 namespaces. Includes distributed storage (MinIO + Longhorn), full observability (Prometheus + Grafana + Loki), federated SSO (Authentik OIDC), secrets management (Vault), CI/CD (Forgejo + Argo CD), messaging (Kafka + kmsvc), and workflow orchestration (Temporal).
Use Cases & Architecture
Why this stack?
This homelab replicates production-grade cloud-native infrastructure on bare metal, enabling:
- Learning & Prototyping — Test distributed systems patterns (HA databases, event-driven messaging, GitOps workflows) before deploying to cloud
- Self-Hosted Services — Run applications (Story Crater, etc.) with zero cloud lock-in; full control over data, compliance, and networking
- Infrastructure as Code — Git-driven cluster state via Helmfile + Forgejo Actions + Argo CD; every change is auditable and reproducible
- Observability Sandbox — Experiment with Prometheus metrics, Loki log aggregation, and custom Grafana dashboards at scale
Typical workflow:
Developer pushes to Forgejo (git forge)
↓
Forgejo Actions CI runs tests + builds OCI image
↓
Image pushed to Forgejo registry (private, on-cluster)
↓
Argo CD detects deployment repo change (pull-based GitOps)
↓
New pods roll out; Grafana alerts on errors/latency
↓
Temporal workflows coordinate long-running operations (e.g., async jobs)
↓
Kafka queues decouple services (fire-and-forget messaging)
↓
All logs + metrics centralized in Grafana for debugging
Architecture principles:
- Immutable OS — Talos Linux (no SSH, declarative machine configs)
- No external dependencies — All data stored locally (MinIO, CloudNativePG, Longhorn)
- High availability — 3-replica databases, multi-node storage, cross-AZ readiness (on bare metal: cross-rack affinity)
- Federated identity — Single Authentik OIDC provider for all services (Grafana, MinIO, Forgejo, Argo CD)
- Secrets at rest — Vault + encrypted etcd; credentials never in logs or ConfigMaps
- Infrastructure-as-code — Every service deployed via Helmfile; one
helmfile applyrecovers from total failure
Quick Start — Deploying the Cluster
1. Bootstrap Talos Nodes
Bootstrap each Talos node with your cluster schematic (see CLAUDE.md or README step 1–8).
2. Set Up Secrets
All secrets are managed via environment variables sourced from .env (gitignored). The helmfile template expands them at deploy time.
Step 1: Copy the template
cp .env.example .env
Step 2: Populate required secrets
Edit .env and fill in cluster configuration. See .env.example for all options:
# Cluster configuration
CLUSTER_DOMAIN=riotpiao.com # Your cluster domain
POSTGRES_HOST=ddb-cluster-rw.ddb.svc.cluster.local
MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
KAFKA_BOOTSTRAP=kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092
REDIS_ADDR=kmsvc-redis-master.sqs.svc.cluster.local:6379
# Service credentials (generate with: openssl rand -hex 32)
MINIO_ROOT_PASSWORD=<random>
GRAFANA_ADMIN_PASSWORD=<random>
AUTHENTIK_SECRET_KEY=<random>
AUTHENTIK_BOOTSTRAP_PASSWORD=<random>
AUTHENTIK_PG_PASSWORD=<random>
Step 3: Load and deploy
# Load .env into current shell
vsource .env
# Preview all changes before deployment
helmfile diff
# Deploy the entire stack
helmfile apply
3. Verify Deployment
# Check all pods are running
kubectl get pods -A
# Confirm key services are ready
kubectl wait deploy/authentik-server -n iam --for=condition=Available --timeout=300s
kubectl wait deploy/grafana -n logging --for=condition=Available --timeout=300s
# Access Grafana
make pf-grafana # localhost:3000 (login: admin / GRAFANA_ADMIN_PASSWORD)
4. First-Time Access
Default Credentials:
- Authentik: https://authentik.$(CLUSTER_DOMAIN) → login:
akadmin/AUTHENTIK_BOOTSTRAP_PASSWORD(change immediately) - Grafana: https://grafana.$(CLUSTER_DOMAIN) → login:
admin/GRAFANA_ADMIN_PASSWORD - MinIO: https://minio.$(CLUSTER_DOMAIN) → login:
MINIO_ROOT_USER/MINIO_ROOT_PASSWORD - Argo CD: https://argocd.$(CLUSTER_DOMAIN) → login via Authentik OIDC
- Forgejo: https://forgejo.$(CLUSTER_DOMAIN) → login via Authentik OIDC
Next Steps:
- Change default passwords in each service
- Configure OIDC redirects (see
k8s/talos-iam/for details) - Set up GitOps: push infrastructure to Forgejo, configure Argo CD
- Review dashboards in Grafana (Prometheus + Loki)
Architecture
192.168.1.0/24 (LAN)
│
┌──────────────────────┼──────────────────────┐
│ │ │
192.168.1.* 192.168.1.* 192.168.1.*
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ talos-cp-1 │ │ talos-worker-1 │ │ talos-worker-2 │
│ Control-Plane │ │ Worker │ │ Worker │
│ + Workloads │ │ (Storage) │ │ (Storage) │
│ (az-a) │ │ (az-b) │ │ (az-c) │
├────────────────┤ ├────────────────┤ ├────────────────┤
│ Pods: │ │ Pods: │ │ Pods: │
│ • ingress-nginx│ │ • kube-system │ │ • kube-system │
│ • authentik │ │ • storage │ │ • storage │
│ • vault │ │ └─ minio-2 │ │ └─ minio-3 │
│ • logging │ │ │ │ │
│ ├─ loki │ │ │ │ │
│ ├─ promtail │ │ │ │ │
│ └─ grafana │ │ │ │ │
│ • monitoring │ │ │ │ │
│ ├─ prom │ │ │ │ │
│ └─ blackbox │ │ │ │ │
│ • storage │ │ │ │ │
│ └─ minio-1 │ │ │ │ │
│ • cicd │ │ │ │ │
│ ├─ forgejo │ │ │ │ │
│ ├─ argocd │ │ │ │ │
│ └─ runner │ │ │ │ │
│ • sqs │ │ • sqs │ │ • sqs │
│ ├─ kafka-0 │ │ ├─ kafka-1 │ │ ├─ kafka-2 │
│ └─ kmsvc │ │ └─ redis │ │ │
│ • ddb │ │ • ddb │ │ • ddb │
│ └─ postgres-0│ │ └─ postgres-1│ │ └─ postgres-2│
│ • temporal │ │ • temporal │ │ • temporal │
│ ├─ server │ │ ├─ cassandra │ │ ├─ cassandra │
│ └─ cassandra │ │ │ -1 │ │ │ -2 │
│ -0 │ │ └─ (replica) │ │ └─ (replica) │
│ • llm │ │ │ │ │
│ └─ ollama │ │ │ │ │
└────────────────┘ └────────────────┘ └────────────────┘
Replication & Fault Tolerance:
Data Layer:
• MinIO: minio-1 ↔ minio-2 ↔ minio-3 (3-way active-active S3)
• PostgreSQL: postgres-0 ↔ postgres-1 ↔ postgres-2 (primary + 2 standbys, HA streaming replication)
• Kafka: kafka-0 ↔ kafka-1 ↔ kafka-2 (3 brokers, RF=3, min-ISR=2, cross-AZ)
• Temporal: cassandra-0 ↔ cassandra-1 ↔ cassandra-2 (3-node distributed)
• Loki: loki → MinIO (chunks stored in s3://loki-chunks, 10-day retention)
Single-Replica Services (Protected by PodDisruptionBudget minAvailable=1):
• Observability: Grafana, Prometheus, Loki (Recreate strategy for RWO PVCs)
• IAM: Authentik, Vault (Recreate strategy for RWO PVCs)
• CI/CD: Argo CD server/repo-server, Forgejo (Recreate strategy for RWO PVCs)
• LLM: Ollama
Networking:
Remote access: UR_OWN.duckdns.org → home IP → cp-1
Stack
| Layer | Technology | Namespace | Purpose |
|---|---|---|---|
| OS | Talos Linux v1.13.3 | — | Immutable, Kubernetes-native OS |
| Kubernetes | v1.36.1 | — | Container orchestration |
| CNI | Cilium (eBPF) | kube-system | Networking, replaces kube-proxy |
| Ingress | Nginx Ingress Controller | ingress-nginx | Reverse proxy, hostname-based routing |
| Block Storage | Longhorn v1.7.0 | longhorn-system | Default StorageClass |
| Object Store | MinIO (multi-AZ) | storage | S3-compatible, site-replicated across az-a/az-b |
| IAM / SSO | Authentik | iam | OIDC provider for Grafana, MinIO, Forgejo, Argo CD |
| Secret Store | HashiCorp Vault | iam | KV secrets backend, JWT auth via Authentik |
| Git Forge (planned) | Forgejo | forge | Git server, built-in OCI registry, Actions CI |
| CI Runner (planned) | Forgejo Actions + DinD | cicd | Privileged build pod; images pushed to Forgejo OCI |
| CD (planned) | Argo CD | argocd | Pull-based GitOps; never holds kubeconfig in CI |
| Log Backend | Loki (SingleBinary) | logging | 10-day retention, backed by MinIO |
| Log Collector | Promtail (DaemonSet) | logging | Scrapes pod logs + Talos journal |
| Metrics | kube-prometheus-stack | monitoring | Prometheus + node-exporter + kube-state-metrics |
| Log/Metrics UI | Grafana | logging | Dashboards for Loki + Prometheus |
| Cluster UI | Portainer CE | dashboard | Container/workload management UI |
| LLM Inference | Ollama | llm | Local LLM model serving (open-source models) |
Repository Structure
homelab/
├── helmfile.yaml # Single source of truth — deploys everything
├── .env.example # Required env vars template (copy to .env, gitignored)
│
├── cluster-config/ # Talos + Kubernetes bootstrap
│ ├── cilium-values.yaml
│ ├── longhorn_bootstrap.sh
│ ├── controlplane.yaml # gitignored — contains secrets
│ ├── worker-1.yaml # gitignored
│ ├── secrets.yaml # gitignored
│ ├── kubeconfig # gitignored
│ └── talosconfig # gitignored
│
├── k8s/
│ ├── ingress/ # Nginx Ingress Controller + all Ingress rules
│ │ ├── nginx-values.yaml
│ │ └── ingress.yaml
│ ├── storage/ # MinIO multi-AZ object store
│ │ ├── minio-az-a-values.yaml
│ │ ├── minio-az-b-values.yaml
│ │ ├── minio-az-a-pvc.yaml
│ │ ├── minio-service.yaml
│ │ ├── minio-legacy-alias.yaml
│ │ └── minio-replication-job.yaml
│ ├── logging/ # Observability stack
│ │ ├── loki-values.yaml
│ │ ├── promtail-values.yaml
│ │ └── grafana-values.yaml
│ ├── monitoring/ # Prometheus stack + alerting + Grafana dashboards-as-code
│ │ ├── prometheus-values.yaml
│ │ ├── blackbox-exporter-values.yaml # Active uptime probes (feeds Service Availability dashboard)
│ │ ├── ingress-alerts.yaml # PrometheusRule: ingress 5xx rate, p95 latency
│ │ └── dashboards/ # ConfigMaps picked up live by Grafana's sidecar
│ │ ├── service-availability.yaml # Uptime probes + cert expiry (operator glance)
│ │ ├── service-golden-signals.yaml # Latency & Golden Signals (ingress RED)
│ │ ├── service-internals.yaml # Per-service deep-dive (MinIO/Forgejo/Argo CD/Vault/Longhorn/certs)
│ │ ├── kube-controller-health.yaml # API server RED + kube-state-metrics controller-health proxy
│ │ ├── hardware-overview.yaml # Per-node CPU/mem/disk/network/load summary
│ │ └── control-plane-logs.yaml # kube-system + add-on logs (Loki)
│ ├── portainer/ # Portainer CE
│ │ └── portainer-values.yaml
│ ├── talos-iam/ # Authentik + Vault IAM
│ │ ├── authentik-values.yaml
│ │ ├── vault-values.yaml
│ │ ├── setup_vault.sh # One-time Vault init (not replaced by Helmfile)
│ │ └── provision_oidc.py # Authentik OIDC provisioning
│ ├── coredns/ # CoreDNS hostname rewrites (in-cluster DNS)
│ │ └── coredns-configmap.yaml
│ ├── talos-ci-cd/ # CI/CD stack (planned — not yet applied)
│ │ ├── talos_version_control.html # Implementation plan + build runbook
│ │ ├── forgejo-values.yaml # Forgejo Helm values (gitea-charts/gitea)
│ │ ├── argocd-values.yaml # Argo CD Helm values
│ │ └── charts/forgejo-runner/ # Local Helm chart for the Actions runner
│ │ ├── Chart.yaml
│ │ ├── values.yaml
│ │ └── templates/
│ │ ├── deployment.yaml # Runner + DinD sidecar, Recreate strategy
│ │ ├── pvc.yaml # runner-reg (1 Gi) + runner-dind (30 Gi)
│ │ └── networkpolicy.yaml # Egress: forge ns + DNS + internet only
│ └── duckdns/ # DuckDNS DDNS updater CronJob
│
├── talos-cli/ # Rust CLI for Vault secret access
├── project_context.md # Authoritative live-state reference
├── refine_cluster.md # Known-issues runbook
└── LOG.md # Append-only change journal
Access
Discover Ingress LoadBalancer IP
# Find the external IP assigned by Cilium LB-IPAM
kubectl get svc -n ingress-nginx ingress-nginx
# Example output:
# LoadBalancer IP: 192.168.1.160 (Cilium LB-IPAM assignment)
Add to /etc/hosts on every client machine (Mac/Linux):
# WireGuard access (remote — via talos-cp-1)
10.6.0.1 grafana.riotpiao.com authentik.riotpiao.com vault.riotpiao.com minio.riotpiao.com prometheus.riotpiao.com portainer.riotpiao.com longhorn.riotpiao.com loki.riotpiao.com forgejo.riotpiao.com temporal.riotpiao.com temporal-grpc.riotpiao.com kmsvc.riotpiao.com
# LAN access (on the home network — use actual LoadBalancer IP from above)
192.168.1.160 grafana.riotpiao.com authentik.riotpiao.com vault.riotpiao.com minio.riotpiao.com prometheus.riotpiao.com portainer.riotpiao.com longhorn.riotpiao.com loki.riotpiao.com forgejo.riotpiao.com temporal.riotpiao.com temporal-grpc.riotpiao.com kmsvc.riotpiao.com
Note: 192.168.1.160 is an example Cilium LB-IPAM assignment. Verify with kubectl get svc -n ingress-nginx ingress-nginx.
There is no real DNS wildcard for *.riotpiao.com — every hostname must be added to /etc/hosts explicitly (as above) before it resolves. Adding a new Ingress host doesn't make it reachable by itself; add the line too.
kubectl Context
Two contexts exist in cluster-config/kubeconfig, pointed at the same cluster over different paths:
| Context | Server | Use when |
|---|---|---|
admin@homelab-cluster |
192.168.1.213:6443 (LAN) |
On the home network |
admin@homelab-cluster-1 |
10.6.0.1:6443 (WireGuard) |
Remote / off-LAN |
If kubectl commands hang or refuse the connection, switch: kubectl config use-context admin@homelab-cluster-1.
Then access services at:
| Service | URL | Credentials |
|---|---|---|
| Grafana | http://grafana.riotpiao.com | admin / GRAFANA_ADMIN_PASSWORD or Authentik SSO |
| Authentik | http://authentik.riotpiao.com | akadmin / see .env |
| Vault | http://vault.riotpiao.com | root token / see setup_vault.sh output |
| MinIO console | http://minio.riotpiao.com | MINIO_ROOT_USER / MINIO_ROOT_PASSWORD |
| Prometheus | http://prometheus.riotpiao.com | no auth |
| Portainer | http://portainer.riotpiao.com | set on first visit |
| Longhorn | http://longhorn.riotpiao.com | no auth |
| Forgejo (planned) | https://forgejo.forge.riotpiao.com | rock / FORGEJO_ADMIN_PASSWORD, or Authentik SSO |
| Argo CD (planned) | kubectl port-forward -n argocd svc/argocd-server 8080:443 |
Authentik SSO (admins only) |
Grafana → "Homelab" folder has the operator dashboards (sidecar-loaded from k8s/monitoring/dashboards/, no restart needed on change):
- Service Availability & Certificate Expiration — uptime probes + cert-manager expiry
- Latency & Golden Signals — ingress request rate/error %/p50-p99 latency
- Kube-Controller Health — API server RED metrics + kube-state-metrics controller-health signals
- Hardware Statistics — per-node CPU/mem/disk/network/load
- Service Internals — per-service deep-dive (MinIO/Forgejo/Argo CD/Vault/Longhorn)
Deploy
# 1. Install helmfile (once)
brew install helmfile
# 2. Set credentials
cp .env.example .env
# edit .env with your passwords
# 3. Deploy everything
helmfile apply
# Deploy a single stack
helmfile apply -l namespace=logging
helmfile apply -l name=grafana
helmfile apply -l namespace=ingress-nginx
# Preview changes before applying
helmfile diff
Bootstrap Order (fresh cluster)
1. Provision nodes: talosctl apply-config (make apply-cp / apply-worker-new)
2. Bootstrap Kubernetes: talosctl bootstrap
3. Install Cilium: helm install cilium -f cluster-config/cilium-values.yaml
4. Install Longhorn: bash cluster-config/longhorn_bootstrap.sh
5. Deploy everything else: helmfile apply
6. Vault init (one-time): bash k8s/talos-iam/setup_vault.sh
7. Authentik OIDC: python3 k8s/talos-iam/provision_oidc.py
8. Label worker: kubectl label node talos-worker-1 node-role.kubernetes.io/worker=
# ── CI/CD (planned — run after step 8) ─────────────────────────────────────
9. Private CA + TLS: see k8s/talos-ci-cd/talos_version_control.html §12 Block 0
10. Deploy Forgejo + runner: helmfile apply -l name=forgejo && helmfile apply -l name=forgejo-runner
11. Authentik SSO for CI: Forgejo + Argo CD OIDC (§12 Block 1.5)
12. Talos node CA trust: talosctl patch machineconfig (§12 Block 2)
13. Deploy Argo CD: helmfile apply -l name=argocd (§12 Block 4)
14. Wire deploy repo: argocd app create + push first manifests (§12 Block 4)
IAM & Auth Flow
Authentik is the central OIDC identity provider. Vault stores secrets and delegates authentication back to Authentik.
User / core-cli
│
│ OAuth2 / OIDC
▼
Authentik (authentik.riotpiao.com)
├── grafana app → Grafana OIDC login (group → Admin/Viewer role)
├── minio app → MinIO OIDC login (group → readwrite/readonly policy)
├── vault-browser → Vault UI OIDC login / `vault login -method=oidc`
└── core-cli-shell → CLI device code flow (public client, no secret)
│
│ JWKS endpoint for JWT validation
▼
HashiCorp Vault (vault.riotpiao.com)
├── auth/jwt — core-cli authenticates with device code JWT
├── auth/oidc — browser/UI login via Authentik
└── secret/ — KV v2: mcp/*, cluster/*, cloud/*
core CLI device code login:
core secrets login # prints URL + code → approve in browser → Vault token cached
core put cluster/DUCKDNS_TOKEN DUCKDNS_TOKEN="abc" # field name = variable name, never `value`
One-time IAM setup (after helmfile apply):
# 1. Provision OIDC apps and groups in Authentik
GRAFANA_URL=http://grafana.riotpiao.com \
MINIO_URL=http://minio.riotpiao.com \
python3 k8s/talos-iam/provision_oidc.py
# 2. Init Vault, wire JWT + OIDC auth, seed secrets
bash k8s/talos-iam/setup_vault.sh
CoreDNS hostname rewrites (k8s/coredns/coredns-configmap.yaml) ensure in-cluster pods (Grafana, Vault, Forgejo runner, Argo CD) resolve internal hostnames to cluster services, avoiding hairpin NAT through LB IPs.
CI/CD Pipeline (planned)
Implementation plan & exact build commands:
k8s/talos-ci-cd/talos_version_control.html
Components
| Component | Helm chart | Namespace | Notes |
|---|---|---|---|
| Forgejo | gitea-charts/gitea (Forgejo image override) |
forge |
Git + OCI registry + Actions engine; SQLite on Longhorn PVC; strategy: Recreate |
| Forgejo runner | local chart charts/forgejo-runner |
cicd |
DinD sidecar; PodSecurity privileged; NetworkPolicy fenced |
| Argo CD | argo/argo-cd |
argocd |
Pull-based CD; single replica; no external ingress (port-forward only) |
Pipeline flow
Developer
│
│ git push
▼
Forgejo (forge ns) ─── webhook ───► Runner (cicd ns)
│ │
│ SSO login (Authentik OIDC) │ ACTIONS_RUNTIME_TOKEN → git checkout
▼ │ ci-registry-token (ci-bot) → docker push → Forgejo OCI
Forgejo UI / Argo CD UI │ ci-deploy-token (ci-bot) → git commit → rock/deploy
│
Forgejo (rock/deploy repo) ◄─────────────┘
│
│ argocd-bot token (repo:read, poll every 3 min)
▼
Argo CD (argocd ns)
│
│ kubectl apply (cluster-admin ServiceAccount — never in CI)
▼
K8s workloads (images from Forgejo OCI)
Key security decisions
- No kubeconfig in CI. The runner can only push commits + OCI images. Argo CD bridges the gap autonomously.
- Scoped machine credentials.
ci-bottokens are narrowly scoped:package:writefor OCI,repo:writeonrock/deployonly. A compromised runner cannot read other repos or call the K8s API. - Private CA TLS. Forgejo self-terminates HTTPS with a homelab CA (EC P-256, 10-year). The CA cert is distributed to Talos nodes via
machineconfigpatch and to the runner via K8s Secret.ca.keynever enters the cluster. - Authentik SSO for humans. All interactive logins (Forgejo UI, Argo CD UI) route through Authentik.
homelab-adminsgroup → Forgejo admin + Argo CDrole:admin;homelab-devsgroup → Forgejo user + no Argo CD access. - Forgejo LB IP pinned. Cilium LB-IPAM annotation
io.cilium/lb-ipam-ips: <LB_IP>fixes the Forgejo LoadBalancer IP so the TLS SAN and DNS entries never need updating. Configure ink8s/talos-ci-cd/forgejo-values.yaml.
Workflow file location
Forgejo Actions uses GitHub Actions syntax. Workflow files live in .forgejo/workflows/ in each source repo:
rock/source/
└── .forgejo/
└── workflows/
├── ci.yml # build + test + push OCI image
└── cd.yml # on: push to main → bump image tag in rock/deploy
Log Data Flow
Pods / Talos journal (both nodes)
│
Promtail (DaemonSet, all nodes) reads /var/log/pods + /var/log/journal
│
▼
Loki (logging ns) indexes + compacts, 10-day retention
│ stores chunks via S3
▼
MinIO frontend service minio.storage.svc.cluster.local:9000
(active-active, round-robin)
│
minio-az-a (cp-1) ↔ minio-az-b (worker-1) ↔ minio-az-c (worker-2)
3-way site replication (bidirectional, automatic)
│
Grafana (logging ns) queries Loki + Prometheus via dashboards
│
Nginx Ingress → grafana.riotpiao.com browser access
Example Applications & Workloads
This cluster runs production-like applications and infrastructure services:
Story Crater Backend
Type: Distributed message-driven application
Namespace: story-crater-backend
Architecture:
- gRPC server + REST gateway (Envoy)
- PostgreSQL database (story_crater in CloudNativePG cluster)
- Kafka topic consumers (via kmsvc message queue)
- Temporal workflow integration for long-running operations
- Prometheus metrics (processed messages, latency, errors)
- Grafana dashboard (message throughput, queue depth, LLM token usage)
Typical flow:
POST /api/messages → gRPC handler
↓
Publish to Kafka topic (Story Crater queue)
↓
Message consumer processes async (may trigger LLM inference)
↓
Results stored in PostgreSQL + emitted as event
↓
Prometheus increments counters (story_crater_messages_handled_total)
↓
Grafana renders message throughput + duration histograms
Infrastructure Services (Essential)
| Service | Purpose | Namespace | Example Use |
|---|---|---|---|
| Authentik | OIDC identity provider | iam | User login, group management, SSO for Grafana/MinIO/Forgejo |
| Vault | Secrets backend | iam | Database passwords, API keys, JWT token validation |
| CloudNativePG | PostgreSQL 3-replica cluster | ddb | Auth database (Authentik), app database (Story Crater) |
| Loki | Log aggregation | logging | Centralize pod logs, Talos kernel logs (10-day retention) |
| Prometheus | Metrics collection | monitoring | Scrape kube-state-metrics, kubelet, ServiceMonitors (every 30s) |
| Grafana | Observability dashboards | logging | Query Prometheus + Loki, alert on latency/error spikes |
| Kafka + kmsvc | Message queue | sqs | Decouple services, async job processing, at-least-once delivery |
| MinIO | S3-compatible object store | storage | Loki log chunks backend, Vault unseal keys, config backups |
| Longhorn | Persistent block storage | longhorn-system | All PVCs (Postgres replicas, Kafka broker disks, MinIO) |
Development Services (Optional)
| Service | Purpose | Namespace | Example Use |
|---|---|---|---|
| Forgejo | Self-hosted git + OCI registry | cicd | Version control, CI Actions runner, private Docker images |
| Argo CD | Pull-based GitOps | cicd | Continuous deployment (deployment repo → Kubernetes) |
| Temporal | Workflow orchestration | temporal | Schedule long-running jobs, retry logic, state machines |
| Portainer | Container management UI | dashboard | Pod inspection, image management, quick debugging |
Monitoring Example: Dashboard Walk-Through
Open Grafana → Homelab folder → "Story Crater Backend — Service Overview":
Row A — Availability & Golden Signals
- Requests/sec (blue = success, red = errors)
- Error rate % (goal: < 0.1%)
- p50/p95/p99 latency (goal: p99 < 500ms)
- Alert: If error rate > 1% for 5 min, page on-call
Row B — Resource Usage
- CPU (request/limit)
- Memory (request/limit)
- Restart count (goal: 0; alerts if > 2)
Row C — Domain-Specific Metrics (Story Crater only)
- Messages processed/sec (bucketed by status: success, dlq, retry)
- Queue depth (Kafka partitions lag)
- Dedup window retention (FIFO redelivery tracking)
- LLM inference tokens used/sec
- External API call latency (e.g., OpenAI)
Row D — Logs
- Live Loki panel: filter by pod + search for errors
- Example:
{namespace="story-crater-backend"} | json | level="error"
Row E — Alert Status (if SLO defined)
- Burn rate (if consuming SLO budget)
- Example: "30-day availability SLO = 99.5%; current burn rate = 0.2x"
Row F — Related Dashboards
- Link to Kafka dashboard (queue depth)
- Link to PostgreSQL dashboard (story_crater DB)
- Link to Temporal dashboard (workflow execution times)
Adding Hardware to the Cluster
Step 1 — Get the Talos image
The image must include the same extensions as the existing nodes (iscsi-tools + util-linux-tools).
Download from the Image Factory using the cluster's schematic ID:
| Format | Use case | URL |
|---|---|---|
| ISO | USB boot (recommended for bare metal) | https://factory.talos.dev/image/613e1592.../v1.13.3/metal-amd64.iso |
| RAW disk image | Write directly to drive via another machine | https://factory.talos.dev/image/613e1592.../v1.13.3/metal-amd64.raw.xz |
| PXE / iPXE | Network boot — no USB needed | https://factory.talos.dev/image/613e1592.../v1.13.3/kernel-amd64 |
Full schematic ID:
613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
Option A — USB Bootable (recommended)
curl -Lo talos-worker.iso \
"https://factory.talos.dev/image/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245/v1.13.3/metal-amd64.iso"
sudo dd if=talos-worker.iso of=/dev/sdX bs=4M status=progress && sync
Option B — In-Memory (diskless / RAM boot via PXE)
Talos runs entirely from RAM. Useful for temporary nodes or hardware where you don't want to touch the existing OS.
# Boot via PXE pointing to:
# Kernel: https://factory.talos.dev/image/613e1592.../v1.13.3/kernel-amd64
# Initrd: https://factory.talos.dev/image/613e1592.../v1.13.3/initramfs-amd64.xz
# Cmdline: talos.platform=metal
Note: in-memory nodes lose state on reboot. Not suitable for Longhorn storage nodes.
Option C — Direct Disk Image (headless / remote)
xz -d talos-worker.raw.xz
sudo dd if=talos-worker.raw of=/dev/sda bs=4M status=progress && sync
Step 2 — Discover hardware in maintenance mode
# Scan your LAN for the new node in maintenance mode
nmap -sn 192.168.1.0/24
# Note: Replace with your actual subnet (e.g., 10.0.1.0/24)
# Discover available disks on the node
talosctl --nodes <maintenance-ip> --talosconfig cluster-config/talosconfig disks --insecure
Step 3 — Prepare the worker config
cp cluster-config/worker-1.yaml cluster-config/worker-N.yaml
Edit exactly these four fields:
| Field | Value |
|---|---|
machine.network.hostname |
talos-worker-N |
machine.network.interfaces[0].addresses |
192.168.1.16N/24 |
machine.install.disk |
disk path from Step 2 |
machine.nodeLabels.topology.kubernetes.io/zone |
az-N |
Step 4 — Apply config
make apply-worker-new N=<num> WN_IP=<maintenance-ip>
Step 5 — Persist the worker IP
# Substitute <WN_MAINTENANCE_IP> with the actual IP discovered in Step 2
echo 'export W<N>_IP=192.168.1.<last-octet>' >> ~/.zshrc && source ~/.zshrc
Step 6 — Set the node-role label
kubectl label node talos-worker-N node-role.kubernetes.io/worker=
Step 7 — Verify
kubectl get nodes -w
kubectl describe node talos-worker-N | grep -A10 Labels
Step 8 — What automatically extends to the new node
| Service | Behaviour |
|---|---|
| Cilium | New pod scheduled automatically |
| Promtail | DaemonSet — starts immediately |
| Nginx Ingress | DaemonSet — starts immediately, port 80/443 available on new node |
| Longhorn | Detects new node, available for replica scheduling |
| MinIO / Loki / Grafana | Stay on existing node (single-replica Deployments) |