homelab/USAGE.md

Cluster Architecture at a Glance

Homelab is a 2-node bare-metal Kubernetes cluster deployed with Talos Linux, designed for self-hosted services, observability, and GitOps-ready CI/CD.

Deployment Stack (18 Helm releases)

Layer	Component	Namespace	Purpose
OS & Networking	Talos Linux v1.13.3	—	Immutable, declarative Linux
CNI	Cilium (eBPF)	kube-system	Advanced networking, no kube-proxy
Ingress	Nginx Ingress Controller	ingress-nginx	Reverse proxy, TLS termination
Certificates	cert-manager + homelab-ca	cert-manager	Self-signed CA, auto-renewal
Storage (Block)	Longhorn v1.7.0	longhorn-system	Persistent volumes, default StorageClass
Storage (Object)	MinIO (3-node, site-repl)	storage	S3-compatible, multi-AZ replication
Database	CloudNativePG (3 replicas)	ddb	PostgreSQL 16 + pgvector
IAM / OIDC	Authentik	iam	Federated OIDC provider for all services
Secrets	HashiCorp Vault	iam	KV secrets backend, JWT auth
Logs	Loki (SingleBinary)	logging	10-day retention, MinIO backend
Log Collection	Promtail (DaemonSet)	logging	Pod + kernel logs → Loki
Metrics	Prometheus + kube-state-metrics	monitoring	Time-series metrics, service discovery
Dashboards	Grafana	logging	Unified UI for Prometheus + Loki
Uptime Monitoring	Blackbox Exporter	monitoring	External endpoint probes
Git Forge	Forgejo (self-hosted)	cicd	Git server, OCI registry, Forgejo Actions
CI Runner	Forgejo Actions Runner	cicd	Privileged build pods, image push
CD	Argo CD	cicd	Pull-based GitOps, declarative deployments
Message Queue	Kafka (Strimzi KRaft mode)	sqs	SQS-like message queue service
Queue Redis	Redis	sqs	In-flight message tracking, dedup
Queue API	kmsvc (gRPC + REST)	sqs	Message service (SendMessage, ReceiveMessage, etc.)
Workflows	Temporal	temporal	Distributed workflow engine
Container UI	Portainer CE	dashboard	Pod/workload management UI
Terminal	Claude Terminal	dev-tools	Persistent dev environment (optional)
Config Reload	Reloader	reloader	Auto-reload pods on ConfigMap/Secret changes

Dependency Chain (Release Order)

cert-manager (root)
  ↓
  ├─→ ingress-nginx
  ├─→ cilium (network policies)
  └─→ cloudnative-pg
        ↓
        ├─→ authentik (DB: authentik)
        │     ├─→ vault (uses Authentik OIDC)
        │     ├─→ forgejo (OIDC login)
        │     ├─→ argocd (OIDC login)
        │     └─→ kmsvc (JWT auth)
        │
        └─→ story-crater-backend (DB: story_crater)
  
storage (MinIO 3-node)
  ├─→ loki (object backend)
  ├─→ vault (unseal keys bucket)
  └─→ monitoring (Prometheus scrape)

monitoring (Prometheus operator)
  ├─→ ingress-nginx (requires ServiceMonitor CRDs)
  ├─→ authentik (requires ServiceMonitor CRDs)
  ├─→ minio-az-a (requires ServiceMonitor CRDs)
  └─→ temporal (requires ServiceMonitor CRDs)

logging (Loki + Grafana)
  ├─→ promtail (pod log collection)
  └─→ grafana (dashboards)

sqs (Kafka + Message Queue)
  ├─→ strimzi-operator
  ├─→ kafka-cluster (KRaft mode, 3 brokers)
  ├─→ kmsvc-redis (in-flight tracking)
  ├─→ queue-crd (operator)
  └─→ management-service (gRPC/REST API)

---

Custom CLI — talos

Homelab cluster control CLI (core/). Manages cluster nodes and Vault secrets.

Secret path convention

All secrets live under cluster/<VARIABLE_NAME>. The field name is always the variable name itself (SCREAMING_SNAKE_CASE), matching the .env key. Example paths:

cluster/ANTHROPIC_API_KEY
cluster/AUTHENTIK_FORGEJO_CLIENT_ID
cluster/AUTHENTIK_ARGOCD_CLIENT_SECRET

talos put — write a secret to Vault

talos put cluster/VARIABLE_NAME VARIABLE_NAME="secret-value"
talos put cluster/FORGEJO_ADMIN_PASSWORD FORGEJO_ADMIN_PASSWORD="$FORGEJO_ADMIN_PASSWORD"

Field name = variable name — never value.

talos get — fetch a secret from Vault

talos get cluster/VARIABLE_NAME --key VARIABLE_NAME        # always specify --key
talos get cluster/FORGEJO_ADMIN_PASSWORD --key FORGEJO_ADMIN_PASSWORD
talos get cluster/VARIABLE_NAME --json                     # full secret as JSON

Note: talos get uses --key (long flag), not a positional arg — unlike talos secrets get.

vsource — load a .env into the shell

zsh function (lives in ~/.zshrc, not in the repo — can reference but cannot run directly). Empty .env values are fetched from Vault at cluster/<KEY>; hardcoded values pass through.

vsource            # loads .env in current directory
vsource .env.local # loads a specific file

.env format — leave secrets empty, vsource resolves them from Vault:

ANTHROPIC_API_KEY=          # fetched from cluster/ANTHROPIC_API_KEY
AUTHENTIK_ARGOCD_CLIENT_ID= # fetched from cluster/AUTHENTIK_ARGOCD_CLIENT_ID
DEBUG=true                  # hardcoded, passed through as-is

Typical workflow for a generated secret

# 1. Store immediately after generation (keeps secrets out of shell history)
talos put cluster/AUTHENTIK_FORGEJO_CLIENT_SECRET AUTHENTIK_FORGEJO_CLIENT_SECRET="<paste>"

# 2. Use via subshell when creating K8s secrets
kubectl create secret generic my-secret \
  --from-literal=client-secret="$(talos get cluster/AUTHENTIK_FORGEJO_CLIENT_SECRET --key AUTHENTIK_FORGEJO_CLIENT_SECRET)"

# 3. Or load into shell via vsource for helmfile/env-driven tools
vsource .env && helmfile apply

IAM Management (Federated OIDC, Phases 1–6 Complete)

Status: ✅ Fully deployed (2026-07-02). Single federated OIDC provider (talos-federation) handles all service auth.

Quick reference:

# View roles and capabilities
talos iam roles list && talos iam roles describe admin

# Service registry (Grafana, MinIO, Forgejo, etc.)
talos iam services list && talos iam services describe grafana

# Agents (admin-bot, ci-bot with auto-rotation)
talos iam agents list && talos iam agents rotate ci-bot

# Role bindings (user → role with TTL)
talos iam bindings grant [email protected] devops --expires 2026-12-31
talos iam bindings list

# Audit trail (90-day retention, 12 event types)
talos iam audit list && talos iam audit export --format json

# OIDC provider sync with Authentik
talos iam providers sync-authentik

See homelab/CLAUDE.md § IAM Management for full reference (roles, services, agents, bindings, audit, providers).

Vault paths: All IAM state stored under cluster/iam/{federation,roles,services,agents,bindings}.

CI/CD Image Registry Authentication (Forgejo + Runner)

After IAM Phase 6 changes: All image push/pull operations via CI runner require JWT token authentication through Authentik → Vault.

Push images to Forgejo registry:

# 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
export REGISTRY_TOKEN=$(talos get cluster/iam/agents/ci-bot --key token)

# 2. Authenticate docker/podman to Forgejo registry
docker login forgejo.riotpiao.homelab.com \
  --username ci-bot \
  --password "$REGISTRY_TOKEN"

# 3. Tag and push image
docker tag myapp:latest forgejo.riotpiao.homelab.com/rock/myapp:latest
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest

Pull images in runner (automatic):

# Inside .forgejo/workflows/*.yml, runner pulls via K8s ServiceAccount
# No explicit login needed — imagePullSecrets injected by runner pod
image: forgejo.riotpiao.homelab.com/rock/myapp:latest

Runner pod setup:

# ServiceAccount (in cicd namespace) has Vault JWT auth injected
kubectl get serviceaccount -n cicd forgejo-runner
kubectl describe sa forgejo-runner -n cicd

# ImagePullSecret auto-mounted from K8s secret:
kubectl get secret -n cicd | grep forgejo-registry

Vault paths for credentials:

cluster/iam/agents/ci-bot          # JWT token for push authentication
cluster/iam/agents/admin-bot       # Alternative admin agent (if needed)

---