homelab/USAGE.md

Cluster Architecture at a Glance

Homelab is a 3-node bare-metal Kubernetes cluster deployed with Talos Linux, designed for self-hosted services, observability, and GitOps-ready CI/CD.

Node Topology (3 control-plane HA, since 2026-07-20)

Node	IP	Zone	Role	Scheduling	Storage
talos-cp-1	192.168.1.213	az-a	control-plane	schedulable (runs all workloads)	sole Longhorn node (sdb/sdc/sdd)
talos-cp-2	192.168.1.163	az-b	control-plane	dedicated (NoSchedule)	none
talos-cp-3	192.168.1.166	az-c	control-plane	dedicated (NoSchedule)	none

• etcd has 3 voting members peering over the LAN (cluster.etcd.advertisedSubnets: 192.168.1.0/24 — without it Talos may advertise on the WireGuard IP and new members hang as learners). Tolerates 1 node loss.
• Only talos-cp-1 runs workloads and holds storage, so stateful services are single-instance (e.g. CNPG ddb-cluster = 1 instance). The kube-apiserver endpoint is single-homed to .213 (no VIP yet).
• Scheduling is declarative: allowSchedulingOnControlPlanes: true + per-node machine.nodeTaints re-adds the control-plane taint on the dedicated nodes only.

Deployment Stack (18 Helm releases)

Layer	Component	Namespace	Purpose
OS & Networking	Talos Linux v1.13.3	—	Immutable, declarative Linux
CNI	Cilium (eBPF)	kube-system	Advanced networking, no kube-proxy
Ingress	Nginx Ingress Controller	ingress-nginx	Reverse proxy, TLS termination
Certificates	cert-manager + homelab-ca	cert-manager	Self-signed CA, auto-renewal
Storage (Block)	Longhorn v1.7.0	longhorn-system	Persistent volumes, default StorageClass
Storage (Object)	MinIO (3-node, site-repl)	storage	S3-compatible, multi-AZ replication
Database	CloudNativePG (1 instance)	ddb	PostgreSQL 16 + pgvector (single-node; see topology)
IAM / OIDC	Authentik	iam	Federated OIDC provider for all services
Secrets	HashiCorp Vault	iam	KV secrets backend, JWT auth
Logs	Loki (SingleBinary)	logging	10-day retention, MinIO backend
Log Collection	Promtail (DaemonSet)	logging	Pod + kernel logs → Loki
Metrics	Prometheus + kube-state-metrics	monitoring	Time-series metrics, service discovery
Dashboards	Grafana	logging	Unified UI for Prometheus + Loki
Uptime Monitoring	Blackbox Exporter	monitoring	External endpoint probes
Git Forge	Forgejo (self-hosted)	cicd	Git server, OCI registry, Forgejo Actions
CI Runner	Forgejo Actions Runner	cicd	Privileged build pods, image push
CD	Argo CD	cicd	Pull-based GitOps, declarative deployments
Message Queue	Kafka (Strimzi KRaft mode)	sqs	SQS-like message queue service
Queue Redis	Redis	sqs	In-flight message tracking, dedup
Queue API	kmsvc (gRPC + REST)	sqs	Message service (SendMessage, ReceiveMessage, etc.)
Workflows	Temporal	temporal	Distributed workflow engine
Container UI	Portainer CE	dashboard	Pod/workload management UI
Terminal	Claude Terminal	dev-tools	Persistent dev environment (optional)
Config Reload	Reloader	reloader	Auto-reload pods on ConfigMap/Secret changes

Dependency Chain (Release Order)

cert-manager (root)
  ↓
  ├─→ ingress-nginx
  ├─→ cilium (network policies)
  └─→ cloudnative-pg
        ↓
        ├─→ authentik (DB: authentik)
        │     ├─→ vault (uses Authentik OIDC)
        │     ├─→ forgejo (OIDC login)
        │     ├─→ argocd (OIDC login)
        │     └─→ kmsvc (JWT auth)
        │
        └─→ story-crater-backend (DB: story_crater)
  
storage (MinIO 3-node)
  ├─→ loki (object backend)
  ├─→ vault (unseal keys bucket)
  └─→ monitoring (Prometheus scrape)

monitoring (Prometheus operator)
  ├─→ ingress-nginx (requires ServiceMonitor CRDs)
  ├─→ authentik (requires ServiceMonitor CRDs)
  ├─→ minio-az-a (requires ServiceMonitor CRDs)
  └─→ temporal (requires ServiceMonitor CRDs)

logging (Loki + Grafana)
  ├─→ promtail (pod log collection)
  └─→ grafana (dashboards)

sqs (Kafka + Message Queue)
  ├─→ strimzi-operator
  ├─→ kafka-cluster (KRaft mode, 3 brokers)
  ├─→ kmsvc-redis (in-flight tracking)
  ├─→ queue-crd (operator)
  └─→ management-service (gRPC/REST API)

---

Custom CLI — core

Homelab cluster control CLI (source: ~/workplace/core/). Manages cluster nodes, Authentik IAM, and Vault secrets.

Setup:

make cli                     # builds + installs to ~/.local/bin/core
core auth login-oob         # authenticate with Authentik (OOB flow)
core nodes                  # verify cluster access

Full documentation: See ~/workplace/core/USAGE.md

---

Authentication

OAuth2 Out-of-Band (OOB) Login

Browser-based login with manual code entry (recommended).

export CORE_CLI_SECRET="cyoVr96FB9oeq3o64DUG0BmoVzMPsOTIWxVEd8ZdTezKrEZYwrKpIRkOwrDQNEtF6QJyNUPH4mjr9jWokQY7KVBWX1KVUXFyyhgAHTqWRRWAYUJ8r3H35wLFiTfn5KsV"
core auth login-oob

Token expires in 8 hours. Check status:

core auth status   # show expiry
core auth clear    # force re-auth on next command

---

Cluster Management

core nodes                    # list cluster nodes
core status <ip>              # Talos node overview
core services <ip>            # list Talos services
core logs <ip>                # stream kubelet logs
core log-svc <ip> <svc>       # logs for specific service (etcd, kubelet, etc.)
core pods clean               # delete Failed/Evicted/Terminating pods

kubectl Context (LAN vs. WireGuard)

cluster-config/kubeconfig has two contexts pointed at the same cluster: admin@homelab-cluster (LAN, 192.168.1.213:6443) and admin@homelab-cluster-1 (WireGuard, 10.6.0.1:6443). If kubectl/core nodes hangs or refuses the connection, you're likely off-LAN — switch contexts:

core config kube-list             # list contexts
core config kube-use admin@homelab-cluster-1   # switch to WireGuard path

Known gap: core config use <talos-context> (the combined talosctl+kubectl switch) only maps to admin@homelab-cluster today — its WireGuard mapping (home-cluster-wire-guard) is stale, that kubectl context doesn't exist. Use core config kube-use admin@homelab-cluster-1 directly until that's fixed.

---

Secret Management (Vault)

All secrets live under cluster/<VARIABLE_NAME>. Field name = variable name (SCREAMING_SNAKE_CASE).

Write Secret to Vault

core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."
core put cluster/FORGEJO_ADMIN_PASSWORD FORGEJO_ADMIN_PASSWORD="secret123"

Key rule: Field name must match variable name — never value=.

Fetch Secret from Vault

core get cluster/ANTHROPIC_API_KEY              # default field
core get cluster/AUTHENTIK_ARGOCD_CLIENT_SECRET # full value

List All Secrets

core secrets list

Load into Shell (vsource)

zsh function in ~/.zshrc — fetches empty .env values from Vault:

# .env format
ANTHROPIC_API_KEY=              # fetched from Vault
AUTHENTIK_ARGOCD_CLIENT_ID=     # fetched from Vault
DEBUG=true                      # hardcoded, passed through

# Usage
vsource                         # loads .env in current dir
vsource .env.prod               # loads specific file
eval "$(vsource .env)" && helmfile apply   # inject + deploy

---

IAM Management (Authentik)

Manage OAuth2 applications, groups, and user access via Authentik.

List Groups

core iam list-groups

Output:

authentik Admins (id: 9d72cbf2-9d52-4d3c-bba9-0068525d7a91)
grafana-admins (id: 5f2e1e79-7d7e-4ef0-9b99-3c6df19c0b88)
minio-admins (id: e640d887-eb85-431b-b5b2-5b6a8a7e0a44)
argocd-admins (id: 22a3c296-0d98-433f-8456-ebc2f1c8d489)
forgejo-admins (id: 4084c8d6-0c12-46af-acf8-7372172b9016)

List OAuth2 Applications

core iam list-apps

Create New Application

core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback"

Returns client ID and secret (save immediately).

Describe Application

core iam describe-app grafana

Shows:

• Client ID
• Client Secret
• Redirect URIs
• Scope claims

Bind Group to Application

core iam bind-app grafana grafana-admins

Members of grafana-admins can log in to Grafana via OIDC.

Create Group

core iam create-group "developers"

Add User to Group

core iam add-member grafana-admins newuser

Rotate Application Secret

core iam rotate-secret grafana

⚠️ Must update deployment after rotating.

---

Port Forwarding

core pf grafana         # localhost:3000 → Grafana
core pf prometheus      # localhost:9090 → Prometheus
core pf minio           # localhost:9001 → MinIO console
core pf iam             # localhost:7000 → Authentik

---

Workflow: Rotate OAuth2 Secret

# 1. Rotate in Authentik
SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')

# 2. Update deployment
vi k8s/logging/grafana-values.yaml
# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"

# 3. Redeploy
helmfile apply -l app=grafana

# 4. Verify
core iam describe-app grafana

---

Workflow: Add User to Service

# 1. Create or verify group exists
core iam list-groups | grep minio-admins

# 2. Add user to group
core iam add-member minio-admins alice

# 3. Verify
# (User will have access next login via OIDC)

---

Vault Integration (Advanced)

Vault paths for IAM state (if using federated OIDC):

cluster/iam/federation/
cluster/iam/roles/
cluster/iam/services/
cluster/iam/agents/
cluster/iam/bindings/

Query via:

core get cluster/iam/roles/admin --key roles

See CLAUDE.md § IAM for full architecture (roles, services, agents, audit).

CI/CD Image Registry Authentication (Forgejo + Runner)

After IAM Phase 6 changes: All image push/pull operations via CI runner require JWT token authentication through Authentik → Vault.

Push images to Forgejo registry:

# 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
export REGISTRY_TOKEN=$(core get cluster/iam/agents/ci-bot --key token)

# 2. Authenticate docker/podman to Forgejo registry
docker login forgejo.riotpiao.com \
  --username ci-bot \
  --password "$REGISTRY_TOKEN"

# 3. Tag and push image
docker tag myapp:latest forgejo.riotpiao.com/rock/myapp:latest
docker push forgejo.riotpiao.com/rock/myapp:latest

Pull images in runner (automatic):

# Inside .forgejo/workflows/*.yml, runner pulls via K8s ServiceAccount
# No explicit login needed — imagePullSecrets injected by runner pod
image: forgejo.riotpiao.com/rock/myapp:latest

Runner pod setup:

# ServiceAccount (in cicd namespace) has Vault JWT auth injected
kubectl get serviceaccount -n cicd forgejo-runner
kubectl describe sa forgejo-runner -n cicd

# ImagePullSecret auto-mounted from K8s secret:
kubectl get secret -n cicd | grep forgejo-registry

Vault paths for credentials:

cluster/iam/agents/ci-bot          # JWT token for push authentication
cluster/iam/agents/admin-bot       # Alternative admin agent (if needed)

---