381 lines
11 KiB
Markdown
381 lines
11 KiB
Markdown
## Cluster Architecture at a Glance
|
|
|
|
**Homelab** is a 3-node bare-metal Kubernetes cluster deployed with Talos Linux, designed for self-hosted services, observability, and GitOps-ready CI/CD.
|
|
|
|
### Node Topology (3 control-plane HA, since 2026-07-20)
|
|
|
|
| Node | IP | Zone | Role | Scheduling | Storage |
|
|
|------|----|----|------|-----------|---------|
|
|
| `talos-cp-1` | 192.168.1.213 | az-a | control-plane | **schedulable** (runs all workloads) | sole Longhorn node (sdb/sdc/sdd) |
|
|
| `talos-cp-2` | 192.168.1.163 | az-b | control-plane | dedicated (`NoSchedule`) | none |
|
|
| `talos-cp-3` | 192.168.1.166 | az-c | control-plane | dedicated (`NoSchedule`) | none |
|
|
|
|
- **etcd** has 3 voting members peering over the LAN (`cluster.etcd.advertisedSubnets: 192.168.1.0/24` — without it Talos may advertise on the WireGuard IP and new members hang as learners). Tolerates 1 node loss.
|
|
- Only `talos-cp-1` runs workloads and holds storage, so stateful services are **single-instance** (e.g. CNPG `ddb-cluster` = 1 instance). The kube-apiserver endpoint is single-homed to `.213` (no VIP yet).
|
|
- Scheduling is declarative: `allowSchedulingOnControlPlanes: true` + per-node `machine.nodeTaints` re-adds the control-plane taint on the dedicated nodes only.
|
|
|
|
### Deployment Stack (18 Helm releases)
|
|
|
|
| Layer | Component | Namespace | Purpose |
|
|
|-------|-----------|-----------|---------|
|
|
| **OS & Networking** | Talos Linux v1.13.3 | — | Immutable, declarative Linux |
|
|
| **CNI** | Cilium (eBPF) | kube-system | Advanced networking, no kube-proxy |
|
|
| **Ingress** | Nginx Ingress Controller | ingress-nginx | Reverse proxy, TLS termination |
|
|
| **Certificates** | cert-manager + homelab-ca | cert-manager | Self-signed CA, auto-renewal |
|
|
| **Storage (Block)** | Longhorn v1.7.0 | longhorn-system | Persistent volumes, default StorageClass |
|
|
| **Storage (Object)** | MinIO (3-node, site-repl) | storage | S3-compatible, multi-AZ replication |
|
|
| **Database** | CloudNativePG (1 instance) | ddb | PostgreSQL 16 + pgvector (single-node; see topology) |
|
|
| **IAM / OIDC** | Authentik | iam | Federated OIDC provider for all services |
|
|
| **Secrets** | HashiCorp Vault | iam | KV secrets backend, JWT auth |
|
|
| **Logs** | Loki (SingleBinary) | logging | 10-day retention, MinIO backend |
|
|
| **Log Collection** | Promtail (DaemonSet) | logging | Pod + kernel logs → Loki |
|
|
| **Metrics** | Prometheus + kube-state-metrics | monitoring | Time-series metrics, service discovery |
|
|
| **Dashboards** | Grafana | logging | Unified UI for Prometheus + Loki |
|
|
| **Uptime Monitoring** | Blackbox Exporter | monitoring | External endpoint probes |
|
|
| **Git Forge** | Forgejo (self-hosted) | cicd | Git server, OCI registry, Forgejo Actions |
|
|
| **CI Runner** | Forgejo Actions Runner | cicd | Privileged build pods, image push |
|
|
| **CD** | Argo CD | cicd | Pull-based GitOps, declarative deployments |
|
|
| **Message Queue** | Kafka (Strimzi KRaft mode) | sqs | SQS-like message queue service |
|
|
| **Queue Redis** | Redis | sqs | In-flight message tracking, dedup |
|
|
| **Queue API** | kmsvc (gRPC + REST) | sqs | Message service (SendMessage, ReceiveMessage, etc.) |
|
|
| **Workflows** | Temporal | temporal | Distributed workflow engine |
|
|
| **Container UI** | Portainer CE | dashboard | Pod/workload management UI |
|
|
| **Terminal** | Claude Terminal | dev-tools | Persistent dev environment (optional) |
|
|
| **Config Reload** | Reloader | reloader | Auto-reload pods on ConfigMap/Secret changes |
|
|
|
|
### Dependency Chain (Release Order)
|
|
|
|
```
|
|
cert-manager (root)
|
|
↓
|
|
├─→ ingress-nginx
|
|
├─→ cilium (network policies)
|
|
└─→ cloudnative-pg
|
|
↓
|
|
├─→ authentik (DB: authentik)
|
|
│ ├─→ vault (uses Authentik OIDC)
|
|
│ ├─→ forgejo (OIDC login)
|
|
│ ├─→ argocd (OIDC login)
|
|
│ └─→ kmsvc (JWT auth)
|
|
│
|
|
└─→ story-crater-backend (DB: story_crater)
|
|
|
|
storage (MinIO 3-node)
|
|
├─→ loki (object backend)
|
|
├─→ vault (unseal keys bucket)
|
|
└─→ monitoring (Prometheus scrape)
|
|
|
|
monitoring (Prometheus operator)
|
|
├─→ ingress-nginx (requires ServiceMonitor CRDs)
|
|
├─→ authentik (requires ServiceMonitor CRDs)
|
|
├─→ minio-az-a (requires ServiceMonitor CRDs)
|
|
└─→ temporal (requires ServiceMonitor CRDs)
|
|
|
|
logging (Loki + Grafana)
|
|
├─→ promtail (pod log collection)
|
|
└─→ grafana (dashboards)
|
|
|
|
sqs (Kafka + Message Queue)
|
|
├─→ strimzi-operator
|
|
├─→ kafka-cluster (KRaft mode, 3 brokers)
|
|
├─→ kmsvc-redis (in-flight tracking)
|
|
├─→ queue-crd (operator)
|
|
└─→ management-service (gRPC/REST API)
|
|
```
|
|
|
|
---
|
|
|
|
## Custom CLI — `core`
|
|
|
|
Homelab cluster control CLI (source: `~/workplace/core/`). Manages cluster nodes, Authentik IAM, and Vault secrets.
|
|
|
|
**Setup:**
|
|
```bash
|
|
make cli # builds + installs to ~/.local/bin/core
|
|
core auth login-oob # authenticate with Authentik (OOB flow)
|
|
core nodes # verify cluster access
|
|
```
|
|
|
|
**Full documentation:** See [~/workplace/core/USAGE.md](../core/USAGE.md)
|
|
|
|
---
|
|
|
|
### Authentication
|
|
|
|
#### OAuth2 Out-of-Band (OOB) Login
|
|
|
|
Browser-based login with manual code entry (recommended).
|
|
|
|
```bash
|
|
export CORE_CLI_SECRET="cyoVr96FB9oeq3o64DUG0BmoVzMPsOTIWxVEd8ZdTezKrEZYwrKpIRkOwrDQNEtF6QJyNUPH4mjr9jWokQY7KVBWX1KVUXFyyhgAHTqWRRWAYUJ8r3H35wLFiTfn5KsV"
|
|
core auth login-oob
|
|
```
|
|
|
|
Token expires in 8 hours. Check status:
|
|
```bash
|
|
core auth status # show expiry
|
|
core auth clear # force re-auth on next command
|
|
```
|
|
|
|
---
|
|
|
|
### Cluster Management
|
|
|
|
```bash
|
|
core nodes # list cluster nodes
|
|
core status <ip> # Talos node overview
|
|
core services <ip> # list Talos services
|
|
core logs <ip> # stream kubelet logs
|
|
core log-svc <ip> <svc> # logs for specific service (etcd, kubelet, etc.)
|
|
core pods clean # delete Failed/Evicted/Terminating pods
|
|
```
|
|
|
|
#### kubectl Context (LAN vs. WireGuard)
|
|
|
|
`cluster-config/kubeconfig` has two contexts pointed at the same cluster:
|
|
`admin@homelab-cluster` (LAN, `192.168.1.213:6443`) and `admin@homelab-cluster-1`
|
|
(WireGuard, `10.6.0.1:6443`). If `kubectl`/`core nodes` hangs or refuses the
|
|
connection, you're likely off-LAN — switch contexts:
|
|
|
|
```bash
|
|
core config kube-list # list contexts
|
|
core config kube-use admin@homelab-cluster-1 # switch to WireGuard path
|
|
```
|
|
|
|
**Known gap:** `core config use <talos-context>` (the combined talosctl+kubectl
|
|
switch) only maps to `admin@homelab-cluster` today — its WireGuard mapping
|
|
(`home-cluster-wire-guard`) is stale, that kubectl context doesn't exist. Use
|
|
`core config kube-use admin@homelab-cluster-1` directly until that's fixed.
|
|
|
|
---
|
|
|
|
### Secret Management (Vault)
|
|
|
|
All secrets live under `cluster/<VARIABLE_NAME>`. Field name = variable name (SCREAMING_SNAKE_CASE).
|
|
|
|
#### Write Secret to Vault
|
|
|
|
```bash
|
|
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."
|
|
core put cluster/FORGEJO_ADMIN_PASSWORD FORGEJO_ADMIN_PASSWORD="secret123"
|
|
```
|
|
|
|
**Key rule:** Field name must match variable name — never `value=`.
|
|
|
|
#### Fetch Secret from Vault
|
|
|
|
```bash
|
|
core get cluster/ANTHROPIC_API_KEY # default field
|
|
core get cluster/AUTHENTIK_ARGOCD_CLIENT_SECRET # full value
|
|
```
|
|
|
|
#### List All Secrets
|
|
|
|
```bash
|
|
core secrets list
|
|
```
|
|
|
|
#### Load into Shell (vsource)
|
|
|
|
zsh function in `~/.zshrc` — fetches empty `.env` values from Vault:
|
|
|
|
```bash
|
|
# .env format
|
|
ANTHROPIC_API_KEY= # fetched from Vault
|
|
AUTHENTIK_ARGOCD_CLIENT_ID= # fetched from Vault
|
|
DEBUG=true # hardcoded, passed through
|
|
|
|
# Usage
|
|
vsource # loads .env in current dir
|
|
vsource .env.prod # loads specific file
|
|
eval "$(vsource .env)" && helmfile apply # inject + deploy
|
|
```
|
|
|
|
---
|
|
|
|
### IAM Management (Authentik)
|
|
|
|
Manage OAuth2 applications, groups, and user access via Authentik.
|
|
|
|
#### List Groups
|
|
|
|
```bash
|
|
core iam list-groups
|
|
```
|
|
|
|
Output:
|
|
```
|
|
authentik Admins (id: 9d72cbf2-9d52-4d3c-bba9-0068525d7a91)
|
|
grafana-admins (id: 5f2e1e79-7d7e-4ef0-9b99-3c6df19c0b88)
|
|
minio-admins (id: e640d887-eb85-431b-b5b2-5b6a8a7e0a44)
|
|
argocd-admins (id: 22a3c296-0d98-433f-8456-ebc2f1c8d489)
|
|
forgejo-admins (id: 4084c8d6-0c12-46af-acf8-7372172b9016)
|
|
```
|
|
|
|
#### List OAuth2 Applications
|
|
|
|
```bash
|
|
core iam list-apps
|
|
```
|
|
|
|
#### Create New Application
|
|
|
|
```bash
|
|
core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback"
|
|
```
|
|
|
|
Returns client ID and secret (save immediately).
|
|
|
|
#### Describe Application
|
|
|
|
```bash
|
|
core iam describe-app grafana
|
|
```
|
|
|
|
Shows:
|
|
- Client ID
|
|
- Client Secret
|
|
- Redirect URIs
|
|
- Scope claims
|
|
|
|
#### Bind Group to Application
|
|
|
|
```bash
|
|
core iam bind-app grafana grafana-admins
|
|
```
|
|
|
|
Members of `grafana-admins` can log in to Grafana via OIDC.
|
|
|
|
#### Create Group
|
|
|
|
```bash
|
|
core iam create-group "developers"
|
|
```
|
|
|
|
#### Add User to Group
|
|
|
|
```bash
|
|
core iam add-member grafana-admins newuser
|
|
```
|
|
|
|
#### Rotate Application Secret
|
|
|
|
```bash
|
|
core iam rotate-secret grafana
|
|
```
|
|
|
|
⚠️ Must update deployment after rotating.
|
|
|
|
---
|
|
|
|
### Port Forwarding
|
|
|
|
```bash
|
|
core pf grafana # localhost:3000 → Grafana
|
|
core pf prometheus # localhost:9090 → Prometheus
|
|
core pf minio # localhost:9001 → MinIO console
|
|
core pf iam # localhost:7000 → Authentik
|
|
```
|
|
|
|
---
|
|
|
|
### Workflow: Rotate OAuth2 Secret
|
|
|
|
```bash
|
|
# 1. Rotate in Authentik
|
|
SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')
|
|
|
|
# 2. Update deployment
|
|
vi k8s/logging/grafana-values.yaml
|
|
# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"
|
|
|
|
# 3. Redeploy
|
|
helmfile apply -l app=grafana
|
|
|
|
# 4. Verify
|
|
core iam describe-app grafana
|
|
```
|
|
|
|
---
|
|
|
|
### Workflow: Add User to Service
|
|
|
|
```bash
|
|
# 1. Create or verify group exists
|
|
core iam list-groups | grep minio-admins
|
|
|
|
# 2. Add user to group
|
|
core iam add-member minio-admins alice
|
|
|
|
# 3. Verify
|
|
# (User will have access next login via OIDC)
|
|
```
|
|
|
|
---
|
|
|
|
### Vault Integration (Advanced)
|
|
|
|
Vault paths for IAM state (if using federated OIDC):
|
|
|
|
```
|
|
cluster/iam/federation/
|
|
cluster/iam/roles/
|
|
cluster/iam/services/
|
|
cluster/iam/agents/
|
|
cluster/iam/bindings/
|
|
```
|
|
|
|
Query via:
|
|
```bash
|
|
core get cluster/iam/roles/admin --key roles
|
|
```
|
|
|
|
**See `CLAUDE.md` § IAM for full architecture** (roles, services, agents, audit).
|
|
|
|
### CI/CD Image Registry Authentication (Forgejo + Runner)
|
|
|
|
**After IAM Phase 6 changes:** All image push/pull operations via CI runner require JWT token authentication through Authentik → Vault.
|
|
|
|
**Push images to Forgejo registry:**
|
|
|
|
```bash
|
|
# 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
|
|
export REGISTRY_TOKEN=$(core get cluster/iam/agents/ci-bot --key token)
|
|
|
|
# 2. Authenticate docker/podman to Forgejo registry
|
|
docker login forgejo.riotpiao.com \
|
|
--username ci-bot \
|
|
--password "$REGISTRY_TOKEN"
|
|
|
|
# 3. Tag and push image
|
|
docker tag myapp:latest forgejo.riotpiao.com/rock/myapp:latest
|
|
docker push forgejo.riotpiao.com/rock/myapp:latest
|
|
```
|
|
|
|
**Pull images in runner (automatic):**
|
|
|
|
```bash
|
|
# Inside .forgejo/workflows/*.yml, runner pulls via K8s ServiceAccount
|
|
# No explicit login needed — imagePullSecrets injected by runner pod
|
|
image: forgejo.riotpiao.com/rock/myapp:latest
|
|
```
|
|
|
|
**Runner pod setup:**
|
|
|
|
```bash
|
|
# ServiceAccount (in cicd namespace) has Vault JWT auth injected
|
|
kubectl get serviceaccount -n cicd forgejo-runner
|
|
kubectl describe sa forgejo-runner -n cicd
|
|
|
|
# ImagePullSecret auto-mounted from K8s secret:
|
|
kubectl get secret -n cicd | grep forgejo-registry
|
|
```
|
|
|
|
**Vault paths for credentials:**
|
|
|
|
```
|
|
cluster/iam/agents/ci-bot # JWT token for push authentication
|
|
cluster/iam/agents/admin-bot # Alternative admin agent (if needed)
|
|
```
|
|
|
|
--- |