docs: add foundation docs and cluster configuration templates
- README: cluster architecture, quick start, use cases - USAGE: stack topology, custom CLI reference - TROUBLESHOOTING: operational safety rules - .env.example: configuration template - Makefile: build shortcuts
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,218 @@
|
||||
## Cluster Architecture at a Glance
|
||||
|
||||
**Homelab** is a 2-node bare-metal Kubernetes cluster deployed with Talos Linux, designed for self-hosted services, observability, and GitOps-ready CI/CD.
|
||||
|
||||
### Deployment Stack (18 Helm releases)
|
||||
|
||||
| Layer | Component | Namespace | Purpose |
|
||||
|-------|-----------|-----------|---------|
|
||||
| **OS & Networking** | Talos Linux v1.13.3 | — | Immutable, declarative Linux |
|
||||
| **CNI** | Cilium (eBPF) | kube-system | Advanced networking, no kube-proxy |
|
||||
| **Ingress** | Nginx Ingress Controller | ingress-nginx | Reverse proxy, TLS termination |
|
||||
| **Certificates** | cert-manager + homelab-ca | cert-manager | Self-signed CA, auto-renewal |
|
||||
| **Storage (Block)** | Longhorn v1.7.0 | longhorn-system | Persistent volumes, default StorageClass |
|
||||
| **Storage (Object)** | MinIO (3-node, site-repl) | storage | S3-compatible, multi-AZ replication |
|
||||
| **Database** | CloudNativePG (3 replicas) | ddb | PostgreSQL 16 + pgvector |
|
||||
| **IAM / OIDC** | Authentik | iam | Federated OIDC provider for all services |
|
||||
| **Secrets** | HashiCorp Vault | iam | KV secrets backend, JWT auth |
|
||||
| **Logs** | Loki (SingleBinary) | logging | 10-day retention, MinIO backend |
|
||||
| **Log Collection** | Promtail (DaemonSet) | logging | Pod + kernel logs → Loki |
|
||||
| **Metrics** | Prometheus + kube-state-metrics | monitoring | Time-series metrics, service discovery |
|
||||
| **Dashboards** | Grafana | logging | Unified UI for Prometheus + Loki |
|
||||
| **Uptime Monitoring** | Blackbox Exporter | monitoring | External endpoint probes |
|
||||
| **Git Forge** | Forgejo (self-hosted) | cicd | Git server, OCI registry, Forgejo Actions |
|
||||
| **CI Runner** | Forgejo Actions Runner | cicd | Privileged build pods, image push |
|
||||
| **CD** | Argo CD | cicd | Pull-based GitOps, declarative deployments |
|
||||
| **Message Queue** | Kafka (Strimzi KRaft mode) | sqs | SQS-like message queue service |
|
||||
| **Queue Redis** | Redis | sqs | In-flight message tracking, dedup |
|
||||
| **Queue API** | kmsvc (gRPC + REST) | sqs | Message service (SendMessage, ReceiveMessage, etc.) |
|
||||
| **Workflows** | Temporal | temporal | Distributed workflow engine |
|
||||
| **Container UI** | Portainer CE | dashboard | Pod/workload management UI |
|
||||
| **Terminal** | Claude Terminal | dev-tools | Persistent dev environment (optional) |
|
||||
| **Config Reload** | Reloader | reloader | Auto-reload pods on ConfigMap/Secret changes |
|
||||
|
||||
### Dependency Chain (Release Order)
|
||||
|
||||
```
|
||||
cert-manager (root)
|
||||
↓
|
||||
├─→ ingress-nginx
|
||||
├─→ cilium (network policies)
|
||||
└─→ cloudnative-pg
|
||||
↓
|
||||
├─→ authentik (DB: authentik)
|
||||
│ ├─→ vault (uses Authentik OIDC)
|
||||
│ ├─→ forgejo (OIDC login)
|
||||
│ ├─→ argocd (OIDC login)
|
||||
│ └─→ kmsvc (JWT auth)
|
||||
│
|
||||
└─→ story-crater-backend (DB: story_crater)
|
||||
|
||||
storage (MinIO 3-node)
|
||||
├─→ loki (object backend)
|
||||
├─→ vault (unseal keys bucket)
|
||||
└─→ monitoring (Prometheus scrape)
|
||||
|
||||
monitoring (Prometheus operator)
|
||||
├─→ ingress-nginx (requires ServiceMonitor CRDs)
|
||||
├─→ authentik (requires ServiceMonitor CRDs)
|
||||
├─→ minio-az-a (requires ServiceMonitor CRDs)
|
||||
└─→ temporal (requires ServiceMonitor CRDs)
|
||||
|
||||
logging (Loki + Grafana)
|
||||
├─→ promtail (pod log collection)
|
||||
└─→ grafana (dashboards)
|
||||
|
||||
sqs (Kafka + Message Queue)
|
||||
├─→ strimzi-operator
|
||||
├─→ kafka-cluster (KRaft mode, 3 brokers)
|
||||
├─→ kmsvc-redis (in-flight tracking)
|
||||
├─→ queue-crd (operator)
|
||||
└─→ management-service (gRPC/REST API)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Custom CLI — `talos`
|
||||
|
||||
Homelab cluster control CLI (`core/`). Manages cluster nodes and Vault secrets.
|
||||
|
||||
### Secret path convention
|
||||
|
||||
All secrets live under `cluster/<VARIABLE_NAME>`. The field name is always the variable name itself (SCREAMING_SNAKE_CASE), matching the `.env` key. Example paths:
|
||||
|
||||
```
|
||||
cluster/ANTHROPIC_API_KEY
|
||||
cluster/AUTHENTIK_FORGEJO_CLIENT_ID
|
||||
cluster/AUTHENTIK_ARGOCD_CLIENT_SECRET
|
||||
```
|
||||
|
||||
### `talos put` — write a secret to Vault
|
||||
|
||||
```bash
|
||||
talos put cluster/VARIABLE_NAME VARIABLE_NAME="secret-value"
|
||||
talos put cluster/FORGEJO_ADMIN_PASSWORD FORGEJO_ADMIN_PASSWORD="$FORGEJO_ADMIN_PASSWORD"
|
||||
```
|
||||
|
||||
Field name = variable name — never `value`.
|
||||
|
||||
### `talos get` — fetch a secret from Vault
|
||||
|
||||
```bash
|
||||
talos get cluster/VARIABLE_NAME --key VARIABLE_NAME # always specify --key
|
||||
talos get cluster/FORGEJO_ADMIN_PASSWORD --key FORGEJO_ADMIN_PASSWORD
|
||||
talos get cluster/VARIABLE_NAME --json # full secret as JSON
|
||||
```
|
||||
|
||||
Note: `talos get` uses `--key` (long flag), not a positional arg — unlike `talos secrets get`.
|
||||
|
||||
### `vsource` — load a `.env` into the shell
|
||||
|
||||
zsh function (lives in `~/.zshrc`, not in the repo — can reference but cannot run directly).
|
||||
Empty `.env` values are fetched from Vault at `cluster/<KEY>`; hardcoded values pass through.
|
||||
|
||||
```zsh
|
||||
vsource # loads .env in current directory
|
||||
vsource .env.local # loads a specific file
|
||||
```
|
||||
|
||||
`.env` format — leave secrets empty, vsource resolves them from Vault:
|
||||
|
||||
```bash
|
||||
ANTHROPIC_API_KEY= # fetched from cluster/ANTHROPIC_API_KEY
|
||||
AUTHENTIK_ARGOCD_CLIENT_ID= # fetched from cluster/AUTHENTIK_ARGOCD_CLIENT_ID
|
||||
DEBUG=true # hardcoded, passed through as-is
|
||||
```
|
||||
|
||||
### Typical workflow for a generated secret
|
||||
|
||||
```bash
|
||||
# 1. Store immediately after generation (keeps secrets out of shell history)
|
||||
talos put cluster/AUTHENTIK_FORGEJO_CLIENT_SECRET AUTHENTIK_FORGEJO_CLIENT_SECRET="<paste>"
|
||||
|
||||
# 2. Use via subshell when creating K8s secrets
|
||||
kubectl create secret generic my-secret \
|
||||
--from-literal=client-secret="$(talos get cluster/AUTHENTIK_FORGEJO_CLIENT_SECRET --key AUTHENTIK_FORGEJO_CLIENT_SECRET)"
|
||||
|
||||
# 3. Or load into shell via vsource for helmfile/env-driven tools
|
||||
vsource .env && helmfile apply
|
||||
```
|
||||
|
||||
### IAM Management (Federated OIDC, Phases 1–6 Complete)
|
||||
|
||||
**Status:** ✅ Fully deployed (2026-07-02). Single federated OIDC provider (`talos-federation`) handles all service auth.
|
||||
|
||||
**Quick reference:**
|
||||
|
||||
```bash
|
||||
# View roles and capabilities
|
||||
talos iam roles list && talos iam roles describe admin
|
||||
|
||||
# Service registry (Grafana, MinIO, Forgejo, etc.)
|
||||
talos iam services list && talos iam services describe grafana
|
||||
|
||||
# Agents (admin-bot, ci-bot with auto-rotation)
|
||||
talos iam agents list && talos iam agents rotate ci-bot
|
||||
|
||||
# Role bindings (user → role with TTL)
|
||||
talos iam bindings grant [email protected] devops --expires 2026-12-31
|
||||
talos iam bindings list
|
||||
|
||||
# Audit trail (90-day retention, 12 event types)
|
||||
talos iam audit list && talos iam audit export --format json
|
||||
|
||||
# OIDC provider sync with Authentik
|
||||
talos iam providers sync-authentik
|
||||
```
|
||||
|
||||
**See `homelab/CLAUDE.md` § IAM Management for full reference** (roles, services, agents, bindings, audit, providers).
|
||||
|
||||
**Vault paths:** All IAM state stored under `cluster/iam/{federation,roles,services,agents,bindings}`.
|
||||
|
||||
### CI/CD Image Registry Authentication (Forgejo + Runner)
|
||||
|
||||
**After IAM Phase 6 changes:** All image push/pull operations via CI runner require JWT token authentication through Authentik → Vault.
|
||||
|
||||
**Push images to Forgejo registry:**
|
||||
|
||||
```bash
|
||||
# 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
|
||||
export REGISTRY_TOKEN=$(talos get cluster/iam/agents/ci-bot --key token)
|
||||
|
||||
# 2. Authenticate docker/podman to Forgejo registry
|
||||
docker login forgejo.riotpiao.homelab.com \
|
||||
--username ci-bot \
|
||||
--password "$REGISTRY_TOKEN"
|
||||
|
||||
# 3. Tag and push image
|
||||
docker tag myapp:latest forgejo.riotpiao.homelab.com/rock/myapp:latest
|
||||
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
|
||||
```
|
||||
|
||||
**Pull images in runner (automatic):**
|
||||
|
||||
```bash
|
||||
# Inside .forgejo/workflows/*.yml, runner pulls via K8s ServiceAccount
|
||||
# No explicit login needed — imagePullSecrets injected by runner pod
|
||||
image: forgejo.riotpiao.homelab.com/rock/myapp:latest
|
||||
```
|
||||
|
||||
**Runner pod setup:**
|
||||
|
||||
```bash
|
||||
# ServiceAccount (in cicd namespace) has Vault JWT auth injected
|
||||
kubectl get serviceaccount -n cicd forgejo-runner
|
||||
kubectl describe sa forgejo-runner -n cicd
|
||||
|
||||
# ImagePullSecret auto-mounted from K8s secret:
|
||||
kubectl get secret -n cicd | grep forgejo-registry
|
||||
```
|
||||
|
||||
**Vault paths for credentials:**
|
||||
|
||||
```
|
||||
cluster/iam/agents/ci-bot # JWT token for push authentication
|
||||
cluster/iam/agents/admin-bot # Alternative admin agent (if needed)
|
||||
```
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user