homelab/USAGE.md

## Cluster Architecture at a Glance

**Homelab** is a 2-node bare-metal Kubernetes cluster deployed with Talos Linux, designed for self-hosted services, observability, and GitOps-ready CI/CD.

### Deployment Stack (18 Helm releases)

| Layer | Component | Namespace | Purpose |
|-------|-----------|-----------|---------|
| **OS & Networking** | Talos Linux v1.13.3 | — | Immutable, declarative Linux |
| **CNI** | Cilium (eBPF) | kube-system | Advanced networking, no kube-proxy |
| **Ingress** | Nginx Ingress Controller | ingress-nginx | Reverse proxy, TLS termination |
| **Certificates** | cert-manager + homelab-ca | cert-manager | Self-signed CA, auto-renewal |
| **Storage (Block)** | Longhorn v1.7.0 | longhorn-system | Persistent volumes, default StorageClass |
| **Storage (Object)** | MinIO (3-node, site-repl) | storage | S3-compatible, multi-AZ replication |
| **Database** | CloudNativePG (3 replicas) | ddb | PostgreSQL 16 + pgvector |
| **IAM / OIDC** | Authentik | iam | Federated OIDC provider for all services |
| **Secrets** | HashiCorp Vault | iam | KV secrets backend, JWT auth |
| **Logs** | Loki (SingleBinary) | logging | 10-day retention, MinIO backend |
| **Log Collection** | Promtail (DaemonSet) | logging | Pod + kernel logs → Loki |
| **Metrics** | Prometheus + kube-state-metrics | monitoring | Time-series metrics, service discovery |
| **Dashboards** | Grafana | logging | Unified UI for Prometheus + Loki |
| **Uptime Monitoring** | Blackbox Exporter | monitoring | External endpoint probes |
| **Git Forge** | Forgejo (self-hosted) | cicd | Git server, OCI registry, Forgejo Actions |
| **CI Runner** | Forgejo Actions Runner | cicd | Privileged build pods, image push |
| **CD** | Argo CD | cicd | Pull-based GitOps, declarative deployments |
| **Message Queue** | Kafka (Strimzi KRaft mode) | sqs | SQS-like message queue service |
| **Queue Redis** | Redis | sqs | In-flight message tracking, dedup |
| **Queue API** | kmsvc (gRPC + REST) | sqs | Message service (SendMessage, ReceiveMessage, etc.) |
| **Workflows** | Temporal | temporal | Distributed workflow engine |
| **Container UI** | Portainer CE | dashboard | Pod/workload management UI |
| **Terminal** | Claude Terminal | dev-tools | Persistent dev environment (optional) |
| **Config Reload** | Reloader | reloader | Auto-reload pods on ConfigMap/Secret changes |

### Dependency Chain (Release Order)

```
cert-manager (root)
  ↓
  ├─→ ingress-nginx
  ├─→ cilium (network policies)
  └─→ cloudnative-pg
        ↓
        ├─→ authentik (DB: authentik)
        │     ├─→ vault (uses Authentik OIDC)
        │     ├─→ forgejo (OIDC login)
        │     ├─→ argocd (OIDC login)
        │     └─→ kmsvc (JWT auth)
        │
        └─→ story-crater-backend (DB: story_crater)
  
storage (MinIO 3-node)
  ├─→ loki (object backend)
  ├─→ vault (unseal keys bucket)
  └─→ monitoring (Prometheus scrape)

monitoring (Prometheus operator)
  ├─→ ingress-nginx (requires ServiceMonitor CRDs)
  ├─→ authentik (requires ServiceMonitor CRDs)
  ├─→ minio-az-a (requires ServiceMonitor CRDs)
  └─→ temporal (requires ServiceMonitor CRDs)

logging (Loki + Grafana)
  ├─→ promtail (pod log collection)
  └─→ grafana (dashboards)

sqs (Kafka + Message Queue)
  ├─→ strimzi-operator
  ├─→ kafka-cluster (KRaft mode, 3 brokers)
  ├─→ kmsvc-redis (in-flight tracking)
  ├─→ queue-crd (operator)
  └─→ management-service (gRPC/REST API)
```

---

## Custom CLI — `core`

Homelab cluster control CLI (source: `~/workplace/core/`). Manages cluster nodes, Authentik IAM, and Vault secrets.

**Setup:**
```bash
make cli                     # builds + installs to ~/.local/bin/core
core auth login-oob         # authenticate with Authentik (OOB flow)
core nodes                  # verify cluster access
```

**Full documentation:** See [~/workplace/core/USAGE.md](../core/USAGE.md)

---

### Authentication

#### OAuth2 Out-of-Band (OOB) Login

Browser-based login with manual code entry (recommended).

```bash
export CORE_CLI_SECRET="cyoVr96FB9oeq3o64DUG0BmoVzMPsOTIWxVEd8ZdTezKrEZYwrKpIRkOwrDQNEtF6QJyNUPH4mjr9jWokQY7KVBWX1KVUXFyyhgAHTqWRRWAYUJ8r3H35wLFiTfn5KsV"
core auth login-oob
```

Token expires in 8 hours. Check status:
```bash
core auth status   # show expiry
core auth clear    # force re-auth on next command
```

---

### Cluster Management

```bash
core nodes                    # list cluster nodes
core status <ip>              # Talos node overview
core services <ip>            # list Talos services
core logs <ip>                # stream kubelet logs
core log-svc <ip> <svc>       # logs for specific service (etcd, kubelet, etc.)
core pods clean               # delete Failed/Evicted/Terminating pods
```

#### kubectl Context (LAN vs. WireGuard)

`cluster-config/kubeconfig` has two contexts pointed at the same cluster:
`admin@homelab-cluster` (LAN, `192.168.1.213:6443`) and `admin@homelab-cluster-1`
(WireGuard, `10.6.0.1:6443`). If `kubectl`/`core nodes` hangs or refuses the
connection, you're likely off-LAN — switch contexts:

```bash
core config kube-list             # list contexts
core config kube-use admin@homelab-cluster-1   # switch to WireGuard path
```

**Known gap:** `core config use <talos-context>` (the combined talosctl+kubectl
switch) only maps to `admin@homelab-cluster` today — its WireGuard mapping
(`home-cluster-wire-guard`) is stale, that kubectl context doesn't exist. Use
`core config kube-use admin@homelab-cluster-1` directly until that's fixed.

---

### Secret Management (Vault)

All secrets live under `cluster/<VARIABLE_NAME>`. Field name = variable name (SCREAMING_SNAKE_CASE).

#### Write Secret to Vault

```bash
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."
core put cluster/FORGEJO_ADMIN_PASSWORD FORGEJO_ADMIN_PASSWORD="secret123"
```

**Key rule:** Field name must match variable name — never `value=`.

#### Fetch Secret from Vault

```bash
core get cluster/ANTHROPIC_API_KEY              # default field
core get cluster/AUTHENTIK_ARGOCD_CLIENT_SECRET # full value
```

#### List All Secrets

```bash
core secrets list
```

#### Load into Shell (vsource)

zsh function in `~/.zshrc` — fetches empty `.env` values from Vault:

```bash
# .env format
ANTHROPIC_API_KEY=              # fetched from Vault
AUTHENTIK_ARGOCD_CLIENT_ID=     # fetched from Vault
DEBUG=true                      # hardcoded, passed through

# Usage
vsource                         # loads .env in current dir
vsource .env.prod               # loads specific file
eval "$(vsource .env)" && helmfile apply   # inject + deploy
```

---

### IAM Management (Authentik)

Manage OAuth2 applications, groups, and user access via Authentik.

#### List Groups

```bash
core iam list-groups
```

Output:
```
authentik Admins (id: 9d72cbf2-9d52-4d3c-bba9-0068525d7a91)
grafana-admins (id: 5f2e1e79-7d7e-4ef0-9b99-3c6df19c0b88)
minio-admins (id: e640d887-eb85-431b-b5b2-5b6a8a7e0a44)
argocd-admins (id: 22a3c296-0d98-433f-8456-ebc2f1c8d489)
forgejo-admins (id: 4084c8d6-0c12-46af-acf8-7372172b9016)
```

#### List OAuth2 Applications

```bash
core iam list-apps
```

#### Create New Application

```bash
core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.homelab.com/callback"
```

Returns client ID and secret (save immediately).

#### Describe Application

```bash
core iam describe-app grafana
```

Shows:
- Client ID
- Client Secret
- Redirect URIs
- Scope claims

#### Bind Group to Application

```bash
core iam bind-app grafana grafana-admins
```

Members of `grafana-admins` can log in to Grafana via OIDC.

#### Create Group

```bash
core iam create-group "developers"
```

#### Add User to Group

```bash
core iam add-member grafana-admins newuser
```

#### Rotate Application Secret

```bash
core iam rotate-secret grafana
```

⚠️ Must update deployment after rotating.

---

### Port Forwarding

```bash
core pf grafana         # localhost:3000 → Grafana
core pf prometheus      # localhost:9090 → Prometheus
core pf minio           # localhost:9001 → MinIO console
core pf iam             # localhost:7000 → Authentik
```

---

### Workflow: Rotate OAuth2 Secret

```bash
# 1. Rotate in Authentik
SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')

# 2. Update deployment
vi k8s/logging/grafana-values.yaml
# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"

# 3. Redeploy
helmfile apply -l app=grafana

# 4. Verify
core iam describe-app grafana
```

---

### Workflow: Add User to Service

```bash
# 1. Create or verify group exists
core iam list-groups | grep minio-admins

# 2. Add user to group
core iam add-member minio-admins alice

# 3. Verify
# (User will have access next login via OIDC)
```

---

### Vault Integration (Advanced)

Vault paths for IAM state (if using federated OIDC):

```
cluster/iam/federation/
cluster/iam/roles/
cluster/iam/services/
cluster/iam/agents/
cluster/iam/bindings/
```

Query via:
```bash
core get cluster/iam/roles/admin --key roles
```

**See `CLAUDE.md` § IAM for full architecture** (roles, services, agents, audit).

### CI/CD Image Registry Authentication (Forgejo + Runner)

**After IAM Phase 6 changes:** All image push/pull operations via CI runner require JWT token authentication through Authentik → Vault.

**Push images to Forgejo registry:**

```bash
# 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
export REGISTRY_TOKEN=$(core get cluster/iam/agents/ci-bot --key token)

# 2. Authenticate docker/podman to Forgejo registry
docker login forgejo.riotpiao.homelab.com \
  --username ci-bot \
  --password "$REGISTRY_TOKEN"

# 3. Tag and push image
docker tag myapp:latest forgejo.riotpiao.homelab.com/rock/myapp:latest
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
```

**Pull images in runner (automatic):**

```bash
# Inside .forgejo/workflows/*.yml, runner pulls via K8s ServiceAccount
# No explicit login needed — imagePullSecrets injected by runner pod
image: forgejo.riotpiao.homelab.com/rock/myapp:latest
```

**Runner pod setup:**

```bash
# ServiceAccount (in cicd namespace) has Vault JWT auth injected
kubectl get serviceaccount -n cicd forgejo-runner
kubectl describe sa forgejo-runner -n cicd

# ImagePullSecret auto-mounted from K8s secret:
kubectl get secret -n cicd | grep forgejo-registry
```

**Vault paths for credentials:**

```
cluster/iam/agents/ci-bot          # JWT token for push authentication
cluster/iam/agents/admin-bot       # Alternative admin agent (if needed)
```

---