homelab/project-usage/core-cli-tools.md

# Core CLI Tools: Decision Guide

The `core` CLI is your primary tool for cluster auth, node operations, Vault secrets management, IAM administration, and object storage. This doc covers **when** to reach for `core` vs `kubectl`/`helmfile`, the two separate auth domains that power different commands, and the complete command inventory.

For exhaustive flag-level detail on each subcommand, see [~/workplace/core/USAGE.md](../../core/USAGE.md).

---

## The Two Auth Domains

The `core` CLI maintains **two independent authentication systems** that gate different command families. Confusing them causes authentication failures.

### Domain 1: Node/Talos Operations (`core auth`)

**Gates:** Node-level commands, Talos service management, cluster status.

**Login mechanism:** `core auth login-oob`  
- Out-of-band device-code flow via Authentik
- Token cached in `~/.core/token` (24h expiry)
- Required before: `core nodes`, `core status <ip>`, `core services <ip>`, `core logs <ip>`, `core log-svc <ip> <svc>`, `core dmesg <ip>`, `core config`, `core upgrade`, `core shutdown`, `core reboot`, `core node`, `core pods clean`

**Check auth state:**
```bash
core auth status          # Show token expiry
core auth clear           # Force re-auth on next command
```

**When to use:**
- Troubleshooting node-level issues (crashes, disk space, network on a specific Talos node)
- Checking Talos service health (etcd, kubelet, scheduler, etc.)
- Upgrading cluster OS or managing node lifecycle
- Reading kernel logs (dmesg) or kubelet logs on specific nodes

### Domain 2: Vault Secrets (`core secrets`)

**Gates:** Secrets management (read/write), secret listing, secret export.

**Login mechanism:** `core secrets login`  
- Interactive login to Vault (opens browser, approves auth)
- Reads/writes to `~/.config/talos/secrets.toml`
- Required before: `core get <path> [--key KEY]`, `core put <path> key=value`, `core secrets list`, `core secrets export`, `core secrets exec`

**Check auth state:**
```bash
core secrets status       # Show Vault token / scopes
core secrets clear        # Force re-auth on next command
```

**When to use:**
- Reading or writing cluster secrets (API keys, database passwords, OAuth client secrets)
- Managing application credentials in Vault (centralized secret store)
- Rotating secrets for services (e.g., OAuth2 client secrets)
- Listing all secrets under a path
- Bootstrapping `.env` files for helmfile deployment (via `vsource`)

---

## Auth Bug Fixes

**Bug #1 (Fixed):** `core status <NODE>` is a **node-ops command**, not an auth verifier. It does NOT verify that you're authenticated to Vault. To verify auth state, use the correct domain-specific command:
- For node access: `core auth status`
- For Vault access: `core secrets status`

**Bug #2 (Fixed):** `core secrets login` and `core auth login-oob` are **NOT interchangeable**. They gate completely different systems:
- `core auth login-oob` gates node/cluster operations
- `core secrets login` gates Vault secret read/write
- You may be authenticated to one domain and not the other. Check state separately.

---

## When to Reach for `core`

| Scenario | Tool | Why |
|----------|------|-----|
| Pod/deployment issue | `kubectl` | Pods, replicas, rollouts, events |
| Package release management | `helmfile` | Install/upgrade Helm charts |
| Node crashes, disk, kernel panic | `core auth` + node commands | Direct access to Talos node state |
| Service unreachable on Talos node | `core status <ip>`, `core services <ip>` | Talos service health |
| Database password rotation | `core secrets` + `core put` | Vault secret write |
| Need API key for app deployment | `core secrets` + `core get` | Fetch from Vault, inject via `vsource` |
| Create OAuth2 app in Authentik | `core iam create-app` | Manage federated identity apps |
| Add user to service (MinIO, Grafana, etc.) | `core iam add-member` | Group membership binding |
| S3 object upload/download | `core bucket` | MinIO operations |

---

## Command Inventory

Complete list of `core` subcommands, organized by domain. See [~/workplace/core/USAGE.md](../../core/USAGE.md) for usage flags and examples.

### Authentication

```bash
# Node/Talos operations auth
core auth login-oob       # Authenticate with Authentik (device code flow)
core auth status          # Check token expiry / auth state
core auth clear           # Clear cached auth token

# Vault secrets auth
core secrets login        # Authenticate to Vault (interactive)
core secrets status       # Check Vault token / scopes / expiry
core secrets clear        # Clear Vault auth token
```

### Cluster & Node Operations

```bash
# Node discovery and status
core nodes                # List all cluster nodes (IP, hostname, status)
core status <ip>          # Talos node overview (resources, uptime)
core services <ip>        # List Talos services (etcd, kubelet, etc.)

# Logs
core logs <ip>            # Stream kubelet logs on node
core log-svc <ip> <svc>   # Logs for specific Talos service (etcd, scheduler, etc.)
core dmesg <ip>           # Kernel logs (ring buffer) from node

# Node lifecycle
core upgrade <ip>         # Upgrade Talos OS on node
core shutdown <ip>        # Graceful shutdown
core reboot <ip>          # Reboot node

# Kubernetes context
core config kube-list     # List available kubeconfig contexts
core config kube-use <ctx>  # Switch to kubeconfig context (LAN vs WireGuard)

# Pod cleanup
core pods clean           # Delete Failed/Evicted/Terminating pods
core node <node-name>     # Get node details (includes pod stats)
```

### Secrets Management (Vault)

```bash
# Read secrets
core get <path>                      # Fetch all fields under path
core get <path> --key KEY_NAME       # Fetch specific field

# Write secrets
core put <path> key=value [key2=value2 ...]  # Store secrets in Vault

# List and export
core secrets list                    # List all secret paths
core secrets export <path>           # Export secrets as shell-sourceable format

# Execute with secrets in environment
core secrets exec -- <command>       # Run command with secrets loaded in env
```

**Convention:** Field name = variable name (SCREAMING_SNAKE_CASE). Never use `value=`.

Example:
```bash
# Write
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."

# Read
core get cluster/ANTHROPIC_API_KEY
```

### IAM Management (Authentik)

```bash
# Groups
core iam list-groups              # List all groups
core iam create-group <name>      # Create new group

# OAuth2 Applications
core iam list-apps                # List all OAuth2 apps
core iam create-app <name> --slug <slug> --redirect-uri <uri>  # Create app
core iam describe-app <app>       # Show client ID, secret, URIs, scope claims
core iam rotate-secret <app>      # Rotate OAuth2 client secret

# User management
core iam add-member <group> <username>  # Add user to group
core iam bind-app <app> <group>   # Grant group access to application

# Cleanup
core iam delete-app <app>         # Delete OAuth2 application
```

### Object Storage (MinIO)

```bash
# List and manage buckets
core bucket list                  # List all buckets
core bucket upload <bucket> <file> [<remote-path>]  # Upload file to S3
core bucket download <bucket> <remote-path> <file>  # Download file from S3
core bucket delete <bucket> <remote-path>  # Delete object from S3
```

### Utilities

```bash
core help                         # Show command help
core pf grafana                   # Port-forward to Grafana (localhost:3000)
core pf prometheus                # Port-forward to Prometheus (localhost:9090)
core pf minio                     # Port-forward to MinIO console (localhost:9001)
core pf iam                       # Port-forward to Authentik (localhost:7000)
```

---

## Access Control Tiers

The cluster uses a tiered access model based on authentication mechanism and network perimeter.

### Tier A: OIDC + RBAC (Authentik-enforced)

Services with federated OIDC login and group-based role assignment.

| Service | Auth Method | Group Claim | Role Binding |
|---------|-------------|------------|--------------|
| **Grafana** | Authentik OIDC | `groups` | Mapped to Admin / Viewer / Viewer |
| **Argo CD** | Authentik OIDC | `groups` | RBAC role binding (policy.csv) |
| **MinIO** | Authentik OIDC | `groups` / custom `policy` | Policy-based access (readwrite / readonly) |
| **Forgejo** | Authentik OIDC | `email` (OAuth login only) | No group enforcement (open git repo) |
| **kmsvc** | Vault JWT | `sub` / `aud` | Audience validation + service scope |

**Provisioning:**
```bash
# After core auth login-oob, bootstrap IAM apps:
bash k8s/talos-iam/bootstrap-iam.sh
```

### Tier B: Network-Perimeter Only (No Authentik Enforcement)

Services with no OIDC support (product limitation). Access is restricted to LAN/WireGuard perimeter only.

| Service | Network Access | Use Case |
|---------|----------------|----------|
| **Portainer** | LAN + WireGuard only | Container UI, workload browsing |
| **Longhorn** | LAN + WireGuard only | Storage volume management |
| **Temporal** | LAN + WireGuard only | Workflow execution (auth TBD) |

All three are reachable **only** via WireGuard/LAN-only Ingress rules. Zero remote access risk, but also zero federated identity. If remote Temporal access is needed, upgrade the chart's auth configuration or replace with an OIDC-compatible workflow platform.

**Accessing Tier B services:**
```bash
# From off-LAN, use WireGuard context
core config kube-use admin@homelab-cluster-1
core pf minio  # connects via 10.6.0.1:9001
```

---

## Workflow: Rotate an OAuth2 Application Secret

```bash
# 1. Rotate secret in Authentik
SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')

# 2. Update Helm values
vi k8s/logging/grafana-values.yaml
# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"

# 3. Redeploy the app
helmfile apply -l app=grafana

# 4. Verify new secret is in use
core iam describe-app grafana | grep client_secret
```

---

## Workflow: Add User to Service

```bash
# 1. Verify group exists (or create it)
core iam list-groups | grep minio-admins

# If not found:
core iam create-group minio-admins

# 2. Add user to group
core iam add-member minio-admins alice

# 3. Bind group to MinIO application
core iam bind-app minio minio-admins

# 4. User has access on next login
# (OIDC login to MinIO → Authentik → group check → MinIO policy applied)
```

---

## Workflow: Rotate a Database Password (Vault)

```bash
# 1. Generate new password
NEW_PASS=$(openssl rand -hex 32)

# 2. Store in Vault
core put cluster/POSTGRES_ADMIN_PASSWORD POSTGRES_ADMIN_PASSWORD="$NEW_PASS"

# 3. Update database user
kubectl exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c \
  "ALTER USER postgres WITH PASSWORD '$NEW_PASS';"

# 4. Update Helm values with new password reference
# (Or if using helmfile hook: helmfile will re-run postInitApplicationSQL with new password)

# 5. Restart pods to pick up new secret
kubectl rollout restart -n <ns> deployment/<app>
```

---

## Common Questions

**Q: I'm getting "not authenticated" on `core nodes`. What do I do?**
A: Run `core auth login-oob`. Node commands use a different auth domain than secrets. After login, `core nodes` should work.

**Q: I have a valid Vault token but `core get` fails. Why?**
A: Check that both auth domains are active:
```bash
core auth status        # Verify node auth is valid
core secrets status     # Verify Vault auth is valid
```
Both must succeed. If one is expired, re-auth that domain.

**Q: Should I use `core bucket` or S3 CLI tools (aws-cli, s3cmd)?**
A: Use `core bucket` for simplicity (no AWS credentials). Use s3cmd/aws-cli if you need advanced sync or bandwidth control. All three talk to the same MinIO backend.

**Q: Can I add a user without creating a group first?**
A: Groups are the unit of access control. Always create the group, then add users to it, then bind it to applications. Single-user bindings are not supported (by design).

**Q: I rotated an OAuth2 secret but the app still fails to authenticate. What's next?**
A: 
1. Verify the new secret is stored: `core iam describe-app grafana`
2. Check the deployment has the new secret: `kubectl get secret -n logging grafana-oidc -o yaml | grep client_secret`
3. Restart the pod: `kubectl rollout restart -n logging deployment/grafana`
4. Check logs: `kubectl logs -n logging deployment/grafana | grep -i oauth`

---

## See Also

- [~/workplace/core/USAGE.md](../../core/USAGE.md) — Exhaustive command reference with flags and examples
- [CLAUDE.md](../CLAUDE.md) § Sign In (Device Code Flow) — Quick reference for `core auth login-oob`
- [CLAUDE.md](../CLAUDE.md) § Quick Shortcuts — One-liners for common tasks
- [CLAUDE.md](../CLAUDE.md) § Integration Checklist — New service onboarding