homelab/project-usage/vault-secrets.md

# Vault: Secret Management & JWT Auth

**Vault:** `https://vault.riotpiao.com`  
**Internal:** `vault.iam.svc.cluster.local:8200`  
**Namespace:** `iam`

## When to Use

- **Store secrets** — Database passwords, API keys, TLS certs
- **Rotate credentials** — Auto-rotate, track rotation history
- **JWT validation** — Verify tokens from Authentik, no external call needed
- **Audit trail** — Who accessed what secret, when

## Quick Start

**1. Login to Vault:**
```bash
# Browser: https://vault.riotpiao.com
# Auth method: OIDC → "Sign in with Authentik" (federated)
# Or: Device code → core secrets login (CLI)

# Via CLI (device code flow)
core secrets login
# → Opens browser, approve device code
# → Token cached in ~/.talos/vault
```

**2. Store a secret:**
```bash
# Field name = variable name (SCREAMING_SNAKE_CASE)
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
core put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
```

**3. Retrieve a secret:**
```bash
# Always use --key flag
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → sk-...

# Full secret as JSON
core get cluster/ANTHROPIC_API_KEY --json
```

**4. Load into shell (helmfile, scripts):**
```bash
vsource .env
# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
# → Passes hardcoded vars as-is (DEBUG=true)

helmfile diff
helmfile apply
```

## Vault Paths (KV v2)

**Naming convention:** `cluster/<VARIABLE_NAME>`

```
cluster/
├── ANTHROPIC_API_KEY            # third-party API
├── STORY_CRATER_DB_PASS         # database password
├── MINIO_ROOT_PASSWORD          # MinIO credentials
├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
├── iam/
│   ├── federation               # Authentik OIDC app config
│   ├── roles/admin              # RBAC role definitions
│   ├── services/grafana         # OAuth2 client info
│   ├── agents/ci-bot            # machine credentials
│   └── bindings/alice           # user→role mappings
└── kubernetes/
    ├── ingress-tls              # TLS cert private keys
    └── pull-secrets             # Docker registry creds
```

## Common Patterns

**Store generated secret immediately (keeps it out of shell history):**
```bash
# Generate & store in one command
core put cluster/GRAFANA_OIDC_CLIENT_SECRET \
  GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
```

**Use in Kubernetes Secret:**
```bash
# Create Secret using Vault secret
kubectl create secret generic grafana-oidc \
  --from-literal=client-secret="$(core get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
  -n logging
```

**Rotate credentials safely:**
```bash
# 1. Generate new secret
NEW_PASS=$(openssl rand -base64 24)

# 2. Store in Vault
core put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"

# 3. Update database user
psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
  -c "ALTER USER story_crater PASSWORD '$NEW_PASS';"

# 4. Update Kubernetes Secret
kubectl patch secret db-credentials -n story-crater-backend \
  --type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"

# 5. Restart pods to pick up new Secret
k rollout restart -n story-crater-backend deployment/app
```

**JWT token from CLI:**
```bash
# After device code login
core secrets login

# Token is cached and auto-renewed
# Use for API calls
VAULT_TOKEN=$(cat ~/.talos/vault)
curl -H "X-Vault-Token: $VAULT_TOKEN" \
  https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY
```

## Monitoring

**Vault status:**
```bash
# Check if sealed
core status

# If sealed (disaster recovery):
# See /TROUBLESHOOTING.md § Vault Sealed
```

**Audit log (who accessed what):**
```bash
# Enable audit logging (already enabled by helmfile)
# Logs stored in Loki under vault namespace

# View recent access
core audit log --limit 50

# Export for compliance
core audit export --format json > vault-audit.json
```

## Security Rules

✅ **DO:**
- Store ALL secrets in Vault (never .env in git)
- Use field name = variable name
- Always use `--key` flag when retrieving
- Rotate credentials on schedule (quarterly)
- Review audit log for anomalies

❌ **DON'T:**
- Commit `.env` with real secrets (only `.env.example`)
- Use positional arguments (must use `--key`)
- Share Vault root token
- Store secrets in pod env (use Secret volumes)

## Troubleshooting

**Cannot login (Authentik OIDC fails):**
```bash
# Check Authentik is running
k get pods -n iam -l app=authentik

# Verify OIDC app in Authentik console
# Applications → Vault OIDC → Check client ID/secret

# Restart Vault to re-sync OIDC config
k rollout restart -n iam statefulset/vault
```

**Vault is sealed:**
```bash
# Check status
core status

# If sealed, use unseal keys (stored in MinIO backup)
# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
```

**Secret not found:**
```bash
# Verify path exists
core list cluster

# Check secret name (case-sensitive)
core get cluster/anthropic_api_key --key anthropic_api_key  # won't work
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY  # correct
```

**vsource not expanding secrets:**
```bash
# Verify .env has empty value
grep ANTHROPIC_API_KEY .env
# → Should be: ANTHROPIC_API_KEY=  (empty, not a value)

# Verify Vault is accessible
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → Should return secret

# Run vsource explicitly
vsource .env
echo $ANTHROPIC_API_KEY  # should be populated
```

See `/TROUBLESHOOTING.md` for full incident guide.