homelab/project-usage/vault-secrets.md

Vault: Secret Management & JWT Auth

Vault: https://vault.riotpiao.homelab.com
Internal: vault.iam.svc.cluster.local:8200
Namespace: iam

When to Use

• Store secrets — Database passwords, API keys, TLS certs
• Rotate credentials — Auto-rotate, track rotation history
• JWT validation — Verify tokens from Authentik, no external call needed
• Audit trail — Who accessed what secret, when

Quick Start

1. Login to Vault:

# Browser: https://vault.riotpiao.homelab.com
# Auth method: OIDC → "Sign in with Authentik" (federated)
# Or: Device code → talos secrets login (CLI)

# Via CLI (device code flow)
talos secrets login
# → Opens browser, approve device code
# → Token cached in ~/.talos/vault

2. Store a secret:

# Field name = variable name (SCREAMING_SNAKE_CASE)
talos put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
talos put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"

3. Retrieve a secret:

# Always use --key flag
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → sk-...

# Full secret as JSON
talos get cluster/ANTHROPIC_API_KEY --json

4. Load into shell (helmfile, scripts):

vsource .env
# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
# → Passes hardcoded vars as-is (DEBUG=true)

helmfile diff
helmfile apply

Vault Paths (KV v2)

Naming convention: cluster/<VARIABLE_NAME>

cluster/
├── ANTHROPIC_API_KEY            # third-party API
├── STORY_CRATER_DB_PASS         # database password
├── MINIO_ROOT_PASSWORD          # MinIO credentials
├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
├── iam/
│   ├── federation               # Authentik OIDC app config
│   ├── roles/admin              # RBAC role definitions
│   ├── services/grafana         # OAuth2 client info
│   ├── agents/ci-bot            # machine credentials
│   └── bindings/alice           # user→role mappings
└── kubernetes/
    ├── ingress-tls              # TLS cert private keys
    └── pull-secrets             # Docker registry creds

Common Patterns

Store generated secret immediately (keeps it out of shell history):

# Generate & store in one command
talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
  GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"

Use in Kubernetes Secret:

# Create Secret using Vault secret
kubectl create secret generic grafana-oidc \
  --from-literal=client-secret="$(talos get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
  -n logging

Rotate credentials safely:

# 1. Generate new secret
NEW_PASS=$(openssl rand -base64 24)

# 2. Store in Vault
talos put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"

# 3. Update database user
psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
  -c "ALTER USER story_crater PASSWORD '$NEW_PASS';"

# 4. Update Kubernetes Secret
kubectl patch secret db-credentials -n story-crater-backend \
  --type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"

# 5. Restart pods to pick up new Secret
k rollout restart -n story-crater-backend deployment/app

JWT token from CLI:

# After device code login
talos secrets login

# Token is cached and auto-renewed
# Use for API calls
VAULT_TOKEN=$(cat ~/.talos/vault)
curl -H "X-Vault-Token: $VAULT_TOKEN" \
  https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY

Monitoring

Vault status:

# Check if sealed
talos status vault

# If sealed (disaster recovery):
# See /TROUBLESHOOTING.md § Vault Sealed

Audit log (who accessed what):

# Enable audit logging (already enabled by helmfile)
# Logs stored in Loki under vault namespace

# View recent access
talos audit log --limit 50

# Export for compliance
talos audit export --format json > vault-audit.json

Security Rules

✅ DO:

• Store ALL secrets in Vault (never .env in git)
• Use field name = variable name
• Always use --key flag when retrieving
• Rotate credentials on schedule (quarterly)
• Review audit log for anomalies

❌ DON'T:

• Commit .env with real secrets (only .env.example)
• Use positional arguments (must use --key)
• Share Vault root token
• Store secrets in pod env (use Secret volumes)

Troubleshooting

Cannot login (Authentik OIDC fails):

# Check Authentik is running
k get pods -n iam -l app=authentik

# Verify OIDC app in Authentik console
# Applications → Vault OIDC → Check client ID/secret

# Restart Vault to re-sync OIDC config
k rollout restart -n iam statefulset/vault

Vault is sealed:

# Check status
talos status vault

# If sealed, use unseal keys (stored in MinIO backup)
# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps

Secret not found:

# Verify path exists
talos list cluster

# Check secret name (case-sensitive)
talos get cluster/anthropic_api_key --key anthropic_api_key  # won't work
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY  # correct

vsource not expanding secrets:

# Verify .env has empty value
grep ANTHROPIC_API_KEY .env
# → Should be: ANTHROPIC_API_KEY=  (empty, not a value)

# Verify Vault is accessible
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → Should return secret

# Run vsource explicitly
vsource .env
echo $ANTHROPIC_API_KEY  # should be populated

See /TROUBLESHOOTING.md for full incident guide.