docs: add comprehensive secret rotation strategy
Covers: - OAuth client secrets (90d, minio/grafana/forgejo/argocd) - Forgejo registry token (90d, CI builds) - MinIO root credentials (180d, operator) - Service account credentials (90d, portfolio/memory agents) - Age master key rotation (365d, multi-phase) - Verification steps and emergency procedures - Calendar checklist and automation opportunities
This commit is contained in:
@@ -0,0 +1,359 @@
|
||||
# Secret Rotation Strategy
|
||||
|
||||
## Overview
|
||||
|
||||
Secrets in this cluster are encrypted with SOPS (age backend). Rotation must maintain:
|
||||
1. **No downtime** - rolling deployments during rotation
|
||||
2. **Audit trail** - git history shows what was rotated when
|
||||
3. **Age key rotation** - age master key itself should rotate periodically
|
||||
4. **Graceful transitions** - old and new secrets coexist briefly during rollout
|
||||
|
||||
---
|
||||
|
||||
## Secret Categories & Rotation Frequency
|
||||
|
||||
| Category | Location | Rotation | Owner |
|
||||
|----------|----------|----------|-------|
|
||||
| **OAuth Client Secrets** | `k8s/argocd/secrets/minio-oidc.enc.yaml`, `minio-oidc`, etc | 90d | FIPS admins |
|
||||
| **Service Account Tokens** | `portfolio-agent-oidc`, `memory-agent-oidc`, etc | 90d | FIPS admins |
|
||||
| **Forgejo Registry Token** | `k8s/argocd/secrets/forgejo-registry-secret.enc.yaml` | 90d | FIPS admins |
|
||||
| **MinIO Root Credentials** | `k8s/argocd/secrets/minio-secrets.enc.yaml` | 180d | FIPS admins |
|
||||
| **Vault Unseal Key** | Vault init output (not in git) | Never | Manual only |
|
||||
| **Age Master Key** | `~/.sops/age/keys.txt` (local, not in repo) | 365d | FIPS admins |
|
||||
|
||||
---
|
||||
|
||||
## Rotation Process by Secret Type
|
||||
|
||||
### Type A: OAuth Client Secrets (minio, grafana, forgejo, etc)
|
||||
|
||||
**Affected:** MinIO, Grafana, Forgejo, ArgoCD
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Provision new secret in Authentik** (manually or via script):
|
||||
```bash
|
||||
# In Authentik UI: Applications → {app} → Providers → {provider} → Client Secret
|
||||
# OR via API:
|
||||
curl -X PATCH https://authentik.riotpiao.com/api/v3/oauth2/applications/{id}/ \
|
||||
-H "Authorization: Bearer $AUTHENTIK_TOKEN" \
|
||||
-d '{"client_secret": "'$(openssl rand -hex 32)'"}'
|
||||
```
|
||||
|
||||
2. **Update k8s Secret** (SOPS-encrypted):
|
||||
```bash
|
||||
# Decrypt, edit, re-encrypt
|
||||
sops k8s/argocd/secrets/minio-oidc.enc.yaml
|
||||
# Change MINIO_IDENTITY_OPENID_CLIENT_SECRET value
|
||||
# Save (sops auto-encrypts on exit)
|
||||
```
|
||||
|
||||
3. **Commit to git**:
|
||||
```bash
|
||||
git add k8s/argocd/secrets/minio-oidc.enc.yaml
|
||||
git commit -m "chore: rotate minio oidc client secret (90d)"
|
||||
git push
|
||||
```
|
||||
|
||||
4. **Monitor rollout**:
|
||||
```bash
|
||||
kubectl rollout status deployment/minio-cluster-az-a -n storage
|
||||
# MinIO pod restarts → picks up new secret → re-connects to Authentik with new creds
|
||||
```
|
||||
|
||||
5. **Delete old secret in Authentik** (optional, but recommended):
|
||||
- Wait 24h after successful rollout to ensure no pod is using old secret
|
||||
- Delete via UI or API
|
||||
|
||||
---
|
||||
|
||||
### Type B: Forgejo Registry Token
|
||||
|
||||
**Affected:** CI pipelines that build & push images
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Generate new token in Forgejo**:
|
||||
```bash
|
||||
# UI: Settings → Access Tokens → Generate new token
|
||||
# OR API:
|
||||
curl -X POST https://forgejo.riotpiao.com/api/v1/user/tokens \
|
||||
-H "Authorization: token $FORGEJO_ADMIN_TOKEN" \
|
||||
-d '{"name": "registry-push-$(date +%s)", "scopes": ["write:registry"]}'
|
||||
```
|
||||
|
||||
2. **Update secret**:
|
||||
```bash
|
||||
sops k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
|
||||
# Update REGISTRY_TOKEN
|
||||
```
|
||||
|
||||
3. **Commit & sync**:
|
||||
```bash
|
||||
git add k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
|
||||
git commit -m "chore: rotate forgejo registry token"
|
||||
git push
|
||||
```
|
||||
|
||||
4. **Delete old token in Forgejo**:
|
||||
```bash
|
||||
curl -X DELETE https://forgejo.riotpiao.com/api/v1/user/tokens/{token_id} \
|
||||
-H "Authorization: token $FORGEJO_ADMIN_TOKEN"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Type C: MinIO Root Credentials
|
||||
|
||||
**Affected:** MinIO console, backup procedures, operator provisioning
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Backup current credentials**:
|
||||
```bash
|
||||
sops -d k8s/argocd/secrets/minio-secrets.enc.yaml > /tmp/minio-secrets-backup-$(date +%Y%m%d).txt
|
||||
# Store securely (password manager, encrypted USB, etc)
|
||||
```
|
||||
|
||||
2. **Update both secrets**:
|
||||
- `minio-secrets.enc.yaml` (operator reads this)
|
||||
- MinIO user password via `mc` or S3 API
|
||||
|
||||
```bash
|
||||
# Via mc:
|
||||
mc alias set minio-local https://minio-api.riotpiao.com $OLD_ACCESS_KEY $OLD_SECRET_KEY
|
||||
mc admin user svc-account add minio-local $OLD_ACCESS_KEY
|
||||
# Then change password in MinIO UI or via API
|
||||
```
|
||||
|
||||
3. **Update k8s Secret**:
|
||||
```bash
|
||||
sops k8s/argocd/secrets/minio-secrets.enc.yaml
|
||||
# Update MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
|
||||
```
|
||||
|
||||
4. **Commit**:
|
||||
```bash
|
||||
git commit -m "chore: rotate minio root credentials (180d)"
|
||||
git push
|
||||
```
|
||||
|
||||
5. **Wait for operator to roll**:
|
||||
```bash
|
||||
kubectl -n storage rollout status statefulset/minio-cluster-az-a
|
||||
```
|
||||
|
||||
6. **Verify with new credentials**:
|
||||
```bash
|
||||
mc alias set minio-new https://minio-api.riotpiao.com $NEW_ACCESS_KEY $NEW_SECRET_KEY
|
||||
mc ls minio-new/vault # Should list objects without error
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Type D: Service Account Credentials (portfolio-agent, memory-agent)
|
||||
|
||||
**Affected:** In-cluster apps using Authentik OAuth
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Regenerate in Authentik**:
|
||||
```bash
|
||||
# Run provisioning script (generates new credentials if not found):
|
||||
export AUTHENTIK_BOOTSTRAP_TOKEN=...
|
||||
python3 scripts/iam/authentik-provision.py
|
||||
|
||||
# Or manually in UI: Applications → {service-account-app} → Credentials
|
||||
```
|
||||
|
||||
2. **Extract new credentials**:
|
||||
```bash
|
||||
# From Authentik API
|
||||
CRED_ID=$(curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/" \
|
||||
-H "Authorization: Bearer $TOKEN" | jq '.[0].pk')
|
||||
curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/$CRED_ID/access_keys/" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
3. **Update k8s Secret**:
|
||||
```bash
|
||||
kubectl -n portfolio edit secret portfolio-agent-oidc
|
||||
# Update CLIENT_ID, CLIENT_SECRET, TOKEN_URL
|
||||
```
|
||||
|
||||
4. **Verify with old credentials (during transition)**:
|
||||
```bash
|
||||
# Old token should still work for 24h (if Authentik supports grace periods)
|
||||
curl -X POST $OLD_TOKEN_URL \
|
||||
-d "grant_type=client_credentials" \
|
||||
-d "client_id=$OLD_CLIENT_ID" \
|
||||
-d "client_secret=$OLD_CLIENT_SECRET"
|
||||
```
|
||||
|
||||
5. **Trigger pod rollout**:
|
||||
```bash
|
||||
kubectl rollout restart deployment/portfolio -n portfolio
|
||||
```
|
||||
|
||||
6. **Delete old credential in Authentik**:
|
||||
- Wait 24h for all pods to restart
|
||||
- Delete via UI or API
|
||||
|
||||
---
|
||||
|
||||
## Age Master Key Rotation (Annual)
|
||||
|
||||
The age encryption key is stored locally and not in git (by design). Rotation is more complex:
|
||||
|
||||
### Prerequisites
|
||||
- All active encryption keys must be available locally
|
||||
- Someone (FIPS admin) must hold current age key
|
||||
|
||||
### Process
|
||||
|
||||
1. **Generate new age key**:
|
||||
```bash
|
||||
age-keygen -o ~/.sops/age/keys-new.txt
|
||||
```
|
||||
|
||||
2. **Update SOPS config to use both old and new keys**:
|
||||
```yaml
|
||||
# .sops.yaml
|
||||
creation_rules:
|
||||
- path_regex: k8s/argocd/secrets/.*\.enc\.yaml
|
||||
key_groups:
|
||||
- age: |
|
||||
age1old... (old key)
|
||||
age1new... (new key)
|
||||
```
|
||||
|
||||
3. **Re-encrypt all secrets with both keys**:
|
||||
```bash
|
||||
for file in k8s/argocd/secrets/*.enc.yaml; do
|
||||
sops -e -i "$file" # Re-encrypt with both keys in config
|
||||
done
|
||||
```
|
||||
|
||||
4. **Commit transition state**:
|
||||
```bash
|
||||
git commit -m "chore: age key rotation - transition phase (both keys active)"
|
||||
git push
|
||||
```
|
||||
|
||||
5. **Distribute new key securely**:
|
||||
- Send new key to all admins via secure channel
|
||||
- Store in password manager / HSM
|
||||
- Backup to encrypted media
|
||||
|
||||
6. **After 30d, remove old key from config**:
|
||||
```yaml
|
||||
# .sops.yaml - only new key now
|
||||
creation_rules:
|
||||
- path_regex: k8s/argocd/secrets/.*\.enc\.yaml
|
||||
key_groups:
|
||||
- age: |
|
||||
age1new... (new key only)
|
||||
```
|
||||
|
||||
7. **Re-encrypt one more time**:
|
||||
```bash
|
||||
for file in k8s/argocd/secrets/*.enc.yaml; do
|
||||
sops -e -i "$file"
|
||||
done
|
||||
git commit -m "chore: age key rotation - removed old key"
|
||||
git push
|
||||
```
|
||||
|
||||
8. **Destroy old key**:
|
||||
```bash
|
||||
# Each admin:
|
||||
rm ~/.sops/age/keys.txt # Old key
|
||||
# Keep keys-new.txt as new active key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification After Rotation
|
||||
|
||||
After each rotation, verify:
|
||||
|
||||
```bash
|
||||
# 1. Secret updated in cluster
|
||||
kubectl get secret minio-oidc -n storage -o jsonpath='{.data.MINIO_IDENTITY_OPENID_CLIENT_SECRET}' | base64 -d | wc -c
|
||||
# Should show new secret length
|
||||
|
||||
# 2. Pod restarted with new secret
|
||||
kubectl logs -n storage minio-cluster-az-a-0 -c minio --tail=20 | grep -i "oidc\|openid"
|
||||
# Should NOT show "Error" or "503"
|
||||
|
||||
# 3. Service working with new credentials
|
||||
curl -s https://minio.riotpiao.com/minio/login.html | grep -i "authentik"
|
||||
# Should have Authentik login option
|
||||
|
||||
# 4. Git audit trail
|
||||
git log --oneline -5 k8s/argocd/secrets/
|
||||
# Should show rotation commits
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Calendar & Checklist
|
||||
|
||||
### Q1 (Jan-Mar)
|
||||
- [ ] Age key rotation (if last rotated 12m ago)
|
||||
- [ ] OAuth secrets: grafana, forgejo, argocd, minio
|
||||
- [ ] Forgejo registry token
|
||||
|
||||
### Q2 (Apr-Jun)
|
||||
- [ ] OAuth secrets rotation #2
|
||||
- [ ] Service account tokens: portfolio-agent, memory-agent
|
||||
|
||||
### Q3 (Jul-Sep)
|
||||
- [ ] OAuth secrets rotation #3
|
||||
- [ ] Forgejo registry token rotation #2
|
||||
|
||||
### Q4 (Oct-Dec)
|
||||
- [ ] MinIO root credentials (180d rotation)
|
||||
- [ ] Age key rotation (if due)
|
||||
- [ ] All OAuth secrets final rotation
|
||||
|
||||
---
|
||||
|
||||
## Automation Opportunities
|
||||
|
||||
Future improvements:
|
||||
|
||||
1. **CertManager-style Secret Controller**: Watch Authentik OAuth apps, auto-rotate client secrets
|
||||
2. **CI Job for registry token**: Forgejo Actions job that rotates token monthly
|
||||
3. **Vault integration**: Store secrets in Vault, sync to k8s (for MinIO, service accounts)
|
||||
4. **Secret replication**: Multi-region backup of encrypted secrets
|
||||
|
||||
---
|
||||
|
||||
## Emergency: Compromised Secret
|
||||
|
||||
If a secret is compromised immediately:
|
||||
|
||||
1. **Disable in external system** (Authentik, Forgejo, etc):
|
||||
```bash
|
||||
# Authentik: mark app as inactive
|
||||
# Forgejo: revoke token immediately
|
||||
# MinIO: change root password via mc admin
|
||||
```
|
||||
|
||||
2. **Update k8s Secret** (even if not synced yet):
|
||||
```bash
|
||||
kubectl patch secret minio-oidc -p '{"data":{"MINIO_IDENTITY_OPENID_CLIENT_SECRET":"'$(echo -n newval | base64)'"}}'
|
||||
```
|
||||
|
||||
3. **Commit rollback to git**:
|
||||
```bash
|
||||
git revert HEAD # If accidental commit
|
||||
# OR update with new secret and commit normally
|
||||
```
|
||||
|
||||
4. **Monitor for misuse**:
|
||||
- Check auth logs in Authentik
|
||||
- Check API Gateway request logs
|
||||
- Check MinIO audit logs
|
||||
|
||||
Reference in New Issue
Block a user