diff --git a/SECRET_ROTATION.md b/SECRET_ROTATION.md new file mode 100644 index 0000000..40c682e --- /dev/null +++ b/SECRET_ROTATION.md @@ -0,0 +1,359 @@ +# Secret Rotation Strategy + +## Overview + +Secrets in this cluster are encrypted with SOPS (age backend). Rotation must maintain: +1. **No downtime** - rolling deployments during rotation +2. **Audit trail** - git history shows what was rotated when +3. **Age key rotation** - age master key itself should rotate periodically +4. **Graceful transitions** - old and new secrets coexist briefly during rollout + +--- + +## Secret Categories & Rotation Frequency + +| Category | Location | Rotation | Owner | +|----------|----------|----------|-------| +| **OAuth Client Secrets** | `k8s/argocd/secrets/minio-oidc.enc.yaml`, `minio-oidc`, etc | 90d | FIPS admins | +| **Service Account Tokens** | `portfolio-agent-oidc`, `memory-agent-oidc`, etc | 90d | FIPS admins | +| **Forgejo Registry Token** | `k8s/argocd/secrets/forgejo-registry-secret.enc.yaml` | 90d | FIPS admins | +| **MinIO Root Credentials** | `k8s/argocd/secrets/minio-secrets.enc.yaml` | 180d | FIPS admins | +| **Vault Unseal Key** | Vault init output (not in git) | Never | Manual only | +| **Age Master Key** | `~/.sops/age/keys.txt` (local, not in repo) | 365d | FIPS admins | + +--- + +## Rotation Process by Secret Type + +### Type A: OAuth Client Secrets (minio, grafana, forgejo, etc) + +**Affected:** MinIO, Grafana, Forgejo, ArgoCD + +**Steps:** + +1. **Provision new secret in Authentik** (manually or via script): + ```bash + # In Authentik UI: Applications → {app} → Providers → {provider} → Client Secret + # OR via API: + curl -X PATCH https://authentik.riotpiao.com/api/v3/oauth2/applications/{id}/ \ + -H "Authorization: Bearer $AUTHENTIK_TOKEN" \ + -d '{"client_secret": "'$(openssl rand -hex 32)'"}' + ``` + +2. **Update k8s Secret** (SOPS-encrypted): + ```bash + # Decrypt, edit, re-encrypt + sops k8s/argocd/secrets/minio-oidc.enc.yaml + # Change MINIO_IDENTITY_OPENID_CLIENT_SECRET value + # Save (sops auto-encrypts on exit) + ``` + +3. **Commit to git**: + ```bash + git add k8s/argocd/secrets/minio-oidc.enc.yaml + git commit -m "chore: rotate minio oidc client secret (90d)" + git push + ``` + +4. **Monitor rollout**: + ```bash + kubectl rollout status deployment/minio-cluster-az-a -n storage + # MinIO pod restarts → picks up new secret → re-connects to Authentik with new creds + ``` + +5. **Delete old secret in Authentik** (optional, but recommended): + - Wait 24h after successful rollout to ensure no pod is using old secret + - Delete via UI or API + +--- + +### Type B: Forgejo Registry Token + +**Affected:** CI pipelines that build & push images + +**Steps:** + +1. **Generate new token in Forgejo**: + ```bash + # UI: Settings → Access Tokens → Generate new token + # OR API: + curl -X POST https://forgejo.riotpiao.com/api/v1/user/tokens \ + -H "Authorization: token $FORGEJO_ADMIN_TOKEN" \ + -d '{"name": "registry-push-$(date +%s)", "scopes": ["write:registry"]}' + ``` + +2. **Update secret**: + ```bash + sops k8s/argocd/secrets/forgejo-registry-secret.enc.yaml + # Update REGISTRY_TOKEN + ``` + +3. **Commit & sync**: + ```bash + git add k8s/argocd/secrets/forgejo-registry-secret.enc.yaml + git commit -m "chore: rotate forgejo registry token" + git push + ``` + +4. **Delete old token in Forgejo**: + ```bash + curl -X DELETE https://forgejo.riotpiao.com/api/v1/user/tokens/{token_id} \ + -H "Authorization: token $FORGEJO_ADMIN_TOKEN" + ``` + +--- + +### Type C: MinIO Root Credentials + +**Affected:** MinIO console, backup procedures, operator provisioning + +**Steps:** + +1. **Backup current credentials**: + ```bash + sops -d k8s/argocd/secrets/minio-secrets.enc.yaml > /tmp/minio-secrets-backup-$(date +%Y%m%d).txt + # Store securely (password manager, encrypted USB, etc) + ``` + +2. **Update both secrets**: + - `minio-secrets.enc.yaml` (operator reads this) + - MinIO user password via `mc` or S3 API + + ```bash + # Via mc: + mc alias set minio-local https://minio-api.riotpiao.com $OLD_ACCESS_KEY $OLD_SECRET_KEY + mc admin user svc-account add minio-local $OLD_ACCESS_KEY + # Then change password in MinIO UI or via API + ``` + +3. **Update k8s Secret**: + ```bash + sops k8s/argocd/secrets/minio-secrets.enc.yaml + # Update MINIO_ROOT_USER / MINIO_ROOT_PASSWORD + ``` + +4. **Commit**: + ```bash + git commit -m "chore: rotate minio root credentials (180d)" + git push + ``` + +5. **Wait for operator to roll**: + ```bash + kubectl -n storage rollout status statefulset/minio-cluster-az-a + ``` + +6. **Verify with new credentials**: + ```bash + mc alias set minio-new https://minio-api.riotpiao.com $NEW_ACCESS_KEY $NEW_SECRET_KEY + mc ls minio-new/vault # Should list objects without error + ``` + +--- + +### Type D: Service Account Credentials (portfolio-agent, memory-agent) + +**Affected:** In-cluster apps using Authentik OAuth + +**Steps:** + +1. **Regenerate in Authentik**: + ```bash + # Run provisioning script (generates new credentials if not found): + export AUTHENTIK_BOOTSTRAP_TOKEN=... + python3 scripts/iam/authentik-provision.py + + # Or manually in UI: Applications → {service-account-app} → Credentials + ``` + +2. **Extract new credentials**: + ```bash + # From Authentik API + CRED_ID=$(curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/" \ + -H "Authorization: Bearer $TOKEN" | jq '.[0].pk') + curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/$CRED_ID/access_keys/" \ + -H "Authorization: Bearer $TOKEN" + ``` + +3. **Update k8s Secret**: + ```bash + kubectl -n portfolio edit secret portfolio-agent-oidc + # Update CLIENT_ID, CLIENT_SECRET, TOKEN_URL + ``` + +4. **Verify with old credentials (during transition)**: + ```bash + # Old token should still work for 24h (if Authentik supports grace periods) + curl -X POST $OLD_TOKEN_URL \ + -d "grant_type=client_credentials" \ + -d "client_id=$OLD_CLIENT_ID" \ + -d "client_secret=$OLD_CLIENT_SECRET" + ``` + +5. **Trigger pod rollout**: + ```bash + kubectl rollout restart deployment/portfolio -n portfolio + ``` + +6. **Delete old credential in Authentik**: + - Wait 24h for all pods to restart + - Delete via UI or API + +--- + +## Age Master Key Rotation (Annual) + +The age encryption key is stored locally and not in git (by design). Rotation is more complex: + +### Prerequisites +- All active encryption keys must be available locally +- Someone (FIPS admin) must hold current age key + +### Process + +1. **Generate new age key**: + ```bash + age-keygen -o ~/.sops/age/keys-new.txt + ``` + +2. **Update SOPS config to use both old and new keys**: + ```yaml + # .sops.yaml + creation_rules: + - path_regex: k8s/argocd/secrets/.*\.enc\.yaml + key_groups: + - age: | + age1old... (old key) + age1new... (new key) + ``` + +3. **Re-encrypt all secrets with both keys**: + ```bash + for file in k8s/argocd/secrets/*.enc.yaml; do + sops -e -i "$file" # Re-encrypt with both keys in config + done + ``` + +4. **Commit transition state**: + ```bash + git commit -m "chore: age key rotation - transition phase (both keys active)" + git push + ``` + +5. **Distribute new key securely**: + - Send new key to all admins via secure channel + - Store in password manager / HSM + - Backup to encrypted media + +6. **After 30d, remove old key from config**: + ```yaml + # .sops.yaml - only new key now + creation_rules: + - path_regex: k8s/argocd/secrets/.*\.enc\.yaml + key_groups: + - age: | + age1new... (new key only) + ``` + +7. **Re-encrypt one more time**: + ```bash + for file in k8s/argocd/secrets/*.enc.yaml; do + sops -e -i "$file" + done + git commit -m "chore: age key rotation - removed old key" + git push + ``` + +8. **Destroy old key**: + ```bash + # Each admin: + rm ~/.sops/age/keys.txt # Old key + # Keep keys-new.txt as new active key + ``` + +--- + +## Verification After Rotation + +After each rotation, verify: + +```bash +# 1. Secret updated in cluster +kubectl get secret minio-oidc -n storage -o jsonpath='{.data.MINIO_IDENTITY_OPENID_CLIENT_SECRET}' | base64 -d | wc -c +# Should show new secret length + +# 2. Pod restarted with new secret +kubectl logs -n storage minio-cluster-az-a-0 -c minio --tail=20 | grep -i "oidc\|openid" +# Should NOT show "Error" or "503" + +# 3. Service working with new credentials +curl -s https://minio.riotpiao.com/minio/login.html | grep -i "authentik" +# Should have Authentik login option + +# 4. Git audit trail +git log --oneline -5 k8s/argocd/secrets/ +# Should show rotation commits +``` + +--- + +## Calendar & Checklist + +### Q1 (Jan-Mar) +- [ ] Age key rotation (if last rotated 12m ago) +- [ ] OAuth secrets: grafana, forgejo, argocd, minio +- [ ] Forgejo registry token + +### Q2 (Apr-Jun) +- [ ] OAuth secrets rotation #2 +- [ ] Service account tokens: portfolio-agent, memory-agent + +### Q3 (Jul-Sep) +- [ ] OAuth secrets rotation #3 +- [ ] Forgejo registry token rotation #2 + +### Q4 (Oct-Dec) +- [ ] MinIO root credentials (180d rotation) +- [ ] Age key rotation (if due) +- [ ] All OAuth secrets final rotation + +--- + +## Automation Opportunities + +Future improvements: + +1. **CertManager-style Secret Controller**: Watch Authentik OAuth apps, auto-rotate client secrets +2. **CI Job for registry token**: Forgejo Actions job that rotates token monthly +3. **Vault integration**: Store secrets in Vault, sync to k8s (for MinIO, service accounts) +4. **Secret replication**: Multi-region backup of encrypted secrets + +--- + +## Emergency: Compromised Secret + +If a secret is compromised immediately: + +1. **Disable in external system** (Authentik, Forgejo, etc): + ```bash + # Authentik: mark app as inactive + # Forgejo: revoke token immediately + # MinIO: change root password via mc admin + ``` + +2. **Update k8s Secret** (even if not synced yet): + ```bash + kubectl patch secret minio-oidc -p '{"data":{"MINIO_IDENTITY_OPENID_CLIENT_SECRET":"'$(echo -n newval | base64)'"}}' + ``` + +3. **Commit rollback to git**: + ```bash + git revert HEAD # If accidental commit + # OR update with new secret and commit normally + ``` + +4. **Monitor for misuse**: + - Check auth logs in Authentik + - Check API Gateway request logs + - Check MinIO audit logs +