# Secret Rotation Strategy

## Overview

Secrets in this cluster are encrypted with SOPS (age backend). Rotation must maintain:
1. **No downtime** - rolling deployments during rotation
2. **Audit trail** - git history shows what was rotated when
3. **Age key rotation** - age master key itself should rotate periodically
4. **Graceful transitions** - old and new secrets coexist briefly during rollout

---

## Secret Categories & Rotation Frequency

| Category | Location | Rotation | Owner |
|----------|----------|----------|-------|
| **OAuth Client Secrets** | `k8s/argocd/secrets/minio-oidc.enc.yaml`, `minio-oidc`, etc | 90d | FIPS admins |
| **Service Account Tokens** | `portfolio-agent-oidc`, `memory-agent-oidc`, etc | 90d | FIPS admins |
| **Forgejo Registry Token** | `k8s/argocd/secrets/forgejo-registry-secret.enc.yaml` | 90d | FIPS admins |
| **MinIO Root Credentials** | `k8s/argocd/secrets/minio-secrets.enc.yaml` | 180d | FIPS admins |
| **Vault Unseal Key** | Vault init output (not in git) | Never | Manual only |
| **Age Master Key** | `~/.sops/age/keys.txt` (local, not in repo) | 365d | FIPS admins |

---

## Rotation Process by Secret Type

### Type A: OAuth Client Secrets (minio, grafana, forgejo, etc)

**Affected:** MinIO, Grafana, Forgejo, ArgoCD

**Steps:**

1. **Provision new secret in Authentik** (manually or via script):
   ```bash
   # In Authentik UI: Applications → {app} → Providers → {provider} → Client Secret
   # OR via API:
   curl -X PATCH https://authentik.riotpiao.com/api/v3/oauth2/applications/{id}/ \
     -H "Authorization: Bearer $AUTHENTIK_TOKEN" \
     -d '{"client_secret": "'$(openssl rand -hex 32)'"}'
   ```

2. **Update k8s Secret** (SOPS-encrypted):
   ```bash
   # Decrypt, edit, re-encrypt
   sops k8s/argocd/secrets/minio-oidc.enc.yaml
   # Change MINIO_IDENTITY_OPENID_CLIENT_SECRET value
   # Save (sops auto-encrypts on exit)
   ```

3. **Commit to git**:
   ```bash
   git add k8s/argocd/secrets/minio-oidc.enc.yaml
   git commit -m "chore: rotate minio oidc client secret (90d)"
   git push
   ```

4. **Monitor rollout**:
   ```bash
   kubectl rollout status deployment/minio-cluster-az-a -n storage
   # MinIO pod restarts → picks up new secret → re-connects to Authentik with new creds
   ```

5. **Delete old secret in Authentik** (optional, but recommended):
   - Wait 24h after successful rollout to ensure no pod is using old secret
   - Delete via UI or API

---

### Type B: Forgejo Registry Token

**Affected:** CI pipelines that build & push images

**Steps:**

1. **Generate new token in Forgejo**:
   ```bash
   # UI: Settings → Access Tokens → Generate new token
   # OR API:
   curl -X POST https://forgejo.riotpiao.com/api/v1/user/tokens \
     -H "Authorization: token $FORGEJO_ADMIN_TOKEN" \
     -d '{"name": "registry-push-$(date +%s)", "scopes": ["write:registry"]}'
   ```

2. **Update secret**:
   ```bash
   sops k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
   # Update REGISTRY_TOKEN
   ```

3. **Commit & sync**:
   ```bash
   git add k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
   git commit -m "chore: rotate forgejo registry token"
   git push
   ```

4. **Delete old token in Forgejo**:
   ```bash
   curl -X DELETE https://forgejo.riotpiao.com/api/v1/user/tokens/{token_id} \
     -H "Authorization: token $FORGEJO_ADMIN_TOKEN"
   ```

---

### Type C: MinIO Root Credentials

**Affected:** MinIO console, backup procedures, operator provisioning

**Steps:**

1. **Backup current credentials**:
   ```bash
   sops -d k8s/argocd/secrets/minio-secrets.enc.yaml > /tmp/minio-secrets-backup-$(date +%Y%m%d).txt
   # Store securely (password manager, encrypted USB, etc)
   ```

2. **Update both secrets**:
   - `minio-secrets.enc.yaml` (operator reads this)
   - MinIO user password via `mc` or S3 API

   ```bash
   # Via mc:
   mc alias set minio-local https://minio-api.riotpiao.com $OLD_ACCESS_KEY $OLD_SECRET_KEY
   mc admin user svc-account add minio-local $OLD_ACCESS_KEY
   # Then change password in MinIO UI or via API
   ```

3. **Update k8s Secret**:
   ```bash
   sops k8s/argocd/secrets/minio-secrets.enc.yaml
   # Update MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
   ```

4. **Commit**:
   ```bash
   git commit -m "chore: rotate minio root credentials (180d)"
   git push
   ```

5. **Wait for operator to roll**:
   ```bash
   kubectl -n storage rollout status statefulset/minio-cluster-az-a
   ```

6. **Verify with new credentials**:
   ```bash
   mc alias set minio-new https://minio-api.riotpiao.com $NEW_ACCESS_KEY $NEW_SECRET_KEY
   mc ls minio-new/vault  # Should list objects without error
   ```

---

### Type D: Service Account Credentials (portfolio-agent, memory-agent)

**Affected:** In-cluster apps using Authentik OAuth

**Steps:**

1. **Regenerate in Authentik**:
   ```bash
   # Run provisioning script (generates new credentials if not found):
   export AUTHENTIK_BOOTSTRAP_TOKEN=...
   python3 scripts/iam/authentik-provision.py
   
   # Or manually in UI: Applications → {service-account-app} → Credentials
   ```

2. **Extract new credentials**:
   ```bash
   # From Authentik API
   CRED_ID=$(curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/" \
     -H "Authorization: Bearer $TOKEN" | jq '.[0].pk')
   curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/$CRED_ID/access_keys/" \
     -H "Authorization: Bearer $TOKEN"
   ```

3. **Update k8s Secret**:
   ```bash
   kubectl -n portfolio edit secret portfolio-agent-oidc
   # Update CLIENT_ID, CLIENT_SECRET, TOKEN_URL
   ```

4. **Verify with old credentials (during transition)**:
   ```bash
   # Old token should still work for 24h (if Authentik supports grace periods)
   curl -X POST $OLD_TOKEN_URL \
     -d "grant_type=client_credentials" \
     -d "client_id=$OLD_CLIENT_ID" \
     -d "client_secret=$OLD_CLIENT_SECRET"
   ```

5. **Trigger pod rollout**:
   ```bash
   kubectl rollout restart deployment/portfolio -n portfolio
   ```

6. **Delete old credential in Authentik**:
   - Wait 24h for all pods to restart
   - Delete via UI or API

---

## Age Master Key Rotation (Annual)

The age encryption key is stored locally and not in git (by design). Rotation is more complex:

### Prerequisites
- All active encryption keys must be available locally
- Someone (FIPS admin) must hold current age key

### Process

1. **Generate new age key**:
   ```bash
   age-keygen -o ~/.sops/age/keys-new.txt
   ```

2. **Update SOPS config to use both old and new keys**:
   ```yaml
   # .sops.yaml
   creation_rules:
     - path_regex: k8s/argocd/secrets/.*\.enc\.yaml
       key_groups:
       - age: |
           age1old... (old key)
           age1new... (new key)
   ```

3. **Re-encrypt all secrets with both keys**:
   ```bash
   for file in k8s/argocd/secrets/*.enc.yaml; do
     sops -e -i "$file"  # Re-encrypt with both keys in config
   done
   ```

4. **Commit transition state**:
   ```bash
   git commit -m "chore: age key rotation - transition phase (both keys active)"
   git push
   ```

5. **Distribute new key securely**:
   - Send new key to all admins via secure channel
   - Store in password manager / HSM
   - Backup to encrypted media

6. **After 30d, remove old key from config**:
   ```yaml
   # .sops.yaml - only new key now
   creation_rules:
     - path_regex: k8s/argocd/secrets/.*\.enc\.yaml
       key_groups:
       - age: |
           age1new... (new key only)
   ```

7. **Re-encrypt one more time**:
   ```bash
   for file in k8s/argocd/secrets/*.enc.yaml; do
     sops -e -i "$file"
   done
   git commit -m "chore: age key rotation - removed old key"
   git push
   ```

8. **Destroy old key**:
   ```bash
   # Each admin:
   rm ~/.sops/age/keys.txt  # Old key
   # Keep keys-new.txt as new active key
   ```

---

## Verification After Rotation

After each rotation, verify:

```bash
# 1. Secret updated in cluster
kubectl get secret minio-oidc -n storage -o jsonpath='{.data.MINIO_IDENTITY_OPENID_CLIENT_SECRET}' | base64 -d | wc -c
# Should show new secret length

# 2. Pod restarted with new secret
kubectl logs -n storage minio-cluster-az-a-0 -c minio --tail=20 | grep -i "oidc\|openid"
# Should NOT show "Error" or "503"

# 3. Service working with new credentials
curl -s https://minio.riotpiao.com/minio/login.html | grep -i "authentik" 
# Should have Authentik login option

# 4. Git audit trail
git log --oneline -5 k8s/argocd/secrets/
# Should show rotation commits
```

---

## Calendar & Checklist

### Q1 (Jan-Mar)
- [ ] Age key rotation (if last rotated 12m ago)
- [ ] OAuth secrets: grafana, forgejo, argocd, minio
- [ ] Forgejo registry token

### Q2 (Apr-Jun)
- [ ] OAuth secrets rotation #2
- [ ] Service account tokens: portfolio-agent, memory-agent

### Q3 (Jul-Sep)
- [ ] OAuth secrets rotation #3
- [ ] Forgejo registry token rotation #2

### Q4 (Oct-Dec)
- [ ] MinIO root credentials (180d rotation)
- [ ] Age key rotation (if due)
- [ ] All OAuth secrets final rotation

---

## Automation Opportunities

Future improvements:

1. **CertManager-style Secret Controller**: Watch Authentik OAuth apps, auto-rotate client secrets
2. **CI Job for registry token**: Forgejo Actions job that rotates token monthly
3. **Vault integration**: Store secrets in Vault, sync to k8s (for MinIO, service accounts)
4. **Secret replication**: Multi-region backup of encrypted secrets

---

## Emergency: Compromised Secret

If a secret is compromised immediately:

1. **Disable in external system** (Authentik, Forgejo, etc):
   ```bash
   # Authentik: mark app as inactive
   # Forgejo: revoke token immediately
   # MinIO: change root password via mc admin
   ```

2. **Update k8s Secret** (even if not synced yet):
   ```bash
   kubectl patch secret minio-oidc -p '{"data":{"MINIO_IDENTITY_OPENID_CLIENT_SECRET":"'$(echo -n newval | base64)'"}}'
   ```

3. **Commit rollback to git**:
   ```bash
   git revert HEAD  # If accidental commit
   # OR update with new secret and commit normally
   ```

4. **Monitor for misuse**:
   - Check auth logs in Authentik
   - Check API Gateway request logs
   - Check MinIO audit logs