homelab/SECRET_ROTATION.md

Secret Rotation Strategy

Overview

Secrets in this cluster are encrypted with SOPS (age backend). Rotation must maintain:

1. No downtime - rolling deployments during rotation
2. Audit trail - git history shows what was rotated when
3. Age key rotation - age master key itself should rotate periodically
4. Graceful transitions - old and new secrets coexist briefly during rollout

---

Secret Categories & Rotation Frequency

Category	Location	Rotation	Owner
OAuth Client Secrets	k8s/argocd/secrets/minio-oidc.enc.yaml, minio-oidc, etc	90d	FIPS admins
Service Account Tokens	portfolio-agent-oidc, memory-agent-oidc, etc	90d	FIPS admins
Forgejo Registry Token	k8s/argocd/secrets/forgejo-registry-secret.enc.yaml	90d	FIPS admins
MinIO Root Credentials	k8s/argocd/secrets/minio-secrets.enc.yaml	180d	FIPS admins
Vault Unseal Key	Vault init output (not in git)	Never	Manual only
Age Master Key	~/.sops/age/keys.txt (local, not in repo)	365d	FIPS admins

---

Rotation Process by Secret Type

Type A: OAuth Client Secrets (minio, grafana, forgejo, etc)

Affected: MinIO, Grafana, Forgejo, ArgoCD

Steps:

1. Provision new secret in Authentik (manually or via script):

   # In Authentik UI: Applications → {app} → Providers → {provider} → Client Secret
   # OR via API:
   curl -X PATCH https://authentik.riotpiao.com/api/v3/oauth2/applications/{id}/ \
     -H "Authorization: Bearer $AUTHENTIK_TOKEN" \
     -d '{"client_secret": "'$(openssl rand -hex 32)'"}'
   

2. Update k8s Secret (SOPS-encrypted):

   # Decrypt, edit, re-encrypt
   sops k8s/argocd/secrets/minio-oidc.enc.yaml
   # Change MINIO_IDENTITY_OPENID_CLIENT_SECRET value
   # Save (sops auto-encrypts on exit)
   

3. Commit to git:

   git add k8s/argocd/secrets/minio-oidc.enc.yaml
   git commit -m "chore: rotate minio oidc client secret (90d)"
   git push
   

4. Monitor rollout:

   kubectl rollout status deployment/minio-cluster-az-a -n storage
   # MinIO pod restarts → picks up new secret → re-connects to Authentik with new creds
   

5. Delete old secret in Authentik (optional, but recommended):

   • Wait 24h after successful rollout to ensure no pod is using old secret
   • Delete via UI or API

---

Type B: Forgejo Registry Token

Affected: CI pipelines that build & push images

Steps:

1. Generate new token in Forgejo:

   # UI: Settings → Access Tokens → Generate new token
   # OR API:
   curl -X POST https://forgejo.riotpiao.com/api/v1/user/tokens \
     -H "Authorization: token $FORGEJO_ADMIN_TOKEN" \
     -d '{"name": "registry-push-$(date +%s)", "scopes": ["write:registry"]}'
   

2. Update secret:

   sops k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
   # Update REGISTRY_TOKEN
   

3. Commit & sync:

   git add k8s/argocd/secrets/forgejo-registry-secret.enc.yaml
   git commit -m "chore: rotate forgejo registry token"
   git push
   

4. Delete old token in Forgejo:

   curl -X DELETE https://forgejo.riotpiao.com/api/v1/user/tokens/{token_id} \
     -H "Authorization: token $FORGEJO_ADMIN_TOKEN"
   

---

Type C: MinIO Root Credentials

Affected: MinIO console, backup procedures, operator provisioning

Steps:

1. Backup current credentials:

   sops -d k8s/argocd/secrets/minio-secrets.enc.yaml > /tmp/minio-secrets-backup-$(date +%Y%m%d).txt
   # Store securely (password manager, encrypted USB, etc)
   

2. Update both secrets:

   • minio-secrets.enc.yaml (operator reads this)
   • MinIO user password via mc or S3 API

   # Via mc:
   mc alias set minio-local https://minio-api.riotpiao.com $OLD_ACCESS_KEY $OLD_SECRET_KEY
   mc admin user svc-account add minio-local $OLD_ACCESS_KEY
   # Then change password in MinIO UI or via API
   

3. Update k8s Secret:

   sops k8s/argocd/secrets/minio-secrets.enc.yaml
   # Update MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
   

4. Commit:

   git commit -m "chore: rotate minio root credentials (180d)"
   git push
   

5. Wait for operator to roll:

   kubectl -n storage rollout status statefulset/minio-cluster-az-a
   

6. Verify with new credentials:

   mc alias set minio-new https://minio-api.riotpiao.com $NEW_ACCESS_KEY $NEW_SECRET_KEY
   mc ls minio-new/vault  # Should list objects without error
   

---

Type D: Service Account Credentials (portfolio-agent, memory-agent)

Affected: In-cluster apps using Authentik OAuth

Steps:

1. Regenerate in Authentik:

   # Run provisioning script (generates new credentials if not found):
   export AUTHENTIK_BOOTSTRAP_TOKEN=...
   python3 scripts/iam/authentik-provision.py
   
   # Or manually in UI: Applications → {service-account-app} → Credentials
   

2. Extract new credentials:

   # From Authentik API
   CRED_ID=$(curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/" \
     -H "Authorization: Bearer $TOKEN" | jq '.[0].pk')
   curl -s "https://authentik.riotpiao.com/api/v3/core/service_accounts/$CRED_ID/access_keys/" \
     -H "Authorization: Bearer $TOKEN"
   

3. Update k8s Secret:

   kubectl -n portfolio edit secret portfolio-agent-oidc
   # Update CLIENT_ID, CLIENT_SECRET, TOKEN_URL
   

4. Verify with old credentials (during transition):

   # Old token should still work for 24h (if Authentik supports grace periods)
   curl -X POST $OLD_TOKEN_URL \
     -d "grant_type=client_credentials" \
     -d "client_id=$OLD_CLIENT_ID" \
     -d "client_secret=$OLD_CLIENT_SECRET"
   

5. Trigger pod rollout:

   kubectl rollout restart deployment/portfolio -n portfolio
   

6. Delete old credential in Authentik:

   • Wait 24h for all pods to restart
   • Delete via UI or API

---

Age Master Key Rotation (Annual)

The age encryption key is stored locally and not in git (by design). Rotation is more complex:

Prerequisites

• All active encryption keys must be available locally
• Someone (FIPS admin) must hold current age key

Process

1. Generate new age key:

   age-keygen -o ~/.sops/age/keys-new.txt
   

2. Update SOPS config to use both old and new keys:

   # .sops.yaml
   creation_rules:
     - path_regex: k8s/argocd/secrets/.*\.enc\.yaml
       key_groups:
       - age: |
           age1old... (old key)
           age1new... (new key)
   

3. Re-encrypt all secrets with both keys:

   for file in k8s/argocd/secrets/*.enc.yaml; do
     sops -e -i "$file"  # Re-encrypt with both keys in config
   done
   

4. Commit transition state:

   git commit -m "chore: age key rotation - transition phase (both keys active)"
   git push
   

5. Distribute new key securely:

   • Send new key to all admins via secure channel
   • Store in password manager / HSM
   • Backup to encrypted media

6. After 30d, remove old key from config:

   # .sops.yaml - only new key now
   creation_rules:
     - path_regex: k8s/argocd/secrets/.*\.enc\.yaml
       key_groups:
       - age: |
           age1new... (new key only)
   

7. Re-encrypt one more time:

   for file in k8s/argocd/secrets/*.enc.yaml; do
     sops -e -i "$file"
   done
   git commit -m "chore: age key rotation - removed old key"
   git push
   

8. Destroy old key:

   # Each admin:
   rm ~/.sops/age/keys.txt  # Old key
   # Keep keys-new.txt as new active key
   

---

Verification After Rotation

After each rotation, verify:

# 1. Secret updated in cluster
kubectl get secret minio-oidc -n storage -o jsonpath='{.data.MINIO_IDENTITY_OPENID_CLIENT_SECRET}' | base64 -d | wc -c
# Should show new secret length

# 2. Pod restarted with new secret
kubectl logs -n storage minio-cluster-az-a-0 -c minio --tail=20 | grep -i "oidc\|openid"
# Should NOT show "Error" or "503"

# 3. Service working with new credentials
curl -s https://minio.riotpiao.com/minio/login.html | grep -i "authentik" 
# Should have Authentik login option

# 4. Git audit trail
git log --oneline -5 k8s/argocd/secrets/
# Should show rotation commits

---

Calendar & Checklist

Q1 (Jan-Mar)

• Age key rotation (if last rotated 12m ago)
• OAuth secrets: grafana, forgejo, argocd, minio
• Forgejo registry token

Q2 (Apr-Jun)

• OAuth secrets rotation #2
• Service account tokens: portfolio-agent, memory-agent

Q3 (Jul-Sep)

• OAuth secrets rotation #3
• Forgejo registry token rotation #2

Q4 (Oct-Dec)

• MinIO root credentials (180d rotation)
• Age key rotation (if due)
• All OAuth secrets final rotation

---

Automation Opportunities

Future improvements:

1. CertManager-style Secret Controller: Watch Authentik OAuth apps, auto-rotate client secrets
2. CI Job for registry token: Forgejo Actions job that rotates token monthly
3. Vault integration: Store secrets in Vault, sync to k8s (for MinIO, service accounts)
4. Secret replication: Multi-region backup of encrypted secrets

---

Emergency: Compromised Secret

If a secret is compromised immediately:

1. Disable in external system (Authentik, Forgejo, etc):

   # Authentik: mark app as inactive
   # Forgejo: revoke token immediately
   # MinIO: change root password via mc admin
   

2. Update k8s Secret (even if not synced yet):

   kubectl patch secret minio-oidc -p '{"data":{"MINIO_IDENTITY_OPENID_CLIENT_SECRET":"'$(echo -n newval | base64)'"}}'
   

3. Commit rollback to git:

   git revert HEAD  # If accidental commit
   # OR update with new secret and commit normally
   

4. Monitor for misuse:

   • Check auth logs in Authentik
   • Check API Gateway request logs
   • Check MinIO audit logs