# Phase 0: External Secrets Operator (ESO) Setup After `terraform apply` installs ESO and the ClusterSecretStore manifest is applied, Vault needs to be configured to accept ESO's Kubernetes auth requests. ## Prerequisites - Vault is initialized and unsealed (run setup_vault.sh first) - ESO pods are Running in external-secrets-system namespace - ClusterSecretStore applied: k8s/external-secrets/clustersecretstore.yaml ## Setup Steps (Manual) Run these steps from the repo root with: ```bash export VAULT_ADDR=http://vault.iam.svc.cluster.local:8200 export VAULT_TOKEN=$(kubectl get secret vault-unseal-keys -n storage -o jsonpath='{.data.key1}' | base64 -d) # or use port-forward + login ``` ### 1. Enable Kubernetes auth method ```bash vault auth enable kubernetes || echo "Kubernetes auth already enabled" ``` ### 2. Configure K8s auth to trust the cluster ```bash # Get the K8s API server address and CA cert K8S_HOST=$(kubectl cluster-info | grep 'Kubernetes master' | awk '/https/ {print $NF}') K8S_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d) SA_TOKEN=$(kubectl get secret -n external-secrets-system $(kubectl get secret -n external-secrets-system | grep external-secrets-webhook | awk '{print $1}') -o jsonpath='{.data.token}' | base64 -d) vault write auth/kubernetes/config \ kubernetes_host="${K8S_HOST}" \ kubernetes_ca_cert="${K8S_CA_CERT}" \ token_reviewer_jwt="${SA_TOKEN}" ``` ### 3. Create external-secrets policy ```bash vault policy write external-secrets - <<'EOF' # external-secrets: read application secrets per namespace + release path "secret/data/iam/*" { capabilities = ["read"] } path "secret/data/storage/*" { capabilities = ["read"] } path "secret/data/cicd/*" { capabilities = ["read"] } path "secret/data/logging/*" { capabilities = ["read"] } path "secret/data/monitoring/*" { capabilities = ["read"] } path "secret/data/ddb/*" { capabilities = ["read"] } path "secret/data/temporal/*" { capabilities = ["read"] } path "secret/data/llm/*" { capabilities = ["read"] } path "secret/data/sqs/*" { capabilities = ["read"] } path "secret/data/story-crater-backend/*" { capabilities = ["read"] } EOF ``` ### 4. Create external-secrets K8s auth role ```bash vault write auth/kubernetes/role/external-secrets \ bound_service_account_names=external-secrets \ bound_service_account_namespaces=external-secrets-system \ policies=external-secrets \ ttl=24h \ max_ttl=24h ``` ### 5. Verify ClusterSecretStore can authenticate ```bash kubectl get clustersecretstore vault-homelab -o yaml # Should show no error events if auth is working ``` ## Seed Initial Secrets Once ESO is configured, seed the `.env` values into Vault for each release: ```bash # IAM realm vault kv put secret/iam/authentik \ secret-key="${AUTHENTIK_SECRET_KEY}" \ bootstrap-password="${AUTHENTIK_BOOTSTRAP_PASSWORD}" \ bootstrap-token="${AUTHENTIK_BOOTSTRAP_TOKEN}" \ postgresql-password="${AUTHENTIK_PG_PASSWORD}" vault kv put secret/iam/vault \ minio-access-key="${MINIO_ROOT_USER}" \ minio-secret-key="${MINIO_ROOT_PASSWORD}" # CI/CD realm vault kv put secret/cicd/forgejo \ admin-password="${FORGEJO_ADMIN_PASSWORD}" vault kv put secret/cicd/argocd \ oidc-client-secret="${AUTHENTIK_ARGOCD_CLIENT_SECRET}" # Logging realm vault kv put secret/logging/grafana \ admin-password="${GRAFANA_ADMIN_PASSWORD}" \ oidc-client-secret="${GRAFANA_OIDC_CLIENT_SECRET}" # Storage realm vault kv put secret/storage/minio \ root-user="${MINIO_ROOT_USER}" \ root-password="${MINIO_ROOT_PASSWORD}" \ oidc-client-secret="${MINIO_OIDC_CLIENT_SECRET}" # SQS realm vault kv put secret/sqs/kmsvc \ kafka-bootstrap="${KAFKA_BOOTSTRAP}" \ redis-addr="${REDIS_ADDR}" # Temporal realm vault kv put secret/temporal/temporal \ oidc-client-id="${AUTHENTIK_TEMPORAL_CLIENT_ID}" \ oidc-client-secret="${AUTHENTIK_TEMPORAL_CLIENT_SECRET}" # LLM realm vault kv put secret/llm/ollama \ oidc-client-id="${AUTHENTIK_OLLAMA_CLIENT_ID}" \ oidc-client-secret="${AUTHENTIK_OLLAMA_CLIENT_SECRET}" ``` Load these from .env automatically: ```bash export $(grep -v '^#' .env | xargs) # Then run the vault kv put commands above ``` ## Verification Test ESO by creating a simple ExternalSecret: ```bash kubectl apply -f - <<'EOF' apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: test-secret namespace: iam spec: refreshInterval: 1h secretStoreRef: name: vault-homelab kind: ClusterSecretStore target: name: test-secret creationPolicy: Owner data: - secretKey: root-user remoteRef: key: storage/minio property: root-user EOF # Check if the Secret was created kubectl get secret test-secret -n iam -o yaml ``` If the Secret has the expected data, ESO is working. Clean up the test: ```bash kubectl delete exs test-secret -n iam ```