homelab/docs/PHASE0-ESO-VAULT-SETUP.md

Phase 0: External Secrets Operator (ESO) Setup

After terraform apply installs ESO and the ClusterSecretStore manifest is applied, Vault needs to be configured to accept ESO's Kubernetes auth requests.

Prerequisites

• Vault is initialized and unsealed (run setup_vault.sh first)
• ESO pods are Running in external-secrets-system namespace
• ClusterSecretStore applied: k8s/external-secrets/clustersecretstore.yaml

Setup Steps (Manual)

Run these steps from the repo root with:

export VAULT_ADDR=http://vault.iam.svc.cluster.local:8200
export VAULT_TOKEN=$(kubectl get secret vault-unseal-keys -n storage -o jsonpath='{.data.key1}' | base64 -d)  # or use port-forward + login

1. Enable Kubernetes auth method

vault auth enable kubernetes || echo "Kubernetes auth already enabled"

2. Configure K8s auth to trust the cluster

# Get the K8s API server address and CA cert
K8S_HOST=$(kubectl cluster-info | grep 'Kubernetes master' | awk '/https/ {print $NF}')
K8S_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d)
SA_TOKEN=$(kubectl get secret -n external-secrets-system $(kubectl get secret -n external-secrets-system | grep external-secrets-webhook | awk '{print $1}') -o jsonpath='{.data.token}' | base64 -d)

vault write auth/kubernetes/config \
  kubernetes_host="${K8S_HOST}" \
  kubernetes_ca_cert="${K8S_CA_CERT}" \
  token_reviewer_jwt="${SA_TOKEN}"

3. Create external-secrets policy

vault policy write external-secrets - <<'EOF'
# external-secrets: read application secrets per namespace + release
path "secret/data/iam/*" {
  capabilities = ["read"]
}
path "secret/data/storage/*" {
  capabilities = ["read"]
}
path "secret/data/cicd/*" {
  capabilities = ["read"]
}
path "secret/data/logging/*" {
  capabilities = ["read"]
}
path "secret/data/monitoring/*" {
  capabilities = ["read"]
}
path "secret/data/ddb/*" {
  capabilities = ["read"]
}
path "secret/data/temporal/*" {
  capabilities = ["read"]
}
path "secret/data/llm/*" {
  capabilities = ["read"]
}
path "secret/data/sqs/*" {
  capabilities = ["read"]
}
path "secret/data/story-crater-backend/*" {
  capabilities = ["read"]
}
EOF

4. Create external-secrets K8s auth role

vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets-system \
  policies=external-secrets \
  ttl=24h \
  max_ttl=24h

5. Verify ClusterSecretStore can authenticate

kubectl get clustersecretstore vault-homelab -o yaml
# Should show no error events if auth is working

Seed Initial Secrets

Once ESO is configured, seed the .env values into Vault for each release:

# IAM realm
vault kv put secret/iam/authentik \
  secret-key="${AUTHENTIK_SECRET_KEY}" \
  bootstrap-password="${AUTHENTIK_BOOTSTRAP_PASSWORD}" \
  bootstrap-token="${AUTHENTIK_BOOTSTRAP_TOKEN}" \
  postgresql-password="${AUTHENTIK_PG_PASSWORD}"

vault kv put secret/iam/vault \
  minio-access-key="${MINIO_ROOT_USER}" \
  minio-secret-key="${MINIO_ROOT_PASSWORD}"

# CI/CD realm
vault kv put secret/cicd/forgejo \
  admin-password="${FORGEJO_ADMIN_PASSWORD}"

vault kv put secret/cicd/argocd \
  oidc-client-secret="${AUTHENTIK_ARGOCD_CLIENT_SECRET}"

# Logging realm
vault kv put secret/logging/grafana \
  admin-password="${GRAFANA_ADMIN_PASSWORD}" \
  oidc-client-secret="${GRAFANA_OIDC_CLIENT_SECRET}"

# Storage realm
vault kv put secret/storage/minio \
  root-user="${MINIO_ROOT_USER}" \
  root-password="${MINIO_ROOT_PASSWORD}" \
  oidc-client-secret="${MINIO_OIDC_CLIENT_SECRET}"

# SQS realm
vault kv put secret/sqs/kmsvc \
  kafka-bootstrap="${KAFKA_BOOTSTRAP}" \
  redis-addr="${REDIS_ADDR}"

# Temporal realm
vault kv put secret/temporal/temporal \
  oidc-client-id="${AUTHENTIK_TEMPORAL_CLIENT_ID}" \
  oidc-client-secret="${AUTHENTIK_TEMPORAL_CLIENT_SECRET}"

# LLM realm
vault kv put secret/llm/ollama \
  oidc-client-id="${AUTHENTIK_OLLAMA_CLIENT_ID}" \
  oidc-client-secret="${AUTHENTIK_OLLAMA_CLIENT_SECRET}"

Load these from .env automatically:

export $(grep -v '^#' .env | xargs)
# Then run the vault kv put commands above

Verification

Test ESO by creating a simple ExternalSecret:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-secret
  namespace: iam
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-homelab
    kind: ClusterSecretStore
  target:
    name: test-secret
    creationPolicy: Owner
  data:
    - secretKey: root-user
      remoteRef:
        key: storage/minio
        property: root-user
EOF

# Check if the Secret was created
kubectl get secret test-secret -n iam -o yaml

If the Secret has the expected data, ESO is working. Clean up the test:

kubectl delete exs test-secret -n iam