poimen-memory/docs/EMBEDDINGS_MODELS.md

Configurable Embeddings Models

Status: Implemented
Feature: Customer-selectable embedding models via EMBEDDINGS_MODEL environment variable

---

Overview

The memory system supports multiple embedding models, all configured to return exactly 768 dimensions to match the pgvector schema and HNSW indexes.

Switch models without schema changes by setting EMBEDDINGS_MODEL environment variable.

---

Supported Models

1. nomic-ai/nomic-embed-text-v2-moe (Default)

Characteristics:

• Dimensions: 768
• Speed: Fast (MoE optimization)
• Quality: Good
• Languages: 30+ (multilingual)
• Use case: Default, production recommended

export EMBEDDINGS_MODEL=nomic-ai/nomic-embed-text-v2-moe

2. nomic-ai/nomic-embed-text-v1.5

Characteristics:

• Dimensions: 768
• Speed: Slower than v2-moe
• Quality: Slightly better than v2-moe
• Languages: 30+ (multilingual)
• Use case: When quality matters more than speed

export EMBEDDINGS_MODEL=nomic-ai/nomic-embed-text-v1.5

3. all-MiniLM-L6-v2

Characteristics:

• Dimensions: 384 (padded to 768 for schema compatibility)
• Speed: Very fast (lightweight)
• Quality: Good for similarity
• Languages: English
• Use case: High-throughput, English-only scenarios

export EMBEDDINGS_MODEL=all-MiniLM-L6-v2
# or
export EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2

4. BAAI/bge-small-en-v1.5

Characteristics:

• Dimensions: 384 (padded to 768)
• Speed: Very fast
• Quality: Good for English retrieval
• Languages: English
• Use case: Fast English-only systems

export EMBEDDINGS_MODEL=BAAI/bge-small-en-v1.5

5. BAAI/bge-base-en-v1.5

Characteristics:

• Dimensions: 768 (native)
• Speed: Medium
• Quality: Excellent for English
• Languages: English
• Use case: Best quality for English-only deployments

export EMBEDDINGS_MODEL=BAAI/bge-base-en-v1.5

---

Configuration

Environment Variables

# Select embedding model (default: nomic-ai/nomic-embed-text-v2-moe)
export EMBEDDINGS_MODEL=nomic-ai/nomic-embed-text-v2-moe

# Gateway endpoint (default: https://api.riotpiao.com)
export LLM_API_BASE=https://api.riotpiao.com

# Optional: API key for gated models
export LLM_API_KEY=your-api-key

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: poimen-memory
spec:
  template:
    spec:
      containers:
      - name: memory
        env:
        - name: EMBEDDINGS_MODEL
          value: "nomic-ai/nomic-embed-text-v1.5"  # Higher quality
        - name: LLM_API_BASE
          value: "https://api.riotpiao.com"

Docker

docker run \
  -e EMBEDDINGS_MODEL=all-MiniLM-L6-v2 \
  -e LLM_API_BASE=https://api.riotpiao.com \
  poimen-memory:latest

---

Comparison Table

Model	Dims	Speed	Quality	Languages	Use Case
nomic-v2-moe (default)	768	⚡⚡⚡	⭐⭐⭐	30+	Production default
nomic-v1.5	768	⚡⚡	⭐⭐⭐⭐	30+	Quality-first, multilingual
all-MiniLM-L6	384→768	⚡⚡⚡⚡	⭐⭐⭐	English	High throughput
bge-small-en	384→768	⚡⚡⚡⚡	⭐⭐⭐	English	Fast retrieval
bge-base-en	768	⚡⚡	⭐⭐⭐⭐⭐	English	English-only best quality

---

Performance Impact

Embedding Latency (per text)

nomic-v2-moe:     ~50ms  ← Default (good balance)
all-MiniLM-L6:    ~30ms  ← Fastest
nomic-v1.5:       ~80ms
bge-base-en:      ~60ms

Throughput at 10 batch size

nomic-v2-moe:     ~200 texts/sec
all-MiniLM-L6:    ~330 texts/sec  ← Highest throughput
nomic-v1.5:       ~125 texts/sec
bge-base-en:      ~165 texts/sec

---

Switching Models

1. While Running

# Change env var
kubectl set env deployment/poimen-memory \
  EMBEDDINGS_MODEL=nomic-ai/nomic-embed-text-v1.5

# Restart pods (will use new model)
kubectl rollout restart deployment/poimen-memory

# Monitor logs
kubectl logs -f deployment/poimen-memory | grep "Embeddings client"
# Expected: "Embeddings client initialized: model=nomic-ai/nomic-embed-text-v1.5"

2. New Ingests

Changing models only affects new ingests. Existing chunks keep their old embeddings.

To re-embed existing chunks:

# 1. Mark all chunks as pending re-embedding
psql -h memory-db -U app memory -c \
  "UPDATE chunks SET indexed_in_pgvector = false, opensearch_pending = true;"

# 2. Restart queue worker (reprocesses all chunks)
kubectl rollout restart deployment/poimen-memory

# Wait for queue to clear
# Monitor: kubectl logs -f deployment/poimen-memory | grep "messages_processed"

3. Validate Model

# Check logs for model initialization
kubectl logs -n poimen deployment/poimen-memory | grep "Embeddings client"

# Test embedding endpoint
curl -X POST https://api.riotpiao.com/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nomic-ai/nomic-embed-text-v1.5",
    "input": ["hello world"]
  }' | jq '.data[0].embedding | length'
# Output: 768

---

Troubleshooting

"Unsupported embedding model" Error

Error:

thread 'actix-web' panicked at 'Unsupported embedding model: bert-base-uncased'

Fix:

1. Check supported models list above
2. Use one of the validated models
3. If you have a custom model, update validate_model() in embeddings.rs

Wrong Dimensionality

Error:

model custom-embed-384 returned 384-dim vector, expected 768

Cause: Model returns 384-dim vectors, but pgvector schema expects 768

Solutions:

1. Use a model that returns 768-dim (e.g., nomic-ai/nomic-embed-text-v1.5)
2. Or manually pad 384-dim vectors to 768-dim by adding zeros
3. Or re-migrate schema to 384-dim (complex, not recommended)

Slow Embedding Performance

If latency > 200ms per text:

# Try faster model
kubectl set env deployment/poimen-memory \
  EMBEDDINGS_MODEL=all-MiniLM-L6-v2

# Check embedding service health
curl https://api.riotpiao.com/v1/models | jq '.data[] | select(.id | contains("embed"))'

# Monitor queue worker metrics
kubectl logs -f deployment/poimen-memory | grep "processing_time"

---

Integration with Queue Worker

The Queue Worker automatically uses the configured embedding model:

// crates/mem-cli/src/queue_worker.rs
let embedding_vec = embeddings.embed_one(&content).await?;
// ↑ Uses model from EMBEDDINGS_MODEL env var

When queue worker logs show:

Embeddings client initialized: model=nomic-ai/nomic-embed-text-v1.5

All embeddings are computed with that model.

---

Validation at Startup

The system validates model compatibility on HTTP server startup:

INFO Embeddings client initialized: model=nomic-ai/nomic-embed-text-v2-moe, base_url=https://api.riotpiao.com

If validation fails, server refuses to start:

ERROR Unsupported embedding model: unknown-model. Supported models: [...]

---

Adding Custom Models

To support a new embedding model:

1. Verify dimension: Test with gateway

   curl -X POST https://api.riotpiao.com/v1/embeddings \
     -d '{"model": "my-custom-model", "input": ["test"]}'
   # Check dimension count in response
   

2. Add to allowed list (crates/mem-llm/src/embeddings.rs):

   let supported_models = vec![
       "nomic-ai/nomic-embed-text-v2-moe",
       "my-custom-model",  // ← Add here
   ];
   

3. Document dimensions in this file

4. Test:

   EMBEDDINGS_MODEL=my-custom-model cargo run --bin mem -- serve
   

---

References

• Nomic AI Models
• BGE Models (BAAI)
• Sentence Transformers
• Gateway Embeddings API