Embedding Dimensions

Overview

Embedding dimensions refer to the length of vector representations produced by embedding models. This is a crucial parameter affecting model capacity, storage requirements, and search performance.

Common Dimension Sizes

Text Embeddings

• 384: Small models (all-MiniLM-L6-v2)
• 512: Medium models (some GTE variants)
• 768: BERT-base, many standard models
• 1024: Larger models (BGE-large, multilingual-e5-large)
• 1536: OpenAI text-embedding-ada-002, text-embedding-3-small
• 3072: OpenAI text-embedding-3-large
• 8192: Some specialized models

Image Embeddings

• 512: CLIP models (typical)
• 1024: Larger vision models
• 2048: High-capacity vision transformers

Trade-offs

Higher Dimensions

Advantages:

• More nuanced semantic representations
• Better task performance
• Higher capacity for complex concepts

Disadvantages:

• More storage (linear scaling)
• Slower distance computations
• Higher memory requirements
• Increased indexing time

Lower Dimensions

Advantages:

• Faster search
• Less storage
• Lower memory footprint
• Faster index building

Disadvantages:

• Less expressive
• Potential information loss
• Lower task performance

Matryoshka Embeddings

Modern approach allowing flexible dimensions:

• Single model supports multiple sizes
• Examples: 64, 128, 256, 512, 1024
• Important information in early dimensions
• Choose dimension at inference time
• Used by: OpenAI, Nomic, Alibaba GTE

Storage Impact

Example: 1M vectors

• 384-dim: ~1.5 GB (float32)
• 768-dim: ~3 GB
• 1536-dim: ~6 GB
• 3072-dim: ~12 GB

With Quantization

• Binary (1-bit): 32x reduction
• int8: 4x reduction
• Enables larger dimension at same cost

Choosing Dimensions

For Your Application

Small Dimensions (128-384):

• Simple semantic matching
• Large-scale deployment
• Mobile/edge applications
• Cost-sensitive scenarios

Medium Dimensions (512-1024):

• General-purpose retrieval
• Balanced performance/cost
• Most production RAG systems

Large Dimensions (1536+):

• Complex semantic understanding
• Multi-lingual scenarios
• Specialized domains
• When accuracy is critical

Dimensionality Reduction

Techniques to reduce dimensions:

• PCA: Principal Component Analysis
• Random Projection: Fast approximation
• Matryoshka Training: Learn multi-scale
• Autoencoders: Neural compression

Model Examples by Dimension

384-dim:

• all-MiniLM-L6-v2
• paraphrase-MiniLM-L6-v2

768-dim:

• BERT-base models
• sentence-transformers defaults

1024-dim:

• BGE-large-en
• multilingual-e5-large
• GTE-large

1536-dim:

• OpenAI ada-002
• OpenAI text-embedding-3-small
• Cohere embed-v3

Pricing

Concept; implementation costs vary by model and platform.