Representation learning approach encoding information at multiple granularities, allowing embeddings to be truncated while maintaining performance. Enables 14x smaller sizes and 5x faster search.
Loading more......
Matryoshka Representation Learning (MRL) is an approach that encodes information at different granularities and allows a single embedding to adapt to the computational constraints of downstream tasks.
Matryoshka embedding models store more important information in earlier dimensions, and less important information in later dimensions. This characteristic allows truncating the original (large) embedding produced by the model, while still retaining enough information to perform well on downstream tasks.
Matryoshka Representations enable adaptive retrieval (AR) which alleviates the need to use full-capacity representations for all data and downstream tasks. The approach works by:
MRL offers significant improvements:
OpenAI's text-embedding-3-large model, when truncated to just 256 dimensions, outperforms their previous text-embedding-ada-002 at 1,536 dimensions on the MTEB benchmark.
Facilitates substantial dimensionality reduction while maintaining comparable performance levels, achieving significant enhancement in computational efficiency and cost-effectiveness.
Supported in: