MMTEB

Overview

MMTEB (Massive Multilingual Text Embedding Benchmark) is a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. It represents the largest multilingual collection of evaluation tasks for embedding models to date.

Key Features

Diverse Task Set

Includes a diverse set of challenging, novel tasks:

• Instruction following
• Long-document retrieval
• Code retrieval
• Traditional NLP tasks (classification, clustering, etc.)

Community-Driven

Created through a large-scale, open collaboration, with contributors including:

• Native speakers from diverse linguistic backgrounds
• NLP practitioners
• Academic and industry researchers
• Enthusiasts

Regional Benchmarks

From the extensive collection of tasks in MMTEB, several representative benchmarks were developed:

• MTEB(Multilingual): Highly multilingual benchmark
• MTEB(Europe): Regional geopolitical benchmark for European languages
• MTEB(Indic): Regional geopolitical benchmark for Indic languages

Performance Findings

While large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters.

Computational Efficiency

Introduces a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings at a fraction of the computational cost.

Pricing

Free to use - open benchmark published February 2025.