Experimental sparse embedding approach combining exact keyword search with transformer intelligence, integrating sparse and dense vector searches for improved RAG results, developed by Qdrant.
Loading more......
BM42 is a new sparse embedding approach that combines the benefits of exact keyword search with the intelligence of transformers. It was developed by Qdrant as a search algorithm combining vector and standard BM25 keyword search methods to get better RAG results.
At the core of BM42's innovation is its hybrid search capability, which seamlessly integrates both sparse and dense vector searches:
As a sparse search technique, it retains the inverse document frequency (IDF) aspect of BM25, equipping BM42 with the core ability to capture rare and out-of-vocabulary terms. The key innovation lies in how it defines token-level relevance within documents.
BM42 reverses the tokenization process after getting the attention vectors, and the attention weights of subwords can be summed to get the attention weight of the word.
Experimental Status: Recent evaluations have raised questions about the validity of BM42, and future developments may address these concerns. BM42 does not outperform BM25 implementation of other vendors and should be considered as an experimental approach which requires further research and development before it can be used in production.
Starting from Qdrant v1.10.0, BM42 can be used in Qdrant via FastEmbed inference.
Free to use as part of Qdrant.