Python SDK from Meta for dense passage retrieval using dual BERT encoders and FAISS indexing, supporting batch embedding generation, async queries via multiprocessing. Enables Python/JS app Q&A pipelines and LangChain retrievers; 9-19% better accuracy than BM25 lexical APIs.
Loading more......
Dense Passage Retriever (DPR) is a set of tools and models for open domain Q&A task developed by Facebook AI Research (now Meta AI). The research shows that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages.
DPR uses:
When evaluated on a wide range of open-domain QA datasets, the dense retriever outperforms a strong Lucene-BM25 system by 9%-19% absolute in terms of top-20 passage retrieval accuracy.
At run-time, DPR applies the question encoder to map the input question to a d-dimensional vector, and retrieves k passages of which vectors are the closest to the question vector using efficient similarity search.
Free and open-source.