SPLADE

Overview

SPLADE (Sparse Lexical and Expansion Model) uses pretrained language models like BERT to identify connections between words/sub-words and uses that knowledge to enhance sparse vector embeddings.

How SPLADE Works

Unlike dense embeddings, SPLADE outputs vectors aligned with a vocabulary (similar to bag-of-words), but with learned weights reflecting semantic and contextual information.

Term Expansion Example

For the keyword "study":

• BM25: Single non-zero value for "study"
• SPLADE: Multiple non-zero values for "study", "learn", "research", "investigate", etc.

Key Advantages

• Semantic Understanding: Leverages context from transformers, improving recall over BM25
• Term Expansion: Includes alternative but relevant terms beyond the original sequence
• Interpretability: Maintains sparse vector format similar to traditional lexical search
• Better Recall: Significantly better performance compared to BM25 in IR evaluation tasks

Technical Specifications

• Vector Dimension: 30,522 (based on BERT vocabulary)
• Architecture: Built on BERT transformers
• Sparsity: Contains many zero values like traditional sparse vectors, but with learned weights

SPLADE vs BM25

| Feature | BM25 | SPLADE | |---------|------|--------| | Non-zero values | Single term only | Multiple related terms | | Semantic understanding | None | Yes (via BERT) | | Term expansion | No | Yes | | Context awareness | No | Yes | | Performance | Baseline | Superior recall |

Use Cases

• Hybrid search (combining with dense vectors)
• Keyword search with semantic understanding
• Entity resolution
• Document retrieval
• Information retrieval systems

Supported Platforms

• Qdrant
• Pinecone
• Chroma
• Various vector databases with sparse vector support

Pricing

Open-source model, free to use.