BM25 (Okapi BM25)

Overview

Okapi BM25 (Best Matching 25) is a ranking function used by search engines to estimate the relevance of documents to a given search query. It represents the gold standard for keyword-based retrieval, developed in the 1970s-1980s by Stephen E. Robertson, Karen Spärck Jones, and others.

Key Characteristics

Bag-of-Words Approach

• Ranks documents based on query term appearances
• Independent of term proximity within document
• Focus on frequency and distribution

Scoring Factors

1. Term Frequency (TF): How often query terms appear in document
2. Inverse Document Frequency (IDF): Rarity of terms across corpus
3. Document Length Normalization: Adjusts for varying document sizes

How BM25 Works

Combines three main components:

Term Frequency Component

• Considers how many times a term appears
• Diminishing returns for repeated terms
• Saturation point prevents over-weighting

Inverse Document Frequency

• Rare terms weighted more heavily
• Common terms ("the", "and") weighted less
• Captures term importance across corpus

Document Length Normalization

• Adjusts for document size
• Prevents bias toward longer documents
• Balances verbosity vs. relevance

Variants and Extensions

BM25F

• Accounts for document structure
• Handles anchor text
• Field-weighted scoring
• Used for structured documents

BM25+

• Improved lower-bound handling
• Better term frequency adjustments
• Enhanced accuracy

BM25L

• Enhanced document length normalization
• Better handling of long documents

BM25 in Modern Search

Industry Standard

• Baseline for multiple ranking algorithms
• Default in many search systems
• Benchmark for new algorithms
• Production-proven reliability

Hybrid Search Integration

Often combined with semantic search:

• BM25: Keyword-based retrieval
• Vector Search: Semantic understanding
• Fusion: Reciprocal Rank Fusion (RRF) to combine results

BM25 and RAG Systems

Complementary to LLMs

• BM25 retrieves relevant documents
• LLM generates context-aware responses
• Combines efficiency with semantic understanding
• Overcomes limitations of pure LLMs

Retrieval Phase

1. BM25 performs fast keyword retrieval
2. Returns top-k relevant documents
3. LLM processes retrieved context
4. Generates informed responses

Advantages

Strengths

• Fast: Efficient computation
• Interpretable: Clear scoring logic
• No Training: Works out-of-the-box
• Effective: Strong baseline performance
• Scalable: Handles large corpora

When BM25 Excels

• Exact keyword matching
• Known terminology searches
• Document retrieval with specific terms
• Large-scale text collections
• Resource-constrained environments

Limitations

No Semantic Understanding

• Treats words as independent tokens
• Misses synonyms and related concepts
• No context awareness
• Limited by vocabulary matching

Mitigations

• Combine with semantic search
• Use in hybrid retrieval systems
• Apply query expansion techniques
• Leverage embedding models for coverage

Implementation

Available In

• Elasticsearch: Native BM25 support
• Vespa: BM25 rank feature
• OpenSearch: Default ranking function
• Apache Solr: BM25 similarity
• Many custom search systems

Python Libraries

• rank-bm25: Pure Python implementation
• Elasticsearch-py: Via Elasticsearch
• Whoosh: Search library with BM25

Parameters

k1 (Term Frequency Saturation)

• Typical value: 1.2 to 2.0
• Controls TF saturation
• Higher = less saturation

b (Length Normalization)

• Typical value: 0.75
• Controls document length impact
• 0 = no normalization, 1 = full normalization

Use Cases

• Search Engines: Traditional web search
• Document Retrieval: Enterprise search
• RAG Systems: First-stage retrieval
• Hybrid Search: Combined with vector search
• Question Answering: Document selection
• Recommendation: Content-based filtering

Modern Context (2026)

BM25 remains relevant:

• Foundation for hybrid search
• Baseline in RAG systems
• Combined with neural methods
• Proven reliability and speed
• Essential component in AI search stacks

Pricing

BM25 is an algorithm, not a product:

• Implemented in open-source libraries
• Available in commercial search platforms
• No licensing fees for the algorithm itself