Late Chunking

Overview

Late Chunking is an embedding technique that reverses the traditional chunking process: instead of chunking text first and then embedding each chunk separately, it embeds the entire document and then chunks the resulting embeddings.

The Traditional Chunking Problem

Limitations of Standard Approach

1. Context Loss: Chunks isolated from surrounding text
2. Boundary Issues: Important information split across chunks
3. Long-Range Dependencies: Contextual relationships destroyed
4. Sub-optimal Representations: Chunks lack document-level context

Example Problem

Traditional chunking of:

"The company's revenue grew by 3% over the previous quarter."

Results in a chunk that doesn't specify:

• Which company?
• Which quarter?
• Which year?

How Late Chunking Works

Process Flow

Traditional Chunking:

1. Split document into chunks
2. Embed each chunk independently
3. Store chunk embeddings

Late Chunking:

1. Embed entire document (all tokens)
2. Apply chunking to embeddings
3. Mean pool within chunk boundaries
4. Store contextualized chunk embeddings

Technical Details

1. Full Document Embedding:

   • Pass entire document through transformer
   • Generate token-level embeddings
   • Preserve all contextual information

2. Late Chunking Application:

   • Define chunk boundaries
   • Apply boundaries to token embeddings
   • Group tokens by chunk

3. Chunk Embedding Extraction:

   • Mean pooling within each chunk
   • Produce final chunk embedding
   • Maintains document context

Key Benefits

Superior Context Preservation

• Each chunk embedding contains full document context
• Long-range dependencies maintained
• Cross-chunk relationships preserved
• Better semantic understanding

Improved Retrieval Quality

• Higher relevance scores
• Better handling of ambiguous queries
• Improved recall on complex questions
• Reduced context-related errors

No Additional Training

• Works with existing long-context models
• Generic technique
• Compatible with various chunking strategies
• Plug-and-play implementation

Performance

Benchmark Results

• Retrieval Tasks: Superior across multiple benchmarks
• Context Quality: Significantly improved
• Edge Cases: Better handling of boundary conditions
• Complex Queries: Improved performance

Use Cases Where It Excels

• Multi-hop reasoning
• Cross-document references
• Temporal relationships
• Entity coreference
• Causal relationships

Implementation

Available In

• Jina Embeddings v3: Native support
• Jina AI API: Available via API
• Elasticsearch: Integration available
• Weaviate: Supported
• Custom Implementations: Open source

Example Usage (Jina API)

import requests

# Document to embed with late chunking
document = "Your long document text here..."

# API call with late chunking
response = requests.post(
    "https://api.jina.ai/v1/embeddings",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "input": [document],
        "model": "jina-embeddings-v3",
        "late_chunking": True,
        "dimensions": 1024
    }
)

chunk_embeddings = response.json()["data"]

Custom Implementation

from transformers import AutoModel, AutoTokenizer
import torch

# Load long-context model
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v3")
tokenizer = AutoTokenizer.from_pretrained("jinaai/jina-embeddings-v3")

def late_chunking(text, chunk_size=512):
    # Tokenize entire document
    tokens = tokenizer(text, return_tensors="pt")
    
    # Embed all tokens
    with torch.no_grad():
        outputs = model(**tokens)
        token_embeddings = outputs.last_hidden_state[0]
    
    # Define chunk boundaries
    chunks = []
    for i in range(0, len(token_embeddings), chunk_size):
        chunk_tokens = token_embeddings[i:i+chunk_size]
        # Mean pooling
        chunk_embedding = chunk_tokens.mean(dim=0)
        chunks.append(chunk_embedding)
    
    return chunks

Comparison with Alternatives

vs Traditional Chunking

• Late Chunking: Full context per chunk
• Traditional: Isolated chunks
• Result: 49% fewer failed retrievals (with Contextual Retrieval)

vs Contextual Retrieval

• Late Chunking: Embedding-level context
• Contextual Retrieval: Prompt-based context
• Compatible: Can be combined

vs Sliding Window

• Late Chunking: Full document context
• Sliding Window: Local context only
• Trade-off: Quality vs complexity

Requirements

Model Requirements

• Long Context Support: Essential
• Mean Pooling: Must use mean pooling
• Token-Level Outputs: Need access to token embeddings

Compatible Models

• Jina Embeddings v3 (native)
• Other long-context embedding models
• Custom transformer models with long context

Best Practices

Document Length

• Optimal: Within model's context window
• Long docs: May need hierarchical approach
• Very long: Consider document splitting first

Chunk Size

• Balance: Context vs granularity
• Typical: 256-512 tokens
• Adjust: Based on use case
• Test: Validate retrieval quality

Integration

• Test: Compare with traditional chunking
• Benchmark: Measure retrieval improvement
• Monitor: Track performance metrics
• Iterate: Tune parameters

Use Cases

Ideal For

• Scientific papers with cross-references
• Legal documents with dependencies
• Technical documentation
• Multi-section articles
• Books and long-form content

When to Use

• Context is crucial
• Cross-chunk relationships important
• Long-range dependencies exist
• Document structure matters

Limitations

Computational Cost

• Higher: Must embed entire document
• Memory: Stores all token embeddings
• Processing: Longer for large documents

Model Constraints

• Requires: Long-context model
• Context Window: Document must fit
• Compatibility: Not all models support

Trade-offs

• Cost: Higher compute vs better quality
• Latency: Slower embedding vs better retrieval
• Complexity: More complex vs simpler traditional

Research Status

Published September 2024, with implementations emerging in 2025-2026. Active integration into major vector search platforms.

Paper Details

• Title: "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models"
• Authors: Jina AI Research
• Publication: ArXiv 2409.04701

Industry Adoption

Current Support (2026)

• Jina AI (native)
• Elasticsearch (integrated)
• Weaviate (supported)
• Custom implementations available

Future Directions

• Broader model support
• Optimization techniques
• Hybrid approaches
• Automatic parameter tuning

Pricing

Jina AI API

• Part of Jina Embeddings v3 pricing
• No additional cost for late chunking
• Pay per token embedded

Self-Hosted

• Open source implementations
• Standard compute costs
• No licensing fees

Related Concepts

• Contextual Retrieval: Anthropic's approach
• Hierarchical Chunking: Multi-level chunks
• Sliding Window: Overlapping chunks
• Sentence Window: Sentence-based context

Resources

• Jina AI blog post
• ArXiv paper: 2409.04701
• GitHub implementation
• Elasticsearch integration docs
• Weaviate documentation