Evolution of Text Retrieval Needs

Traditional BM25 scoring has long been a cornerstone of information retrieval systems. However, the emergence of RAG (Retrieval-Augmented Generation) systems has fundamentally changed the landscape. Classic assumptions about document-query relationships no longer hold in modern contexts due to:

• Shift from pure lexical matching to semantic understanding
• Significant differences in typical document lengths (much shorter in RAG contexts)
• New requirements for combining lexical and semantic search capabilities

BM42 Architecture Overview

Developed by Qdrant, BM42 combines classical BM25 with atention-based weighting through a hybrid approach:

1. Maintains BM25's IDF (Inverse Document Frequency) calculation within Qdrant's engine for real-time updates
2. Integrates attention mechanisms from Transformer models to assess token importance:
   • Analyzes attention matrices from [CLS] token to identify significant terms
   • Generates token importance weights through multi-head attention patterns
3. Combines IDF values with attention weights in a unified scoring formula
4. Implements subword token merging using WordPiece tokenization:
   1. Splits words into subword units (e.g., "programming" → "pro", "##gram", "##ming")
   2. Aggregates attention weights across subword components
   3. Example: [0.03, 0.05, 0.07] → 0.15 for complete word representation

Performance Characteristics

BM42 demonstrates particular strengths in specific scenarios:

• Significantly improved accuracy for short document retrieval
• Effective handling of typical RAG use cases
• Memory-efficient sparse vector representations (average 5.6 elemants per document)

However, performance may degrade with:

• Extremely long queries
• Highly complex query patterns

Implementation Workflow

Core implementation involves two primary components:

1. index_bm42.py - For document vectorization and storage
2. evaluate_bm42.py - For query processing and similarity search

Required Dependencies

tqdm
qdrant-client>=1.10.0
fastembed>=0.3.3
tantivy
ipdb

Qdrant Setup

docker run --rm -d --network=host qdrant/qdrant:v1.10.0

Indexing Process

client.create_collection(
    collection_name=DATASET,
    vectors_config={},
    sparse_vectors_config={
        "bm42": models.SparseVectorParams(
            modifier=models.Modifier.IDF
        )
    }
)

Creates a collection with sparse vector configuration using IDF modifier for efficient retrieval.

Evaluation Process

Query processing involves:

1. Loading pre-trained sparse embedding model: ``` model = SparseTextEmbedding( model_name="Qdrant/bm42-all-minilm-l6-v2-attentions" )

2. Executing search with configurable result limit: ``` result = client.query_points( collection_name=DATASET, query=sparse_vector, using="bm42", with_payload=True, limit=limit )

Data Structure Example

{
    "2254": {
        "_id": "2254",
        "text": "Who were the Aztec?",
        "metadata": {},
        "doc_ids": ["101", "103"]
    }
}