rank_bm25

rank_bm25 implements several variants of the BM25 ranking function — Okapi BM25, BM25L, BM25+, and BM25-Adpt — used to score how relevant a document is to a search query based on term frequency and document length, without requiring a full search-engine stack like Elasticsearch or Lucene. Each variant is exposed as a class that indexes a tokenized corpus and returns relevance scores or top-N matches for a query.

Because it’s a small, dependency-light (NumPy-only) pure-Python library rather than a hosted search service, rank_bm25 is frequently used as a lightweight lexical-retrieval baseline or as the sparse-retrieval half of hybrid search pipelines that combine BM25 with dense vector/embedding search — a common pattern in retrieval-augmented generation (RAG) systems.

What You Get

• BM25Okapi, BM25L, BM25Plus, and BM25Adpt classes implementing different BM25 scoring variants
• get_scores() to score every document in the corpus against a tokenized query
• get_top_n() to retrieve the top-N most relevant documents directly
• A simple, transparent scoring model based on term frequency and inverse document frequency with length normalization
• Minimal dependencies (NumPy only), making it easy to drop into any Python pipeline

Common Use Cases

• Adding a lightweight keyword/lexical search baseline to a Python application without deploying Elasticsearch or a vector database
• Implementing the sparse-retrieval half of a hybrid search pipeline alongside dense embedding-based retrieval in a RAG system
• Re-ranking a small candidate document set by lexical relevance before or after a semantic-similarity pass
• Teaching or prototyping information-retrieval concepts where a transparent, inspectable scoring formula matters more than production scale

Under The Hood

Architecture - The entire library lives in a single rank_bm25.py module: a BM25 base class handles corpus indexing (term frequencies, document lengths, average document length, inverse document frequency computation), and each variant (BM25Okapi, BM25L, BM25Plus, BM25Adpt) subclasses it to override only the scoring formula, keeping the differences between BM25 variants explicit and easy to compare.

Tech Stack - Pure Python with a single runtime dependency, NumPy, used for vectorized score computation across the corpus; there is no compiled extension, external index, or service dependency, so the whole library installs and runs anywhere NumPy does.

Code Quality - Test coverage in tests/test_loading.py is comparatively thin, focused mainly on corpus loading/scoring smoke tests rather than exhaustive coverage of every BM25 variant’s edge cases; the small single-file scope makes the implementation easy to read and verify manually against the published BM25 formulas, but the project has seen infrequent maintenance since its last 0.2.2 release.

API Design - Usage is a two-step pattern: construct BM25Okapi(tokenized_corpus) once to index the corpus, then call .get_scores(tokenized_query) or .get_top_n(tokenized_query, corpus, n) as many times as needed — a minimal, easy-to-learn API, though callers are responsible for their own tokenization since the library does no text preprocessing itself.