fastembed

FastEmbed is a lightweight, ONNX Runtime-based Python library for generating text, image, sparse, and late-interaction embeddings without pulling in PyTorch or GPU-sized dependencies. Maintained by the Qdrant team, it ships pre-optimized ONNX versions of popular embedding models (BAAI/bge, Jina, SPLADE, ColBERT, CLIP-style vision models) and downloads them on demand from Hugging Face, so a project only pays for the models it actually uses.

Beyond dense text embeddings, FastEmbed covers sparse retrieval (SPLADE, BM42, BM25-style scoring via mmh3 and Rust stemmers), late-interaction ColBERT-style models, multimodal ColPali embeddings, image embeddings, and cross-encoder reranking, all through the same small, generator-based API. It’s built to pair naturally with Qdrant (via qdrant-client[fastembed]) but works as a standalone embedding layer for any vector store or RAG pipeline.

What You Get

• Ready-to-use ONNX builds of popular embedding models (BAAI/bge family, Jina, SPLADE, ColBERT, CLIP-style vision encoders) downloaded on demand from Hugging Face.
• A single consistent API (embed(), query_embed(), passage_embed()) shared across text, image, sparse, late-interaction, multimodal, and reranking model classes.
• Data-parallel batch encoding via a built-in multiprocessing worker pool, with an optional fastembed-gpu package for CUDA execution providers.
• First-class custom model support through add_custom_model(), so any compatible Hugging Face checkpoint can be registered without forking the library.

Common Use Cases

• Generating document and query embeddings for a RAG pipeline without a PyTorch dependency.
• Adding semantic search to a serverless function where deployment package size is constrained.
• Building hybrid dense+sparse retrieval by combining FastEmbed’s dense and SPLADE/BM42 sparse embeddings.
• Reranking a retriever’s top-k candidates with FastEmbed’s ONNX-based cross-encoders.

Under The Hood

Architecture FastEmbed layers its embedding backends behind small modality-specific base classes (TextEmbeddingBase, image/image_embedding_base.py, sparse/sparse_embedding_base.py, late_interaction/late_interaction_embedding_base.py) that all compose the generic ModelManagement[T] mixin in common/model_management.py for Hugging Face Hub download/caching and the OnnxModel[T] class in common/onnx_model.py for session loading and inference. Actual model execution happens on ort.InferenceSession, with per-modality subclasses such as text/onnx_embedding.py wiring specific pooling/normalization pipelines around a shared _load_onnx_model helper. Large batches are pushed onto a separate multiprocessing worker pool defined in parallel_processor.py (ParallelWorkerPool / Worker), decoupling encode-time parallelism from the model classes themselves. Swapping the core inference runtime would require touching OnnxModel, OnnxTextModel/OnnxImageModel, and every _get_worker_class() implementation, but the public TextEmbedding/ImageEmbedding/SparseTextEmbedding classes would stay stable since they only depend on the base-class embed()/query_embed()/passage_embed() contract.

Tech Stack FastEmbed targets Python 3.10+ and is built on ONNX Runtime (onnxruntime) for model inference, huggingface_hub for model download/caching, tokenizers for text tokenization, numpy for array operations, and mmh3 plus py-rust-stemmers (a Rust-backed stemmer) for sparse/BM25-style embeddings. Packaging and dependency management run through Poetry, with per-Python-version dependency constraints pinned for numpy, onnxruntime, and pillow. Distribution ships two variants, fastembed and fastembed-gpu, that differ only in which onnxruntime execution provider is installed, and documentation is built with mkdocs-material plus Jupyter notebooks converted via mknotebooks.

Code Quality Testing lives entirely under tests/, with dozens of test functions spread across ten files (test_text_onnx_embeddings.py, test_sparse_embeddings.py, test_image_onnx_embeddings.py, test_late_interaction_embeddings.py, test_late_interaction_multimodal.py, test_text_cross_encoder.py, test_multi_gpu.py, and others) exercising every embedding modality against real ONNX inference rather than mocks. CI (python-tests.yml, ci.yml, type-checkers.yml) runs pytest, ruff, mypy, and pyright with typeCheckingMode set to strict against tests/type_stub.py, and a pre-commit config enforces the same checks locally. The codebase is fully type-annotated with modern union syntax and Generic[T] base classes, naming stays consistent across the modality packages (text/, image/, sparse/, late_interaction/, rerank/), and errors surface as explicit exceptions or warnings (e.g. the mutually-exclusive cuda/providers warning in onnx_model.py) rather than being swallowed.

API Design FastEmbed’s public surface is deliberately small: TextEmbedding(), SparseTextEmbedding(), ImageEmbedding(), LateInteractionTextEmbedding(), LateInteractionMultimodalEmbedding(), and TextCrossEncoder() all expose the same embed()/embed_image()/embed_text()/rerank() shape, so switching between dense, sparse, late-interaction, multimodal, and reranking workloads requires no new mental model. add_custom_model() lets a caller register an arbitrary Hugging Face checkpoint (pooling type, normalization, dimension, ONNX file path) as a first-class model without subclassing, which is unusual generosity for a library that ships its own curated model catalog. embed() returns a generator rather than a materialized list, keeping memory bounded when embedding large corpora, and the library’s tight dependency footprint with no PyTorch requirement is itself a developer-experience decision aimed at fast cold starts in serverless and edge deployments.