tantivy

Tantivy is a Rust library for building full-text search engines, strongly inspired by Apache Lucene’s architecture and design vocabulary. It is not a standalone search server like Elasticsearch or Solr — it’s a crate embedded directly into an application, giving developers indexing, BM25-ranked querying, faceted search, fast fields, and aggregations without running a separate service.

Tantivy indexes a large corpus into immutable, independently-searchable segments identified by UUID, supports incremental indexing and background segment merging, and is used to build both embedded search inside applications and larger distributed systems — most notably Quickwit, a distributed search engine built directly on top of Tantivy by the same organization.

What You Get

• Full-text indexing and BM25-ranked search with a natural query language (boolean operators, phrase queries)
• Configurable tokenizers with stemming for 17 Latin languages, plus third-party CJK tokenizer support
• Single- and multi-valued fast fields (columnar doc-values) for u64/i64/f64/bytes/date/ip/bool types, powering aggregations and range queries
• Faceted search, range queries, and an aggregation collector (histograms, range buckets, average/stats metrics)
• Pluggable Directory trait with built-in MmapDirectory and RamDirectory implementations
• Incremental, multithreaded indexing with background segment merging and a Searcher Warmer API

Common Use Cases

• Embedding full-text search directly into a Rust application without standing up a separate search server
• Building a custom search engine or search-as-a-service backend, as Quickwit does on top of Tantivy
• Powering search inside desktop tools, CLIs, or databases where startup time and low memory overhead matter (Tantivy starts in under 10ms)
• Implementing faceted, filterable search UIs (e-commerce category counts, real-estate range aggregations) backed by fast fields

Under The Hood

Architecture - Tantivy organizes an index as a collection of independent, immutable segments (each a UUID-named set of files, one per SegmentComponent), tracked via an atomically-updated meta.json, as documented in the repo’s own ARCHITECTURE.md. src/core/ holds the high-level Index/Segment/Searcher orchestration; deletes are applied via per-segment alive-bitset files rather than in-place mutation; background merge threads periodically combine segments to bound their count and reclaim tombstoned documents; and a Searcher snapshots a fixed set of SegmentReaders so results stay consistent regardless of concurrent commits or merges. This is a genuine Cargo workspace, splitting concerns into member crates: columnar/, sstable/, stacker/, bitpacker/, common/, ownedbytes/, tokenizer-api/, and query-grammar/, each independently versioned and reusable.

Tech Stack - Nearly pure Rust (99.98%) targeting the 2021 edition with MSRV 1.86, and a substantial dependency graph (~123 direct dependencies) reflecting its scope: SIMD-accelerated integer compression, mmap-backed storage, LZ4/Zstd document-store compression, and a query-grammar parser as a dedicated workspace member. Language-specific tokenization is extensible via third-party crates (tantivy-jieba, cang-jie, lindera, Vaporetto) rather than bundled in core.

Code Quality - Testing spans a dedicated tests/ directory including failpoints (fault-injection tests) and compat_tests_data (cross-version on-disk format compatibility fixtures), plus src/functional_test.rs and src/compat_tests.rs at the crate root, alongside per-module unit tests throughout src/. The project runs an OpenSSF Scorecard badge and Codecov coverage tracking, and with 195 contributors, ~3,599 commits, and ~35 commits/month it shows sustained, active maintenance rather than sporadic upkeep.

API Design - The public API mirrors Lucene’s vocabulary deliberately (IndexWriter, Searcher, Schema, Collector) so developers with Lucene/Elasticsearch background transfer knowledge directly, while the schema-first document model (fields declared up front, indexed into up to three data structures — docstore, fast field, inverted index) trades some flexibility for compact, well-compressed on-disk formats. The maintainers’ own ARCHITECTURE.md and companion blog posts substitute for exhaustive API docs on the trickier internals like commit/merge visibility semantics.