grenad

grenad is a Rust library from the Meilisearch team for working with large sets of immutable key-value pairs. It provides a Writer that serializes sorted key-value entries into compact, optionally compressed files, a Reader to iterate them back, a Sorter that sorts arbitrary key-value input (spilling to disk when needed), and a Merger that combines many sorted sources into one stream.

It is designed as a low-level building block for search indexes and databases where data is written once and read many times. Blocks can be compressed with Snappy, zlib, LZ4, or zstd, and merge functions let callers control how duplicate keys are resolved during sorting and merging.

What You Get

• A Writer that serializes sorted key-value pairs into compact files
• A Reader for sequential and prefix iteration over stored entries
• A Sorter that sorts arbitrary input, spilling to disk beyond a memory budget
• A Merger that merges many sorted sources with custom merge functions
• Pluggable block compression via Snappy, zlib, LZ4, or zstd

Common Use Cases

• Building immutable on-disk indexes for a search engine
• External sorting of key-value data too large to fit in memory
• Merging many pre-sorted key-value files into a single stream

Under The Hood

Architecture - The crate layers cleanly from bytes upward: varint.rs, block.rs, and block_writer.rs handle the on-disk block format; writer.rs and the reader/ module implement serialization and iteration; sorter.rs builds an external sorter that spills runs and feeds them into merger.rs, which resolves duplicate keys through a pluggable merge_function.rs. compression.rs abstracts the codec choice and metadata.rs/error.rs round out file headers and error handling.

Tech Stack - Written in Rust (edition 2018). Core dependencies are bytemuck and byteorder for byte-level encoding and either for iterator plumbing; compression codecs (flate2, lz4_flex, snap, zstd), tempfile for spill files, and rayon for parallelism are optional features. Benchmarks use criterion and tests use quickcheck.

Code Quality - The module boundaries are tight and each concern (block, writer, reader, sorter, merger, compression) lives in its own file. The dev-dependency set includes quickcheck for property tests, criterion benchmarks, and even a pinned prior version (grenad-0-4) for cross-version compatibility testing, signalling a mature, correctness-focused codebase maintained by the Meilisearch team.

API Design - The public surface follows a builder-and-stream pattern: configure a Writer or Sorter, push entries, then read through a Reader or fold sources with a Merger and a merge closure. This is a lower-level API than a turnkey KV store, so it assumes familiarity with sorted-file and external-sort concepts, but the type names map directly to the four documented operations and docs.rs covers each with examples.