parquet2

parquet2 is a from-scratch Rust implementation of the Apache Parquet file format, built as a safer and faster alternative to the official parquet crate. It forbids unsafe code entirely (#![forbid(unsafe_code)]), deliberately decouples IO-intensive reading from CPU-intensive decompression and decoding, and leaves parallelism strategy up to the caller rather than baking in a threading model.

The crate reads and writes Parquet metadata, row groups, and both V1 and V2 pages, and supports the format’s major compression codecs (Snappy, Gzip, LZ4, Zstd, Brotli) and encodings (PLAIN, RLE/bit-packing hybrid, dictionary, delta encodings). It does not commit to an in-memory representation of decoded data — consumers convert pages into whatever array format they need, which is how downstream projects like arrow2 and Polars use it as their low-level Parquet IO layer. An optional async feature adds non-blocking metadata and page reads.

What You Get

• Metadata, row-group, and column-chunk readers/writers for the Parquet binary format
• V1 and V2 page reading and writing with dictionary support
• Built-in compression codecs behind feature flags: Snappy, Gzip, LZ4, Zstd, Brotli
• Decoders/encoders for PLAIN, RLE/bit-packing hybrid, dictionary, and delta encodings
• Optional async metadata and page streaming via the async feature
• Bloom filter reading/writing support behind the bloom_filter feature

Common Use Cases

• Serving as the low-level Parquet IO layer beneath a columnar array library (as arrow2 does)
• Reading Parquet metadata and statistics without materializing full column data
• Building custom, memory-conscious Parquet readers that control their own parallelism and buffering
• Writing Parquet files from Rust services that need specific compression/encoding choices
• Cross-validating Parquet output against files produced by pyarrow or Spark in integration tests

Under The Hood

Architecture The crate is organized around a strict separation between metadata parsing, page IO, and decoding: read::metadata parses the Thrift-encoded footer into FileMetaData/RowGroupMetaData/ColumnChunkMetaData structures, read::page::PageReader and ColumnIterator (in src/read/mod.rs) stream compressed pages off a Read + Seek source by seeking to each column chunk’s byte range, and compression::decompress plus the Decompressor/BasicDecompressor wrappers turn those bytes into decoded pages only when the caller asks for the next item. Because PageReader and Decompressor both implement FallibleStreamingIterator rather than eagerly materializing a Vec, callers can hand compressed pages to worker threads and decompress in parallel without the crate imposing a threading model itself — the write module mirrors this with column_chunk.rs, page.rs, and row_group.rs performing the inverse encode/compress/write sequence. parquet_bridge.rs is the seam that isolates the underlying Thrift-generated parquet-format-safe types from the crate’s own public enums (Compression, Encoding), keeping the wire-format library swappable without touching the read/write API.

Tech Stack parquet2 is a pure-Rust, 2021-edition crate with parquet-format-safe (a safe reimplementation of the Thrift-generated Parquet format types) and streaming-decompression as its core non-optional dependencies. Compression codecs are each gated behind a Cargo feature mapping to a dedicated crate — snap (Snappy), flate2 (Gzip), lz4/lz4_flex, zstd, and brotli — so a consumer only compiles in the codecs it needs, and xxhash-rust backs the optional bloom_filter feature. Async support is opt-in via futures and async-stream behind the async feature flag rather than a hard dependency on a specific runtime. There is no build tooling beyond Cargo itself; integration tests generate fixture files with pyarrow through a Python virtualenv invoked from CI, and criterion covers the two benchmarked hot paths (bit-packing and RLE decoding).

Code Quality The crate ships unit tests inline under #[cfg(test)] blocks in roughly two dozen source files (e.g. read/mod.rs, compression.rs) plus a separate tests/it integration suite that exercises read and write paths against real Parquet fixtures, some generated by pyarrow and validated in CI via the coverage.yml workflow using cargo llvm-cov. Error handling is centralized in a single non-exhaustive Error enum (src/error.rs) with explicit variants for out-of-spec data, missing features, unsupported functionality, and over-allocation, plus From conversions from the underlying IO/Thrift/compression error types — there is no silent error swallowing, and results are threaded through a crate-wide Result<T> alias. The whole crate compiles under #![forbid(unsafe_code)], which is an unusually strong safety guarantee for a binary-format parser that would traditionally reach for raw pointer casts.

API Design The public API favors explicit iterator types over hidden allocation: get_page_iterator and ColumnIterator require the caller to pass reusable scratch buffers, and Decompressor::into_buffers() hands ownership of those buffers back so a long-running reader can avoid repeated allocation — this is a deliberate low-level tradeoff that gives performance-sensitive consumers full control at the cost of more boilerplate than a batteries-included reader would need. Module-level docs explain the read -> compressed page -> decompressed page -> decoded bytes -> deserialized pipeline and the README walks through a multi-threaded reading example, but there is no single high-level “read this file into memory” convenience function — the crate documents itself as a toolkit for building readers, not a ready-to-use one, which matches its role as the IO layer underneath arrow2 rather than an end-user library.