lz4-rs

lz4 is a Rust crate providing bindings to the LZ4 compression library, giving Rust code fast, near-linear-scalability compression and extremely fast decompression. It ships two complementary APIs: a streaming frame API built around Encoder/Decoder types that implement std::io::Write and std::io::Read respectively, so compression drops directly into existing I/O pipelines, and a block-level API (lz4::block) that mirrors the python-lz4 block module for whole-buffer compress/decompress calls with optional size-prefix framing.

Under the hood, the crate vendors the upstream C lz4 library as a git submodule inside a companion lz4-sys crate, compiled via the cc build-dependency, and exposes the raw frame and block C APIs through FFI. Originally maintained at bozaro/lz4-rs, the project’s official home moved to 10xGenomics, which continues to track new upstream LZ4 releases, add features like favor_dec_speed, and support WebAssembly (wasm32-unknown-unknown) targets via a small libc shim.

What You Get

• Streaming Encoder/Decoder - Encoder<W: Write> and Decoder<R: Read> types that wrap any writer/reader and transparently compress or decompress the byte stream as it passes through.
• Configurable EncoderBuilder - a builder for block size, block mode (linked vs independent), block/content checksums, compression level, auto-flush, and favor_dec_speed.
• Block-level compress/decompress API - lz4::block::{compress, decompress, compress_to_buffer, decompress_to_buffer} for whole-buffer operations, with an optional 4-byte size prefix, modeled after python-lz4’s block module.
• Selectable compression modes - CompressionMode::FAST(accel), HIGHCOMPRESSION(level), and DEFAULT for trading off speed against ratio at the block-API level.
• Vendored, versioned C library - lz4-sys pins and compiles a specific upstream LZ4 release (currently 1.10.0) via the cc crate, so builds are self-contained and don’t depend on a system-installed liblz4.
• WebAssembly target support - a wasm-shim header set and wasm_shim.rs module let the crate build for wasm32-unknown-unknown, verified in CI.

Common Use Cases

• Compressing files or archives - piping file contents through Encoder/Decoder to produce or read .lz4-compressed files, as shown in the crate’s own lz4 CLI binary.
• In-pipeline stream compression - wrapping network sockets or other Read/Write streams so data is compressed or decompressed transparently as it’s written or read.
• One-shot buffer compression - using the block API to compress a single in-memory buffer (e.g. a cache value or IPC payload) without the overhead of the streaming frame format.
• Interop with other LZ4 block-format tools - the block API’s size-prefix option mirrors python-lz4’s format, easing exchange of block-compressed data between Rust and Python services.
• Embedding in genomics/bioinformatics pipelines - the crate’s current maintainer, 10xGenomics, uses it as a building block in data-processing pipelines that need fast, low-overhead compression.

Under The Hood

Architecture The crate is cleanly layered: lz4-sys is a separate crate that vendors the upstream LZ4 C sources as a git submodule and compiles them via a build.rs using the cc crate, exposing raw FFI bindings (frame and block C functions, error codes) with libc for C types. The top-level lz4 crate builds three layers on top of that FFI surface: liblz4.rs re-exports the raw bindings and adds check_error/version() helpers; encoder.rs/decoder.rs implement the streaming frame API as Encoder<W>/Decoder<R> structs that hold an LZ4FCompressionContext/LZ4FDecompressionContext and implement std::io::Write/Read, with Drop impls freeing the native context; and block/mod.rs implements a separate, simpler block-level compress/decompress API against the raw LZ4_compress_*/LZ4_decompress_safe functions, independent of the frame layer. A change to the vendored C library’s frame API would only affect encoder.rs/decoder.rs, while a change to the block functions would only affect block/mod.rs - the two API surfaces don’t share implementation code beyond the FFI bindings.

Tech Stack Rust edition 2018. lz4-sys depends on libc for C-compatible types and cc as a build-dependency to compile the vendored liblz4/lib/{lz4,lz4frame,lz4hc,xxhash}.c sources at -O3, with target-specific flags (e.g. -march=rv64g for riscv64, -fno-tree-vectorize for 32-bit MinGW, /MT for Windows static CRT builds). The top-level lz4 crate depends only on lz4-sys, plus rand as a dev-dependency for randomized round-trip tests. CI (GitHub Actions) builds and tests on Ubuntu, macOS, and Windows (32- and 64-bit, including static CRT), plus a wasm32-unknown-unknown build/check using a small wasm-shim of libc headers.

Code Quality Each core module carries a #[cfg(test)] suite: encoder.rs and decoder.rs include round-trip tests against randomly generated data, a Send-bound smoke test, an issue_45 regression test that reads decompressed output through several small buffer sizes to catch truncation bugs, and an ErrorWrapper/RetryWrapper pair that injects transient read errors to verify the decoder’s retry behavior. block/mod.rs tests cover truncation correctness, all CompressionMode variants, and both prefixed and unprefixed formats, including a fixed byte-array regression test for interop with python-lz4 output. CI enforces cargo fmt --check before building. No async code, minimal external error typing (a single LZ4Error(String) wraps the C library’s error string) - reasonable for a low-level FFI-wrapping crate, but there’s no linter (clippy) step visible in CI.

API Design The streaming API leans on standard Rust idioms - Encoder/Decoder implement Write/Read directly, so they compose with any other stream-processing code without a bespoke trait, and EncoderBuilder uses the common builder pattern with chainable setters. The block API mirrors python-lz4’s block module naming and semantics (compress, decompress, optional size prefix) specifically to ease porting logic between the two ecosystems. Documentation is limited mostly to the README (included verbatim as the crate’s rustdoc front page) and a doctested example in block/mod.rs; individual functions have doc comments describing error conditions, but there’s no broader usage guide or set of runnable examples beyond the README’s compress/decompress sample and the bin/lz4.rs CLI.