lz4_flex

lz4_flex is a complete, pure-Rust rewrite of LZ4 compression, benchmarked as the fastest LZ4 implementation available in Rust — outperforming both a C-binding wrapper (lzzzz) and other pure-Rust alternatives (lz-fear) on both compression and decompression throughput. It supports both the LZ4 block format (for compressing whole chunks in memory) and the LZ4 frame format (for streaming compression/decompression via std::io::Read/Write), so it fits both simple in-memory use and larger streaming pipelines.

The crate exposes safe and unsafe code paths behind feature flags — safe-encode/safe-decode are enabled by default and forbid unsafe code entirely, while disabling them unlocks additional performance headroom for callers who’ve fuzzed and trust their inputs. It also supports no_std environments (block format only) and 32-bit targets, and is actively fuzzed (cargo fuzz) and checked with Miri to catch undefined behavior in the unsafe code paths.

What You Get

• Block format functions (compress_prepend_size, decompress_size_prepended, compress_into, decompress_into) for whole-chunk in-memory compression
• Frame format FrameEncoder/FrameDecoder types implementing std::io::Write/Read for streaming compression across multiple blocks
• Feature-flagged safe-encode/safe-decode (default) that forbid unsafe code, with an unsafe fast path available via --no-default-features
• no_std support for the block format, including stack-only allocation via compress_into_with_table for environments without an allocator
• Fuzz targets (corrupted-input decompression, buffer-leak, roundtrip) and Miri-checked unsafe code for correctness under adversarial input

Common Use Cases

• Compressing data in a performance-sensitive Rust service where a C binding or FFI overhead is undesirable
• Streaming compression/decompression of large files or network payloads via the LZ4 frame format’s Read/Write implementations
• Embedded or no_std targets that need LZ4 block compression without a heap allocator
• Applications that need compression the community has stress-tested and fuzzed against corrupted or adversarial compressed input

Under The Hood

Architecture - The crate is split into block and frame modules mirroring the two LZ4 wire formats. block/compress.rs and block/decompress.rs (or block/decompress_safe.rs when the safe-decode feature is on) implement the core LZ4 sequence encoding/decoding, backed by a custom hash table (block/hashtable.rs) for finding duplicate byte sequences, and a sink.rs abstraction (forced forbid(unsafe_code) when both safe features are enabled) that centralizes buffer-writing so the unsafe fast paths and safe paths share one interface. The frame module layers the streaming frame format — including its own header parsing (frame/header.rs) and an xxhash32 checksum via the twox-hash crate — on top of the block primitives. Custom fastcpy.rs/fastcpy_unsafe.rs modules provide hand-tuned memory-copy routines used by the hot compression/decompression loops.

Tech Stack - Nearly 100% Rust with zero mandatory runtime dependencies (only twox-hash, optional, gated behind the frame feature). Dev dependencies include criterion-adjacent binggan for benchmarking, proptest for property-based testing, and comparison crates (lzzzz, lz-fear, snap) used purely for the README’s competitive benchmarks. rust-version = "1.81" and edition = "2021" keep the MSRV explicit.

Code Quality - The crate enforces strict lint policy at the top of lib.rs: #![deny(warnings)], #![deny(missing_docs)], #![deny(unsafe_op_in_unsafe_fn)], plus warnings for unreachable-pub items and missing Debug impls. Deprecated top-level re-exports (block::compress aliases) carry explicit #[deprecated] migration notes rather than being silently removed. Beyond unit tests, the project runs Miri (for undefined-behavior detection in unsafe blocks), cargo fuzz targets specifically for corrupted/adversarial decompression input and output-buffer-leak detection, and maintains a benches/ suite with tracked SVG performance regression charts — a notably rigorous quality bar for a compression library where subtle bugs can silently corrupt data.

API Design - The two-format split (block vs frame) is made explicit in both the module structure and doc comments, steering users toward the frame format by default while still exposing the simpler block API for callers who understand the size constraints. Top-level deprecated re-exports guide users away from ambiguous naming toward the namespaced block:: functions, and feature flags (safe-encode, safe-decode, no_std, alloc) are additive and clearly documented, letting a single crate serve constrained embedded targets and performance-critical server code from the same codebase.