rust-brotli

rust-brotli is a from-scratch Rust implementation of the Brotli compression algorithm, built to be a drop-in replacement for Google’s C brotli library at compression levels 0-11. It ships both a compressor and decompressor with io::Read and io::Write wrapper types for idiomatic streaming usage, plus a lower-level manual-memory-management API for no_std environments such as embedded devices, kernels, or sandboxed workers where the caller supplies its own allocator.

Beyond core compression, the crate implements Brotli stream concatenation (both a fast bare-byte mode and a size-optimized catbrotli tool), multithreaded compression via a worker pool, an FFI layer for linking from C code in place of libbrotli.so, and a validation mode that double-checks a compressed stream can be decompressed with the same settings. It is maintained by Dropbox and is a common dependency wherever Rust projects need Brotli support without pulling in the full C toolchain.

What You Get

• Read/Write stream wrappers - CompressorReader/Compressor and Decompressor/DecompressorWriter types that implement io::Read and io::Write for drop-in streaming compression and decompression.
• no_std manual memory management API - a three-step BrotliState + BrotliDecompressStream loop for environments without the Rust standard library, with pluggable custom allocators.
• C-compatible FFI layer - an ffi-api feature exposing a C ABI that can replace libbrotli.so directly, including custom allocator support and panic-safe boundaries via catch_unwind.
• Stream concatenation tooling - both a zero-processing bare-byte concatenation mode and a catbrotli binary for size-optimized concatenation of independently compressed chunks.
• Multithreaded compression - a worker-pool based CompressMulti path for compressing large inputs across multiple threads.
• Bit-identical compatibility - compression levels 0-9 produce byte-identical output to the reference C brotli implementation, useful for cross-language interop and reproducible builds.

Common Use Cases

• Embedding Brotli in no_std binaries - firmware, kernels, or sandboxed (e.g. seccomp-jailed) processes that need compression without a syscall-capable allocator.
• Replacing a system libbrotli dependency - projects that want a memory-safe Rust implementation behind the same C FFI surface as Google’s reference library.
• HTTP body compression - Rust web servers and clients streaming Brotli-encoded request/response bodies via the io::Read/io::Write wrappers.
• Chunked/streaming compression pipelines - systems (like Dropbox’s own Broccoli sync engine) that compress data incrementally and concatenate the resulting streams later.
• Cross-implementation fuzzing and validation - using the crate’s validation mode to confirm compressed output decompresses correctly before shipping it.

Under The Hood

Architecture The crate splits along the compress/decompress boundary: src/enc/ holds the encoder (encode.rs, metablock.rs, block_splitter.rs, entropy_encode.rs, static_dict.rs) organized around a metablock-at-a-time pipeline of LZ77 backward-reference search, block splitting, and entropy coding, while decompression is re-exported from the separate brotli-decompressor crate. An ffi/ module (gated by the ffi-api feature) wraps both directions behind a C ABI in compressor.rs/decompressor.rs/broccoli.rs, with catch_unwind boundaries so Rust panics don’t unwind across extern "C". A concat/ module implements the BroCatli stream-concatenation state machine. The crate is #![no_std] at the root, with a std feature that layers io::Read/io::Write wrappers and the default heap allocator (alloc-stdlib) on top of the no_std core; every allocation-touching type is generic over an Allocator trait, so callers can supply stack, pool, or custom allocators without recompiling internals. Tech Stack Pure Rust, edition 2015, MSRV 1.59.0, built with cargo. Runtime dependencies are minimal and deliberately no_std-first: alloc-no-stdlib for the allocator trait, alloc-stdlib (optional, std feature) for a heap-backed allocator, brotli-decompressor (a sibling crate under the same repo/org) for the decompression path, and an optional sha2 dependency gated behind a validation feature. Optional Cargo features (ffi-api, simd, benchmark, seccomp, portable-float) toggle nightly-only or platform-specific code paths, keeping the default build stable-Rust and dependency-light. CI covers Travis (Linux) and AppVeyor (Windows, both MSVC and GNU targets, 32- and 64-bit), and a c/ subdirectory with its own make-based build produces a libbrotli.so for C consumers. Code Quality Tests exist as #[test]-annotated unit tests embedded in source (e.g. src/enc/test.rs) rather than a separate tests/ integration directory, exercising encoder correctness against known inputs; there’s also a src/bin/integration_tests.rs and validate.rs binary for end-to-end round-trip and cross-implementation checks. Error handling favors explicit Result/status-enum returns (e.g. BrotliFileNotCraftedForConcatenation, Result<BroCatli, ()>) over panics on malformed input, with recent changelog entries specifically hardening against integer-underflow and NULL-pointer panics at FFI boundaries. Naming follows the original C brotli API closely (non-idiomatic Rust casing is explicitly allowed via #![allow(non_snake_case)]), which trades Rust-idiom polish for close parity with the reference implementation and easier porting of fixes between the two. What Makes It Unique Unlike bindings that FFI into Google’s C brotli, this is an independent from-scratch Rust port that stays bit-identical to the C encoder at levels 0-9 while adding capabilities the reference implementation doesn’t expose as directly: a fully allocator-pluggable no_std core suitable for kernels and seccomp-jailed processes, multithreaded compression via a worker pool, and a stream-concatenation feature (BroCatli/Broccoli) built specifically to support incremental, chunked compression workloads like Dropbox’s own sync infrastructure.