async-compression

async-compression provides streaming compression and decompression adaptors that wrap Rust’s asynchronous IO traits, letting you compress or decompress data as it flows through a Tokio or futures-io stream instead of buffering it all in memory first. It wraps well-known synchronous compression crates (flate2, brotli, zstd, bzip2, lzma, lz4) behind a consistent async Encoder/Decoder API for both AsyncRead and AsyncWrite sides.

The crate is organized as a small Cargo workspace: compression-core holds shared primitives, compression-codecs wraps each algorithm’s C/Rust bindings behind a common trait, and the top-level async-compression crate exposes the ergonomic tokio::bufread/tokio::write and futures::bufread/futures::write module surface that application code actually imports. Nothing is enabled by default — consumers opt into exactly the IO implementation and algorithms they need via Cargo features, keeping compile times and binary size down for the common case of needing just one codec.

What You Get

• Async Encoder/Decoder wrappers for AsyncRead, AsyncBufRead, and AsyncWrite under both tokio and futures-io feature flags
• Nine compression algorithms (brotli, bzip2, deflate, deflate64, gzip, lz4, xz/lzma, zlib, zstd) behind granular, opt-in Cargo features
• A BufWriter adaptor that batches small writes before compression to avoid pathological per-byte overhead
• zstd multithreaded encoding support (zstdmt) for CPU-bound compression on multi-core hosts
• A stable, low-level compression-codecs/compression-core split so downstream crates can build their own IO adaptors on the same primitives

Common Use Cases

• Streaming gzip/deflate compression of HTTP response bodies in an async web server without buffering the whole payload
• Decompressing large zstd or xz archives downloaded over the network as the bytes arrive, instead of loading them fully into memory
• Building async file-upload or backup pipelines that compress data on the fly before writing to disk or object storage
• Wrapping database dump or log-shipping pipelines so compression happens inline with async socket IO

Under The Hood

Architecture The workspace splits responsibility into three crates: compression-core defines the shared Level and low-level utility types; compression-codecs implements a common Encode/Decode-style trait per algorithm, delegating to each algorithm’s underlying (mostly C-backed) crate; and the top-level async-compression crate (crates/async-compression/src/lib.rs) exposes the public API, split into generic, tokio, and futures modules that implement the actual poll_read/poll_write state machines shared across both async IO ecosystems via generic/{bufread,write}/{encoder,decoder}.rs. This generic-core-plus-per-runtime-shim pattern lets the crate support both tokio and futures-io from a single implementation of the compression state machine, avoiding logic duplication between the two IO trait families. Tech Stack Pure Rust, 2018 edition, workspace-managed with resolver = "2". Depends on pin-project-lite for safe pinning of internal state, and optionally on tokio (>=1.24.2) and futures-io (0.3) as the two async IO trait providers, with each compression algorithm (flate2, brotli, zstd, bzip2, xz2, lzma-rs, lz4) pulled in only when its corresponding feature is enabled. Code Quality The repo carries 24+ integration test files under crates/*/tests, covering per-algorithm round-trips across both tokio and futures implementations, and CI runs cargo fmt, clippy --no-deps, cargo hack check --feature-powerset (to catch feature-combination breakage), and a wasm32 build target — a notably rigorous CI matrix for a crate with this many optional feature flags. Workspace lints deny missing_debug_implementations and rust_2018_idioms. API Design The crate mirrors the shape of the wrapped IO trait so usage is idiomatic: GzipEncoder::new(reader) reads exactly like a tokio::io::AsyncRead wrapper because it is one, keeping the learning curve low for anyone already familiar with tokio::io. The tradeoff is the feature-flag surface: consumers must know to enable both an implementation feature (tokio or futures-io) and an algorithm feature (gzip, zstd, etc.) before anything compiles, which is well documented in the crate-level docs but is an extra step versus a zero-config crate.