zstd

zstd provides safe, idiomatic Rust bindings around Facebook’s Zstandard (zstd) compression library, exposing Read/Write stream wrappers alongside simple one-shot encode/decode helper functions. It builds on the lower-level zstd-safe and zstd-sys crates in the same workspace, compiling the vendored C library as part of the build so consumers get zstd compression without needing a system dependency.

The crate is one of the most widely used compression bindings in the Rust ecosystem, powering compression layers in databases, build tools, and network protocols where zstd’s balance of speed and ratio is preferred over gzip or bzip2.

What You Get

• Streaming Encoder/Decoder types implementing standard Read/Write traits
• One-shot helper functions (encode_all, decode_all, copy_encode, copy_decode)
• Dictionary training and dictionary-based compression via the zdict_builder feature
• Multi-threaded compression support via the zstdmt feature
• Optional bindgen-based binding regeneration and a pkg-config feature to link a system zstd

Common Use Cases

• Compressing log files, backups, or build artifacts before storage or transfer
• Adding transparent zstd compression to a custom network protocol or RPC layer
• Compressing/decompressing data streams inside a CLI tool (mirrors the zstd/zstdcat examples in the repo)
• Training a shared dictionary to improve compression ratio on many small, similar payloads

Under The Hood

Architecture — The repository is a small Cargo workspace: the top-level zstd crate (this package) provides ergonomic Read/Write wrappers (src/stream/) and bulk one-shot helpers (src/bulk/) on top of zstd-safe, a safe-but-thin layer that itself wraps zstd-safe/zstd-sys, the raw FFI bindings generated against the vendored zstd C sources (included as a git submodule). This three-tier layering (raw FFI → safe wrapper → ergonomic API) is a common pattern for Rust bindings to C libraries and keeps unsafe code isolated to the -sys crate. Tech Stack — Pure Rust wrapper crate targeting edition 2018 with a 1.64 MSRV; it depends only on zstd-safe (path dependency within the workspace) for the runtime API, with clap, humansize, partial-io, and walkdir as dev-dependencies used by the CLI examples and test harness. The C library itself is compiled from source via a build script in zstd-sys. Code Quality — The crate has a small dedicated tests/ directory plus doctests embedded in the stream/bulk modules, and CI badges show Linux/Windows/macOS/wasm build matrices. Error handling follows the underlying C library’s return-code conventions surfaced as io::Result, which is idiomatic for a Read/Write-based API. API Design — The crate intentionally mirrors std::io::copy-style ergonomics (copy_encode/copy_decode) alongside lower-level Encoder/Decoder types, so users needing a quick one-liner and users needing fine-grained streaming control both have a natural entry point; feature flags (zstdmt, zdict_builder, experimental) keep the default build lean while exposing advanced functionality opt-in.