zstd-rs

ruzstd is a from-scratch Rust implementation of the Zstandard (zstd) compression format as defined in RFC 8878. It provides a complete decompression implementation via FrameDecoder and the higher-level StreamingDecoder, both usable in no_std environments through the crate’s own Read/Write traits. Because it never links against the C zstd library, it avoids unsafe FFI and cross-compilation friction for platforms like WASM and embedded targets.

The crate also ships a working compressor (FrameCompressor, compress/compress_to_vec) that currently covers the Uncompressed and Fastest compression levels plus checksums, with more levels and dictionary support still in progress. An optional dict_builder feature generates raw-content dictionaries that benchmark within 0.2% of the reference implementation’s dictionary size, and the workspace bundles a small CLI for exercising the library directly from the terminal.

What You Get

• FrameDecoder & StreamingDecoder - a low-level frame decoder for fine-grained, incremental decoding plus a high-level io::Read-based streaming wrapper for simple end-to-end decompression.
• no_std support - the crate is #![no_std] by default and only pulls in std behind the std feature, so it runs in embedded and WASM targets.
• Compression path - FrameCompressor and compress/compress_to_vec helpers produce valid zstd frames at the Uncompressed and Fastest levels, with checksum support.
• Dictionary builder - an optional dict_builder feature generates raw-content dictionaries competitive with the reference zstd implementation’s dictionary size.
• Bundled CLI - the workspace’s cli crate provides small zstd/streaming binaries for exercising the decoder/encoder from the command line.

Common Use Cases

• Decompressing zstd-compressed assets - reading .zst files, HTTP payloads, or archive entries in a pure-Rust binary that must avoid linking a C library.
• Embedded/WASM decompression - unpacking zstd data in no_std targets where the official C bindings aren’t an option.
• Streaming large payloads - decoding multi-gigabyte frames incrementally via FrameDecoder::decode_blocks instead of buffering the whole result in memory.
• Building custom dictionary tooling - using the dict_builder feature to generate raw-content dictionaries for domain-specific compression ratios.

Under The Hood

Architecture The crate splits decompression and compression into separate decoding/encoding modules over a shared blocks/common/bit_io layer. On the decode side, StreamingDecoder wraps the lower-level FrameDecoder, which drives a BlockDecoder state machine through ready/decoding/failed states, dispatching RLE, Raw, and Compressed block types into dedicated literals- and sequence-section decoders that write into a scratch-owned ring buffer. On the encode side, FrameCompressor owns a match generator and per-symbol FSE tables to emit block and frame headers. The crate is no_std at its root, gating all std-only code behind a feature flag and swapping between two Read/Write implementations depending on target — the dominant architectural constraint is that core decode/encode types stay allocator-only rather than assuming a standard library is present.

Tech Stack A two-crate Cargo workspace — the ruzstd library and a small cli binary — targeting a pinned minimum-supported Rust version and verified against it in CI. Dependencies are deliberately minimal: an optional hashing crate for checksums and an optional random-number crate for the dictionary builder, plus internal-only shims used only when the crate is vendored into the Rust standard library itself. Dev-dependencies include a benchmarking harness, a general-purpose RNG, and the official C-backed zstd crate, used as a differential-testing oracle rather than a runtime dependency. There’s no web framework, database, or ORM in sight — this is a pure algorithmic format implementation with a thin CLI wrapper on top.

Code Quality Testing is layered and unusually rigorous for a compression crate: conformance tests run against well-formed corpus-generated fixtures, encoder round-trip tests validate compress/decompress symmetry, dictionary tests cover the optional builder, and a fuzz-regression suite replays previously discovered crashes; a separate fuzzing setup runs continuously against the decoder. CI checks the full feature powerset for compilation, linting, and tests, verifies the pinned MSRV for both crates, enforces formatting and zero-warning clippy runs with and without default features, and runs an undefined-behavior sanitizer against the ring buffer and a targeted reader edge case. Errors are modeled as explicit typed enums rather than strings or panics, and the block decoder tracks a failure state to prevent misuse after an unrecoverable error.

API Design The public API deliberately offers two tiers: a one-line, io::Read-based streaming wrapper for the common case, and a lower-level frame decoder with an explicit decoding-strategy type for callers who need to bound memory or interleave decoding with other work — both are documented with runnable examples. A window-size limit makes the memory-safety trade-off explicit rather than silently accepting attacker-controlled window sizes, with the doc comment calling out the denial-of-service risk directly. The no_std/std split is exposed through a single re-exported io module so the same call sites work in both environments. The project’s own documentation is candid about the remaining gap: the streaming wrapper only handles a single frame per stream today, and the encoder hasn’t yet reached zstd’s higher compression levels or dictionary support.