python-zstandard

python-zstandard provides Python bindings to Facebook’s Zstandard (zstd) compression library. Rather than a thin wrapper around a handful of functions, it exposes nearly the entire zstd C API — one-shot and streaming compression/decompression, dictionary training and reuse, multi-threaded compression, content-size-aware framing, and fine-grained control over compression parameters like window size and strategy.

The project ships two interchangeable backends: a CPython C extension for maximum performance, and a CFFI implementation that works on PyPy and other alternative interpreters. A Rust-based backend is also under active development. All backends implement the same Python-level API, so code written against one works unmodified against the others, with the import policy configurable via an environment variable for testing and deployment flexibility.

Because it operates on any object implementing Python’s buffer protocol (bytes, bytearray, memoryview, mmap, io.BytesIO), the library integrates naturally with existing I/O code, and its stream reader/writer classes are drop-in compatible with Python’s io module, including support for opening zstd-compressed files directly via a zstandard.open() function analogous to gzip.open().

What You Get

• One-shot compress()/decompress() calls for simple in-memory buffer operations, plus streaming ZstdCompressionReader/Writer and ZstdDecompressionReader/Writer classes that implement Python’s io interfaces
• A ZstdCompressor/ZstdDecompressor pair backed by reusable contexts, avoiding the overhead of recreating compression state for repeated operations
• Dictionary support via train_dictionary() and ZstdCompressionDict, useful for compressing many small, structurally similar payloads (e.g. JSON documents) far better than compressing them independently
• Multi-threaded compression via a threads= argument that splits input into segments processed by a native (non-GIL-bound) thread pool
• Fine-grained ZstdCompressionParameters (window_log, strategy, checksum, content-size flags) for tuning ratio versus speed trade-offs
• A zstandard.open() helper mirroring gzip.open()/bz2.open() for transparent file-level compression and decompression

Common Use Cases

• Compressing build artifacts, backups, or log archives where zstd’s speed/ratio balance beats gzip or bzip2
• Streaming compression of large files or network payloads without buffering the entire dataset in memory
• Compressing many small, similar records (e.g. API responses, log lines, JSON blobs) using a trained dictionary to dramatically improve ratio on small inputs
• Multi-threaded compression of large datasets in data pipelines to reduce wall-clock time by using all available CPU cores
• Drop-in replacement for gzip/bz2 in file formats and storage layers that need better throughput or compression ratio at similar CPU cost

Under The Hood

Architecture The top-level zstandard/__init__.py module acts as a dispatcher rather than an implementation: at import time it selects one of three backends — backend_c (a CPython C extension wrapping the vendored zstd C source under zstd/), backend_cffi (a pure-CFFI module for PyPy and other interpreters), or an in-development backend_rust (via the pyo3-based crate under rust-ext/) — controlled by interpreter detection or the PYTHON_ZSTANDARD_IMPORT_POLICY environment variable, then re-exports that backend’s symbols as the public API. Each backend independently implements the same class surface (ZstdCompressor, ZstdDecompressor, stream reader/writer pairs, ZstdCompressionDict), so the Python-facing contract is identical regardless of which native layer executes it; this backend-parity requirement is enforced by running the full test suite against both cext and cffi policies in CI. Compression/decompression contexts are long-lived objects that wrap zstd’s native context structs, and streaming classes (ZstdCompressionReader/Writer) layer Python’s io interfaces on top of chunked calls into those contexts.

Tech Stack The C backend is built via setuptools with a vendored copy of the zstd reference implementation (zstd/zstd.c, zstd.h) compiled directly into the extension, avoiding a system libzstd dependency. The CFFI backend depends on cffi~=1.17 (or cffi>=2.0 on newer Python) and is implemented entirely in zstandard/backend_cffi.py (~4,500 lines), re-declaring the same C structures via CFFI’s ABI. The nascent Rust backend uses pyo3 and Cargo (Cargo.toml, rust-ext/src/) to bind the same C library through Rust. Packaging spans pyproject.toml (PEP 621 metadata, cibuildwheel config for prebuilt wheels across Linux/macOS/Windows and CPython/PyPy) plus a Justfile for local dev tasks.

Code Quality The tests/ directory contains 31 test modules covering every public class and operation mode, including dedicated fuzzing tests (test_compressor_fuzzing.py, test_decompressor_fuzzing.py, test_data_structures_fuzzing.py) built on the hypothesis property-based testing library, run via pytest with pytest-xdist for parallelism. CI (.github/workflows/test.yml) runs this suite against both the C and CFFI backends on every push, and separate workflows (typing.yml, ruff.yml) enforce mypy static typing and ruff linting (with E4/E7/E9/F/I rule sets) as required checks — a rigorous setup for a project with native-extension complexity.

API Design The library’s central design choice is offering both a simple one-shot API (compress()/decompress()) and a full streaming API (ZstdCompressionReader/Writer) that mirrors Python’s built-in io module conventions, plus a zstandard.open() entry point deliberately modeled on gzip.open()/bz2.open() so it can be swapped in with minimal code changes. Accepting any buffer-protocol object (bytes, bytearray, memoryview, mmap) rather than requiring bytes conversions keeps large-payload usage allocation-free, and the explicit ZstdCompressionParameters object exposes zstd’s advanced tuning knobs (window size, strategy, checksums) in a discoverable, typed way rather than through opaque integer flags.