python-zstd

python-zstd exposes Yann Collet’s Zstandard (zstd) compression algorithm to Python as a thin C extension. It wraps libzstd’s one-shot compress and decompress calls directly, giving Python code access to zstd’s balance of high compression ratio and speed without the overhead of a pure-Python implementation or a heavier wrapper generator.

The module deliberately stays minimal: no dictionary training, no streaming API, just compress, decompress, and a set of introspection helpers for checking which libzstd build is linked. For projects that need dictionaries or streaming, the README itself points to the more full-featured python-zstandard project as an alternative.

What You Get

• One-shot ZSTD_compress/ZSTD_uncompress functions with zlib-style aliases (compress/decompress, dumps/loads, encode/decode)
• Multi-threaded compression via a threads parameter, auto-tuned to CPU core count by default
• Compression levels from -100 (fastest) to 22 (highest ratio), with optional strict-mode range validation
• Support for concatenated/multi-frame and streamed data in ZSTD_uncompress
• Build-time flags to link system libzstd, use bundled sources, enable ASM optimizations, or trim binary size
• Runtime introspection functions (ZSTD_version_loaded, ZSTD_max_threads_count, ZSTD_legacy_support, ZSTD_with_asm) for verifying what the linked libzstd build supports

Common Use Cases

• Compressing API response or request bodies before sending them over the wire to cut bandwidth
• Shrinking values before storing them in a cache like Redis or Memcached to fit more into limited memory
• Archiving logs or dataset dumps at a high compression ratio directly from Python data pipelines
• Producing standard zstd-format output that’s readable by non-Python zstd tooling for cross-language interchange

Under The Hood

Architecture The module is a single CPython C extension (src/python-zstd.c, ~865 lines) exposing a flat set of C functions (py_zstd_compress, py_zstd_compress_mt2, py_zstd_uncompress, py_zstd_check, version getters) directly as the top-level zstd module — there’s no internal Python package layer, no class hierarchy, and no separation between “core” and “bindings”; util.c/util.h and debug.c/debug.h are the only supporting translation units, handling buffer-sizing/threading helpers and conditional debug logging. The bundled libzstd lives as a git submodule pointing at facebook/zstd, and setup.py’s build_ext override is where the real architectural complexity sits — dozens of conditional compiler-flag branches (—external, —libzstd-bundled, —legacy, —libzstd-use-asm, —speed1..—speed-max, —small) decide at build time whether to link system libzstd or compile the vendored sources, and with which codegen options. Because the Python surface is a thin C shim over libzstd’s one-shot calls, changing the core abstraction means updating the C-API glue directly or bumping the vendored submodule.

Tech Stack Language is C (53%) glued to Python (46%) via the raw CPython C-API (Python.h, bytesobject.h) rather than a wrapper generator like Cython or cffi, targeting CPython 2.7 through 3.15 plus free-threaded builds per an extensive GitHub Actions wheel-build matrix. It vendors Facebook’s zstd as a git submodule and can alternatively link a system libzstd >=1.4.0 via —external; setuptools (pinned <72 for test compatibility) drives build_ext with per-platform ASM (BMI2) and LTO toggles. There’s no runtime dependency beyond libzstd itself — this is a leaf compression utility — and CI is entirely wheel-building per Python-version/architecture combination rather than an integration pipeline.

Code Quality Tests live under tests/ using stdlib unittest (base.py, test_compress.py, test_decompress.py, test_resize_and_formatting.py, test_speed.py, test_version.py) with dozens of test_* methods covering compression levels, multi-threading, and streamed/concatenated-frame decompression — a real, functional/black-box suite that gracefully skips on very old Python. Error handling on the Python side is a single custom zstd.Error exception raised from C for out-of-range levels or thread counts; there are no type hints (moot for a C extension) and code style leans toward terse, comment-light C. CI is exclusively wheel-building matrices, not a lint/format gate — no flake8, black, or mypy config exists in the repo.

API Design The API deliberately mirrors Python’s built-in pickle/zlib naming (compress/decompress, dumps/loads, and since 1.5.6.2, encode/decode) so it drops into existing zlib-shaped code with a rename, and ZSTD_check returns a tri-state (0/1/2) distinguishing “not compressed” from “compressed” from “streamed” — a small but useful ergonomic touch many simple bindings skip. Its explicit threads parameter on ZSTD_compress and a battery of introspection calls (ZSTD_max_compression_level, ZSTD_with_asm, ZSTD_is_debug_enable) give callers visibility into exactly what libzstd build they’re running against, unusually thorough for what the README calls a “simple and blunt” binding. The tradeoff, stated openly in the docs, is that dictionary and streaming compression are explicitly out of scope.