python-lz4

lz4 provides Python bindings for the LZ4 compression library, exposing three format-specific APIs: a frame format binding for interoperable compressed streams, a block format binding for raw fixed-size buffers, and an experimental streaming format binding built on a double-buffer strategy. The frame API mirrors the standard library’s gzip, bz2, and lzma modules, with matching context managers, file handle support (LZ4FrameFile, lz4.frame.open), and incremental compression via begin/compress_chunk/flush.

Under the hood, the package builds C extension modules (lz4._version, lz4.block._block, lz4.frame._frame, plus lz4.stream._stream when built with PYLZ4_EXPERIMENTAL=1) directly against bundled or system liblz4 sources, and releases the GIL during compression and decompression so multi-threaded callers aren’t blocked. It ships as prebuilt wheels for CPython 3.9 through 3.14, including free-threaded builds, with an extensive pytest suite covering both formats.

What You Get

• Frame API compress()/decompress() functions plus LZ4FrameCompressor/LZ4FrameDecompressor context managers for incremental compression
• File-like LZ4FrameFile and lz4.frame.open() for reading and writing .lz4 files as a drop-in replacement for gzip/bz2/lzma
• Block API for raw, non-framed LZ4 compression when the caller already tracks the uncompressed size itself
• Experimental stream API (opt-in via PYLZ4_EXPERIMENTAL) exposing LZ4’s double-buffer streaming mode for block-by-block compression
• GIL-released, thread-safe C extension bindings with prebuilt wheels for CPython 3.9-3.14, including free-threaded builds

Common Use Cases

• Compressing application logs or telemetry payloads before writing them to disk or shipping them over the network
• Adding LZ4-compressed file support to a data pipeline that already handles gzip/bz2/lzma via the same file-object API
• Speeding up interprocess or on-disk caching layers where compression speed matters more than compression ratio
• Interoperating with other language LZ4 frame implementations (Java, Go, Rust) via the standardized frame format

Under The Hood

Architecture The package is a thin binding layer built as separate C extension modules (lz4._version, lz4.block._block, lz4.frame._frame, and an optional lz4.stream._stream gated behind PYLZ4_EXPERIMENTAL) each wrapping the corresponding LZ4 C API surface, bundled under lz4libs/ or linked against a system liblz4 via pkgconfig. Thin Python wrapper modules (lz4/block/init.py, lz4/frame/init.py) re-export the C symbols and layer higher-level Python classes on top — LZ4FrameCompressor/LZ4FrameDecompressor and the file-like LZ4FrameFile/lz4.frame.open() built on the standard library’s private compression-stream base class (imported conditionally to handle the Python 3.14 compression package split). There’s no shared abstraction between block/frame/stream — each format’s Python and C code is independent, so setup.py’s build-time branching (system vs bundled liblz4, MSVC vs unix compiler flags, the experimental-package toggle) is the main structural complexity rather than the runtime code itself.

Tech Stack A CPython C extension with a bundled copy of the upstream LZ4 reference implementation, or a dynamic link against a system liblz4 detected via pkgconfig. Build tooling is classic setuptools with setuptools_scm for git-tag-derived versioning, and tox for orchestrating test, lint, and docs environments. CI is a GitHub Actions workflow building and publishing wheels. Documentation is Sphinx-based with doctested examples, hosted on Read the Docs, and the package declares support through the newest CPython releases, including free-threaded builds.

Code Quality An extensive pytest suite is organized per format (block, frame, stream) with dedicated fixtures, run through tox with coverage reporting. The C extensions raise a dedicated exception type for compression failures rather than generic ones. Linting is configured and enforced separately in CI. There’s no static type-stub coverage, which is typical for a package whose logic mostly lives in C rather than Python, but the test coverage across formats is thorough.

What Makes It Unique The frame API deliberately mirrors the standard library’s own compression module conventions (functions, file objects, context managers), which gives it an unusually low learning curve for anyone who has used those modules before. Combined with dropping the GIL during compression and shipping prebuilt wheels across a wide CPython version range, it reads as a considered, ergonomic wrapper rather than a raw C binding exposed as-is.