python-snappy

python-snappy exposes Google’s Snappy compression algorithm to Python, wrapping the C libsnappy library (or a pure-Python fallback via cramjam/similar when the C library isn’t available) behind a simple compress/decompress API. Snappy favors speed over maximum compression ratio, making it a common choice in systems where compression is applied on every read/write and CPU overhead matters more than a few extra bytes saved.

Beyond one-shot block compression, the package supports streaming compression/decompression for large data that shouldn’t be buffered entirely in memory, and includes format helpers for framed Snappy streams used by tools like Hadoop and Kafka. It’s maintained under the intake GitHub organization (part of the broader Intake/fsspec-adjacent data tooling ecosystem) after being transferred from its original author, and remains the standard Snappy binding most Python data-processing libraries depend on when Snappy-compressed data needs to be read or written.

What You Get

• Simple snappy.compress()/snappy.decompress() functions for one-shot block compression
• Streaming compressor/decompressor classes for processing data too large to buffer entirely in memory
• Support for the framed Snappy stream format used by Hadoop, Kafka, and other big-data tooling
• A CLI entry point (python -m snappy) for compressing/decompressing files from the command line
• C-extension-backed performance via the underlying libsnappy library where available

Common Use Cases

• Compressing/decompressing Kafka message payloads that use Snappy as the configured compression codec
• Reading and writing Snappy-compressed Parquet or Avro data in data-lake pipelines
• Speeding up on-disk or in-transit compression in latency-sensitive services where CPU overhead of stronger codecs (gzip, zstd) isn’t acceptable
• Interoperating with Hadoop ecosystem tools that emit Snappy-framed compressed streams
• Compressing large log or telemetry payloads before shipping them over the network

Under The Hood

Architecture - The Python-facing API lives in src/snappy/snappy.py (~360 lines), which wraps the underlying libsnappy C library’s compress/decompress calls and layers Python-friendly streaming compressor/decompressor classes on top of the block-oriented C API; snappy_formats.py (~115 lines) implements the framed-stream format compatible with Hadoop/Kafka’s Snappy usage, and __main__.py exposes a small CLI wrapper. Tech Stack - A thin C-extension binding (built via build_snappy.sh against the system or vendored libsnappy) exposed through a pure-Python API layer, packaged with classic setuptools; no other runtime dependencies beyond the C library itself. Code Quality - The repo includes test_snappy.py and test_formats.py at the root covering both the block-compression API and the framed-format helpers, though the project shows low recent commit activity, meaning newer Python versions or packaging changes may take time to land. API Design - The API is deliberately minimal — compress/decompress functions plus a couple of streaming classes — mirroring the interface of Python’s built-in zlib/gzip modules, so swapping Snappy in for another codec in existing compression code typically requires only changing the import and function names.