blake3-py

blake3-py wraps the official Rust implementation of BLAKE3 with PyO3 bindings, giving Python programs access to the full BLAKE3 feature set: incremental hashing, keyed hashing (MAC), key derivation, extendable-output digests, and memory-mapped multithreaded hashing of large files. The public API deliberately mirrors Python’s standard hashlib module (update(), digest(), hexdigest(), copy()), so it’s a near drop-in replacement wherever a fast, modern hash is needed instead of SHA-2 or SHA-3.

Because the heavy lifting happens in the underlying Rust blake3 crate (with rayon for parallelism and mmap for zero-copy file reads), the Python layer stays thin — its job is safely bridging Python buffer objects (bytes, bytearray, memoryview, NumPy arrays) into Rust slices without violating the GIL’s aliasing guarantees. Prebuilt wheels are published for most platforms, so most users never need a Rust toolchain to install it.

What You Get

• A blake3() hasher object with the same update()/digest()/hexdigest()/copy() shape as hashlib, so it drops into existing hashing code with minimal changes
• Keyed hashing mode (key=) for building MACs from a 32-byte key, and key-derivation mode (derive_key_context=) for deriving application-specific subkeys
• Extendable-output hashing via digest(length=..., seek=...), letting callers pull an arbitrary number of output bytes from any offset
• Multithreaded hashing (max_threads=) backed by rayon, plus update_mmap() for memory-mapped, multithreaded hashing of large files without loading them fully into memory
• Prebuilt binary wheels for the common OS/architecture matrix (Linux, macOS, Windows, multiple CPU targets, manylinux and musllinux) so most installs need no Rust toolchain

Common Use Cases

• Content-addressed storage and deduplication systems that need a fast, collision-resistant hash to fingerprint file contents
• Hashing large files or datasets where BLAKE3’s multithreaded, memory-mapped mode gives a meaningful speedup over single-threaded SHA-2
• Message authentication (MAC) using BLAKE3’s built-in keyed mode instead of composing HMAC with a separate hash
• Deriving multiple independent subkeys from one master secret via BLAKE3’s key-derivation mode
• Drop-in replacement for hashlib.sha256()/sha3_256() in performance-sensitive Python code paths

Under The Hood

Architecture The crate is a thin PyO3 extension (src/lib.rs) exposing a single #[pyclass] Blake3Class with #[pymethods] for new, update, update_mmap, copy, reset, digest, and hexdigest; all cryptographic work is delegated to the upstream blake3 Rust crate (aliased as upstream_blake3), with a BytesPyBuffer enum bridging Python’s buffer protocol (both signed and unsigned byte buffers, e.g. bytes, bytearray, NumPy arrays) into safe &[u8] slices while respecting the GIL and buffer-locking semantics — the module explicitly documents the unsafe-code invariants around concurrent mutation and GIL release for large inputs (GIL_MINSIZE threshold). What breaks if the core abstraction changes: any modification to the buffer-bridging logic risks undefined behavior from aliased mutable buffers, so that layer is deliberately conservative and heavily commented.

Tech Stack Built with pyo3 0.29 (extension-module feature) and packaged via maturin, targeting Python 3.8+ (including free-threaded builds). It depends on the upstream blake3 crate 1.5.5 with mmap and rayon features enabled for parallel, memory-mapped hashing, plus hex for hex-encoding output. pyproject.toml declares dynamic metadata resolved by maturin and a typing_extensions runtime dependency for Python <=3.11. CI builds wheels across a wide OS/architecture matrix (Linux manylinux/musllinux, macOS x86_64/aarch64, Windows x86/x86_64) via GitHub Actions and PyO3/maturin-action.

Code Quality Tests live in tests/test_blake3.py (nearly 500 lines) and run against official BLAKE3 test vectors (test_vectors.json) covering hashing, keyed hashing, key derivation, extendable output, incremental updates, buffer types (including NumPy arrays and non-contiguous buffers), and multithreaded/mmap paths; a separate tests/example.py doubles as a runnable usage example. src/lib.rs uses typed Rust error conversions (PyValueError, PyBufferError, PyOverflowError) rather than silent failures, and extensive inline comments explain non-obvious safety invariants. CI runs the test suite across the full supported Python version matrix.

API Design The library’s defining choice is API compatibility with hashlib: anyone who has used Python’s standard hash objects can use blake3() with zero new concepts, while opt-in keyword arguments (key, derive_key_context, max_threads) expose BLAKE3-specific capabilities without cluttering the common case. digest(length=..., seek=...) cleanly extends the interface for BLAKE3’s extendable-output mode, and update_mmap() adds a memory-mapped fast path for whole-file hashing — both are additive rather than replacing the familiar surface, keeping the barrier to entry low despite the library wrapping a lower-level Rust/C implementation.