xxhash

xxhash is a Python C-extension binding for Yann Collet’s xxHash library, exposing xxh32, xxh64, and the newer XXH3 (xxh3_64 / xxh3_128) hash variants through a hashlib-compatible interface. Because the actual hashing happens in native C code, it is dramatically faster than Python’s built-in hash functions or pure-Python implementations, while remaining a drop-in replacement for hashlib.md5-style usage (update(), digest(), hexdigest(), copy(), reset()).

The library also provides oneshot functions (xxh64_hexdigest(), xxh3_128_intdigest(), etc.) that avoid allocating a persistent hash-state object on the heap when only a single digest is needed, and ships prebuilt wheels for CPython, PyPy, and GraalPy across major platforms via cibuildwheel. With over 40 million weekly PyPI downloads, it is one of the most widely used non-cryptographic hashing libraries in the Python ecosystem, commonly used for checksums, cache keys, and deduplication where cryptographic strength is not required.

What You Get

• hashlib-compatible streaming hash objects (xxh32(), xxh64(), xxh3_64(), xxh3_128()) with update(), digest(), hexdigest(), intdigest(), copy(), and reset()
• Oneshot digest functions (xxh64_hexdigest(bytes, seed=0), etc.) that skip heap allocation for single-shot hashing
• Optional seeding to produce different, predictable digests from the same input
• Prebuilt wheels across CPython, PyPy, GraalPy, and Pyodide via cibuildwheel, so most installs need no C compiler
• Full type stubs (py.typed, .pyi files) for static type checking

Common Use Cases

• Generating fast cache keys or deduplication fingerprints for large datasets where MD5/SHA speed is a bottleneck
• Computing checksums for data integrity checks in ETL and data-processing pipelines
• Hashing large files or byte streams quickly in build tools, package managers, or content-addressed storage
• Replacing Python’s built-in hash() in performance-sensitive hash-table or bloom-filter implementations that need stable cross-run hashing

Under The Hood

Architecture: The Python-facing surface lives in xxhash/__init__.py (a thin re-export layer with full type stubs in __init__.pyi), backed by a single CPython C extension (src/_xxhash.c, ~2,260 lines) that implements the module’s classes and oneshot functions directly against the vendored deps/xxhash/xxhash.c reference implementation. This keeps the Python layer essentially zero-overhead — nearly all logic, including streaming state management, lives in C.

Tech Stack: Built with setuptools (setup.py + pyproject.toml), compiling the bundled xxHash C source by default; an XXHASH_LINK_SO environment variable switches to linking against a system-installed libxxhash.so instead of the vendored copy. cibuildwheel is configured to build wheels for CPython, PyPy (including EOL versions), GraalPy, and Pyodide across platforms, minimizing the need for end users to compile anything.

Code Quality: The tests/ directory is unusually thorough for a small binding library — separate suites for each algorithm (test_xxh32.py, test_xxh64.py, test_xxh3_64.py, test_xxh3_128.py), plus dedicated tests for thread safety (test_thread_safety.py), interpreter-level behavior (test_interpreters.py), hashlib compatibility (test_hashlib_compat.py), fast-call argument handling, and even pyright stub correctness (test_stubs_pyright.py). Strict mypy configuration (disallow_untyped_calls, disallow_incomplete_defs, warn_return_any) is enabled in pyproject.toml, signaling a high bar for type-checked correctness despite the C extension core.

API Design: The library deliberately mirrors hashlib’s object interface, so any developer familiar with hashlib.md5() can pick up xxhash.xxh64() with zero learning curve; oneshot functions (xxh64_hexdigest()) are provided as an ergonomic shortcut for the common single-call case. The one documented rough edge is a historical endianness change in digest() (little-endian before v0.3.0, big-endian after), which the README calls out explicitly as a caveat for anyone reading older code examples.