crc32c

crc32c is a Rust implementation of the CRC-32-Castagnoli checksum algorithm, the variant used by protocols like iSCSI, Btrfs, and other storage formats that need a fast, well-distributed integrity check. It detects CPU capabilities at runtime and transparently switches between an SSE 4.2 hardware-accelerated path on x86-64, an ARMv8 CRC-extension path on aarch64, and a table-driven software fallback everywhere else, so callers get the fastest available implementation without any conditional compilation of their own.

Beyond the core crc32c()/crc32c_append() functions, the crate ships a std::hash::Hasher implementation for drop-in use with hashing containers, Read/Write wrappers that checksum data as it streams through, and a crc32c_combine() function ported from zlib’s CRC-combine algorithm for merging the checksums of two byte ranges without re-reading either one.

What You Get

• crc32c() and crc32c_append() functions for one-shot and incremental checksum computation
• Automatic runtime CPU-feature detection choosing SSE 4.2, ARM CRC, or software implementations
• A Crc32cHasher implementing std::hash::Hasher for use with hashing collections
• Crc32cReader and Crc32cWriter wrappers that checksum data transparently while streaming
• crc32c_combine() for merging two checksums (and their lengths) without re-reading the data

Common Use Cases

• Verifying data integrity for storage formats and protocols that standardize on CRC-32C (iSCSI, Btrfs, ext4 metadata)
• Checksumming log-structured message formats that use CRC-32C framing
• Streaming integrity checks on file uploads/downloads via the Read/Write wrapper types
• Merging per-chunk checksums from parallel or distributed processing using crc32c_combine()

Under The Hood

Architecture — The crate dispatches at runtime through crc32c_append(), which checks is_x86_feature_detected!("sse4.2") on x86-64 or is_aarch64_feature_detected!("crc") on aarch64 (the latter gated behind an armsimd cfg requiring Rust ≥1.80 or nightly) and falls through to a portable software implementation otherwise. The hardware paths (hw_x86_64.rs, hw_aarch64.rs) use _mm_crc32_u8/_mm_crc32_u64 (or the aarch64 equivalent) three-wide to exploit instruction-level parallelism, processing data in LONG/SHORT-sized chunks defined in hw_tables.rs before falling back to the software table for the remainder. The software path (sw.rs) uses an 8-KiB lookup table generated at compile time by build.rs into OUT_DIR/sw.table, consumed byte-at-a-time and 8-bytes-at-a-time via util::split(), which partitions a buffer into unaligned head, 8-byte-aligned middle, and unaligned tail regions. combine.rs is a line-by-line Rust port of zlib’s GF(2)-matrix CRC-combine algorithm, letting crc32c_combine(crc1, crc2, len2) compute the checksum of two concatenated buffers from their individual checksums and the second buffer’s length alone.

Tech Stack — Pure Rust, edition 2018, with zero runtime dependencies for the core library. build.rs depends on rustc_version to detect the compiler channel/version at build time (needed to gate the aarch64 CRC intrinsics, stabilized in Rust 1.80). Dev-dependencies (rand, criterion) are used only for the tests/simple.rs integration test and the benches/rand.rs Criterion benchmark, and are excluded from the published crate via Cargo’s exclude field.

Code Quality — The crate has 9 #[test] functions spread across src/hasher.rs, src/io.rs, and tests/simple.rs, covering Hasher/Read/Write round-trips against known checksum constants and a brute-force loop (buffer lengths 0–12) validating that crc32c_combine() always agrees with crc32c_append() on concatenated buffers. unsafe usage is confined to the SIMD intrinsic call sites in hw_x86_64.rs/hw_aarch64.rs, each behind a #[target_feature] boundary reached only after a runtime feature check, so the public API itself is entirely safe. There is no fuzz-testing harness and the README’s CI badge points at a defunct Travis build, but naming and doc-comment coverage on public items are consistent throughout.

API Design — The public surface is deliberately small: crc32c() and crc32c_append() cover the common case in two calls, Crc32cHasher slots into anything expecting std::hash::Hasher, and Crc32cReader/Crc32cWriter wrap existing Read/Write types with zero new methods to learn beyond .crc32c(). crc32c_combine() is a more specialized addition for distributed/chunked workloads. Getting started requires nothing beyond adding the crate and calling one function, as shown directly in the README.