xxh3

xxh3 is a Go implementation of the XXH3 algorithm from the xxHash family, ported to match the upstream v0.8.0 output exactly. It exposes simple Hash and HashString functions for one-shot 64-bit and 128-bit hashing, plus a streaming Hasher that implements Go’s standard hash.Hash and hash.Hash64 interfaces for incremental writes.

Under the hood the package dispatches to hand-written SIMD kernels for AVX-512, AVX2, SSE2, and NEON, generated with avo and selected at runtime via CPU feature detection, falling back to a portable scalar implementation when no vector instructions are available. This makes it one of the fastest non-cryptographic hash implementations available for Go, commonly used for checksums, deduplication keys, hash table keys, and content-addressable storage where throughput matters more than cryptographic security.

What You Get

• One-shot Hash and HashString functions for direct 64-bit hashing of byte slices and strings
• Hash128 and Hash128Seed for 128-bit hash output via the Uint128 type
• A streaming Hasher type implementing hash.Hash and hash.Hash64 for incremental writes
• Seeded hashing variants (HashSeed, NewSeed) for keyed or salted hash use cases
• Runtime CPU-feature dispatch across AVX-512, AVX2, SSE2, and NEON kernels with a scalar fallback

Common Use Cases

• Fast checksums for deduplication and content-addressable storage
• Hash table and map key generation where speed matters more than cryptographic guarantees
• Streaming hashes of large files or network payloads via the Hasher interface
• Seeded or keyed hashing for partitioning, sharding, or hash-flooding resistance

Under The Hood

Architecture The core entry point, hashAny in hash64.go, dispatches purely on input length: separate branches for 0-16, 17-128, 129-240, and 241+ bytes each mix the input against a fixed 192-byte key table with a different strategy, while the 241+ branch delegates to one of accumAVX512, accumAVX2, accumSSE, accumNEON, or accumScalar chosen at call time via cached CPU-feature flags. hasher.go layers a streaming Hasher on top of the same primitives, buffering into a block-plus-stripe byte array and calling the equivalent accumBlock* variants as data arrives, then converging through the identical merge-and-avalanche formula used by the one-shot path in Sum64/Sum128. The package is intentionally flat: aside from internal/table (shared lookup data) and internal/compare (a cgo-backed reference implementation used only in tests), everything lives in one package, and the avo/ subdirectory is a separate module used purely for code generation, excluded from normal builds via a replace directive.

Tech Stack A Go 1.22 module with a minimal runtime dependency footprint: github.com/klauspost/cpuid/v2 for CPU feature detection and golang.org/x/sys as an indirect dependency; github.com/zeebo/assert is test-only. The assembly kernels are generated through avo, a Go DSL for writing x86/ARM assembly, invoked via go generate and checked into the repo as .s files rather than generated at build time. A Makefile wraps common developer tasks and CI regenerates the assembly to catch drift.

Code Quality Correctness is validated unusually rigorously: compat_test.go contains hundreds of kilobytes of test vectors cross-checked against a cgo binding to the real upstream C xxHash library in internal/compare, so results are verified bit-for-bit against the reference implementation rather than only self-consistently. hasher_test.go asserts the streaming Hasher produces identical output to the one-shot functions across many input sizes, and escape_test.go guards against unwanted heap allocations. CI runs the test suite across three operating systems, three Go versions, and both amd64 and arm64, plus a separate job that regenerates the avo-produced assembly and fails the build on any diff. The package leans on unsafe.Pointer for zero-copy access to string and slice headers, which is deliberate and idiomatic for a hot-path hashing library but raises the bar for casual contributions; there is no CONTRIBUTING guide.

What Makes It Unique Most Go hash libraries offer at most a scalar or single-SIMD-tier implementation; this package generates and runtime-selects among four distinct vectorized kernels (AVX-512, AVX2, SSE2, NEON) plus a scalar fallback, verified against upstream C output to guarantee cross-language interoperability. The result is a hash function whose observed throughput scales with the host CPU’s capabilities without any build-time configuration, at the cost of maintaining several parallel, code-generated implementations of the same algorithm.