xxhash

xxhash implements XXH64, the 64-bit variant of the xxHash non-cryptographic hashing algorithm, as a small, dependency-free Go module. The package exposes a minimal API — a one-shot Sum64/Sum64String pair plus a Digest type implementing hash.Hash64 for streaming writes — so it drops into existing Go code with almost no integration cost. On amd64 and arm64 it dispatches to hand-tuned assembly for a significant throughput boost over the pure-Go path, while a purego build tag keeps a portable fallback available for every other architecture.

Because it is fast, allocation-free, and has zero third-party dependencies, xxhash/v2 has become a default choice for internal hashing needs across the Go ecosystem — it is vendored or imported directly by projects like InfluxDB, Prometheus, VictoriaMetrics, Badger, and Ristretto for tasks such as cache-key generation, sharding, and checksum computation where cryptographic strength is unnecessary and speed is the priority.

What You Get

• A Sum64(b []byte) uint64 and Sum64String(s string) uint64 pair for one-shot hashing with no setup
• A Digest type implementing the standard library’s hash.Hash64 interface for streaming/incremental hashing
• Hand-written amd64 and arm64 assembly implementations for maximum throughput, selected automatically at build time
• A purego build tag that forces the portable pure-Go path on any architecture, including ones without assembly support
• MarshalBinary/UnmarshalBinary support on Digest for persisting and resuming hash state
• A small xxhsum CLI utility (in xxhsum/) for hashing files or stdin from the command line

Common Use Cases

• Generating fast, well-distributed cache keys for in-memory or distributed caches
• Computing checksums for data integrity checks where cryptographic security is not required
• Sharding or partitioning keys across nodes based on a fast, uniform hash distribution
• Deduplicating byte slices or strings in high-throughput data pipelines
• Implementing custom hash-map or bloom-filter style structures that need a fast non-cryptographic hash function

Under The Hood

Architecture The package has a flat, single-package layout with no internal layering — xxhash.go holds the core algorithm state (Digest), the round/mergeRound helper functions, and the prime constants, while xxhash_asm.go and xxhash_other.go use Go build tags to select between assembly (xxhash_amd64.s, xxhash_arm64.s) and a portable Go implementation of the same two exported functions (Sum64, writeBlocks) at compile time — a build-tag dispatch pattern rather than a runtime branch, so choosing the fast path costs nothing at run time. The Digest struct carries only its four accumulator words, a small internal buffer, and a length counter, and both Write and Sum64 mutate this state directly with no heap allocations. Because the abstraction boundary is simply “these two functions must exist and agree,” changing the core hashing algorithm would require updating three parallel implementations — amd64 assembly, arm64 assembly, and the portable Go fallback — in lockstep.

Tech Stack go.mod declares Go 1.11 compatibility with zero external dependencies; the module targets amd64 and arm64 via hand-written Go assembly fed straight to the Go toolchain’s own assembler, with xxhash_unsafe.go using unsafe.Pointer for zero-copy string-to-byte conversion on supported platforms and xxhash_safe.go as the non-unsafe counterpart used elsewhere. CI runs go test -bench across recent Go versions on Linux, macOS, and Windows for native amd64/arm64, plus a QEMU-emulated matrix covering 386/arm/arm64 to validate the pure-Go path on architectures without hand-written assembly. There is no build system beyond the standard go build/go test; the only shipped binary is the small xxhsum CLI in its own subdirectory.

Code Quality Test files provide table-driven tests and benchmarks, and the CI matrix runs them under both the assembly and purego build tags to guarantee the fast and portable implementations stay bit-identical. Error handling is deliberately minimal — Write never fails, matching the hash.Hash contract — with errors reserved for MarshalBinary/UnmarshalBinary on malformed state. Naming follows standard-library hashing conventions (Sum64, Digest, Reset), and the internals favor small, individually documented helper functions over dense one-off code. No linter configuration is present, but the source is consistently gofmt-formatted and idiomatic.

API Design The public surface is intentionally tiny — two free functions (Sum64, Sum64String) plus one struct satisfying the standard library’s hash.Hash64 interface — so most callers need nothing beyond the package doc comment to get started, and existing code written against hash.Hash64 works unmodified. The build-tag-driven fast/portable dispatch and the purego escape hatch stay invisible to callers, so ergonomics never trade off against performance. The tradeoff is a narrow, unconfigurable scope: this package implements XXH64 only, not the newer XXH3 or 32-bit variants, so projects needing multiple hash families must reach for a different module.