compress

klauspost/compress is a collection of independent, high-performance compression packages for Go. It provides a pure-Go implementation of Zstandard (zstd) for both streaming and one-shot encode/decode, S2 as a faster and more compressible drop-in for Snappy, and optimized reimplementations of the standard library’s flate, gzip, zip, zlib, and lzw packages that are typically faster than their stdlib counterparts while remaining API-compatible.

The project also ships gzhttp, HTTP client and server middleware for transparent gzip/zstd request and response compression, and lower-level huff0/FSE entropy coders that the higher-level codecs build on. Hand-written amd64 and arm64 assembly accelerates the hot paths, with noasm and nounsafe build tags available to fall back to portable Go when needed.

What You Get

• A pure-Go Zstandard (zstd) encoder/decoder with streaming, dictionary, and concurrent-block support
• S2, a high-throughput Snappy-compatible format with better compression ratios and concurrent stream support
• Drop-in replacements for compress/flate, compress/gzip, archive/zip, and compress/zlib that are faster than the standard library
• gzhttp middleware for transparently compressing HTTP client and server traffic (gzip and zstd)
• huff0 and FSE low-level entropy coders for building custom compression schemes
• amd64/arm64 assembly optimizations with noasm/nounsafe build tags for portable fallback

Common Use Cases

• Swapping compress/gzip or archive/zip imports for the package’s faster drop-in equivalents with no code changes beyond the import path
• Compressing and decompressing Zstandard-formatted data (e.g. artifacts, logs, or network payloads) from pure Go without cgo
• Adding transparent gzip/zstd compression to an HTTP server or client via the gzhttp middleware wrappers
• Replacing github.com/golang/snappy with S2 for faster, more compressible, concurrent-stream-capable encoding
• Building custom compression formats on top of the huff0/FSE entropy coders

Under The Hood

Architecture The repository is organized as a flat set of independent, single-purpose codec packages (zstd/, s2/, flate/, gzip/, zip/, zlib/, snappy/, lzw/, huff0/, fse/, gzhttp/, dict/) rather than a layered framework — there’s no central orchestrator, each package is usable standalone. zstd/encoder.go and decoder.go expose NewWriter/NewReader constructors that take variadic functional options (EOption/DOption), accumulated into an encoderOptions/decoderOptions struct via a setDefault-then-apply pattern also used across the other codecs. flate/ deliberately mirrors the standard library’s compress/flate types and tiers its match-finding across separate level1.go through level6.go files by compression-effort tier, which is what lets it serve as a drop-in replacement. gzhttp wraps net/http’s Handler and RoundTripper interfaces to add compression as HTTP middleware rather than a separate transport. The higher-level codecs (zstd, flate) both depend on the huff0 and fse packages for Huffman and finite-state-entropy table construction, so those two packages are the shared low-level layer everything else in the repo builds on.

Tech Stack Pure Go module targeting Go 1.24+ (go.mod) with zero third-party dependencies — go.sum is empty, everything is standard library plus the package’s own internal helpers (internal/le for byte-order loads, internal/cpuinfo for feature detection). Hot paths in zstd, s2, flate, and the entropy coders have hand-written amd64 and arm64 assembly gated behind build tags, with noasm and nounsafe tags to force portable Go fallbacks. No build tooling beyond the Go toolchain itself; gen.sh regenerates some tables from source.

Code Quality Test coverage is extensive and spread across nearly every subpackage, including dedicated fuzz tests and an ossfuzz integration for the more security-sensitive decoders. CI runs go vet plus the full test suite across three Go versions and four operating systems, and separately re-runs tests under the noasm and nounsafe build tags to verify the portable Go fallback paths stay correct, not just the assembly-accelerated ones. Error handling follows idiomatic Go — errors are returned rather than panicking on malformed input, which matters for codecs parsing untrusted compressed data. There is no dedicated linter config, but a CodeQL workflow and an OpenSSF Scorecard check run on every change.

API Design The public API follows the standard library’s io.Reader/io.Writer conventions closely, so the flate, gzip, zip, and zlib packages can typically replace their stdlib counterparts by changing only the import path. Encoder and decoder construction uses the functional-options pattern (NewWriter(w, opts…), NewReader(r, opts…)), which keeps the common case to a single call while still exposing advanced tuning knobs. Each subpackage ships its own README and godoc examples (example_test.go), so the getting-started cost for any one codec is low even though the repository as a whole covers many formats.