lz4

lz4 is a pure Go library that implements both the LZ4 streaming frame format and the low-level LZ4 block format, based on the reference C implementation. It exposes io.Reader/io.Writer-compatible Writer and Reader types for streaming compression and decompression, plus direct CompressBlock/UncompressBlock functions for callers that manage framing themselves.

The library is widely used as a dependency inside larger Go systems (databases, container tooling, log shippers) that need fast, low-overhead compression without cgo. Decoding is backed by hand-written Assembly on amd64, arm, and arm64, with a portable Go fallback selectable via the noasm build tag, so it runs anywhere Go runs while still being fast on common architectures.

A functional-options API (BlockSizeOption, ChecksumOption, ConcurrencyOption, CompressionLevelOption, LegacyOption) lets callers tune block size, checksums, compression level, and goroutine-based concurrency without changing call sites. A companion lz4c command-line tool ships in the repo for compressing and decompressing files directly from the shell.

What You Get

• Streaming Writer/Reader types that implement io.Writer/io.Reader for drop-in use with any Go I/O pipeline
• Low-level CompressBlock/UncompressBlock functions for callers that manage their own framing
• Functional options for block size, checksums, compression level (Fast through Level9), and concurrency
• Assembly-optimized decoding on amd64/arm/arm64 with a portable Go fallback via the noasm build tag
• Concurrent multi-goroutine compression via ConcurrencyOption for throughput on multi-core hosts
• A standalone lz4c CLI tool for compressing/decompressing files from the command line

Common Use Cases

• Compressing log or event streams before writing them to disk or shipping them over the network
• Reducing storage footprint for database write-ahead logs, snapshots, or column blocks
• Decompressing LZ4-framed data produced by other tools or the reference C implementation
• Building command-line utilities or services that need fast, low-latency compression without a C toolchain
• Interoperating with Linux kernel images or other consumers of the legacy LZ4 frame format

Under The Hood

Architecture The public API in writer.go/reader.go is a thin, state-machine-governed (aState) wrapper around three internal packages: internal/lz4block (block-level compress/uncompress and buffer pooling), internal/lz4stream (frame descriptor, block sequencing, and concurrent block dispatch via channels), and internal/xxh32 (the XXH32 checksum used for block and content integrity). Writer.Write/Writer.ReadFrom buffer input into block-sized chunks and either compress them inline or hand them to a goroutine-per-block pipeline gated by a channel when ConcurrencyOption is greater than one, decoupling I/O from compression scheduling without exposing that complexity to callers.

Tech Stack The module targets Go 1.17+ with zero non-stdlib runtime dependencies; a separate go.mod under cmd/lz4c isolates the CLI tool’s own dependency graph from the library. Hand-written Assembly (decode_amd64.s, decode_arm.s, decode_arm64.s) backs block decoding on supported architectures, with decode_other.go providing a pure-Go fallback selected automatically off those architectures or explicitly via the noasm build tag. CI (GitHub Actions) matrices across Ubuntu, macOS, and Windows on three recent Go versions.

Code Quality The repository carries _test.go files alongside nearly every source file (block, stream, reader, writer, checksum), plus a fuzz/ directory with corpus data for fuzz testing decompression against malformed input. CI runs the full suite with -race and again under the noasm tag on every OS/Go-version combination, giving real coverage of both the Assembly and portable code paths under concurrent access. Errors are returned as typed sentinel values from a dedicated lz4errors package rather than ad hoc strings, and public options validate their inputs (e.g. block size, compression level) before mutating internal state.

What Makes It Unique Unlike compression libraries that only offer a single code path, lz4 maintains parallel Assembly and portable Go decoders behind one build-tag-selected implementation, so the same API is fast on common server architectures and still fully functional on anything else Go compiles to. Its explicit support for the legacy LZ4 frame variant used by compressed Linux kernel images is a narrow but genuinely uncommon compatibility feature not found in most general-purpose compression libraries.