pgzip

pgzip is a fully gzip-compatible drop-in replacement for Go’s standard compress/gzip package that parallelizes both compression and decompression across CPU cores. On the write side, input is split into fixed-size blocks (1MB by default) that are compressed concurrently by a pool of flate.Writer instances, then written out in strict order so the resulting file is a completely standard gzip stream readable by any gzip implementation. On the read side, a background goroutine decompresses ahead of the consumer and feeds decoded blocks through a buffered channel, so Read calls rarely block on decompression as long as the reader can keep up.

Because the exported API mirrors compress/gzip almost exactly (NewWriter, NewReader, Header, compression level constants), adopting pgzip is usually a single import-line change. The package targets workloads compressing or decompressing more than roughly 1MB at a time — the block-splitting overhead makes it slower than the standard library on small payloads, but the author’s own benchmarks show over 100x throughput improvement on multi-gigabyte corpora on a many-core machine, with only a small compression-ratio penalty from splitting the input into independently-compressed blocks.

What You Get

• A drop-in Writer/Reader pair matching compress/gzip’s API, so most call sites only need an import swap
• Concurrent block compression via a pool of flate.Writer instances, scaling with runtime.GOMAXPROCS
• Read-ahead decompression on a background goroutine so Read rarely blocks waiting on the decompressor
• SetConcurrency(blockSize, blocks) and NewReaderN to tune block size and parallelism explicitly
• Aggressive buffer reuse via sync.Pool on both write and read paths to reduce GC pressure on large workloads
• Output that is a fully standard gzip file, readable by any RFC 1952-compliant gzip reader

Common Use Cases

• Bulk file or archive compression - CLI tools and backup pipelines that gzip large files or tarballs and want to use all available cores instead of one
• Log and data pipeline compression - services that compress multi-megabyte batches of logs or exported data before writing to disk or object storage
• Fast decompression of large gzip inputs - readers processing big gzip files (e.g. datasets, backups) that benefit from read-ahead decompression overlapping with consumer processing
• HTTP/network payload compression at scale - servers or proxies compressing large response bodies where single-threaded gzip would bottleneck request latency

Under The Hood

Architecture The package is a small, single-directory Go library split across two files: gzip.go (the Writer) and gunzip.go (the Reader). The Writer splits input into fixed-size blocks (defaultBlockSize = 1MB) and hands each full block to a compressBlock goroutine that compresses it with a pooled flate.Writer (dictFlatePool); a dedicated goroutine started on first Write drains a buffered results channel of reserved slots in strict order, so blocks can finish compressing out of order internally while the bytes written to the underlying io.Writer stay in the original sequence. The Reader mirrors this with a doReadAhead goroutine that continuously decompresses ahead of the caller into a blockPool-backed buffer and pushes finished blocks through a buffered readAhead channel, so Read mostly just drains that channel instead of blocking on decompression. Both sides defend concurrency correctness with mutex-guarded error state (errMu/pushError) and explicit channel-based shutdown (closeReader/closeErr) to avoid goroutine leaks when a Reader is closed early or a Writer hits an error mid-stream.

Tech Stack pgzip is pure Go with a single external dependency, github.com/klauspost/compress (the same author’s optimized flate/deflate implementation), used via flate.NewWriterDict and flate.NewReader instead of the standard library’s compress/flate. There is no build tooling beyond go build/go test; go.mod pins Go 1.23 and compress v1.18.1. CI runs on GitHub Actions with a standard Go test workflow plus a separate CodeQL workflow for static security analysis. The library has no runtime or server component — it’s imported directly into whatever binary needs parallel gzip.

Code Quality Test coverage is substantial relative to the implementation’s size: roughly 1,600 lines of tests (gunzip_test.go, gzip_test.go, a race-specific gzip_norace_test.go, and gzip_unreliable_test.go) against about 500 lines of source, covering round-trips, Reset, multistream handling, WriteTo/read-ahead edge cases, truncated input, and regression tests referencing specific historical bug numbers (e.g. TestReadAfterWriteToNoDeadlock, TestWriteToDoesNotReturnEOF). Errors are explicit sentinel values (ErrChecksum, ErrHeader) propagated through a checkError/pushError pair rather than swallowed, and exported types carry Go-doc-style comments. There’s no linter config beyond go vet in CI, but the CodeQL workflow adds static security scanning on every push.

API Design The core design goal — and its main strength — is being a genuinely drop-in replacement: the exported Writer/Reader/Header surface mirrors compress/gzip almost exactly, so adopting pgzip is typically a single import swap rather than an API migration. Extra knobs (SetConcurrency, NewReaderN, UncompressedSize) are additive and only need to be touched when tuning block size or parallelism beyond the sane defaults (GOMAXPROCS-based block count, 1MB blocks). Parallel block-splitting gzip is an established technique used by tools like pigz, so the innovation here is less algorithmic novelty and more disciplined execution: buffer pooling on both read and write paths, ordered output despite out-of-order internal compression, and read-ahead decompression that overlaps with consumer processing.