parquet-go

parquet-go is a Go module for working with Apache Parquet, the columnar storage format widely used in data processing and analytics pipelines. It was originally built at Twilio Segment to handle Parquet at scale in latency- and cost-sensitive environments, and is now maintained by the open source community.

The library exposes generic, type-safe APIs (GenericWriter[T], GenericReader[T], GenericBuffer[T]) built on struct tags, alongside lower-level primitives (parquet.File, RowGroup, ColumnChunk, Page) for applications that need direct control over columnar layout. It supports schema evolution and conversion between schema versions, sorting and merging row groups, bloom filters for fast point lookups, and on-disk page buffering for datasets larger than available memory.

Performance is a first-class concern: hot paths such as bloom filters, byte scanning, page statistics, and encodings are SIMD-accelerated on amd64, with build-time controls (GOAMD64=v3/v4, an experimental GOEXPERIMENT=simd path) to trade portability for throughput. The project also supports parallel column writes so wide tables can be encoded across multiple CPU cores.

What You Get

• Generic, struct-tag-driven GenericWriter[T] / GenericReader[T] / GenericBuffer[T] APIs for idiomatic Go read/write of Parquet rows
• Lower-level parquet.File, RowGroup, ColumnChunk, and Page interfaces for building custom storage or query engines on top of Parquet
• Schema evolution support via parquet.Convert, translating rows between schema versions with CopyRows
• Bloom filter generation and lookup (SplitBlockFilter, BloomFilter.Check) for fast point-lookup queries
• Row group sorting (SortingColumns) and merging (MergeRowGroups) for building sorted, compacted datasets
• On-disk page buffering (ColumnPageBuffers, FileBufferPool) so writers can spill to disk instead of holding entire files in memory
• Support for the Parquet VARIANT logical type, including shredded variants for faster typed queries over semi-structured data
• SIMD-accelerated encodings and bloom filters on amd64, with GOAMD64 build tuning and an experimental archsimd-based path

Common Use Cases

• Writing analytics event data or logs directly to Parquet files from Go services for downstream querying in data lakes
• Building custom data processing or storage engines that need direct access to Parquet’s columnar layout (row groups, column chunks, pages)
• Converting and merging Parquet files across evolving application schemas without rewriting all historical data
• Adding fast point-lookup filtering over large Parquet datasets using embedded bloom filters
• Streaming large datasets to Parquet with bounded memory by using on-disk page buffers instead of buffering entire files in memory
• Interoperating with the wider Parquet ecosystem (Spark, Apache Arrow, Java Parquet tooling) from Go-based data pipelines

Under The Hood

Architecture The package is organized around a small set of core abstractions defined at the repository root: Schema (schema.go) derives a columnar schema from Go struct tags or an explicit declaration, File (file.go) parses the Thrift-encoded footer and exposes RowGroups()/ColumnChunks(), and RowGroup/ColumnChunk/Page (row_group.go, column.go, page.go) form the layered abstraction that both the high-level generic writer/reader and low-level consumers build on. GenericWriter[T]/GenericReader[T] (writer.go, reader.go, both several thousand lines) sit on top of this layer, denormalizing Go structs into column buffers and back. Encoding logic is isolated under encoding/ (plain, dictionary, RLE, delta, byte-stream-split, bitpacked), and OS/CPU-specific fast paths live under internal/bytealg, internal/unsafecast, and sparse/, keeping the public API decoupled from the low-level bit-twiddling. Changing the core Page/Value representation would ripple through nearly every encoding and every writer/reader path, since almost all higher-level features are built by composition over these primitives rather than duplication.

Tech Stack The module targets Go 1.24+ (go.mod) with minimal runtime dependencies: klauspost/compress and andybalholm/brotli/pierrec/lz4 for codec support, google/uuid, google.golang.org/protobuf, and parquet-go/bitpack and parquet-go/jsonlite (sibling packages from the same maintainers) for bit-packing and JSON-lite handling respectively. The format/ directory contains generated Thrift bindings for the Parquet file format metadata. Build tooling is plain go build/go test with a purego build tag to disable assembly/SIMD paths, and the project increasingly relies on GOAMD64 levels and an experimental GOEXPERIMENT=simd build for its hottest kernels rather than external build tools.

Code Quality Testing is extensive: 100 _test.go files across the repository, including fuzz tests under encoding/fuzz, and CI (.github/workflows) that cross-validates output against the reference Java Parquet implementation by building parquet-java from source and comparing results, run across both amd64 and ARM runners and with/without the purego tag. The PARQUETGODEBUG=tracebuf=1 facility adds runtime leak detection for internal buffer reference counting, indicating deliberate attention to memory-safety edge cases in a library that does manual buffer pooling. Error handling is idiomatic Go (explicit multi-value returns), and CONTRIBUTING.md documents an explicit code-review policy distinguishing changes that can bypass review from those that require it.

What Makes It Unique Unlike most Parquet bindings for Go, which wrap existing C/C++ implementations, parquet-go is a from-scratch pure-Go implementation with hand-written and, increasingly, portable-SIMD-accelerated kernels for bloom filters, byte scanning, and encodings — letting it cross-compile and run without cgo while still competing with native-code implementations on performance. Its explicit support for on-disk page buffering, parallel column writes, and shredded VARIANT columns addresses large-dataset and semi-structured-data use cases that many Parquet libraries leave to the caller.