fastparquet

fastparquet is a Python library that implements the Apache Parquet columnar storage format directly in Python (with Cython-accelerated encoding/decoding routines), so projects can read and write Parquet files without depending on a compiled Arrow toolchain. It integrates tightly with pandas, exposing a ParquetFile object for reading row groups and column subsets, and a write() function for producing single-file or partitioned (hive/drill-style) datasets with a choice of compression codecs.

Because it speaks fsspec, fastparquet can read and write against local disk, S3, GCS, HDFS, and other remote filesystems with the same API, which is why it is used implicitly by Dask and other big-data-adjacent Python tooling. It supports dictionary-encoded categoricals, nullable pandas extension types, partitioned datasets with _metadata/_common_metadata sidecar files, and most of the compression codecs in common use (gzip, snappy, brotli, lz4, zstandard).

The project’s own README now notes that, with pandas 3.0 depending explicitly on pyarrow, fastparquet’s raison d’être has diminished and it is being wound down — but it remains actively maintained enough to support pandas 2.x users and existing Dask integrations, and its small install footprint and long track record still make it a relevant option for constrained environments where a full Arrow dependency isn’t wanted.

What You Get

• A ParquetFile class for opening single files, _metadata-based dataset directories, or lists of files, with lazy row-group and column access
• A write() function supporting single-file output, row-group chunking, and hive/drill-style partitioned datasets
• Cython-accelerated encoding/decoding (speedups, cencoding) for the hot paths of reading and writing pages
• Built-in support for gzip, snappy, brotli, lz4, and zstandard compression codecs, with optional lzo support
• fsspec-based filesystem abstraction, so the same API reads/writes local disk, S3, GCS, HDFS, and other remote stores
• Handling of pandas-specific metadata: categoricals, nullable Int/UInt/boolean extension types, and index preservation across round-trips

Common Use Cases

• Reading and writing Parquet datasets from pandas pipelines without pulling in a full PyArrow/Arrow C++ dependency
• Serving as the Parquet I/O engine underneath Dask DataFrame’s read_parquet/to_parquet in existing Dask deployments
• Working with hive/spark-partitioned Parquet directory trees (directory-encoded partition columns, _metadata sidecar files)
• Appending to or selectively rewriting row groups in existing Parquet datasets rather than rewriting a file wholesale
• Running in environments where compiled Arrow binaries are hard to install but a lightweight, mostly-pure-Python Parquet reader/writer is acceptable

Under The Hood

Architecture fastparquet splits cleanly into a read path (api.py, core.py) and a write path (writer.py), both sitting on shared building blocks in schema.py, encoding.py/cencoding.pyx, converted_types.py, and compression.py. api.ParquetFile is the entry point for reading: it parses Thrift-encoded file metadata via thrift_structures.py/cencoding.ThriftObject, exposes row-group and column-level access, and delegates actual page decoding to core.read_data/_read_page, which handles decompression and definition/repetition-level decoding before handing raw arrays back up for conversion into a pandas DataFrame in dataframe.py. writer.write() mirrors this in reverse — converting DataFrame columns into typed Parquet pages, encoding them through the same Cython routines, and assembling Thrift metadata for single-file or hive-partitioned output. The dependency direction is deliberate: core.py and writer.py both import from api.py for shared file-scheme helpers, while the Cython extensions (speedups.pyx, cencoding.pyx) are isolated behind plain-Python wrapper functions so the rest of the codebase never touches compiled internals directly. Swapping the on-disk encoding scheme would mean touching converted_types.py and both core.py/writer.py symmetrically, since encode and decode logic isn’t unified behind one dispatch table.

Tech Stack The library targets CPython 3.10-3.14 (per ci/environment-py3*.yml) and depends on pandas>=1.5.0, numpy, cramjam>=2.3 (a Rust-backed compression binding replacing the older pure-Python codec wrappers), fsspec for filesystem abstraction, and packaging. Two Cython extension modules (fastparquet.speedups, fastparquet.cencoding) are compiled at install time via setup.py’s custom build_ext, with numpy.get_include() wired in and an x86-64-v2 -march probe for optional SIMD-friendly codegen; a pure-Python/C fallback path exists for platforms without Cython at build time. Compression codecs (gzip, snappy, brotli, lz4, zstandard, optional lzo) are dispatched through compression.py. When PyArrow is present, encoding.py conditionally uses Arrow-backed string arrays (_USE_ARROW_STRINGS) and Arrow byte-array unpacking for extra throughput, but PyArrow itself is not a hard dependency, matching the project’s positioning as an Arrow-independent Parquet engine.

Code Quality The fastparquet/test/ directory holds a substantial pytest suite (test_api.py, test_read.py, test_output.py, test_partition_filters_specialstrings.py, test_pd_optional_types.py, test_timestamp_conversion_960.py, and others) exercising round-trips, partitioning, nullable dtypes, and specific historical regressions, run with pytest --cov=fastparquet in CI. CI (.github/workflows/main.yaml) matrices across Python 3.10-3.14 and separately re-runs a subset against a pandas-3.x/PyArrow-backed environment, which is a meaningful quality signal given the pandas 3.0 transition the README calls out. Error handling favors a dedicated ParquetException for domain errors surfaced from util.py, though much of the low-level Cython/Thrift-parsing code has limited inline documentation and no static type annotations, and the module-level evolve.py file is present but empty, suggesting an abandoned or in-progress feature.

API Design The public surface is intentionally small and pandas-idiomatic: ParquetFile(path).to_pandas(columns=..., categories=...) for reading and write(path, df, ...) for writing cover the overwhelming majority of use cases in one or two calls, with keyword arguments (row_group_offsets, compression, file_scheme, partition_on) layered on for advanced control rather than requiring separate builder objects. Docstrings on ParquetFile and write() are detailed and parameter-by-parameter, which lowers the barrier for someone coming from pandas.read_parquet/to_parquet (which can delegate to this engine). The tradeoff is that some behavior — such as pandas_nulls type coercion rules or hive-partition value coercion — is documented in prose within docstrings rather than through named, discoverable enums or constants, so getting non-default behavior right requires reading documentation carefully rather than relying on autocomplete alone.