PyArrow

PyArrow is the official Python implementation of Apache Arrow, a language-independent columnar memory format built for fast analytics and zero-copy data interchange. It wraps the project’s C++ core in a Cython layer, exposing Arrow’s Table, Array, ChunkedArray, and RecordBatch primitives directly to Python while keeping the heavy numerical work in native code.

Beyond in-memory representation, pyarrow bundles readers and writers for Parquet, ORC, CSV, and Feather/IPC, a compute module of vectorized kernels, a Dataset API for scanning partitioned files on local or cloud filesystems, Arrow Flight RPC for streaming data over the network, and interoperability layers for pandas, NumPy, and the C Data Interface/DLPack standards used by other Arrow-aware tools.

What You Get

• Columnar Array/Table/ChunkedArray/RecordBatch types backed by the Arrow C++ memory model
• File format I/O for Parquet, ORC, CSV, JSON, and the Arrow IPC/Feather formats
• A vectorized pyarrow.compute kernel library for filtering, aggregation, and string/temporal operations
• A pyarrow.dataset API for scanning partitioned datasets across local disk, S3, GCS, Azure, and HDFS
• Zero-copy interop with pandas, NumPy, and any library implementing the Arrow C Data Interface or DLPack
• Arrow Flight RPC client/server support for streaming Arrow data over the network

Common Use Cases

• Loading and writing large Parquet datasets faster and with lower memory overhead than pandas alone
• Passing columnar data between Python, R, Java, and C++ processes without serialization overhead
• Building data pipelines that filter and aggregate with pyarrow.compute before or instead of pandas
• Reading partitioned data lakes (Hive-style directories on S3/GCS) via the Dataset API
• Bridging data between DuckDB, Polars, pandas, and ML frameworks through the shared Arrow format

Under The Hood

Architecture

pyarrow wraps the Arrow C++ engine via a Cython bridge layer (lib.pyx/lib.pxd, table.pxi, array.pxi) that exposes cdef classes wrapping shared_ptr<CTable>/CArray/CChunkedArray objects declared in includes/libarrow.pxd; higher-level Python modules (dataset.py, compute.py, csv.py, the parquet/ package) sit on top of these primitives and delegate to native kernels, so nearly all data movement and computation happens in C++ while Python mostly marshals calls and metadata — changing a core abstraction like Table requires touching both the Cython .pxi/.pyx layer and its C++ counterpart in the sibling cpp/ tree.

Tech Stack

Built with scikit-build-core, CMake, and Cython >=3.1 as declared in pyproject.toml, targeting the Arrow C++ libraries (libarrow, libarrow_dataset, libarrow_flight, libparquet, libgandiva) with NumPy as the primary Python-side runtime dependency and optional pandas for interop; requires Python >=3.11, builds cross-platform wheels via a dedicated CI workflow, and exposes optional bindings for GLib, Flight, Gandiva, and Substrait behind separate extension modules.

Code Quality

Extensive test coverage lives under pyarrow/tests (dozens of modules spanning arrays, compute, datasets, Parquet/ORC, Flight, CUDA, and Cython-extension interop), run with pytest and supplemented by hypothesis-based property tests; C++ Status codes are translated into a typed Python exception hierarchy (ArrowInvalid, ArrowTypeError, ArrowKeyError, and friends in error.pxi) rather than being swallowed, docstrings embed runnable doctest examples validated via numpydoc, and CI builds and tests across multiple Python versions and operating systems.

API Design The public API favors pandas-like ergonomics (pa.table(), pa.array(), .to_pandas()) while keeping columnar/chunked internals explicit through distinctly named types, so zero-copy behavior is opted into rather than hidden; construction functions follow a consistent per-format naming scheme (read_csv/write_csv, read_parquet/write_parquet) and compute kernels are exposed as both free functions and array/table methods, keeping the common “read format, compute, write format” path low on boilerplate, though the still-evolving separate type-stub package means static typing support lags the runtime API.