Arrow

arrow is the official Rust implementation of Apache Arrow, a cross-language columnar in-memory data format designed for fast analytics and zero-copy data interchange. It provides statically typed array implementations (Int32Array, StringArray, and dozens more), a shared memory layout compatible with Arrow implementations in C++, Python, Java, and other languages, and a rich set of compute kernels for filtering, casting, sorting, and arithmetic over columnar data.

Maintained by the Apache Software Foundation and used as the foundational data layer for projects like DataFusion and Ballista, arrow re-exports and unifies a family of focused sub-crates (arrow-array, arrow-buffer, arrow-schema, arrow-ipc, arrow-csv, arrow-json, and more) behind a single dependency, giving Rust programs a fast, memory-safe way to represent, transform, and exchange large in-memory datasets.

What You Get

• Statically typed array types (Int32Array, StringArray, ListArray, StructArray, dictionary and run-end-encoded arrays, and more) built on a shared ArrayData/Buffer memory model
• Compute kernels for arithmetic, comparison, casting, sorting, filtering, and string operations over columnar arrays
• Readers/writers for CSV, JSON, and the Arrow IPC (stream/file) format, plus optional Arrow Flight and C Data Interface (FFI) support for cross-process and cross-language interchange
• A RecordBatch/Schema abstraction for representing tabular data, shared across the whole Arrow Rust ecosystem (DataFusion, Ballista, object_store consumers)
• Feature flags to opt in to only what you need (csv, json, ipc, ipc_compression, prettyprint, ffi, pyarrow, chrono-tz, async), keeping compile times and binary size down

Common Use Cases

• Building an analytical query engine or dataframe library - use arrow’s array types and compute kernels as the execution substrate, the way DataFusion does
• Exchanging large datasets between processes or languages with zero copy - serialize RecordBatches over Arrow IPC or Arrow Flight and read them back without deserialization overhead
• Reading/writing columnar data from CSV or JSON at scale - use the built-in csv/json readers and writers to convert row-oriented sources into Arrow’s columnar layout for fast downstream analytics
• Interfacing Rust with Python’s pyarrow via the C Data Interface - use the ffi/pyarrow feature to pass Arrow arrays between Rust and Python without copying memory

Under The Hood

Architecture: The arrow crate (arrow/src/lib.rs) is primarily a facade: it re-exports and wires together a family of focused workspace crates — arrow-array (typed array implementations, builders, iterators), arrow-data (the untyped ArrayData/Buffer memory representation and MutableArrayData transform), arrow-buffer (raw byte buffers, bit-packed boolean buffers, bigint/interval scalars), arrow-schema (DataType/Field/Schema), arrow-ord/arrow-select/arrow-string/arrow-arith (compute kernels), and optional arrow-csv/arrow-json/arrow-ipc/arrow-pyarrow readers and writers gated behind Cargo features (csv, json, ipc, ffi, pyarrow, async). Data flows from a RecordBatch (a Schema plus a Vec of Arc<dyn Array>) through zero-copy operations on shared Buffer allocations; format readers/writers convert between this columnar in-memory layout and external representations, and the FFI/pyarrow modules expose the same ArrayData layout across the C Data Interface for cross-language interop without copying memory.

Tech Stack: Pure Rust, edition 2024, MSRV 1.85, built with Cargo’s workspace resolver v2 to keep feature unification from leaking dev-dependencies into library builds. Core dependencies are the sibling arrow-* crates plus half (f16 support) and rand (test utilities, optional). Optional integrations layer in chrono/chrono-tz (timezone-aware temporal types), serde (dev-only), pyo3-based FFI for Python interop (arrow-pyarrow), and lz4/zstd for IPC stream compression. CI (rust.yml, arrow.yml, arrow_flight.yml, parquet.yml, miri.yaml, docs.yml) runs cargo fmt —check, lint gates, Miri for undefined-behavior detection given the crate’s extensive use of unsafe pointer arithmetic in buffer code, and a monthly release cadence (minor releases with no breaking changes, majors quarterly) documented in CONTRIBUTING.md.

Code Quality: Testing is extensive: a workspace-wide grep finds roughly 1,557 #[test] functions across the core crates (arrow-array, arrow-buffer, arrow-data, arrow-cast), plus dedicated integration test files in arrow/tests/ covering arithmetic, array casting, array equality, array transforms, array validation, CSV round-tripping, schema handling, and timezone behavior. The project also runs Miri in CI to catch undefined behavior in its low-level buffer/pointer code, and gates PRs on cargo fmt —all — —check for consistent formatting across all crates. Error handling is centralized in a single ArrowError enum (arrow/src/error.rs) shared across the workspace rather than crate-specific error types, and the codebase enforces a documented panic-vs-Result policy (panics only for unreachable/unrecoverable states, Result for invalid user input) — a level of process rigor typical of an Apache-governed foundational project.

API Design: The public API favors static typing and zero-cost abstractions: concrete array types (Int32Array, StringArray, etc.) implement shared traits (Array, ArrayAccessor, ArrayIter) so generic code can be written once and monomorphized per type, while the AsArray extension trait and as_any().downcast_ref() pattern support type-erased dyn Array handling when the concrete type isn’t known ahead of time. Getting started requires minimal boilerplate — Int32Array::from(vec![…]) — and the crate ships runnable doctests directly in lib.rs demonstrating both the typed and type-erased paths. Feature flags (default: csv, ipc, json) keep the base dependency footprint small while making advanced capabilities (ffi, pyarrow, async, chrono-tz, ipc_compression) fully opt-in, and docs.rs is configured to build with all-features so the complete API surface is discoverable in one place.