Apache DataFusion

Apache DataFusion is a Rust query engine built on Apache Arrow that gives developers a fully working, high-performance analytical database out of the box — complete with a SQL parser and planner, a fluent DataFrame API, a cost-based optimizer, and a columnar, vectorized, multi-threaded execution engine. Rather than being a single-purpose database, it is designed as a toolkit: every stage of query processing (data sources, functions, logical/physical optimizer rules, and the planner itself) can be extended or replaced, which is why it underpins projects ranging from InfluxDB IOx to distributed engines like Ballista and Comet.

It ships built-in readers for CSV, Parquet, JSON, and Avro, supports user-defined scalar/aggregate/window functions, and reuses the broader Arrow/DataFusion ecosystem (datafusion-python, datafusion-java, Substrate/Substrait interop) so teams can start with a production-grade engine and specialize it for their workload instead of writing a query engine from scratch.

What You Get

• A SessionContext entry point exposing both a SQL interface (via sql()) and a fluent, chainable DataFrame API (filter, aggregate, limit, collect) over the same execution engine.
• Built-in, pushdown-aware readers for CSV, Parquet, JSON, and Avro, plus an object_store-backed abstraction for reading from local disk or cloud storage (S3, GCS, Azure).
• A cost-based optimizer pipeline — AnalyzerRules for type coercion and semantic checks, OptimizerRules for logical rewrites like predicate/projection pushdown, and PhysicalOptimizerRules for join and sort strategy selection.
• Extension points at every layer: custom TableProviders for new data sources, user-defined ScalarUDF/AggregateUDF/WindowUDF functions, custom ExecutionPlan operators, and a pluggable QueryPlanner for alternate query languages.
• A columnar, streaming, multi-threaded execution engine built directly on Apache Arrow RecordBatches, so results interoperate natively with the rest of the Arrow ecosystem (Python, Java, Substrait, Arrow Flight).

Common Use Cases

• Building a domain-specific query engine or analytical database without writing a SQL parser, planner, or execution engine from scratch.
• Adding an embedded SQL/DataFrame layer to a Rust data pipeline or ETL tool that needs to query CSV/Parquet/JSON files directly.
• Powering the query layer of a distributed system (as InfluxDB IOx, Ballista, and DataFusion Comet do) by extending DataFusion’s planner and execution operators.
• Serving as the Rust core behind higher-level language bindings (DataFusion Python, DataFusion Java) for teams that want Arrow-native analytics from non-Rust applications.

Under The Hood

Architecture — DataFusion’s execution model has two front doors that converge on the same engine: SQL strings are parsed by sqlparser into an AST, then converted by SqlToRel into a LogicalPlan (with name/type resolution, i.e. “binding”); the DataFrame API builds the same LogicalPlan directly via LogicalPlanBuilder. From there, AnalyzerRules enforce semantic rules like type coercion, OptimizerRules rewrite the plan for efficiency (projection/filter pushdown, etc.), a PhysicalPlanner lowers the LogicalPlan into an ExecutionPlan tree, and PhysicalOptimizerRules pick concrete join/sort strategies before the plan is executed as a columnar, streaming, multi-threaded pipeline over Arrow RecordBatches — the same four-stage pipeline (parse → logical plan → physical plan → execute) is documented and diagrammed directly in the crate’s own rustdoc (datafusion/core/src/lib.rs) and formalized in the project’s SIGMOD 2024 paper. The top-level datafusion crate is itself a thin façade over ~30 workspace crates (datafusion-common, datafusion-catalog, datafusion-expr, datafusion-optimizer, datafusion-physical-plan, datafusion-functions*, datafusion-datasource*, etc.), letting downstream projects depend on just the pieces they need.

Tech Stack — Rust 2024 edition with an MSRV of 1.94.0, built on arrow/arrow-schema 59.1.0 as the in-memory columnar format, sqlparser for SQL parsing, tokio for async execution, and object_store for pluggable local/cloud storage backends; optional features gate Parquet, Avro, compression codecs (zstd, bzip2, flate2, xz2), cryptographic functions, and Parquet Modular Encryption so consumers only pay for what they use. The workspace commits its Cargo.lock and keeps dependencies current via scheduled Dependabot PRs.

Code Quality — The datafusion/core crate alone carries 139+ Rust test files spanning tests/sql, tests/dataframe, tests/optimizer, tests/physical_optimizer, tests/fuzz_cases, tests/memory_limit, tests/tpc-ds, and a dedicated sqllogictest crate for spec-level SQL correctness, backed by insta snapshot testing, rstest parameterized tests, and criterion benchmarks. Lint discipline is enforced at compile time — #![deny(clippy::clone_on_ref_ptr, clippy::or_fun_call, clippy::unnecessary_lazy_evaluations)] and #![warn(missing_docs, clippy::needless_borrow)] — and public API changes follow a published deprecation/API-health policy rather than breaking silently.

API Design — A single SessionContext is the entry point for both SQL and DataFrame workflows, and the DataFrame builder methods (filter, aggregate, limit, collect) read as a fluent query pipeline rather than raw engine internals. The crate’s own rustdoc embeds runnable, tested doctests for both the SQL and DataFrame paths, and the separate datafusion-examples directory holds dozens of runnable examples grouped by topic (UDFs, custom data sources, FFI, Arrow Flight, query planning, Substrait), which meaningfully lowers the cost of the engine’s otherwise large public surface area.