Polars

Polars is a DataFrame library whose query engine is written entirely in Rust and exposed to Python (and Node.js, R) through native bindings, giving pandas-scale ergonomics with multi-threaded, vectorized execution and a columnar Apache Arrow memory model. It offers both an eager API for immediate execution and a lazy API that builds a query plan, optimizes it (predicate/projection pushdown, common subexpression elimination), and executes it in parallel — including out-of-core streaming for datasets larger than RAM. Its expression-based syntax (pl.col(...)) lets the same code run efficiently whether the data is a few rows or hundreds of gigabytes.

What You Get

• Eager (pl.DataFrame) and lazy (pl.LazyFrame) APIs sharing one consistent expression syntax built around pl.col()
• A query optimizer that applies predicate pushdown, projection pushdown, and common subexpression elimination before executing a lazy query
• Out-of-core streaming execution for datasets that don’t fit in memory
• Native readers/writers for CSV, Parquet, JSON, Arrow IPC, Delta Lake, and database connections, with automatic schema inference
• Zero-copy interop with Apache Arrow, NumPy, and pandas for mixed pipelines

Common Use Cases

• Replacing pandas in ETL and data-cleaning pipelines that are I/O- or CPU-bound and would benefit from multi-core execution
• Processing datasets larger than available RAM using the lazy streaming engine instead of chunking manually
• Feature engineering pipelines for ML where query-plan optimization reduces redundant computation across chained transformations
• Interactive data analysis in notebooks where fast groupby/join/window operations matter for iteration speed

Under The Hood

Architecture: the repository is a Cargo workspace under crates/ (polars-core, polars-lazy, polars-io, polars-arrow, polars-expr, polars-compute, and more) implementing the actual columnar engine and lazy query optimizer in Rust, with a separate py-polars/ crate using PyO3 to expose that engine as the polars Python package — the Python layer is a thin binding over the same core used by the Rust, Node.js, and R distributions. Tech Stack: Rust for the engine (Cargo workspace, Cargo.toml/Cargo.lock at the root), Apache Arrow as the in-memory columnar format, PyO3/maturin for the Python extension build, with the Python package requiring Python 3.10+ and declaring polars version 1.43.0 in py-polars/pyproject.toml. Code Quality: extremely well-tested, with 455 Python test files under py-polars/tests/ covering the DataFrame/LazyFrame API surface plus extensive Rust unit tests in each crate; CI enforces rustfmt/clippy (per rustfmt.toml/clippy.toml) and dependency auditing via deny.toml. API Design: the expression API (pl.col("x").filter(...).sum()) is designed to be identical whether called eagerly or lazily, which minimizes the learning curve for pandas users switching over, though the strict typing and different null-handling semantics (Arrow-style nulls vs. NaN) require some relearning; the lazy/eager duality is powerful but does add a concept newcomers must learn before choosing the right one for a workload.