Ibis

Ibis is the portable Python dataframe library. It gives you a single, lazy dataframe expression API that runs against more than 20 execution backends — from fast local engines like DuckDB and Polars to cloud warehouses such as Snowflake, BigQuery, and Databricks, and distributed systems like Spark and Trino.

Instead of learning a new API per engine or hand-writing SQL for each database, you write Ibis expressions once and switch backends by changing a single line. Ibis compiles those expressions to each backend’s native SQL or dataframe operations, letting you iterate locally on small data and deploy the same code remotely against production-scale systems.

What You Get

• A single lazy dataframe expression API portable across 20+ backends
• Local execution via DuckDB by default with no separate database to run
• One-line backend switching between local engines and cloud warehouses
• The ability to compose Python dataframe code and raw SQL together
• An interactive mode for iterative, notebook-style data exploration

Common Use Cases

• Writing analytics once and running it on DuckDB locally, then Snowflake or BigQuery in production
• Replacing per-database SQL with a unified, type-checked Python API
• Exploring and transforming large warehouse tables without pulling data into memory
• Building portable data pipelines that stay backend-agnostic

Under The Hood

Architecture — Ibis separates expression construction from execution. User-facing tables and columns are built as immutable expression trees (ibis/expr) over an internal typed operation graph (ibis/common), validated against schemas expressed through ibis/formats. Each backend under ibis/backends implements a compiler that lowers that shared operation graph into the engine’s native SQL or dataframe calls, so the same expression can target DuckDB, Postgres, Snowflake, Spark, and many others. Selectors (ibis/selectors.py), an interactive display layer (ibis/interactive.py), and format converters bridge to pandas, PyArrow, and Polars.

Tech Stack — Pure Python packaged with a modern pyproject/uv workflow. It relies on Apache Arrow (PyArrow) for zero-copy data interchange, a SQL-generation layer for backend compilers, and optional per-backend driver dependencies installed via extras (for example ibis-framework[duckdb]). DuckDB serves as the default local engine.

Code Quality — Ibis is a large, mature project with strong community engagement, very active development, and extensive CI covering its many backends (separate main and backend test workflows). It maintains high codecov coverage, a broad ibis/tests suite plus per-backend tests, and clear internal boundaries between the expression API, operation graph, and backend compilers.

API Design — The API is deliberately dataframe-like and chainable, familiar to pandas users while remaining lazy and typed. Portability is a first-class design goal: connecting to a different backend is a single call, and the same expressions compile everywhere. Interactive mode, rich documentation on ibis-project.org, and bundled example datasets make onboarding smooth despite the breadth of supported systems.