deltalake

deltalake is the official Python binding for delta-rs, a from-scratch Rust implementation of the Delta Lake open table format. Rather than embedding a JVM and Spark runtime, it exposes the format’s transaction log, schema enforcement, time travel, and ACID guarantees directly to Python through a native extension module built with PyO3 and maturin, backed by Apache Arrow via arro3-core.

The package targets the same audience that already reaches for pandas, Polars, or DuckDB: DeltaTable(path) and write_deltalake() read and write tables in a couple of lines, while lower-level primitives like Transaction, WriterProperties, and PyMergeBuilder are available for teams building custom ingestion or merge/upsert pipelines. Optional Cargo-feature-gated backends add S3, Azure, GCS, HDFS, and lakeFS storage support, plus Unity Catalog and Glue catalog integrations, and OpenTelemetry tracing is built in for observing operation timings in production.

What You Get

• A DeltaTable class for opening, querying, time-traveling, and inspecting Delta tables by version or timestamp
• write_deltalake() and convert_to_deltalake() for writing new tables or converting existing Parquet directories into Delta format
• Merge/upsert support via PyMergeBuilder for building custom ingestion and CDC pipelines
• Pluggable storage backends (S3, Azure Blob, GCS, HDFS, lakeFS) and catalog integrations (Unity Catalog, AWS Glue) enabled as optional Cargo features
• Built-in OpenTelemetry tracing (init_tracing) for inspecting operation-level performance in production
• Zero-copy interop with Apache Arrow via arro3-core, with optional pandas and pyarrow extras

Common Use Cases

• Reading and writing versioned, ACID-compliant tables from a data pipeline without standing up Spark
• Converting an existing directory of Parquet files into a Delta table for schema enforcement and time travel
• Building custom merge/upsert (CDC) ingestion jobs against S3, Azure, or GCS-backed lakes
• Querying Delta tables directly from pandas, Polars, or DataFusion in local analytics and notebook workflows
• Interoperating with Delta tables written by Spark/Databricks jobs from a lightweight Python-only service

Under The Hood

Architecture The package is organized as layered Rust crates under a single Cargo workspace, wrapped by a thin Python surface. crates/core implements the Delta protocol on top of the external delta_kernel (buoyant_kernel) dependency, with logstore, operations, protocol, and table modules handling transaction-log reads/writes, writer/merge/optimize/vacuum operations, and table state respectively. Storage- and catalog-specific crates (aws, azure, gcp, hdfs, opendal, lakefs, catalog-glue, catalog-unity) plug into core as optional Cargo features, and a deltalake facade crate re-exports the combination behind one dependency. The python/ subproject builds a PyO3 extension module (deltalake._internal) from Rust sources in python/src/ (lib.rs, writer.rs, query.rs, schema.rs, merge.rs, datafusion.rs), and the Python package (table.py, writer/, query.py, schema.py, transaction.py) is a thin typed proxy over that native module, converting to/from Arrow via arro3-core. Because the kernel and core protocol layer sit beneath every storage backend and every Python-facing class, a protocol change cascades through the entire stack.

Tech Stack Rust (edition 2024) across a multi-crate Cargo workspace, built into a Python wheel via maturin/PyO3. Table-format logic is delegated to delta_kernel (the buoyant_kernel fork), columnar data uses arrow-rs, and storage IO is handled by the object_store crate family with per-backend crates for S3, Azure, GCS, HDFS, and lakeFS; DataFusion is available as an optional query-engine feature. The Python package’s only hard runtime dependency is arro3-core (a Rust-backed Arrow implementation), with pandas and pyarrow offered as opt-in extras, and dependencies/dev tooling are managed with uv dependency groups.

Code Quality The Python test suite spans 51 files and roughly 19,000 lines under python/tests/, segmented by pytest markers for optional backends and integrations (s3, azure, lakefs, pyspark, datafusion, pyarrow, benchmark), with testcontainers-backed MinIO used for storage-integration tests. pyproject.toml enables strict mypy (disallow_untyped_defs, disallow_any_generics, disallow_subclassing_any) and ruff with pycodestyle/pyflakes/isort/annotation rule sets, and the package ships a py.typed marker. CI is split across dedicated GitHub Actions workflows for build, python build, integration tests, benchmarks, docs, and typo/DCO checks, giving each concern its own gate rather than one monolithic pipeline.

What Makes It Unique Most mature Delta Lake tooling assumes a JVM and Spark; deltalake instead reimplements the protocol natively in Rust and exposes it to Python as an ordinary importable package, so a pandas or Polars user gets ACID table semantics, schema enforcement, and time travel with an ordinary pip install and no cluster. The public API deliberately mirrors familiar dataframe-library ergonomics (DeltaTable(path), write_deltalake(...), .to_pandas()) for the common path while still surfacing advanced primitives (Transaction, CommitProperties, WriterProperties, BloomFilterProperties) for teams that need fine-grained control over commits and file layout, and it bundles OpenTelemetry tracing directly into the core rather than leaving observability to the caller.