delta-spark

delta-spark is the PyPI package that provides the Python APIs for Delta Lake, an open-source storage framework that brings a Lakehouse architecture to your data lake. It runs on top of Apache Spark and adds ACID transactions, scalable metadata handling, and unified streaming plus batch processing to Parquet-based tables.

With delta-spark you configure a SparkSession for Delta and work with tables through the DeltaTable API — performing upserts (merge), updates, deletes, schema enforcement and evolution, and time-travel queries against historical versions. It is the Python entry point into the broader Delta Lake project, which also spans Scala, Flink, Trino, and a native kernel.

What You Get

• The DeltaTable Python API for creating, reading, and mutating Delta Lake tables via Spark
• ACID transactions over data-lake tables with schema enforcement and evolution
• Merge (upsert), update, and delete operations expressed in Python against large tables
• Time-travel queries to read previous table versions or timestamps
• Helpers to configure a Delta-enabled SparkSession and unify streaming and batch reads/writes

Common Use Cases

• Building reliable, ACID-compliant data-lake tables on top of Apache Spark
• Performing upserts and GDPR-style deletes on big-data tables with merge operations
• Auditing or reproducing analyses via time travel to earlier table versions

Under The Hood

Architecture — The Python package lives in the python/delta/ directory of the delta-io/delta monorepo. tables.py implements the user-facing DeltaTable class, which is a thin, well-typed Python facade over Delta Lake’s JVM implementation invoked through PySpark’s Py4J bridge — Python calls translate into operations on the Scala/JVM Delta core that manages the transaction log and Parquet data files. pip_utils.py wires the Delta Maven artifact into a SparkSession, a connect/ module supports Spark Connect, and exceptions/ surfaces Delta-specific errors. The heavy lifting (transaction log, ACID protocol) resides in the surrounding Scala modules (spark/, storage/, kernel/).

Tech Stack — The Python distribution targets Python 3.10+ and depends on pyspark (4.x) and importlib_metadata; version is derived from the repo’s version.sbt. The wider project is built with sbt across Scala, with connectors for Flink, Trino, Hive, and a native Rust/Java kernel.

Code Quality — The monorepo is a mature, Linux Foundation-governed project with 477+ contributors, extensive test suites (python/delta/tests/, integration tests, benchmarks), mypy typing (py.typed, mypy.ini), and CI-enforced version/tag verification. The Python layer is small and disciplined, delegating correctness-critical logic to the battle-tested JVM core.

API Design — For PySpark users the API is idiomatic: DeltaTable.forPath(...), .merge(...), .update(...), .delete(...), and .history() read naturally, and Delta tables plug into ordinary Spark read/write calls. The main friction is the required SparkSession configuration and understanding Spark itself, which raises the learning curve; comprehensive docs at docs.delta.io mitigate this.