dbt-spark

dbt-spark is the official dbt adapter that lets dbt Core compile and run SQL and Python models directly against Apache Spark. It implements Spark’s connection layer across three distinct paths — Thrift server (Hive/PyHive), ODBC (Databricks and generic ODBC endpoints), and an in-process PySpark session — and translates dbt’s materialization strategies (table, view, incremental, snapshot) into Spark-specific SQL and job submission calls.

The adapter handles authentication and connection pooling per method, retries and reconnects around dropped Thrift/ODBC sessions, and exposes Python model execution through job-cluster and all-purpose-cluster helpers for Databricks-backed Spark deployments. It ships as one of the plugin adapters in the dbt-adapters monorepo alongside dbt-bigquery, dbt-snowflake, and dbt-redshift, all built on the shared dbt-adapters base classes.

For teams running dbt against Spark Thrift servers, Databricks SQL warehouses, or a local PySpark session, dbt-spark is the layer that turns dbt’s platform-agnostic model definitions into connection-specific execution against the cluster.

What You Get

• Three connection methods - Thrift server (via PyHive/thrift_sasl), ODBC (pyodbc, for Databricks and generic endpoints), and an in-process PySpark session, selectable per profile.
• Materialization support - table, view, incremental (with merge/insert_overwrite/append strategies), snapshot, and clone macros compiled into Spark SQL.
• Python model execution - job-cluster and all-purpose-cluster helper classes that submit Python models as Databricks jobs and poll them to completion.
• Connection resilience - built-in retry and reconnect handling for dropped Thrift/ODBC sessions during long-running queries.
• Grants and schema introspection - macros for listing schemas/relations without caching overhead and applying grants after materialization.
• SASL/SSL auth handling - configurable auth (NONE/LDAP/CUSTOM/KERBEROS) with sensible username defaults to avoid common connection failures.

Common Use Cases

• dbt on a Spark Thrift server - Connect dbt to a self-managed Spark cluster’s Thrift server for SQL-based transformations.
• dbt on Databricks via ODBC - Run dbt models against Databricks SQL warehouses using the ODBC driver path.
• Local PySpark development - Use the session connection method to run and test dbt models against an in-process Spark session without a remote cluster.
• Python model pipelines - Submit dbt Python models as Spark jobs on Databricks job clusters or all-purpose clusters for ML/feature-engineering workloads.
• Incremental warehouse-style ELT on Spark - Build incremental and snapshot models against Spark-backed lakehouse tables as part of a larger dbt project.

Under The Hood

Architecture dbt-spark implements the standard dbt-adapters plugin contract: SparkAdapter (impl.py) extends SQLAdapter, SparkConnectionManager (connections.py) extends SQLConnectionManager, and SparkRelation/SparkColumn model Spark’s relation and type system. The connection layer branches into three independent code paths gated by optional imports (PyHive/thrift_sasl for Thrift, pyodbc for ODBC, pyspark for session mode), each producing a common cursor-like interface so the rest of the adapter is connection-method-agnostic. Python model execution is handled by a separate python_submissions.py module with JobClusterPythonJobHelper and AllPurposeClusterPythonJobHelper classes that submit jobs to the Databricks Jobs API and poll for completion — a clean separation between SQL execution (connections.py) and job-based Python execution (python_submissions.py). SQL materialization logic lives in Jinja macros under include/spark/macros/, following dbt’s standard adapter-plugin split between Python connection/type logic and SQL-generating macros.

Tech Stack Python 3.10+, built on dbt-adapters>=1.24 and dbt-common>=1.10 (the shared base classes dbt-core and all adapters depend on), with dbt-core pinned only for backwards-compatible installs, not as a functional dependency. Connection methods pull in PyHive[hive_pure_sasl] and thrift/thrift_sasl for the Thrift path, pyodbc for ODBC, and pyspark for the session path — all as optional extras (ODBC, PyHive, session, all) so users only install what their connection method needs. Packaged with Hatchling, tested with pytest (pytest-xdist for parallel runs) against a docker-compose environment running a local Spark Thrift server and Postgres-backed Hive Metastore.

Code Quality The repo has both a unit test suite (tests/unit/ — adapter, credentials, macros, telemetry, polling) and a functional test suite (tests/functional/adapter/) that exercises real materializations. Code is formatted with black and type-checked with mypy (both advertised via README badges and enforced via mypy.ini at the monorepo root). Optional imports are wrapped in try/except ImportError blocks with explicit None fallbacks and an inline comment noting the deliberate mypy trade-off, which is a pragmatic but visible compromise on strict typing. Error handling distinguishes connection-loss exceptions from query failures explicitly via a dedicated CONNECTION_LOST_EXCEPTIONS tuple built once at import time.

What Makes It Unique Unlike single-connection-method adapters, dbt-spark supports three genuinely different transport mechanisms (Thrift, ODBC, in-process PySpark session) behind one adapter interface, letting the same dbt project target a self-managed Spark cluster, a Databricks SQL warehouse, or a local Spark session by changing only the profile config. Its Python model support goes further than most SQL-only adapters by submitting models as actual Databricks jobs (job-cluster or all-purpose-cluster) rather than only supporting SQL, making it one of the more operationally flexible adapters in the dbt-adapters monorepo.