Databricks SQL Connector for Python

The Databricks SQL Connector is Databricks’ official Python client for querying Databricks clusters and SQL warehouses, implementing the Python DB API 2.0 specification (PEP 249) so it drops into existing DB-API-based tooling (SQLAlchemy, pandas.read_sql, generic ETL frameworks) without a custom integration. It communicates over Thrift with no ODBC/JDBC driver dependency, uses Arrow as its data-exchange format for high-throughput fetches (fetchmany_arrow), and supports Cloud Fetch for parallel result-set retrieval directly from cloud storage, along with OAuth, PAT, and Kerberos/Negotiate authentication and HTTP/HTTPS proxy support.

What You Get

• A PEP 249 (DB API 2.0) compliant connect()/cursor() interface that works with existing SQL tooling expecting a standard Python DB-API client
• Arrow-native result fetching (fetchmany_arrow) wrapped in an ArrowQueue for efficient batch retrieval of large query results
• Cloud Fetch support for parallel, direct-from-cloud-storage retrieval of large result sets, bypassing the Thrift channel for bulk data
• Multiple authentication methods: Personal Access Tokens, OAuth (including Databricks-native OAuth), and Kerberos/Negotiate
• Built-in HTTP/HTTPS proxy support with basic and Kerberos/Negotiate proxy authentication

Common Use Cases

• Running SQL queries against Databricks SQL warehouses from Python ETL scripts, orchestrators (Airflow), or notebooks outside the Databricks runtime
• Loading Databricks query results directly into pandas or Arrow-based dataframes for downstream analysis
• Building BI/reporting tools or internal dashboards that query Databricks via a standard DB-API interface (often through SQLAlchemy)
• Connecting from environments where installing a native ODBC/JDBC driver isn’t practical (containers, serverless functions, restricted environments)

Under The Hood

Architecture: the connector is organized under src/databricks/sql/, with dedicated subpackages for auth (PAT/OAuth/Kerberos credential providers), backend (Thrift protocol implementation and connection handling), cloudfetch (parallel direct-from-storage result retrieval), thrift_api (generated Thrift service stubs), parameters (query parameterization), and telemetry; the top-level Connection/Cursor classes implement PEP 249 while delegating transport and auth concerns to these subpackages. Tech Stack: Python 3.8+ managed via Poetry (pyproject.toml/poetry.lock), depending on thrift for the wire protocol (pinned below 0.23.0 due to a documented packaging regression noted directly in the pyproject comments) and optionally pyarrow for Arrow-native fetch methods. Code Quality: 64 test files split between tests/unit and tests/e2e, plus a DBR LTS Install CI workflow that installs the built wheel on real Databricks Runtime LTS clusters specifically to catch dependency-pinning regressions before release — evidence of an unusually disciplined release-validation process for a client library. API Design: because it implements the standard DB API 2.0 interface, existing DB-API-aware code (SQLAlchemy dialects, pandas.read_sql, generic SQL runners) works against it with minimal changes, keeping day-to-day usage boilerplate low even though the underlying connection/auth configuration for Databricks-specific concepts (SQL warehouses, cluster HTTP paths, OAuth scopes) requires reading Databricks-specific docs.