Databricks SDK for Python

The Databricks SDK for Python is the official client library for programmatically working with the Databricks Data Intelligence Platform. It exposes the full breadth of Databricks REST APIs — workspace and account level — as typed Python clients, covering jobs, clusters, notebooks, repos, SQL warehouses, Unity Catalog, model serving, and more.

A single WorkspaceClient (or AccountClient) gives you dataclass-typed request and response models, unified authentication across Databricks-native tokens, OAuth, and cloud provider credentials (Azure, AWS, GCP), and helpers for long-running operations and pagination. Generated from the same API specifications as the other Databricks SDKs, it stays current with the platform and is the standard way to automate Databricks from Python.

What You Get

• WorkspaceClient and AccountClient entry points covering the full Databricks REST API surface
• Typed dataclass request and response models generated from the official API specs
• Unified authentication across PAT tokens, OAuth, and Azure/AWS/GCP cloud credentials
• Built-in pagination and long-running-operation waiters for jobs, clusters, and deployments
• Consistent behavior with the Databricks SDKs for Java and Go, kept in sync with platform releases

Common Use Cases

• Automating job, cluster, and notebook lifecycle management from Python scripts and CI
• Administering Unity Catalog objects, permissions, and workspace configuration programmatically
• Deploying and querying model-serving endpoints and SQL warehouses

Under The Hood

Architecture The package lives under databricks/sdk, with a large generated service/ subpackage where each module (jobs, clusters, catalog.py, serving, billing.py, and dozens more) contains the typed dataclasses and client methods for one API area. A _base_client.py centralizes HTTP transport, retries, and error handling; config.py and the credentials machinery resolve authentication; and WorkspaceClient/AccountClient aggregate the service clients behind a single object. Most of service/ is code-generated from the shared Databricks API spec, which keeps the three official SDKs consistent.

Tech Stack Python 3.10+ built with setuptools. Runtime dependencies are deliberately lean: requests for HTTP, google-auth for GCP identity, protobuf, and urllib3 (with pinned exclusions for known CVEs). Dev tooling uses pytest with coverage and xdist, requests-mock and pyfakefs for testing, and ruff for linting; uv.lock pins the dev environment.

Code Quality The repo carries an extensive tests directory, codecov reporting, CVE-aware dependency pinning, a SECURITY.md, and a DCO contribution process. Because service modules are generated, naming and structure are uniform across the whole API surface, and the read-only-mirror release model ensures every published version matches an internal reviewed build.

API Design The two-client entry model plus per-service namespacing makes discovery straightforward — w.clusters.create(...), w.jobs.run_now(...) — and dataclass models give editor autocompletion and type checking. Waiters and automatic pagination remove common boilerplate. The main learning curve is the sheer breadth of services and Databricks’ authentication options rather than any awkwardness in the API itself; ReadTheDocs documentation and an examples directory ease onboarding.