dbt-bigquery

dbt-bigquery is the official dbt adapter that lets dbt Core compile and run SQL and Python models directly against Google BigQuery. It implements BigQuery’s connection and authentication layer, translates dbt’s materialization strategies (table, view, incremental, materialized view) into BigQuery-specific DDL/DML, and exposes BigQuery-specific configuration such as partitioning, clustering, labels, and reservations.

Beyond standard SQL models, dbt-bigquery adds first-class support for BigQuery’s platform-specific capabilities: Python models executed via Dataproc or BigQuery DataFrames (BigFrames), Dataproc Serverless batch submission, multiple authentication methods (OAuth, service account, service account JSON, and Workload Identity Federation), and catalog integrations for BigLake/Iceberg metastore tables. It is maintained by dbt Labs as part of the dbt-adapters monorepo, alongside sibling adapters for Snowflake, Redshift, Postgres, Athena, and Spark.

What You Get

• A BigQuery connection manager that authenticates via OAuth, service account keys, service account JSON, or Workload Identity Federation
• BigQuery-native materializations for tables, views, incremental models, and materialized views, including clustering and partitioning config
• Python model execution on BigQuery through Dataproc clusters, Dataproc Serverless batches, or BigQuery DataFrames (BigFrames)
• Job retry and polling logic tuned for BigQuery’s asynchronous query execution and rate-limit behavior
• Catalog integration support for BigLake and Iceberg metastore tables alongside standard BigQuery datasets
• Fine-grained BigQuery configuration surfaces: labels, access entries, table expiration, reservations, and query priority

Common Use Cases

• Running dbt Core transformation pipelines directly against a BigQuery data warehouse
• Building incremental and partitioned BigQuery models for large, frequently-updated datasets
• Executing Python-based dbt models on BigQuery via Dataproc or BigFrames for workloads that don’t fit in SQL
• Managing BigQuery-specific access controls, labels, and reservations from within dbt model configuration
• Connecting BigQuery to the broader dbt Core / dbt Cloud ecosystem for testing, documentation, and orchestration

Under The Hood

Architecture dbt-bigquery plugs into dbt-core’s adapter interface, implemented primarily in impl.py (BigQueryAdapter, ~1,500 lines), connections.py (BigQueryConnectionManager), credentials.py, and relation.py. The connection manager owns a google.cloud.bigquery.Client per thread, wraps job submission in retry/backoff logic that short-circuits on terminal job failure states rather than polling for the full timeout window, and exposes available-decorated methods (e.g. nest_column_data_types, get_table_ref_from_relation) that dbt’s Jinja macros call at compile time. Python model execution is split into a separate python_submissions.py module implementing PythonJobHelper subclasses for Dataproc cluster jobs, Dataproc Serverless batches, and BigFrames, decoupling SQL model execution from the heavier, lazily-imported Python/ML dependencies. Relation and catalog handling (partitioning, clustering, BigLake/Iceberg catalogs) is factored into relation.py and a relation_configs package, keeping BigQuery-specific DDL generation separate from the connection layer. Tech Stack The adapter is Python 3.10+ and depends on dbt-common and dbt-adapters (the shared base-adapter framework also used by the Snowflake, Redshift, and Postgres adapters in this monorepo), plus the official google-cloud-bigquery[pandas], google-cloud-storage, google-cloud-dataproc, google-api-core, and google-auth SDKs for talking to BigQuery, GCS, and Dataproc. google-cloud-aiplatform and nbformat support the BigFrames/notebook-based Python model path. Packaging uses Hatch/Hatchling with a dynamic version sourced from __version__.py, and the monorepo wires local sibling packages (dbt-adapters, dbt-tests-adapter) via editable installs for development. Code Quality The repo has an extensive unit test suite (100+ files under tests/unit) plus a separate tests/functional suite built on dbt-tests-adapter’s shared adapter test contract, run via pytest with pytest-xdist for parallelism and pytest-mock/freezegun for mocking time and API calls. Type checking is enforced via a monorepo-wide mypy.ini with namespace_packages and explicit_package_bases configured to resolve imports across sibling adapter packages. Formatting and linting run through pre-commit with black (line-length 99, multi-version target) plus a custom dbt-core-in-adapters-check hook, and CI (hatch run ci:unit-tests / integration-tests) runs the same suites with ddtrace instrumentation and pytest-split sharding. What Makes It Unique What distinguishes dbt-bigquery from most dbt adapters is its handling of BigQuery’s asynchronous job model and its Python-model story: retry logic distinguishes terminal job failures (short-circuit immediately) from transient rate-limit errors (back off with a bounded deadline), avoiding both hung polling loops and premature failures during bursts of concurrent jobs. Its Python model support spans three execution backends (Dataproc clusters, Dataproc Serverless, and BigFrames) with heavy ML dependencies lazy-imported so a plain dbt parse stays fast, and its catalog integration layer bridges standard BigQuery datasets with BigLake/Iceberg metastore-backed tables, letting a single dbt project mix conventional and lakehouse-style storage.