dbt-core

dbt-core (distributed on PyPI as dbt-core) is the open-source engine behind dbt, the tool that popularized “transformation as software engineering” for the analytics workflow. Instead of writing ad-hoc SQL scripts or stored procedures, analysts and data engineers write modular SELECT statements as version-controlled “models,” and dbt compiles, orders, and executes them against the warehouse as tables or views, resolving dependencies automatically via a ref()-based DAG.

The framework owns the full lifecycle of a transformation project: parsing and compiling Jinja-templated SQL, building a dependency graph, selecting a subset of nodes to run, executing them in the correct order against a configured warehouse adapter, testing the results, and generating documentation and lineage artifacts. It ships as a CLI (dbt run, dbt test, dbt build, dbt docs generate) built on top of a pluggable adapter architecture (dbt-adapters) so the same project logic works across Snowflake, BigQuery, Redshift, Postgres, Databricks, and dozens of community-maintained adapters.

dbt-core is maintained by dbt Labs and has become the de facto standard for the “transform” step of modern ELT stacks, with a large ecosystem of packages (via dbt deps), a hosted commercial product (dbt Cloud), and a very large practitioner community. Note that as of this cataloging, the GitHub repository’s default branch has moved on to an in-development Rust rewrite (dbt Core v2.0 / the Fusion engine); the dbt-core PyPI package documented here still ships the mature, actively maintained Python v1 implementation from the 1.latest branch.

What You Get

• A CLI (dbt run, dbt test, dbt build, dbt docs generate, dbt compile) that drives the entire transformation lifecycle from a single dbt_project.yml
• A ref()/source()-based dependency graph that automatically orders model execution and lets you select subsets of the DAG to run (--select, --exclude)
• Built-in and custom data tests (uniqueness, not-null, relationships, accepted values, or arbitrary SQL assertions) that run against real warehouse data
• Auto-generated documentation and column-level lineage graphs served as a static docs site
• A pluggable adapter interface (dbt-adapters) so the same project works across Snowflake, BigQuery, Redshift, Postgres, Databricks, Spark, and many community adapters
• Package management via dbt deps, letting teams reuse shared model libraries and macros published as dbt packages

Common Use Cases

• Transforming raw data landed by an EL tool (Fivetran, Airbyte, Stitch) into clean, tested, business-ready warehouse tables
• Building a dimensional/analytics layer (marts, dimensions, facts) on top of raw source tables with materialized incremental models
• Enforcing data quality with automated tests that fail CI when a model produces nulls, duplicates, or referential integrity violations
• Generating and publishing living data documentation and column-level lineage for a warehouse so downstream consumers can self-serve
• Standing up a reusable macro/package library so multiple teams share transformation logic instead of duplicating SQL

Under The Hood

Architecture dbt-core is organized as a layered pipeline under core/dbt/: parser/ reads .sql and .yml files from a project into an in-memory Manifest, context/ resolves Jinja rendering (including the ref()/source() functions that establish the dependency graph), graph/ builds and topologically selects a networkx-backed DAG of nodes via selector_methods.py and selector_spec.py, and task/ (with runners/) executes the selected nodes against the target warehouse through the dbt-adapters plugin interface, materializing each node according to its declared strategy (view, table, incremental, ephemeral). The cli/ package wraps this pipeline behind a Click-based command surface (cli/main.py), while plugins/ implements an opt-in bundled-plugin manager (e.g. dbt-state) gated by explicit CLI flags or project config. Warehouse-specific execution is deliberately kept out of dbt-core itself and delegated to separately versioned adapter packages, so the core stays warehouse-agnostic while adapters evolve independently.

Tech Stack The project is pure Python (3.10+), built with hatchling and versioned from dbt/__version__.py. Core dependencies include Jinja2 for templating, click for the CLI, networkx for graph operations, jsonschema and mashumaro/pydantic for schema validation and (de)serialization, agate for lightweight in-memory tabular results, protobuf for artifact schemas, and dbt Labs’ own dbt-common, dbt-adapters, and dbt-protos packages that were split out of the core repo to support the multi-adapter ecosystem. Dependency pins are unusually disciplined, with the pyproject.toml documenting exactly why each package is pinned to patch, minor, or major-version ranges.

Code Quality The project has an extensive test suite (367+ test files) split into tests/unit and tests/functional, run via pytest with pythonpath = core. Pre-commit enforces black formatting, mypy static typing (scoped to core/dbt/), and flake8 linting, and GitHub Actions CI runs on every push. Error handling is centralized in exceptions.py with a typed, code-numbered exception hierarchy (e.g. ContractBreakingChangeError) rather than bare exceptions, and type hints are used throughout the core modules.

What Makes It Unique dbt-core’s distinguishing choice is treating the SQL transformation layer as a first-class software artifact: Jinja-templated models compiled through a dependency graph, tested with declarative assertions, and documented automatically, all decoupled from the underlying warehouse via a pluggable adapter contract. That combination — DAG-aware SQL compilation plus a stable adapter interface — is what let a single open-source project become the shared transformation layer across dozens of otherwise incompatible cloud warehouses, and it’s the reason the current in-development Rust rewrite (Fusion) is being built to preserve that same model/test/docs contract while replacing the execution engine underneath it.