OpenLineage Integration Common

openlineage-integration-common is the shared Python library underpinning OpenLineage’s integration ecosystem, published from the integration/common subpath of the OpenLineage monorepo. It supplies the core building blocks — Dataset and Source abstractions, DbTableSchema/DbColumn models, and redaction-aware facet helpers — that higher-level integrations for dbt, BigQuery, Great Expectations, Redshift, and Snowflake reuse instead of reimplementing lineage plumbing themselves.

The package’s largest surface area is its dbt provider, which parses dbt artifacts (manifest.json, run_results.json, catalog.json) and dbt Cloud/local run output into OpenLineage RunEvents, including a structured-logs extraction path for newer dbt versions. Around that sit thinner adapters for BigQuery job/error facets, Great Expectations validation-result facets, and Redshift/Snowflake dataset metadata, all built on OpenLineage’s client-side facet and event models.

What You Get

• Dataset/Source model classes with redaction-aware serialization for building OpenLineage facets consistently across integrations
• A production-grade dbt artifact processor that turns manifest.json/run_results.json/catalog.json (or structured dbt Cloud logs) into OpenLineage RunEvents
• Ready-made provider adapters for BigQuery, Great Expectations, Redshift Data API, and Snowflake dataset/job facets
• DbTableSchema/DbColumn models plus SQL parsing utilities shared with openlineage_sql for column-level lineage

Common Use Cases

• Extracting lineage from dbt Core or dbt Cloud runs without dbt’s own OpenLineage integration built in
• Emitting BigQuery job-level lineage facets (query text, error info, statistics) from a custom Python integration
• Attaching Great Expectations validation results as OpenLineage data-quality facets on existing pipelines
• Building a new first-party OpenLineage integration for a Python-based tool without re-deriving dataset/schema models from scratch

Under The Hood

Architecture The package centers on a handful of core abstractions — Dataset, Source, and Field in dataset.py, and DbTableSchema/DbColumn in models.py — that wrap OpenLineage’s client-side event and facet models (openlineage.client.event_v2, openlineage.client.facet_v2) with a redaction-aware convenience layer via RedactMixin. Provider-specific adapters (bigquery.py, snowflake.py, redshift_data.py, and the dbt/ and great_expectations/ subpackages) sit on top of this shared layer and translate each tool’s native output into those same OpenLineage facets, so adding a new integration means writing one more provider module rather than a parallel copy of the dataset/schema plumbing. The dbt provider is by far the largest surface, with DbtLocalArtifactProcessor parsing static manifest/run_results/catalog artifacts and a separate structured_logs.py path consuming dbt’s newer structured JSON log stream — a dual-mode design that reflects dbt’s own evolving instrumentation surface rather than a single clean data flow. Optional heavyweight dependencies (google-cloud-bigquery, great_expectations, boto3) are lazy-imported behind small factory functions like get_bq_client(), keeping the package importable without every provider’s dependencies installed.

Tech Stack Built for Python 3.10+ using modern union-type syntax (str | None) throughout, packaged with uv’s build backend and namespace packages, and linted with ruff plus mypy (namespace_packages, explicit_package_bases). Runtime dependencies are minimal — attrs for facet classes, pyyaml, and the sibling openlineage-python/openlineage_sql packages resolved via uv workspace sources — while provider-specific dependencies (google-cloud-bigquery, great_expectations, sqlalchemy, boto3) are pushed into pip extras (bigquery, great_expectations, redshift) so consumers only install what they use. CI runs through the monorepo’s CircleCI pipeline, and a tox matrix exercises the dbt provider against multiple dbt-core versions to catch dbt API drift.

Code Quality Test coverage is organized per provider (tests/bigquery, tests/dbt, tests/great_expectations, tests/redshift_data, tests/sql, plus top-level dataset/models/snowflake/utils test files) using pytest with fixtures and mocking, and a dedicated tox environment runs the dbt tests against more than one dbt-core version. Error handling favors explicit typed exceptions over silent failures — for example UnsupportedDbtCommand is raised from the dbt processor rather than swallowed — and naming is consistent with the rest of the OpenLineage Python codebase (RunFacet/DatasetFacet suffixes, Provider-suffixed classes). Static analysis is enforced via ruff (with import-sorting) and mypy in namespace-package mode, and comment density is moderate — module-level license headers are universal but inline documentation is sparser inside the larger dbt processor files.

API Design For an internal shared library rather than a standalone product, the developer experience is solid: provider adapters follow a consistent constructor-then-build pattern, and Source/Dataset expose ergonomic factory methods so integration authors don’t hand-assemble facet dictionaries. Nothing here is architecturally novel relative to other lineage/observability client libraries — the value is consolidation, not a new pattern — but the extras-based optional-dependency split and the dual manifest-vs-structured-logs dbt extraction path are pragmatic, well-executed answers to supporting many dbt/warehouse versions from one package.