Airbyte Python CDK

The Airbyte Python CDK (Connector Development Kit) is a framework for building Airbyte API source and destination connectors. It supplies base classes and helpers for talking to HTTP APIs (REST, GraphQL, and more) or generic Python sources, handling pagination, incremental sync, state management, and the Airbyte protocol so connector authors focus on API specifics.

It also includes a low-code path where connectors are described entirely through a declarative YAML manifest, plus a concurrent source engine for high-throughput reads and support for file-based sources like S3, GCS, and Azure.

What You Get

• Base classes for HTTP and generic Python sources with built-in pagination, retries, and incremental sync
• A low-code declarative CDK that constructs connectors from a YAML manifest
• A concurrent source engine for reading streams in parallel across partitions
• A file-based CDK for S3, GCS, Azure, and similar object stores
• Airbyte protocol models and a CLI runtime for executing connectors

Common Use Cases

• Building a custom Airbyte source connector for a third-party REST or GraphQL API
• Authoring a connector declaratively via YAML with the low-code CDK
• Scaling high-volume syncs with concurrent, partition-parallel stream reads

Under The Hood

Architecture

The CDK is organized under airbyte_cdk into layered subsystems: sources define the AbstractSource/Source base classes and stream abstractions; sources/declarative implements the low-code engine that parses a YAML manifest into models (retrievers, requesters, paginators, record selectors) and executes them; sources/concurrent_source provides a partition-parallel read engine; sources/file_based handles object-store parsing; and destinations plus models expose the Airbyte protocol. A CLI under airbyte_cdk/cli runs the source-declarative-manifest execution logic.

Tech Stack

Python packaged with Poetry (pyproject.toml, poetry.lock), targeting modern Python versions. It depends on Pydantic for protocol and manifest models, requests/backoff for HTTP with retries, and PyYAML/jsonschema for manifest parsing and validation. Tooling includes Ruff for linting, mypy for typing, and pytest for the extensive unit_tests suite; a Dockerfile packages the SDM runtime.

Code Quality

This is a large, actively released codebase (hundreds of modules, frequent releases) with a comprehensive unit_tests directory covering sources, declarative components, and error handling. It enforces static typing via mypy.ini and style via ruff.toml, and the declarative layer is heavily model-validated with Pydantic, giving strong guarantees despite the broad surface area.

API Design

The framework offers two very different ergonomics: an imperative Python API for full-control connectors and a declarative YAML manifest for the common case, which the project recommends starting from the Connector Builder UI. The imperative path has a steep learning curve given the many abstractions, but extensive docs at docs.airbyte.com and the low-code option significantly lower the barrier for typical API connectors.