PyAirbyte

PyAirbyte brings the power of Airbyte’s connector ecosystem to Python developers. It lets you extract data from hundreds of Airbyte source connectors — SaaS APIs, databases, and files — directly from Python, then cache or load that data into DuckDB, Snowflake, BigQuery, Postgres, or pandas DataFrames without deploying the full Airbyte platform.

Designed for data engineers, analysts, and notebook users, PyAirbyte handles connector execution, incremental sync state, record processing, and caching so you can build lightweight ELT pipelines and ad-hoc data workflows entirely in code.

What You Get

• Access to hundreds of Airbyte source connectors from a single get_source() call
• Built-in caches backed by DuckDB, Snowflake, BigQuery, and Postgres
• Incremental sync with automatic state tracking between runs
• Direct conversion of synced streams into pandas DataFrames for analysis
• A CLI and MCP integration for running connectors outside pure-Python contexts

Common Use Cases

• Pulling data from a SaaS API into a notebook for exploratory analysis
• Building lightweight ELT pipelines that load into DuckDB or a cloud warehouse
• Prototyping a data integration before deploying it on the full Airbyte platform
• Feeding third-party source data into pandas-based data science workflows

Under The Hood

Architecture — The airbyte package is organized around sources, caches, and writers. get_source() resolves a connector from the Airbyte registry (registry.py) and runs it via executors in _executors, streaming Airbyte-protocol messages through _message_iterators.py and record processors in _processors. Synced records land in a cache (caches/) — DuckDB by default — and can be read out as datasets or pandas DataFrames. Separate modules handle cloud integration, secrets, progress reporting, and an MCP server.

Tech Stack — Python built on the Airbyte CDK and airbyte-protocol models, with pydantic v2 for typing, DuckDB and SQLAlchemy engines for local caching, and first-class connectors for Snowflake, BigQuery, and Postgres. It uses orjson and pyarrow for fast record handling and cyclopts for its CLI. Dependencies are managed with uv.

Code Quality — The repository ships an extensive tests/ suite, typed code with a py.typed marker, dedicated exceptions.py error handling, and pyrefly type checking. Connectors, caches, and processors are cleanly separated, and the project maintains examples/ and docs/ directories.

API Design — The public API is intentionally small: get_source(), select_streams(), read(), and to_pandas() cover the common path, mirroring how analysts think about data movement. Rich progress output and sensible DuckDB defaults keep the getting-started boilerplate minimal.