ConnectorX

ConnectorX loads data from SQL databases into Python and Rust DataFrames in the fastest and most memory-efficient way possible. A single cx.read_sql(...) call streams query results from sources like PostgreSQL, MySQL, SQLite, SQL Server, Oracle, and Redshift directly into pandas, Arrow, Polars, Modin, or Dask.

Written in Rust with a C++ acceleration layer, ConnectorX avoids the overhead of traditional Python database clients by writing results straight into destination memory. It supports client-side parallelism through partitioned reads and offers experimental federated queries that join tables across multiple databases in one statement.

What You Get

• One-line read_sql loading from many SQL databases into DataFrames
• Destinations for pandas, PyArrow, Polars, Modin, and Dask
• Parallel partitioned reads by evenly splitting a numeric column across threads
• A high-performance Rust core with a C++ acceleration layer
• Experimental federated queries that join tables across multiple databases

Common Use Cases

• Loading large SQL query results into pandas or Polars for analysis
• Speeding up ETL extraction steps that bottleneck on database reads
• Parallelizing extraction of big tables via partitioned reads
• Joining data across separate databases with a single federated query

Under The Hood

Architecture - The repository is a Cargo/Rust workspace split into connectorx (the core engine), connectorx-cpp (a C++ acceleration layer), and connectorx-python (the PyO3 bindings shipped as the connectorx wheel). The core defines source readers per database protocol and destination writers per DataFrame format, then streams data source-to-destination without materializing intermediate Python objects. Partitioned reads split a numeric column into ranges and assign one thread per partition. Federated queries are planned and pushed down where joins share a data source.

Tech Stack - Primarily Rust, with C++ for hot paths and a thin Python layer via PyO3. It leans on Apache Arrow as an intermediate columnar representation and integrates with pandas, Polars, Modin, and Dask on the destination side. Builds are orchestrated with Cargo and a Justfile.

Code Quality - The project is a mature, actively developed workspace with 80+ contributors, CI, dedicated benchmark suites (benchmarks/, Benchmark.md), and clear source/destination modularization. Performance claims are backed by published, reproducible benchmarks.

API Design - The Python surface is intentionally minimal — most work is a single cx.read_sql(conn, query, ...) call — with optional partition_on/partition_num and return_type parameters. This keeps the common path trivial while exposing parallelism and destination choice for advanced users.