redshift_connector

redshift_connector is AWS’s official pure-Python database driver for Amazon Redshift, implementing the Python Database API Specification v2.0 (PEP 249). It provides native IAM and identity-provider authentication, Redshift-specific data type handling, and first-class integration with pandas and NumPy so query results can flow directly into DataFrames or arrays.

Because it ships from AWS itself and is maintained alongside the Redshift service, it stays current with cluster-side features like IAM role chaining, Redshift Serverless, and native IdP federation, making it the reference client for teams building data pipelines, BI tooling, or ETL jobs against Redshift from Python.

What You Get

• A DB-API 2.0 compliant connect()/Cursor interface for executing SQL against Redshift clusters
• Built-in IAM authentication and identity-provider (IdP) federation, including AWS profile and access-key based connections
• Direct fetch_dataframe() and fetch_numpy_array() helpers for pandas and NumPy integration
• Support for Redshift-specific data types and bulk write_dataframe() inserts
• Configurable paramstyle (qmark, numeric, named, format, pyformat) at module or cursor level

Common Use Cases

• Connecting Python ETL and data pipeline jobs to Amazon Redshift clusters
• Querying Redshift from data science notebooks and loading results into pandas DataFrames
• Authenticating to Redshift using IAM roles or identity providers instead of static credentials
• Powering Redshift integrations in tools like Apache Airflow, Querybook, and SQLAlchemy-Redshift

Under The Hood

Architecture - The driver centers on core.py, which implements the Redshift/PostgreSQL wire protocol directly (message framing, type OIDs, result parsing), with connection.py-style connect() building a Connection object that owns a Cursor factory in cursor.py. Authentication is split into an auth/ and plugin/ subpackage that pluggably resolves IAM credentials, browser-based IdP redirects, and native SAML/JWT plugins before handing a token to iam_helper.py and idp_auth_helper.py, which negotiate temporary Redshift credentials via the AWS API prior to opening the actual database socket.

Tech Stack - Pure Python 3.8+ with no C extensions; core runtime dependencies are boto3/botocore for AWS API calls, scramp for SCRAM authentication, packaging, and beautifulsoup4/requests for browser-based IdP flows, with optional pandas and numpy extras enabling fetch_dataframe()/fetch_numpy_array(). Packaging uses classic setup.py/setup.cfg rather than a pyproject.toml-based build.

Code Quality - The test/ directory contains roughly 75 test files split into unit and integration suites, run via pytest with coverage reporting (--cov=redshift_connector) wired into custom setup.py commands; redshift_connector/ totals over 10,000 lines with type hints (py.typed marker present) and dedicated modules per concern (pg_types.py, error.py, objects.py), suggesting deliberate separation rather than a monolithic client file.

API Design - The public surface deliberately mirrors PEP 249 (connect(), Cursor.execute()/executemany()/fetchall()), which keeps the learning curve low for anyone who has used psycopg2 or sqlite3, while IAM-specific parameters (iam=True, cluster_identifier, db_user) are layered on as optional keyword arguments rather than a separate API, and the configurable paramstyle (qmark/numeric/named/format/pyformat) adds flexibility at the cost of a small amount of extra cognitive overhead for newcomers.