AWS SDK for pandas

AWS SDK for pandas (formerly awswrangler) is a Python library that connects pandas DataFrames to a broad range of AWS analytics and data services. With concise, high-level functions it reads and writes Parquet, CSV, JSON, and Excel on S3; runs and loads Athena queries; manages the Glue Catalog; and moves data in and out of Redshift, Timestream, DynamoDB, OpenSearch, Neptune, and relational databases like PostgreSQL, MySQL, and SQL Server.

An AWS Professional Services open-source initiative, it abstracts away boilerplate around IAM, chunking, and format handling, and can scale from a laptop to distributed execution on Ray and Modin.

What You Get

• DataFrame read/write for S3 in Parquet, CSV, JSON, and Excel with dataset and partitioning support
• Athena query execution and result loading directly into pandas DataFrames
• Glue Data Catalog management including databases, tables, and partitions
• Connectors for Redshift, Timestream, DynamoDB, OpenSearch, Neptune, and PostgreSQL/MySQL/SQL Server
• Optional distributed execution at scale via Ray and Modin

Common Use Cases

• Building S3 data-lake ETL pipelines that write partitioned Parquet datasets from pandas
• Querying Athena and loading results into DataFrames for analysis
• Loading and unloading bulk data between pandas and Redshift or relational databases

Under The Hood

Architecture

The library is organized under the awswrangler package into per-service modules (s3, athena, catalog, redshift, timestream, dynamodb, opensearch, neptune, postgresql, mysql, sqlserver, and others), each exposing DataFrame-oriented functions. Underneath, it builds on boto3 for AWS API calls and PyArrow for columnar Parquet I/O, with a distributed layer that swaps pandas for Modin on Ray when the ray extra is enabled, keeping the public API identical across single-node and distributed execution.

Tech Stack

Python packaged with modern tooling (pyproject.toml, uv.lock). Core dependencies include boto3/botocore for AWS access, pandas and PyArrow for data handling, and optional extras for database drivers (redshift-connector, pymysql, psycopg, pyodbc), OpenSearch, and Ray/Modin for scale. Quality tooling includes Ruff, mypy, and a large pytest suite, with infrastructure-as-code test fixtures under test_infra.

Code Quality

A mature, heavily used codebase (thousands of stars, Apache-2.0, AWS-backed) with an extensive tests directory, documented architecture decision records under adr/, and strict static checking via mypy and Ruff. Public functions are thoroughly type-annotated and documented, and the project maintains tutorials and readthedocs reference, reflecting production-grade engineering discipline.

API Design

The API is deliberately ergonomic: import awswrangler as wr and call service-namespaced functions that take and return DataFrames, minimizing boilerplate around sessions, pagination, and file formats. Rich docstrings, a large tutorial notebook collection, and clear versioned optional-dependency guidance make onboarding smooth even across its wide service surface.