DataComPy

DataComPy is a Python library for comparing two DataFrames or tables across Pandas, Polars, Spark, and Snowflake/Snowpark backends. It was originally built as a friendlier replacement for SAS’s PROC COMPARE, going beyond a simple equality check by reporting statistics, column-level mismatches, and rows unique to each side.

Comparisons are configurable: you can join on one or more key columns, set absolute and relative tolerances for numeric matches, ignore case or whitespace, and then render a human-readable report summarizing how the two datasets differ. The same API shape works across each supported DataFrame engine, so the comparison logic stays consistent whether you are on a laptop with Pandas or a cluster with Spark.

What You Get

• A Compare class (and backend equivalents) that diffs two DataFrames on configurable join keys
• Support for multiple engines: Pandas, Polars, Spark, and Snowflake/Snowpark via install extras
• Configurable numeric tolerances (absolute and relative) plus options to ignore case and whitespace
• Human-readable text reports summarizing matched rows, mismatched columns, and rows unique to each side
• Programmatic access to intersection rows, mismatch DataFrames, and match statistics for further processing

Common Use Cases

• Validating a data pipeline migration by comparing old and new output tables
• Reconciling two datasets from different systems on shared key columns
• Regression-testing ETL jobs to confirm a refactor did not change results

Under The Hood

Architecture — Under datacompy/, a shared base.py defines the comparison contract while backend modules (pandas.py, polars.py, spark.py, snowflake.py) implement it per engine, so each Compare variant follows the same join-diff-report flow using that engine’s native operations. A comparator/ package and report.py with Jinja-style templates/ render the human-readable output, and utility.py holds shared helpers. This keeps the diff semantics uniform across very different execution backends.

Tech Stack — Pure Python packaged via pyproject.toml, linted with Ruff, targeting NumPy/Pandas at the core with optional Polars, Spark (PySpark), and Snowflake/Snowpark backends pulled in through install extras. It also integrates with the Fugue/Dask ecosystem per its topics.

Code Quality — The repo is well maintained: a substantial tests/ suite, a benchmarks/ directory, pytest configuration, CODEOWNERS, and a ROADMAP indicate disciplined engineering by Capital One’s open-source team. The recent v1 release with an explicit deprecation policy for the 0.19.x line shows active version management.

API Design — The developer experience is simple and consistent: construct a Compare with two DataFrames and join keys, optionally set tolerances, then call reporting/accessor methods. Because each backend mirrors the same interface, switching engines requires minimal code change, and documentation on the GitHub Pages site covers each backend with examples.