chDB

chDB embeds the ClickHouse query engine inside a Python process, giving developers a fast, in-process OLAP SQL engine with no server to install or manage. It exposes a pandas-compatible DataStore API alongside a full SQL interface, letting you query Parquet, CSV, JSON, Arrow, ORC, and 60+ other formats directly from disk, S3, or other databases such as MySQL and PostgreSQL. Because ClickHouse runs in-process, chDB minimizes data-copy overhead between C++ and Python and can outperform single-threaded pandas workflows on large datasets while keeping a familiar developer experience.

What You Get

• A pandas-compatible DataStore API with lazy evaluation and automatic SQL generation for familiar dataframe-style code
• A full SQL interface via chdb.query() and a stateful Session API for creating databases, tables, and views
• Python DB-API 2.0 support through chdb.dbapi, plus a connection/cursor API for traditional database workflows
• Native readers for Parquet, CSV, JSON, Arrow, ORC and 60+ formats, and connectors for S3, MySQL, PostgreSQL, SQLite, MongoDB, and more
• A clickhouse-connect backend entry point so chDB can act as a drop-in local backend for existing ClickHouse client code

Common Use Cases

• Running fast local SQL analytics over Parquet/CSV files without spinning up a ClickHouse server
• Accelerating pandas-style data exploration on datasets too large for comfortable in-memory processing
• Joining and querying data across multiple sources (S3, MySQL, PostgreSQL) from a single Python process
• Embedding OLAP query capability inside notebooks, ETL scripts, or lightweight data services

Under The Hood

Architecture - chDB’s core is a Python package (chdb/__init__.py) that wraps a compiled ClickHouse engine binary, exposing query(), a connect()/cursor API, and a stateful Session class layered on top of the same embedded engine; the datastore.py module builds a separate lazy-evaluation layer (a QueryPlanner) that translates pandas-style DataFrame operations into ClickHouse SQL before falling back to pandas for unsupported operations, and cc_backend.py/cc_extension.py register chDB as a pluggable backend for the external clickhouse-connect library via Python entry points. Tech Stack - The public interface is pure Python (requires-python >=3.9) built with setuptools, depending on chdb-core (the compiled ClickHouse engine), pandas, and pyarrow; optional extras add clickhouse-connect integration, and the project also ships an agents subpackage (descriptors.py, safety.py, tool.py, smolagents.py) providing typed tool descriptors for LLM agent frameworks. Code Quality - The tests/ directory includes dedicated suites for clickhouse_connect integration and for the agents conformance/safety layer (150+ test functions found), alongside format-specific tests referenced from the README (e.g. tests/format_output.py); the codebase favors explicit, documented public APIs (connection/cursor objects, DB-API 2.0 compliance) over implicit magic. API Design - The library offers multiple deliberately-scoped entry points for different audiences: a one-line chdb.query() for quick SQL, a datastore pandas-compatible API for dataframe users who want zero syntax changes, a DB-API 2.0 driver for tools expecting a standard Python database interface, and a Session class for multi-statement stateful workflows — this layering keeps the barrier to entry low while still supporting advanced use.