sqlparser

sqlparser is a Rust crate providing a lexer and recursive-descent parser for SQL that conforms to the ANSI/ISO SQL standard while supporting a growing set of vendor-specific dialects (PostgreSQL, MySQL, Snowflake, BigQuery, ClickHouse, DuckDB, Databricks, Hive, MSSQL, Oracle, and more). It parses SQL text into a strongly-typed AST and can regenerate the original SQL from that AST, preserving syntax (aside from comments and whitespace normalization) for tooling that needs faithful round-tripping.

As an Apache Software Foundation project living under the DataFusion umbrella, it deliberately stays a pure syntax parser with no semantic analysis, making it a reusable foundation rather than a full query engine. It’s the SQL front-end for Apache DataFusion, Polars, GlueSQL, Opteryx, PRQL, and several other query engines and SQL analysis tools listed in its own README.

What You Get

• A Parser::parse_sql() entry point that turns SQL text into a Vec of typed AST Statement nodes
• Built-in Dialect implementations for Postgres, MySQL, Snowflake, BigQuery, ClickHouse, DuckDB, Databricks, Hive, MSSQL, Oracle, Redshift, SQLite, and more, plus a GenericDialect
• Syntax round-trip via Display/to_string(), recovering the original SQL (modulo comments/whitespace) from the parsed AST
• Optional visitor feature for recursively walking the AST, and serde feature for serializing AST nodes
• A Spanned trait for recovering source locations from AST nodes, useful for diagnostics tooling

Common Use Cases

• Providing the SQL front-end for a query engine or database, as done by Apache DataFusion, Polars, and GlueSQL
• Building SQL linters, formatters, or migration tools that need to parse, transform, and re-emit SQL
• Static analysis of SQL queries embedded in application code (e.g. detecting unsafe patterns or unsupported dialect features)
• Building custom SQL dialects or database proxies (e.g. ParadeDB, CipherStash Proxy, JumpWire) that need to intercept and rewrite SQL

Under The Hood

Architecture - The crate is organized into tokenizer.rs (lexing), parser/ (the Pratt/recursive-descent parser producing the AST), ast/ (the typed AST node definitions and their Display impls for round-tripping), keywords.rs (a large reserved/non-reserved keyword table), and dialect/ (per-database Dialect trait implementations that toggle parsing rules). This separation lets a single tokenizer and core grammar serve dozens of dialects by having each Dialect impl override specific parsing hooks rather than forking the whole parser.

Tech Stack - Nearly pure Rust (99.5% of the repo), zero mandatory external dependencies for the core parser, with optional serde (AST serialization) and recursive (stack-overflow protection on deeply nested queries, on by default via recursive-protection) as the only real feature-gated dependencies. It’s built and released under the Apache Software Foundation’s governance as part of the broader Apache DataFusion project.

Code Quality - Testing is extensive and dialect-segmented: 19+ dedicated test files under tests/ (e.g. sqlparser_postgres.rs, sqlparser_snowflake.rs, sqlparser_clickhouse.rs, sqlparser_regression.rs) plus a sqlparser_bench micro-benchmark suite for tracking parser performance across changes. The project enforces cargo fmt and cargo clippy in CI and requires tests on every PR, per its contributing guidelines — the README states PRs without tests are not reviewed.

API Design - Parser::parse_sql(&dialect, sql) is the entire entry point for the common case, and the crate deliberately avoids semantic analysis so it accepts syntactically valid-but-semantically-wrong SQL (like duplicate column names), keeping it usable as a base for many different downstream semantic layers. Round-trip formatting (ast[0].to_string() reproducing the original SQL) is a notable ergonomic feature for tooling authors who need to both parse and re-emit SQL, though the project is explicit that it does not accept major API-breaking refactors, favoring incremental dialect additions instead.