polyglot-sql

polyglot-sql is a Rust crate that tokenizes, parses, generates, and transpiles SQL across more than 30 dialects — from Postgres and MySQL to BigQuery, Snowflake, ClickHouse, and DuckDB. It exposes a fully-typed AST, a fluent query builder, an AST visitor toolkit for walking and transforming queries, and schema-aware validation, and is inspired by Python’s sqlglot but compiled to native code and WebAssembly instead of interpreted.

Beyond dialect translation, the crate produces column-lineage graphs and OpenLineage-compatible payloads, plus compact query-analysis summaries (projections, base tables, CTEs, set-operation branches) for tools that need query facts without walking a full AST. Configurable complexity guards (max input bytes, tokens, AST nodes, set-operation chain depth) protect callers from pathological inputs, and an opt-in stacker-based stack-growth guard hardens deeply nested queries on native targets.

The same core ships as a TypeScript/WASM SDK (@polyglot-sql/sdk), a PyO3-backed Python package (polyglot-sql on PyPI), a C FFI shared/static library, and a Go SDK built on PureGo over that FFI layer — so the Rust crate is one distribution of a single shared parsing/transpilation engine used across four language ecosystems.

What You Get

• Cross-dialect transpilation - translate a single SQL statement into any of 30+ target dialects (Postgres, MySQL, BigQuery, Snowflake, ClickHouse, DuckDB, Trino, and more) via transpile().
• Typed AST parsing and generation - Parser::parse_sql builds a fully-typed Expression tree, and Generator::sql renders it back into dialect-correct SQL text.
• Fluent query builder - construct SELECT/WHERE/ORDER BY/LIMIT queries programmatically via a chained builder API instead of string concatenation.
• Column lineage and OpenLineage output - trace column-level lineage through a query and emit OpenLineage-compatible columnLineage facets and job/run event payloads.
• Schema-aware validation - validate queries against a JSON-defined table/column schema with syntax, semantic, and type checks.
• AST visitor and transform utilities - walk, rename, qualify, and rewrite query nodes (tables, columns, limits, filters) programmatically.
• Compact query analysis - analyze_query returns projections, base tables, CTE facts, and set-operation branch metadata without building a full lineage graph.
• Complexity guard rails and stack-safety hardening - configurable byte/token/AST-node/set-operation limits plus optional stack-growth protection for deeply nested SQL.

Common Use Cases

• Migrating a data warehouse - transpile a large body of legacy SQL from one warehouse dialect (e.g. Redshift) to another (e.g. Snowflake or BigQuery) automatically.
• Building data-lineage tooling - a data-catalog or governance tool ingests SQL and calls lineage()/openlineage() to produce column-level lineage graphs for downstream consumers.
• Cross-database query layers - an application accepts one SQL dialect from users and transpiles it to whichever backend dialect the underlying database actually speaks.
• Static SQL analysis in CI - a lint or review tool parses SQL changes and uses analyze_query to flag unqualified columns, wildcard projections, or unresolved table references before merge.
• Multi-language SQL tooling - teams embed the same transpilation engine in a Rust service, a TypeScript frontend (via the WASM SDK), and a Python data pipeline (via the PyPI bindings) without maintaining three separate parsers.

Under The Hood

Architecture The crate follows a tokenizer → parser → generator pipeline: tokens.rs lexes SQL text, parser.rs runs a recursive-descent parser that builds a fully-typed Expression AST, and generator.rs renders that AST back into dialect-specific SQL text. Dialect-specific behavior is configured through a Dialect/DialectType abstraction implemented across 36 modules in src/dialects/ (e.g. postgres.rs, mysql.rs, bigquery.rs), with a normalization/ submodule handling cross-dialect canonicalization. Feature-gated modules extend the core pipeline: ast-tools/traversal.rs for programmatic AST rewriting, the semantic group (resolver.rs, scope.rs, schema.rs, validation.rs) for schema-aware analysis, lineage.rs/openlineage.rs layered on top of semantic for column-lineage tracing, query_analysis.rs combining semantic+generate for compact query facts, planner.rs for query planning, and builder/ (engine.rs, plan.rs) for the fluent query-construction API. Cargo feature flags (all-dialects, generate, transpile, semantic, openlineage, planner, stacker) let downstream bindings opt into only the pipeline stages they need — the WASM build disables the native-only stacker stack-growth guard via default-features = false. Because every dialect module, the generator, and all semantic/lineage passes pattern-match directly over the shared Expression enum, that type is the crate’s central point of coupling.

Tech Stack A pure Rust 2021-edition Cargo workspace (polyglot-sql core plus polyglot-sql-ast-derive, polyglot-sql-function-catalogs, polyglot-sql-wasm, polyglot-sql-ffi, and polyglot-sql-python) built on serde/serde_json for AST (de)serialization, thiserror for typed errors, unicode-segmentation for correct multi-byte tokenization, and an optional stacker dependency for stack-growth protection on deeply nested queries. WASM bindings use wasm-bindgen/js-sys/web-sys with default features disabled; the Python crate uses PyO3 built via maturin; the FFI crate exposes a C ABI (shared/static libraries plus a generated header) consumed by a companion Go SDK through PureGo. Build tooling is a Makefile-driven workflow (make build-all, make develop-python, make build-python) alongside a pnpm workspace for the TypeScript SDK and playground, with CI defined in .github/workflows/ci.yml. The release profile favors binary size (opt-level = "z", LTO, panic = "abort", symbol stripping), with a separate native_release profile for full-speed native builds.

Code Quality The crate carries an extensive test suite — 36 integration test files under crates/polyglot-sql/tests/, covering identity roundtrips ported from sqlglot’s own fixture suite, dialect-specific regressions (ClickHouse, Fabric, Oracle, Postgres/SQLite interplay), custom dialect registration, deep-nesting inputs, and explicit error-handling cases — plus a benches/ suite for performance-regression tracking via criterion and stats_alloc. Errors are modeled as a typed, #[non_exhaustive] thiserror::Error enum (Tokenize/Parse/Generate/Unsupported variants carrying line/column/span data) rather than stringly-typed panics, and complexity guards return dedicated E_GUARD_* error codes instead of silently truncating oversized input. No CONTRIBUTING.md is present in the repository, but CI is configured, and the workspace’s feature-gated compilation across five sub-crates exercises a large surface of conditional-compilation correctness on every run.

API Design The public surface is small and composable: Parser::parse_sql returns typed Expression nodes, Generator::sql renders one back to text, and transpile() chains both into a single call — a first dialect conversion takes three lines. Builder methods (select, from, where_, order_by, limit) read close to the SQL they generate, and schema-aware entry points (lineage_with_schema, resolve_column, analyze_query) follow a consistent _with_schema naming convention across the crate. Module-level rustdoc on lib.rs documents pipeline stages and feature-flag boundaries up front, and the README carries quick-start snippets for all four supported ecosystems (Rust, TypeScript, Python, Go), lowering the barrier for consumers arriving from any of them. The one point of friction is the Go SDK, which requires separately building or downloading a native FFI shared library and pointing POLYGLOT_SQL_FFI_PATH at it — a manual step none of the other three bindings need.