sqllogictest

sqllogictest is a Rust crate implementing the Sqllogictest test format originally designed for SQLite, extended with retries, environment-variable substitution, let variable binding, and precise SQLSTATE error matching. You implement a small DB trait around your database, then hand .slt test files to a Runner (async, via run_file/run_async) or wire them into cargo test with the harness! macro, which globs test files and runs them through libtest-mimic.

The workspace also ships sqllogictest-bin, a standalone CLI that runs .slt suites directly against Postgres or MySQL with colorful diffs, JUnit reports, an --override mode to regenerate expected output, and parallel execution isolated per database. It’s used in the test suites of RisingLight, RisingWave, Apache DataFusion, Databend, and CnosDB.

What You Get

• A DB trait abstraction — implement one run(&mut self, sql: &str) method to plug any database into the runner
• An async Runner with run_file/run_async/run for executing parsed .slt records programmatically
• A full sqllogictest parser supporting statements, queries, system shell commands, and let variable binding
• Extensions beyond the original SQLite dialect: retry-with-backoff, environment-variable substitution, and exact/regex/SQLSTATE error matching
• A harness! macro that globs .slt files and wires them into a native cargo test binary via libtest-mimic
• The sqllogictest-bin CLI for running suites directly against Postgres or MySQL with colorful diffs and JUnit output

Common Use Cases

• Verifying SQL correctness in a database engine’s own test suite
• Regression-testing query planners and optimizers against pinned EXPLAIN output
• Cross-engine compatibility testing against Postgres and MySQL via the CLI
• Golden-file testing, using --override to auto-update expected output after intentional changes

Under The Hood

Architecture The workspace splits cleanly into a parser (parser.rs, ~1,570 lines) that turns .slt text into a Vec<Record<T>> of typed statement/query/system/let records, and a runner (runner.rs, ~2,500 lines) that walks those records against a caller-supplied DB implementation, diffing actual output against expected results with the similar crate for colorful diffs. Location tracks file/line (with an upper chain for included files) so failures point at exact source positions. The Connections abstraction (connection.rs) lets a single test file address multiple named database connections, and substitution.rs implements the $VAR/${VAR:default} environment-variable interpolation used when control substitution on is set. A separate sqllogictest-bin crate layers a CLI (main.rs, ~1,120 lines) and sqllogictest-engines crate (Postgres/MySQL/external-process backends) on top of the core library, keeping the parser/runner reusable by any embedding project. Tech Stack Pure Rust on the 2021 edition, async via async-trait and futures (with futures::executor::block_on bridging sync call sites), regex and subst for pattern/variable extraction, md-5 for result-set hashing (rowsort/large-result digesting), tempfile for isolated per-test scratch directories, owo-colors for terminal diff coloring, and libtest-mimic to integrate glob-discovered .slt files into cargo test without a custom test runner. Dependencies are current, minimal in count, and scoped tightly to their single responsibility. Code Quality The project dogfoods itself: a dedicated tests workspace member (tests/Cargo.toml) runs multiple [[test]] binaries (harness, custom_type, validator, test_dir_escape, system_command, let, substitution) plus .slt fixture files under tests/slt/, exercising the parser and runner through their own format. Errors are modeled with thiserror and a type-erased Arc<dyn Error + Send + Sync> for uniform propagation across arbitrary DB::Error types, and public enums like RecordOutput are marked #[non_exhaustive] for forward-compatible evolution. Naming is consistent and narrowly scoped (QueryExpect, StatementExpect, RetryConfig). API Design The public surface is deliberately small: implement one DB trait method, construct a Runner::new(...), and call run_file. The harness! macro removes remaining boilerplate for the common case of “run every .slt file matching a glob as its own cargo test case.” Doc comments on lib.rs include a complete working example, and the README doubles as a cookbook covering every .slt extension (retry, substitution, let, system commands, SQLSTATE matching) with runnable snippets, keeping the learning curve low for anyone already familiar with SQLite’s original sqllogictest format.