Criterion.rs

Criterion.rs is a statistics-driven microbenchmarking library for Rust. It measures the performance of your code with statistical rigor, detecting and quantifying regressions and improvements even when they are small, so you can optimize with confidence.

Unlike a naive timing loop, Criterion.rs runs each benchmark many times, models the measurement noise, and reports confidence intervals along with a clear verdict on whether performance changed since the last run. It works on stable Rust and can render detailed HTML reports and gnuplot charts.

What You Get

• A stable-Rust benchmark harness driven by criterion_group! and criterion_main! macros
• Statistical change detection that reports whether performance improved, regressed, or stayed within noise
• HTML reports and gnuplot charts visualizing distributions, regressions, and trends over time
• Parameterized and grouped benchmarks via BenchmarkGroup for comparing implementations
• Async benchmark support for Tokio, async-std, smol, and futures executors

Common Use Cases

• Catching performance regressions in CI before they reach production
• Comparing multiple implementations of a hot function to pick the fastest
• Measuring the impact of an optimization with statistical confidence
• Profiling parameterized workloads across a range of input sizes

Under The Hood

Architecture — Criterion.rs is structured around a sampling-and-analysis pipeline. Benchmarks are declared with the criterion_group!/criterion_main! macros (src/macros.rs), which build a runner that repeatedly invokes each routine (src/routine.rs, src/bencher.rs) to collect timing samples. Those samples flow through a statistics layer (src/stats, src/analysis, src/kde.rs, src/estimate.rs) that fits models, computes confidence intervals, and compares against saved baselines persisted on disk (src/fs.rs). Results are then emitted through pluggable reporters (src/report.rs, src/html, src/csv_report.rs) that produce console output, HTML pages, and gnuplot charts via the companion criterion-plot crate.

Tech Stack — Written in Rust (2021 edition, MSRV 1.86). Core dependencies include serde/serde_json/ciborium for serialization, clap for its runner CLI arguments, regex and walkdir for filtering and file management, tinytemplate for HTML report rendering, and oorandom/num-traits/cast for the statistics. Optional features add rayon for parallelism, csv output, and async executor bridges for tokio, smol, and futures.

Code Quality — The codebase is mature and modular, splitting statistics, measurement, reporting, and I/O into focused modules. It ships an integration test suite (tests/criterion_tests.rs) plus in-tree unit tests, and has 138 contributors with an actively maintained CHANGELOG. Naming is descriptive and the public surface is deliberately small relative to the internal machinery.

API Design — The developer-facing API is intentionally minimal: two macros plus Criterion, Bencher, and BenchmarkGroup. Getting started requires only a few lines in Cargo.toml and a benchmark file, and the extensive online book plus docs.rs reference lower the learning curve. The black_box helper and closure-based iter pattern keep benchmark authoring ergonomic while preventing the optimizer from eliding the measured work.