SacreBLEU

SacreBLEU provides hassle-free computation of shareable, comparable, and reproducible BLEU scores for machine-translation evaluation. It wraps the original reference BLEU implementation with sensible defaults and emits a short version string that lets researchers know exactly how a score was produced, eliminating the ambiguity that comes from divergent tokenization and preprocessing across decoders.

Beyond BLEU, it supports chrF, chrF++, and Translation Error Rate (TER), automatically downloads and manages standard WMT test sets, and includes statistical-significance testing via paired bootstrap resampling and approximate randomization. It works both as a command-line tool and as an importable Python library.

What You Get

• Reproducible BLEU scores matching the official WMT mteval-v13a.pl reference
• Additional metrics: chrF, chrF++, and Translation Error Rate (TER)
• Automatic download, processing, and tokenization of standard WMT test sets
• A short version string encoding exactly how each score was computed
• Statistical significance testing via paired bootstrap resampling and approximate randomization
• Both a command-line interface and an importable Python API

Common Use Cases

• Reporting comparable BLEU scores in machine-translation research papers
• Evaluating translation model output against standard WMT benchmarks
• Computing chrF or TER alongside BLEU for a fuller quality picture
• Running significance tests to compare two translation systems

Under The Hood

Architecture - SacreBLEU is organized around metric implementations (BLEU, chrF/chrF++, TER) and a test-set registry that maps names like wmt14 to downloadable, cached datasets. A tokenization layer applies WMT-standard tokenizers (including Japanese and Chinese variants) to detokenized text before scoring, and a signature module builds the version string that encodes the exact configuration used. The same core is exposed through both a CLI entry point and importable functions such as corpus_bleu.

Tech Stack - The package is pure Python with a small dependency footprint, distributed on PyPI. As of v2.0.0 the default output format switched to JSON for easier machine parsing, with a -f text flag or SACREBLEU_FORMAT=text environment variable to restore the legacy textual output.

Code Quality - The project is mature (created in 2017), with 46 contributors and an established release history. It is validated against the official mteval reference to guarantee score parity, which is the central correctness guarantee of the tool.

API Design - The API is deliberately minimal and opinionated: sensible BLEU defaults mean researchers get correct, comparable scores without configuration, and the emitted version string makes results self-documenting. The CLI accepts test-set names directly, and the Python functions mirror the CLI behavior for programmatic use.