whatlang

whatlang is a Rust library that identifies the natural language of a piece of text, along with its writing script (Latin, Cyrillic, etc.), using a trigram-based statistical model derived from the Franc JavaScript library. It supports 70 languages, returns a confidence score and an is_reliable() boolean derived from trigram-density and score-separation heuristics, and ships as a pure-Rust, dependency-light crate with no external model files to load.

The library is used as a direct or transitive dependency in notable Rust search infrastructure projects such as Sonic and Meilisearch, and it exposes optional feature flags for serde serialization, enum-map integration, and arbitrary-based fuzz testing. With over 2.4 million total crates.io downloads, it’s a well-established choice for lightweight language identification where a full neural-network-based classifier (like CLD3) would be overkill.

What You Get

• A detect(text) function returning language, script, and confidence in one call
• Support for 70 languages and recognition of writing script (Latin, Cyrillic, Arabic, etc.) alongside the language itself
• An is_reliable() signal derived from trigram density and score separation between the top two candidate languages
• Language blacklist/whitelist support for constraining detection to an expected subset of languages
• Optional feature flags for serde serialization, enum-map trait implementations, and arbitrary fuzz-testing support

Common Use Cases

• Auto-detecting the language of user-submitted search queries or documents before routing to language-specific search indexes (as used in Meilisearch)
• Tagging scraped or ingested text content with a detected language for downstream NLP or filtering pipelines
• Filtering or routing multilingual user-generated content (comments, reviews) by detected language
• Adding lightweight language detection to a Rust CLI or service without depending on a heavyweight neural-network classifier

Under The Hood

Architecture: The crate implements a trigram-based statistical classifier (per Cavnar & Trenkle’s 1994 N-gram text categorization approach) as a derivative work of the JavaScript Franc library. Source is organized under src/ into alphabets/, scripts/, trigrams/, combined/, and core/ modules, separating script detection, language-specific trigram frequency tables, and the core scoring/combination logic that merges script and trigram signals into a final Info result (lang, script, confidence, reliability).

Tech Stack: Rust 2024 edition with a minimal dependency set — hashbrown for fast hash maps is the only required dependency, while enum-map, serde, and arbitrary are optional feature-gated integrations. Benchmarks use the bencher crate (benches/example.rs), and fuzz-style property testing uses arbtest.

Code Quality: The project ships a documented comparison table against CLD2/CLD3 alternatives and an explicit explanation (with a plotted threshold function) of how is_reliable() is computed, which is unusually transparent for a statistical classifier library. Test coverage is comparatively thin (2 dedicated test files) relative to the size of the generated trigram data tables, and the project has seen very low recent commit activity (0 commits/month in the trailing period) despite continued adoption — worth noting for dependency-risk evaluation, though the trigram-based algorithm and 70-language dataset don’t need frequent updates to remain useful.

API Design: The single-call detect(text) entry point, returning a small Info struct with .lang(), .script(), and .confidence() accessors, makes the common case trivial to use immediately. Advanced use (blacklisting languages, accessing intermediate detector state) is available through additional functions and the dev feature flag, kept separate from the primary API so casual users aren’t exposed to the more advanced internals.