charabia

charabia is the multilingual tokenizer that powers Meilisearch, detecting a text’s script and language and then applying a specialized pipeline to segment it into tokens and normalize them. It splits sentences into words while handling the particularities of each writing system, from CamelCase in Latin scripts to dictionary-based segmentation for Chinese, Japanese, Korean, Thai, and Khmer.

Beyond segmentation, charabia normalizes tokens through Unicode compatibility decomposition, lowercasing, diacritic removal, and script-specific rules so that queries and documents match reliably regardless of surface form. Its feature-gated design lets you compile in only the languages you need.

What You Get

• Automatic script and language detection before tokenization
• Specialized segmenters for Latin, Chinese, Japanese, Korean, Thai, Khmer, and more
• Token normalization via Unicode decomposition, lowercasing, and diacritic removal
• A Tokenize trait offering a one-call ergonomic entry point on string slices
• Feature flags to include only the language pipelines your application needs

Common Use Cases

• Tokenizing documents and search queries for a full-text search engine
• Normalizing multilingual text so queries match regardless of accents or case
• Segmenting CJK and Thai text that lacks whitespace word boundaries
• Preprocessing user input into consistent tokens for indexing pipelines

Under The Hood

Architecture - The charabia crate splits into detection (script/language identification via whatlang), segmenter (per-language word splitting), and normalizer (Unicode and script-specific cleanup), tied together by tokenizer.rs and the Tokenize trait in lib.rs. A Token type in token.rs carries byte offsets and metadata. A companion irg-kvariants crate handles Chinese kVariant conversion.

Tech Stack - Rust (edition 2021), depending on aho-corasick, fst, whatlang, unicode-normalization, and optional language backends like jieba-rs (Chinese), lindera (Japanese/Korean), and pinyin. Organized as a Cargo workspace with heavy use of feature flags.

Code Quality - Maintained by the Meilisearch team as a production dependency, the crate is well modularized per language with benchmark-tracked throughput figures in the README and dictionary fixtures backing each segmenter. Development activity has slowed but the project is mature and widely used.

API Design - The headline text.tokenize() call makes basic usage trivial, while a builder-based Tokenizer exposes fine control over segmentation and normalization. Feature flags keep binaries lean by excluding unused languages.