Tokenizers

Tokenizers is Hugging Face’s Rust implementation of today’s most widely used text tokenization algorithms — BPE, WordPiece, Unigram, and byte-level variants — built for training new vocabularies and encoding text at very high throughput. The core library is written entirely in Rust for speed and memory efficiency, with official bindings for Python, Node.js, and (community-maintained) Ruby, so the same fast tokenizer implementation can be trained and used consistently across a project’s Rust services and Python/JS model pipelines.

It underlies tokenization for the Hugging Face transformers ecosystem and countless independent NLP/LLM projects, handling not just encoding but the surrounding pipeline: normalization, pre-tokenization, truncation/padding, special-token insertion, and alignment tracking back to the original text so token-to-character offsets are always recoverable.

What You Get

• Rust implementations of BPE, WordPiece, Unigram, and byte-level BPE tokenizer algorithms
• Vocabulary training from raw text corpora, not just inference on pretrained vocab files
• Official Python and Node.js bindings (plus a community Ruby binding) sharing the same Rust core
• Composable pipeline stages: normalizers, pre-tokenizers, models, and post-processors that can be mixed and matched
• Offset/alignment tracking so every output token maps back to its exact position in the original string
• Built-in truncation, padding, and special-token handling for feeding models directly

Common Use Cases

• Tokenizing training and inference text for transformer-based language models at high throughput
• Training a custom BPE/WordPiece/Unigram vocabulary for a domain-specific or non-English corpus
• Sharing one tokenizer implementation across a Python training pipeline and a Rust/Node.js production inference service
• Recovering exact character offsets for named-entity recognition or span-extraction tasks that need token-to-text alignment

Under The Hood

Architecture The crate’s Rust source under tokenizers/src/ is organized around a Tokenizer struct defined in tokenizer/mod.rs (nearly 1,900 lines) that orchestrates a pipeline of normalizers/, pre_tokenizers/, models/ (the actual BPE/WordPiece/Unigram/WordLevel algorithms), and processors/ (post-processing such as adding special tokens), each implemented as a trait so stages are swappable; decoders/ mirrors this for turning token ids back into text. lib.rs re-exports the public surface and wires the crate together, while the bindings/python and bindings/node directories wrap this same core via PyO3/napi so all language bindings share one battle-tested implementation rather than reimplementing tokenization logic per language. Tech Stack Pure Rust (edition 2018) at the core with no heavyweight runtime dependencies, using PyO3 for the Python binding and napi-rs/N-API for the Node.js binding; the crate is published to crates.io as tokenizers and to PyPI as the tokenizers package (via the Python binding), keeping one source of truth. Code Quality Testing is extensive and multi-layered: tests/ includes training.rs, serialization.rs, offsets.rs, unigram.rs, added_tokens.rs, and a documentation.rs test that verifies README examples actually compile and run, plus a common/ shared-fixture module; CI runs a dedicated Rust workflow (badge referenced in the README) on every push. API Design The public API centers on a small number of composable builder types (Tokenizer::new, .with_normalizer, .with_pre_tokenizer, .encode/.encode_batch, .train) that read as a clear declarative pipeline, and the crate ships runnable examples/ (encode_batch.rs, serialization.rs) alongside the documentation tests, keeping the learning curve low despite the algorithmic complexity underneath.