sentencepiece

SentencePiece is Google’s unsupervised text tokenizer and detokenizer for neural network-based text generation systems, including large language models, where the vocabulary size must be fixed before training begins. It implements both Byte-Pair-Encoding (BPE) and the unigram language model as interchangeable subword segmentation algorithms, and trains end-to-end directly from raw sentences with no external pre-tokenization step.

Because it treats input as a raw stream of Unicode characters and escapes whitespace as an explicit meta-symbol, SentencePiece produces lossless, fully reversible tokenization independent of language-specific word boundaries. That makes it equally effective on space-delimited languages and on languages like Chinese, Japanese, and Thai that have no explicit word segmentation. The C++ core exposes Python and other language bindings so the exact same .model file produces identical tokenization everywhere it’s loaded.

What You Get

• A trainer (SentencePieceTrainer) that fits a BPE or unigram subword vocabulary directly from raw text files or in-memory sentence iterables
• A processor (SentencePieceProcessor) that encodes text to subword pieces, integer IDs, or serialized protobufs, and decodes losslessly back to the original string
• Subword regularization and BPE-dropout for on-the-fly sampling of alternate segmentations during training, to make downstream models more robust to noisy input
• A self-contained .model file that bundles the vocabulary, segmentation model, and normalization rules so tokenization is bit-for-bit identical across C++, Python, and other bindings
• Command-line binaries (spm_train, spm_encode, spm_decode) for training and running models outside of Python
• GIL-released C++ encoding paths for real multi-threaded throughput from Python, plus optional NumPy and protobuf output formats

Common Use Cases

• Building the tokenizer for a from-scratch LLM or NMT model where vocabulary must be fixed and reproducible before training starts
• Tokenizing multilingual corpora, including CJK and Thai text with no natural word boundaries, without language-specific pre-processors
• Augmenting training data via subword regularization to make sequence models more robust to spelling variation and segmentation noise
• Reproducing an exact tokenization pipeline across a C++ inference server and a Python training/research environment from one shared .model file

Under The Hood

Architecture The library is a thin, language-independent core written in C++ (src/sentencepiece_processor.cc, model_interface.h) that implements the segmentation models — bpe_model.cc and char_model.cc trainers/inference — behind a shared ModelInterface, with normalizer.cc handling the Unicode normalization and whitespace-escaping rules that make detokenization lossless. The Python package (python/src/sentencepiece/init.py) is a thin wrapper: a pybind11 extension module (sentencepiece_pybind.cc, ~1700 lines) exposes the C++ SentencePieceProcessor/SentencePieceTrainer classes, and init.py adds Pythonic ergonomics on top — a dispatch table mapping (is_batch, return_type) tuples to the right C++ method, lazy-loaded protobuf and NumPy integrations, and iterator/sequence coercion helpers. The tokenization behavior itself lives entirely in the C++ layer; changing it means changing bpe_model.cc or char_model.cc, not the Python bindings.

Tech Stack The core is C++ (96% of the codebase) built via both CMake and Bazel, depending on abseil-cpp and protobuf-lite for its internal data structures and serialization, with the .model file itself defined as a protobuf schema (sentencepiece_model.proto). Python bindings use pybind11 with setuptools’ build_ext machinery in python/setup.py, which locates prebuilt abseil/protobuf static libraries at build time. Optional dependencies are loaded lazily at runtime: protobuf for out_type='proto' and NumPy for return_type='numpy', so neither is a hard install requirement. CI (.github/workflows) runs separate CMake, Bazel, cross-build, and wheel-building pipelines plus OSS-Fuzz continuous fuzzing (cifuzz.yml).

Code Quality The C++ core has 23 *_test.cc files exercising the trainers, models, normalizer, and processor directly, run through the CMake/Bazel CI matrix. The Python package has a smaller but targeted test suite (python/test/sentencepiece_test.py, numpy_test.py, gil_release_test.py, typing_smoke.py) including an explicit test that verifies the GIL is actually released during batch encoding — an unusual and welcome test for a native-extension package. Type stubs (__init__.pyi, py.typed) are shipped for static-typing consumers. Comment density in the C++ core is moderate rather than exhaustive, but public APIs and non-obvious algorithmic choices are documented at the top of key files.

API Design The Python API is intentionally small — two classes, SentencePieceTrainer and SentencePieceProcessor — with encode/decode methods that accept a single string or a batch (list of strings) transparently, and an out_type/return_type parameter pair that switches between piece strings, integer IDs, serialized protobuf, or NumPy arrays without changing call shape. This dispatch-table design keeps the common case (sp.encode(text, out_type=str)) a one-liner while still exposing lower-level protobuf output for pipelines that need alignment/score metadata. Optional dependencies (protobuf, NumPy) fail with a clear, actionable ImportError only when the corresponding output format is actually requested, rather than at import time.