BlingFire

BlingFire (Beyond Language Understanding, FInite state machine and REgular expression manipulation) is a text tokenization library built by Microsoft’s Bing search team and used internally for operations like tokenization, multi-word expression matching, unknown-word guessing, and stemming/lemmatization. The Python package wraps a compiled C++ core through ctypes, shipping precompiled binaries for Windows, macOS, and Linux so there is no build step for end users.

The library supports four tokenization algorithm families behind one uniform API: pattern-based tokenization, WordPiece (BERT-style), SentencePiece Unigram LM, and SentencePiece BPE, plus byte-BPE variants for GPT-2/RoBERTa and induced syllabification for hyphenation. Precompiled models are bundled for common targets (BERT base/cased/Chinese/multilingual, XLNET, XLM-RoBERTa, GPT-2, RoBERTa) so a typical call needs no external model files. Microsoft’s published benchmarks show it running 4-5x faster than Hugging Face’s tokenizers and roughly 10x faster than spaCy’s default tokenization, since each call runs as a single C++ operation without Python’s per-token overhead or GIL contention in multithreaded use.

What You Get

• Default NLTK-compatible word and sentence tokenization with no model file or setup required
• Precompiled WordPiece models for BERT base, cased, Chinese, and multilingual variants
• SentencePiece Unigram LM and BPE tokenization, including byte-BPE for GPT-2 and RoBERTa
• A text_to_ids API that returns padded/truncated numpy arrays ready for PyTorch or TensorFlow batching
• Custom model loading (load_model/free_model) for training and using project-specific tokenization models
• Multilingual hyphenation/syllabification via induced patterns across many languages

Common Use Cases

• Preprocessing text for BERT, XLNET, or RoBERTa fine-tuning pipelines without pulling in the full Hugging Face tokenizers stack
• High-throughput tokenization in production NLP services where per-request latency matters
• Multithreaded batch tokenization where Python’s GIL would otherwise bottleneck a pure-Python tokenizer
• Sentence and word segmentation for search, text analytics, or Bing-style linguistic pipelines
• URL and text normalization using the bundled URI-specific Unigram LM models

Under The Hood

Architecture The project separates a reusable C++ tokenization runtime (blingfireclient.library, headers in inc/, implementation in src/) from a compile-time model-building toolchain (blingfirecompile.library) and CLI tools (blingfiretools/); the Python package (dist-pypi/blingfire/__init__.py) is a thin ctypes wrapper that loads a platform-specific shared library (libblingfiretokdll.so/.dylib/blingfiretokdll.dll) and calls flat C functions like TextToWords, TextToSentences, and TextToIds, marshaling UTF-8 byte buffers across the FFI boundary rather than exposing any C++ classes to Python. Core abstractions (FALDB, FADictInterpreter_t, FARSDfa_pack_triv, FAMealyDfa_pack_triv) implement finite-state-machine and dictionary lookups over packed binary model images, so changing the core FSM representation would require rebuilding every bundled .bin model file, not just relinking code.

Tech Stack The core is C++11, built via CMake (CMakeLists.txt) across Windows/MSVC, macOS, and Linux/GCC toolchains, with an Emscripten target for a WASM build. The Python distribution’s only runtime dependency is NumPy, used solely to wrap the text_to_ids output as an array; there is no C-extension build step for pip installs since binaries are precompiled and bundled per-platform in dist-pypi/blingfire/. CI runs through Azure Pipelines (azure-pipelines.yml), and the repo also produces NuGet and native library artifacts alongside the PyPI package.

Code Quality Testing lives in C++ tools under blingfiretools/ (test_fsm, test_wre, test_ldb, any_test) rather than as a conventional pytest suite for the Python wrapper — there is no dedicated Python test directory, so Python-level regressions rely on the underlying C++ tests and manual verification against the documented example outputs in the README. Error handling in the Python layer is minimal by design: FFI calls return -1 or an oversized length on failure, and the wrapper functions return an empty string in that case rather than raising, which can silently mask truncation on very large inputs. The C++ layer favors packed, memory-mapped-friendly data structures (_pack suffixed classes) with a custom FAException type for internal errors, but this is not surfaced through the Python bindings.

API Design The Python surface is deliberately minimal and function-based rather than object-oriented: default-model functions (text_to_words, text_to_sentences) require zero setup and no model handle, while custom-model functions (text_to_words_with_model, text_to_ids) take an opaque handle returned by load_model and require an explicit free_model call, an unusual manual-memory-management pattern for a Python API that isn’t wrapped in a context manager. Getting started needs only pip install blingfire and a single import, with no configuration files, and the README documents five worked examples covering default tokenization, custom models, BERT-specific tokenization, PyTorch batch preparation, and hyphenation.