TensorFlow Text

TensorFlow Text provides a collection of text-related ops and classes for TensorFlow 2.x that handle the preprocessing commonly required by text-based models — tokenization, Unicode normalization, wordshape features, and n-gram/sliding-window generation. Because these ops execute inside the TensorFlow computation graph rather than as separate Python preprocessing scripts, the same tokenization logic runs identically at training time and at inference time, eliminating a common source of train/serve skew.

The library ships several tokenizer implementations (WhitespaceTokenizer, UnicodeScriptTokenizer, and subword tokenizers among others), integrates with tf.data pipelines and the Keras API, and is version-locked to specific TensorFlow releases (matching minor versions must be installed together).

What You Get

• Multiple graph-native tokenizers, including WhitespaceTokenizer and UnicodeScriptTokenizer
• Unicode normalization and Unicode-aware string splitting ops
• Wordshape feature extraction and n-gram/sliding-window generation ops
• Direct integration with tf.data pipelines and the Keras API for end-to-end text models
• Consistent train/serve preprocessing since ops execute inside the TensorFlow graph itself

Common Use Cases

• Tokenizing raw text inside a tf.data input pipeline so training and serving use identical logic
• Building NLP models that need Unicode-aware normalization and splitting across many languages
• Extracting wordshape or n-gram features as part of a text feature-engineering pipeline
• Deploying a text model where preprocessing must be embedded in the exported SavedModel graph, not a separate Python step

Under The Hood

Architecture - The package is organized as tensorflow_text/core (C++ kernel implementations of tokenizers and text ops, compiled as custom TensorFlow ops) and tensorflow_text/python (the Python-facing wrappers that expose these ops as tf.data/Keras-compatible classes); a tftext.bzl Bazel macro file and top-level WORKSPACE mirror TensorFlow’s own build tooling since the ops must be compiled against a matching TensorFlow ABI. Tech Stack - C++ for the performance-critical tokenization/normalization kernels, exposed through a Python API; built with Bazel and tightly version-pinned to specific TensorFlow releases (the README explicitly warns to match TF Text’s minor version to the installed TensorFlow version). Code Quality - Ships a CODEOWNERS file indicating an actively reviewed contribution process, an examples/ directory demonstrating tokenizer and Keras usage, and docs/ with generated API reference documentation; as an official TensorFlow-org project it follows Google’s internal review and testing standards before code lands upstream. API Design - Tokenizers follow a consistent tokenize()/detokenize() interface across implementations, and because the ops return standard TensorFlow tensors (including RaggedTensor for variable-length token sequences), they compose directly with existing tf.data.Dataset.map() calls and Keras preprocessing layers without custom glue code.