tiktoken

tiktoken is the byte pair encoding (BPE) tokenizer OpenAI uses internally for its GPT family of models. It converts text into the integer token sequences a model actually consumes, and back again, so applications can measure prompt length, truncate context windows, and estimate API costs before ever making a request. The core encoding loop is implemented in Rust and exposed to Python through PyO3 bindings, which is what gives it a 3-6x speed advantage over comparable pure-Python or tokenizers-based tokenizers on realistic workloads.

The library ships several named encodings (o200k_base, cl100k_base, p50k_base, r50k_base, gpt2) that map directly onto specific OpenAI model families, plus a tiktoken.encoding_for_model() helper that resolves the right encoding from a model name automatically. Vocabularies and merge tables are downloaded on first use and cached locally (respecting TIKTOKEN_CACHE_DIR), so repeated calls don’t re-fetch data. A tiktoken_ext plugin mechanism lets consumers register entirely custom encodings without forking the package.

Beyond basic encode/decode, the API exposes batch and threaded variants (encode_batch, decode_batch), a numpy-returning encode_to_numpy for zero-copy token buffers, offset-aware decoding for mapping tokens back to source text spans, and an explicit allowed_special/disallowed_special control surface so callers can decide whether user-supplied text is allowed to smuggle in special tokens like <|endoftext|>. It is the de facto standard for token counting across the Python LLM tooling ecosystem, used directly by OpenAI’s own SDKs and by most third-party libraries that need to budget context windows against OpenAI models.

What You Get

• Prebuilt encodings (o200k_base, cl100k_base, p50k_base, r50k_base, gpt2) covering every current and legacy OpenAI model family
• encoding_for_model() and a maintained model-name-to-encoding lookup table, so callers don’t need to track which encoding a given model uses
• A Rust-implemented core (CoreBPE, via PyO3) that is 3-6x faster than comparable Python tokenizers on real workloads
• Batch and multi-threaded encode_batch/decode_batch APIs, plus a numpy-array encoding path that avoids Python list overhead
• Fine-grained control over special-token handling via allowed_special/disallowed_special, preventing accidental encoding of control tokens like <|endoftext|>
• An extensible plugin system (tiktoken_ext) for registering custom encodings without modifying the library itself
• Local caching of downloaded vocabulary/merge files, configurable via TIKTOKEN_CACHE_DIR, so repeated use doesn’t re-fetch data
• An _educational submodule that trains a toy BPE tokenizer from scratch and visualises the merge process, for learning how BPE works

Common Use Cases

• Counting tokens in a prompt before sending it to the OpenAI API, to stay under context-window limits or estimate cost
• Truncating or chunking long documents to fit inside a fixed token budget for retrieval-augmented generation pipelines
• Building custom rate limiters or cost estimators for applications that call GPT models at scale
• Mapping model output tokens back to source text offsets for highlighting or streaming UIs
• Registering a custom encoding for a fine-tuned or non-standard vocabulary via the tiktoken_ext plugin mechanism

Under The Hood

Architecture

The package is split cleanly across a language boundary: tiktoken/core.py defines the public Encoding class, a thin Python wrapper that resolves special-token handling, batching, and error normalisation, then delegates the actual byte-pair merge loop to a compiled _tiktoken extension backed by src/lib.rs’s CoreBPE struct. Encoding names are resolved lazily through tiktoken/registry.py, which discovers ENCODING_CONSTRUCTORS from tiktoken_ext-namespaced plugin modules (tiktoken_ext/openai_public.py ships the built-in OpenAI encodings) and caches constructed Encoding instances behind a lock. tiktoken/model.py provides pure lookup tables (MODEL_TO_ENCODING, MODEL_PREFIX_TO_ENCODING) mapping model name strings to encoding names, so adding support for a new model release is a table update rather than an architectural change. This layering means the hot path (merge algorithm) lives entirely in Rust while extensibility (custom encodings, model mappings) stays in easily-patched Python data structures.

Tech Stack

The Python side targets 3.9+ with regex and requests as required runtime dependencies (an optional blobfile extra supports authenticated cloud storage reads for private vocab files); the native side is a Rust crate (edition 2024) built with pyo3 behind an optional python feature flag, using fancy-regex for the lookahead-capable tokenisation regex and rustc-hash’s FxHashMap in place of the standard hasher for measurable speedups on the encoder/decoder maps. Wheels are built across Python 3.9 through 3.15 (including free-threaded builds) and multiple OS/architecture combinations via cibuildwheel, with Rust toolchain bootstrap handled per-platform in CI.

Code Quality

Tests live under tests/ and combine ordinary pytest assertions against known encode/decode outputs with hypothesis-driven property tests (via test_helpers.py’s ENCODING_FACTORIES) that fuzz arbitrary text through round-trip encode/decode checks — a good fit for a tokenizer, where correctness on unusual Unicode and edge-case byte sequences matters more than typical branch coverage. The Rust core also carries its own #[cfg(test)] unit tests for the byte-pair-merge internals. The Python package ships a py.typed marker and consistent type hints throughout the public API; no linter or formatter configuration is checked into the repo, so style enforcement appears to happen outside the public tree.

What Makes It Unique

Unlike general-purpose tokenizer libraries that approximate a model’s tokenisation, tiktoken is maintained by OpenAI specifically to reproduce the exact token boundaries their models were trained on, which matters for precise cost estimation and for any workflow (like log-prob analysis or fine-tuning data prep) that depends on exact token alignment. Its performance work is unusually well-documented in source comments — the Rust module explains, with links to upstream issues, why a heap-based merge algorithm was chosen for long inputs, why rayon was rejected in favour of Python-controlled threading, and why a thread-local regex clone works around a regex crate contention bug — giving it a rare level of engineering transparency for a tokenisation library.