tiktoken-rs

tiktoken-rs is a ready-made Rust tokenizer library for working with OpenAI GPT models and the tiktoken byte-pair encoding (BPE) scheme. It wraps a vendored copy of OpenAI’s tiktoken core and adds ergonomic, Rust-native helpers for encoding text, counting tokens, and computing how many tokens remain in a model’s context window.

Built for LLM application developers, it ships all of OpenAI’s current encodings — o200k_harmony, o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base — behind simple constructor functions and cached singletons, plus optional async-openai integration for measuring chat completion request sizes.

What You Get

• Ready-made constructors for every current OpenAI encoding (o200k_harmony, o200k_base, cl100k_base, p50k_base, p50k_edit, r50k_base)
• Cached singleton accessors that initialize each tokenizer only once for cheap repeated use
• Token-counting helpers including get_chat_completion_max_tokens and get_text_completion_max_tokens with per-model context sizes
• A CoreBPE type for direct encode/decode, including encode_with_special_tokens
• Optional async-openai feature integration for measuring chat completion request messages

Common Use Cases

• Counting tokens in prompts or chat messages before sending them to an OpenAI model
• Estimating remaining context-window budget to avoid exceeding a model’s max tokens
• Splitting or truncating text to fit within token limits in an LLM pipeline
• Building cost/usage estimates for OpenAI API calls without network round-trips

Under The Hood

Architecture - The crate is a thin, well-layered wrapper around a vendored copy of OpenAI’s tiktoken. vendor_tiktoken.rs holds the CoreBPE BPE engine (with a BinaryHeap-based merge implementation and fancy-regex splitting); tiktoken_ext/openai_public.rs defines each named encoding (o200k_base, cl100k_base, etc.); singleton.rs wraps each in a std::sync::LazyLock for one-time init; model.rs maps model names to context sizes and encodings; and api.rs composes these into user-facing helpers like get_chat_completion_max_tokens. lib.rs re-exports the public surface flatly.

Tech Stack - Pure Rust on the 2024 edition (MSRV 1.85). Core dependencies are fancy-regex and regex for tokenization patterns, rustc-hash (FxHashMap) for fast rank lookups, base64 and bstr for loading vocab assets, and anyhow for error propagation. Optional features gate async-openai (chat-message token counting) and dhat (heap profiling). A git submodule pins upstream tiktoken 0.13.0 for syncing the vendored core.

Code Quality - Solid. The library is comprehensively tested with ~75 test functions across tests/tiktoken.rs, tests/model.rs, and inline src tests, plus doctests on the public helpers. The vendored core is deliberately kept close to upstream with modifications limited to visibility and lint suppression for easy syncing. Errors are surfaced as Result types rather than panicking in the core paths, and public functions carry thorough rustdoc with runnable examples.

API Design - Ergonomic and discoverable. Encodings are exposed as plain constructor functions returning Result<CoreBPE> alongside *_singleton() variants for repeated use, so getting a token count is a two-line call. Naming mirrors OpenAI’s own encoding names, reducing translation friction, and the README documents model-to-encoding and context-size tables. The main friction is upstream-driven: encode returns a Result and requires an explicit special-tokens set, which the docs call out directly.