tiktoken

tiktoken is the JavaScript/WASM port of OpenAI’s Python tiktoken library, giving Node.js, Bun, and browser-based applications the same byte-pair-encoding tokenizer used internally by GPT models. It ships prebuilt encodings for every OpenAI encoding scheme (gpt2, r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base) so developers can count tokens, truncate prompts, or estimate API costs without making a network call.

The repository builds two distinct npm packages from a shared Rust core compiled to WebAssembly: tiktoken, a full-featured WASM build with 1-to-1 feature parity with the Python original, and js-tiktoken, a pure-JavaScript port for environments like Vercel Edge Functions or Cloudflare Workers where WASM support is limited or undesired. Both expose the same encode/decode/encoding_for_model API surface, and a lightweight tiktoken/lite entry point lets consumers load only the encoding they need instead of bundling every vocabulary.

What You Get

• Precompiled BPE encodings for every OpenAI tokenizer scheme (gpt2, r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base) available via a single get_encoding() call
• A model-aware encoding_for_model() helper that maps any current OpenAI model name, including gpt-4o, gpt-4.1, gpt-5, and the o-series reasoning models, to its correct encoding
• A lightweight tiktoken/lite plus tiktoken/load entry point for loading only the ranks you need, sized for edge runtimes like Cloudflare Workers and Vercel Edge Functions
• A pure-JavaScript sibling package (js-tiktoken) for runtimes where WASM isn’t well supported, with matching encode/decode semantics
• Support for custom vocabularies and special tokens by constructing a Tiktoken instance directly from your own BPE ranks and pattern string

Common Use Cases

• Counting tokens client-side or server-side before sending a prompt to the OpenAI API, to stay under context-window limits
• Estimating and tracking per-request API costs by converting text length into exact billed token counts
• Truncating or chunking long documents to fit a fixed token budget for RAG pipelines and chat history windows
• Running consistent tokenization in edge functions (Vercel, Cloudflare Workers) where the Python tiktoken library isn’t available

Under The Hood

Architecture tiktoken is built as a Rust/WASM monorepo with a clear layered design: a shared core crate (core/src/lib.rs) implements the byte-pair-merge tokenization algorithm (CoreBPENative) independent of any binding target, a wasm/ crate wraps that core with wasm-bindgen to expose a Tiktoken class (wasm/src/lib.rs) compiled via wasm-pack into the published tiktoken npm package, and a separate pure-JavaScript js-tiktoken package (under js/) reimplements the same encode/decode surface for WASM-constrained runtimes like Vercel Edge Functions. Turborepo and Yarn workspaces coordinate builds across these packages plus a legacy-compatibility alias package (wasm/alias), and Changesets drives versioned, independent npm publishes for each.

Tech Stack The tokenizer core relies on fancy_regex for pattern matching and rustc_hash::FxHashMap for fast rank lookups, bridged to JavaScript via wasm-bindgen and gloo-utils; the WASM build is produced with wasm-pack build --target bundler, then post-processed by TypeScript scripts (scripts/postprocess.ts, scripts/alias.ts) that generate the final dist/ output. Tests run under Vitest, and the README documents explicit integration recipes for Node.js, Bun, Vite, Next.js, Create React App (via Craco), Vercel Edge Runtime, Cloudflare Workers, and Electron, reflecting broad target-runtime support.

Code Quality Vitest test files under wasm/test/ (test_simple_public.test.ts, init_error.test.ts) cover the public API’s encode/decode round-trip across multiple encodings and its error-throwing behavior on invalid model/encoding names, and CI (.github/workflows/build_js.yml) enforces yarn build and yarn test on every pull request. Error handling in the Rust layer is explicit, converting internal failures into typed JsError/unwrap_throw exceptions rather than swallowing them, though test coverage is comparatively thin for a tokenizer with this much downstream usage, and no dedicated lint step is wired into CI.

API Design The public API deliberately mirrors the original Python tiktoken almost one-to-one (get_encoding, encoding_for_model, .encode()/.decode()), which lowers the learning curve for teams porting existing Python token-counting logic into JavaScript. A tiktoken/lite entry point paired with a registry-driven load() helper is a JS-specific addition for loading a single encoding’s ranks on demand in bundle-size-constrained edge environments, and the README’s per-runtime setup guides are unusually thorough; the main rough edge is the requirement to manually call .free() to release WASM memory, an unfamiliar pattern for typical JS developers.