tiktoken-go

tiktoken-go is a faithful Go reimplementation of OpenAI’s Python tiktoken library, the byte-pair-encoding (BPE) tokenizer used by GPT models. It reproduces the same encodings (o200k_base, cl100k_base, p50k_base, p50k_edit, r50k_base) and model-to-encoding mappings as the original, so Go services can compute exact token counts for prompts and chat messages before sending them to an OpenAI-compatible API.

The library downloads the same public BPE rank files the Python implementation uses, caches them locally (configurable via TIKTOKEN_CACHE_DIR), and exposes a small Encode/Decode API. It also supports a pluggable BpeLoader interface, so callers who don’t want a network fetch on first use can supply an offline-embedded loader instead.

What You Get

• GetEncoding and EncodingForModel entry points that return a ready-to-use *Tiktoken for any supported model name or encoding name
• Encode, EncodeOrdinary, and Decode methods matching the semantics of the original Python tiktoken API
• Built-in support for all five OpenAI encodings: o200k_base, cl100k_base, p50k_base, p50k_edit, and r50k_base
• A pluggable BpeLoader interface, with a companion offline loader project for embedding the BPE dictionary instead of downloading it at runtime
• Local disk caching of downloaded BPE rank files via TIKTOKEN_CACHE_DIR / DATA_GYM_CACHE_DIR

Common Use Cases

• Counting tokens in a prompt or chat history before calling an OpenAI-compatible chat completion endpoint, to stay under a model’s context window
• Estimating per-request cost in a Go backend that bills based on OpenAI token usage
• Truncating or chunking long documents to a fixed token budget prior to embedding or summarization
• Building token-aware rate limiters or usage dashboards for multi-tenant AI features written in Go

Under The Hood

Architecture tiktoken-go is a single flat package (no internal subpackages) split by responsibility: encoding.go defines the five supported OpenAI encodings (o200k_base, cl100k_base, p50k_base, p50k_edit, r50k_base) plus the model-name-to-encoding lookup tables, load.go owns fetching and disk-caching the raw .tiktoken BPE rank files behind a swappable BpeLoader interface, core_bpe.go implements the actual byte-level BPE merge/encode/decode primitives (CoreBPE), bpe.go holds the low-level bytePairMerge/bytePairEncode merge algorithm, and tiktoken.go exposes the public Tiktoken type with Encode/EncodeOrdinary/Decode. Encodings are lazily initialized and memoized in a package-level map guarded by a sync.RWMutex, so repeated calls to GetEncoding for the same encoding name reuse the already-built CoreBPE. The design mirrors OpenAI’s Python tiktoken closely enough that changing the core merge algorithm would require re-validating against the upstream Python test vectors in test/.

Tech Stack Written in plain Go (module targets Go 1.19) with a small, stable dependency set: github.com/dlclark/regexp2 for .NET-style regex features Go’s native regexp package lacks (needed to match tiktoken’s original regex patterns), github.com/google/uuid for temp-file naming during cached downloads, and github.com/stretchr/testify for test assertions. There is no build tooling beyond the Go toolchain itself, no HTTP framework, and no database — the only network dependency is a plain net/http GET against OpenAI’s public blob storage to fetch BPE rank files, which are then cached to local disk.

Code Quality Tests exist and use testify’s assertion style (tiktoken_test.go, regex_test.go), covering encode/decode round-trips and model-to-encoding resolution against known token sequences, with a test/ directory of cross-checked fixtures against the original Python implementation. There is no CI configuration in the repository (no .github/workflows), so test execution is left to contributors running go test locally rather than being enforced automatically. Error handling is idiomatic Go (explicit error returns from loader and encoding functions), though Encode does still panic on a disallowed special token rather than returning an error, which callers need to guard against explicitly.

What Makes It Unique Its value is faithfulness rather than novelty: it is a line-for-line-equivalent port of OpenAI’s official Python tiktoken, giving Go services exact token counts without shelling out to Python or calling an API — something no comparable Go-native tokenizer covers as completely across all five OpenAI encodings. The pluggable BpeLoader interface (with a separate tiktoken-go-loader companion project for fully offline, embedded-dictionary use) is a small but genuinely useful addition over the reference implementation for teams that can’t allow runtime network calls to OpenAI’s blob storage.