TokenLens

TokenLens is a typed registry of LLM model metadata — context window sizes, pricing, and provider aliases — paired with utilities that answer the questions AI apps constantly need to ask at runtime: does this input fit the model’s context window, what will this call cost, and how much budget remains for the next turn. It normalizes usage objects across providers and SDKs (including native Vercel AI SDK usage fields) into a consistent shape.

The registry can be fetched live from the community-maintained models.dev dataset or used as a static, bundled snapshot, and model IDs can be given in several flexible forms (provider/model, provider:id, or a bare model name) with automatic normalization. It’s framework-agnostic but designed to slot in naturally alongside the Vercel AI SDK.

What You Get

• getModels()/fetchModels() for a static or live-fetched catalog of provider/model metadata sourced from models.dev
• getContext(modelId, providers) returning input/output/total context-window caps for a given model
• getTokenCosts(modelId, usage, providers) returning a USD cost breakdown (input, output, reasoning, cache read/write)
• getUsage() combining context and cost into one call for a model + usage object
• Flexible model ID resolution accepting provider/model, provider:id, or a bare model name, with automatic version-dot normalization
• listModels() for filtering the catalog by provider and status (e.g. stable models only)

Common Use Cases

• Checking whether a prompt plus expected completion will fit inside a target model’s context window before sending it
• Estimating the USD cost of an LLM call from token usage returned by a provider or the Vercel AI SDK
• Deciding when to compact or summarize a long conversation before it exceeds context limits
• Computing remaining token budget for the next turn in a multi-turn agent conversation
• Normalizing usage/cost reporting across multiple LLM providers in one dashboard or logging pipeline

Under The Hood

Architecture: The published tokenlens package (in the packages/tokenlens workspace of a Turborepo monorepo) exposes registry.ts for provider/model catalog access, context.ts for context-window budget calculations, async.ts/source.ts for fetching the live models.dev dataset, and conversation.ts for higher-level conversation-usage utilities; models/ holds the bundled static snapshot used when live fetching isn’t wanted or available. Sibling workspace packages (core, models, fetch, helpers, provider-tests) split out lower-level pieces the main package composes.

Tech Stack: TypeScript throughout (97% of the codebase), built with a pnpm + Turborepo monorepo setup, Biome for linting/formatting, and Lefthook for git hooks. The published package ships as ESM (type: module) with per-provider subpath exports (tokenlens/providers/*).

Code Quality: Test coverage is comparatively light for a young project — 4 .test.ts files at the time of review — and the README documents an in-progress API consolidation: a large set of earlier helpers (normalizeUsage, estimateCost, shouldCompact, and others) are explicitly marked deprecated in favor of three focused functions (getContext, getTokenCosts, getUsage), signaling active API stabilization rather than a settled surface.

API Design: The current focused API (getContext/getTokenCosts/getUsage) is deliberately narrow and composable, each taking a model ID, usage object, and provider catalog and returning a small, typed result object — a clear improvement in ergonomics over the larger, now-deprecated helper set the project is migrating away from.