TensorZero

TensorZero is an open-source LLMOps platform that collapses the usual sprawl of separate gateway, observability, evaluation, and optimization tools into a single self-hosted stack. At its center is a Rust-based gateway that speaks the OpenAI API format and routes to every major model provider — Anthropic, AWS Bedrock, Azure, Google, Groq, Mistral, Together AI, vLLM, and more — while adding retries, fallbacks, load balancing, caching, and tool use on top.

Every inference and piece of feedback the gateway sees is stored in your own database, powering the open-source TensorZero UI for debugging individual calls, monitoring aggregate metrics, and replaying historical inferences with new prompts or models. That same data feeds the platform’s evaluation and optimization layers: heuristic and LLM-judge evaluations for both single inferences and multi-step workflows, and optimization techniques like supervised fine-tuning, dynamic in-context learning, and automated prompt engineering (GEPA).

A built-in experimentation layer adds adaptive A/B testing and principled routing across prompts, models, and inference strategies, so teams can turn production feedback into a genuine data flywheel rather than manually tuning prompts by hand. The project is used in production from early-stage AI startups up to Fortune 10 companies, and its authors report it handles roughly 1% of global LLM API spend today.

What You Get

• A Rust-based LLM gateway with sub-millisecond p99 latency overhead that routes to Anthropic, OpenAI, AWS Bedrock/SageMaker, Azure, GCP Vertex, Google AI Studio, Groq, Mistral, Together AI, vLLM, TGI, SGLang, xAI, and any OpenAI-compatible endpoint
• Built-in retries, fallbacks, load balancing, granular timeouts, response caching, rate limiting, and usage/cost tracking at the gateway layer
• An open-source TensorZero UI plus programmatic APIs for browsing inferences, building evaluation datasets, and replaying historical requests against new prompts or models
• Heuristic and LLM-judge-based evaluations for both individual inferences and complete multi-step workflows
• Optimization workflows for supervised fine-tuning, dynamic in-context learning, best/mixture-of-N sampling, and automated prompt engineering via GEPA
• Adaptive A/B testing and experimentation primitives that route production traffic across variants with statistical rigor
• OpenTelemetry trace export and Prometheus metrics export for integration with existing observability stacks
• Official OpenAI-compatible SDKs/clients for multiple languages, so existing OpenAI SDK code can point at TensorZero with a base URL change

Common Use Cases

• Standardizing on a single API in front of many LLM providers to avoid vendor lock-in and simplify failover
• Capturing production inference and human feedback data to build datasets for fine-tuning and evaluation
• Running heuristic or LLM-judge evaluations before shipping a new prompt, model, or inference strategy
• A/B testing prompts and models in production with adaptive experimentation instead of static traffic splits
• Building agentic and multi-hop workflows (e.g. RAG) where structured outputs, tool use, and observability all need to compose
• Fine-tuning smaller, cheaper models on production data collected through the gateway to replace expensive frontier-model calls

Under The Hood

Architecture TensorZero is organized as a Rust workspace under crates/ with clear separation between a thin HTTP layer and business logic: the gateway crate wires up an axum router and CLI (main.rs, router, routes) and delegates almost everything to tensorzero-core, which owns config loading, the provider abstraction, inference orchestration, function/variant/tool resolution, evaluations, optimization, experimentation, and a delegating_connection layer that abstracts over Postgres and ClickHouse as the primary datastore, with Valkey used for caching. The project’s own AGENTS.md codifies this boundary explicitly (“API handler will be a thin function … calls a function to perform business logic”). A separate “Autopilot” automated-engineer feature is composed from independent autopilot-client/autopilot-tools/autopilot-worker crates built on a durable-tools job system, layered on top of the same gateway rather than tangled into it. The open-source TensorZero UI is a fully separate React Router application under ui/app whose route and component folders mirror the platform’s feature areas (observability, evaluations, optimization, experimentation, datasets, playground, autopilot), consuming Rust-generated TypeScript types rather than hand-maintained API clients.

Tech Stack The core is async Rust on tokio with axum for HTTP, sqlx for Postgres access, a ClickHouse client for analytical storage, and Valkey (a Redis-compatible fork) for caching, plus tower-http, mimalloc as the global allocator, and metrics-exporter-prometheus for metrics. Rust structs are exported to TypeScript via ts-rs to keep the UI in sync with backend types. The frontend is a pnpm-workspace React Router app tested with Vitest and Storybook, with end-to-end tests. Additional crates expose Python (tensorzero-python) and Node (tensorzero-node) bindings so the same Rust core powers native clients in multiple languages, alongside official OpenAI-compatible SDK wrappers for Go and Node. Deployment is Docker-first (single gateway container, docker compose run for evaluations), with CI split across GitHub Actions and Buildkite.

Code Quality Nearly every crate carries its own tests/ directory, and the workspace standardizes on cargo nextest with the googletest crate (#[gtest], matches_pattern!, expect_that!) rather than ad hoc assertions. AGENTS.md documents concrete, enforced conventions: mandatory cargo fmt and cargo clippy -- -D warnings, structured Error/ErrorDetails types instead of swallowed errors, explicit typed conversions instead of round-tripping through serde_json::Value, RFC 3339 as the standard datetime format, and a preference for .expect("reason") over bare .unwrap(). The frontend has its own Vitest unit and Playwright-style e2e suites, and CodeQL scanning runs in CI. This is a codebase with an unusually explicit, actively maintained style guide rather than implicit conventions.

What Makes It Unique Most open-source LLM gateways (LiteLLM, Portkey) are Python-first; TensorZero’s Rust core is built specifically to keep gateway overhead in the sub-millisecond range at high QPS, and that performance budget is treated as a first-class constraint rather than an afterthought. Its differentiation is less any single feature than the integration: gateway, observability, evaluation, optimization, and experimentation share one data model, so evaluation results and production feedback can feed fine-tuning and dynamic in-context learning without exporting data between separate tools. Built-in support for automated prompt engineering (GEPA) and an “Autopilot” layer that consumes the platform’s own observability data to drive further optimization extend that flywheel further than most comparable stacks attempt.

Self-Hosting

Licensing Model Apache-2.0 licensed — the entire TensorZero LLMOps platform (gateway, UI, evaluations, optimization, experimentation) is open source with no license keys or feature gates in the self-hosted deployment.

Self-Hosting Restrictions None found in the repository — no ee/, enterprise/, or pro/ directories, and no license-check or feature-flag gating tied to a paid tier in the source.

Cloud vs Self-Hosted TensorZero the platform is self-hosted only; there is no hosted SaaS version of the core gateway/observability/evaluation stack itself. A separate product, TensorZero Autopilot (an automated AI engineer built on top of TensorZero), is called out in the project’s FAQ as a complementary paid offering, but it is distinct from — and not required to use — the open-source platform.

License Key Required No — the self-hosted platform requires no license key for any documented feature.