Braintrust Python SDK

The braintrust package is the official Python SDK for the Braintrust platform, a service for evaluating and shipping AI products. It provides an Eval() function for scoring model outputs against expected results, a logging and tracing layer for capturing spans across an AI application, a bundled CLI (braintrust eval) for running evaluation scripts, and built-in integrations spanning OpenAI, OpenAI Agents, OpenTelemetry, and Temporal. As an SDK for one external service, it exists specifically to connect Python applications to the Braintrust API for eval tracking, dataset management, and observability.

What You Get

• An Eval() API for defining datasets, tasks, and scoring functions, runnable via the bundled braintrust eval CLI command
• A tracing/logging layer (logger.py, trace.py, span_types.py) for capturing structured spans from AI application code
• Built-in integrations for OpenAI, OpenAI Agents, OpenTelemetry, and Temporal, installable as optional extras
• A pytest plugin (registered via the pytest11 entry point) for running evals as part of a standard test suite
• A local dev server (devserver) and sandboxing utilities for running evals in an isolated environment

Common Use Cases

• Writing code-first evaluations for LLM applications and running them via braintrust eval in CI
• Instrumenting a production AI application with tracing spans that flow into the Braintrust dashboard
• Wrapping OpenAI or OpenAI Agents calls to automatically capture prompts, completions, and scores
• Exporting traces via OpenTelemetry so Braintrust telemetry sits alongside a team’s existing observability stack

Under The Hood

Architecture - The SDK lives under py/src/braintrust/ in the monorepo, with framework.py/framework2.py implementing the Eval() execution engine, logger.py/trace.py/span_types.py/span_identifier_v1..v4.py implementing the versioned span/tracing data model, queue.py/merge_row_batch.py handling batched log delivery, and separate integrations/, wrappers/, otel/, and contrib/ packages isolating third-party framework hooks (OpenAI, OpenAI Agents, OpenTelemetry, pytest) from the core client. Tech Stack - Pure Python (>=3.10), built with setuptools, depending on GitPython, requests, jsonschema, sseclient-py (for streaming), wrapt (for instrumentation wrapping), and typing_extensions, with optional extras (cli, otel, openai-agents, temporal, performance) that pull in boto3, OpenTelemetry SDK packages, temporalio, and orjson only when needed. Code Quality - Tests are colocated with source modules (test_logger.py, test_trace.py, test_framework.py, test_otel.py, etc.) rather than in a separate tests directory, run via noxfile.py across a dependency matrix, with VCR cassettes (cassettes/) for replaying HTTP interactions in tests and a scripts/check-stale-cassettes.py guard against test drift. API Design - The core Eval() function keeps the common case to a few lines (data/task/scores), while power users reach the same tracing primitives used internally for OpenAI wrapping, OpenTelemetry export, and Temporal workflow instrumentation, giving a consistent low-boilerplate entry point with depth available via extras.