Weave

Weave is Weights & Biases’ open-source toolkit for developing, debugging, and evaluating Generative AI applications. A single @weave.op decorator instruments any Python function — from raw LLM API calls to custom RAG pipelines and multi-step agents — capturing structured traces of inputs, outputs, latency, and nested call trees without changing how the function is invoked.

Beyond tracing, Weave provides a rigorous evaluation framework for building apples-to-apples comparisons across model versions and prompts, a scorer library for automated and human-in-the-loop feedback, versioned datasets and prompts through its object/ref system, and built-in autopatching for over twenty LLM providers and agent frameworks (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, DSPy, and more). It’s built for teams who need to move from experimentation to production without losing visibility into how their AI systems actually behave.

What You Get

• Zero-boilerplate tracing via the @weave.op decorator that captures inputs, outputs, exceptions, and nested call trees for any Python function
• An evaluation framework (weave.flow) for scoring model/prompt variants against datasets with built-in and custom scorers
• Automatic autopatching for 20+ LLM providers and agent frameworks (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, DSPy, Bedrock, Vertex AI) with zero manual instrumentation
• Versioned objects, datasets, and prompts through Weave’s ref/object system, so every trace links back to the exact code and data that produced it
• A pluggable trace server backend supporting ClickHouse, S3, Azure Blob, and GCS for self-hosted or BYOB (bring-your-own-bucket) storage

Common Use Cases

• Debugging a RAG or agent pipeline by inspecting the full call tree of retrieval, prompt construction, and generation steps
• Running side-by-side evaluations of different prompts or models against a fixed dataset before shipping a change
• Auto-instrumenting an existing OpenAI/Anthropic/LangChain-based app with a single autopatch call to get tracing without rewriting code
• Building a human-in-the-loop feedback loop by attaching scorers and annotations to production traces
• Tracking prompt and dataset versions across experiments so results stay reproducible

Under The Hood

Architecture Weave is organized around a core tracing layer (weave/trace/) built on a global WeaveClient and a thread-local call-context stack (context/call_context.py) that tracks the currently executing op so nested calls can be linked into a tree; the @weave.op decorator in trace/op.py wraps arbitrary functions, capturing inputs/outputs through a content-addressed ref system (trace/refs.py, object_record.py) so every logged value is versioned and reproducible. Autopatching lives in weave/integrations/, where a shared patcher.py abstraction lets each of the 20+ supported SDKs (OpenAI, Anthropic, LangChain, Bedrock, Vertex AI, DSPy, CrewAI, and more) register its own monkeypatch without touching core tracing logic. Evaluation and scoring sit in a separate weave/flow/ layer (Evaluation, Scorer, leaderboard, monitor) that consumes the same op/ref primitives, while a distinct trace_server and trace_server_bindings package implements the storage/query backend that can run against ClickHouse, S3, Azure, or GCS — letting the client and server evolve independently.

Tech Stack Weave targets Python 3.10+ and is built with Pydantic v2 for schema validation, httpx (via gql[httpx]) for API communication, and OpenTelemetry (opentelemetry-api/sdk/otlp-proto-http) for span emission in its conversation SDK. The trace server optionally depends on ClickHouse, Redis, Kafka, and cloud storage SDKs (boto3, azure-storage-blob, google-cloud-storage) to support self-hosted deployments, and Sentry (sentry-sdk) is used for client-side error reporting. Dozens of extras-based dependency groups (openai, anthropic, langchain, crewai, dspy, bedrock, vertexai, and others) let consumers install only the integrations they need. The project builds with Hatchling and pins dependencies via uv.lock, indicating uv-based dependency management.

Code Quality The repository ships an extensive test suite of roughly 300 test files under tests/, with pytest markers distinguishing trace-server-backed, remote-only, and Stainless-only test runs, plus a comprehensive .github/workflows/ set (PR checks, nightly tests, dedicated Node/trace-server test jobs) and pre-commit hooks. Ruff is configured with an unusually strict rule set (bugbear, pydoclint, tryceratops, pyupgrade, isort) and mypy is configured for static type checking, with error handling centralized in typed helpers like trace/errors.py that extract structured messages from HTTP error responses rather than swallowing them.

API Design The headline ergonomic choice is that tracing requires no restructuring of existing code: weave.init("project") plus @weave.op on any function is enough to get full call trees, and autopatch settings extend that same zero-touch model to entire third-party SDKs. The API surface (init, op, publish, get, Evaluation, Scorer) stays small and consistent even though the underlying system spans tracing, evaluation, versioned object storage, and a pluggable server — a deliberate choice to keep day-to-day usage simple while the more advanced dataset/eval/self-hosting features stay opt-in.