AgentOps

AgentOps is a Python SDK that instruments AI agents so developers can observe, evaluate, and debug them from prototype to production. With a two-line setup it automatically captures LLM calls, tool usage, token costs, latency, session replays, and errors, then streams them to the AgentOps dashboard for analysis.

It ships built-in instrumentation for the most popular agent stacks, including OpenAI, the OpenAI Agents SDK, Anthropic, CrewAI, LangChain, AutoGen/AG2, Mistral, Ollama, and Groq. Because it is built on OpenTelemetry-style instrumentation, AgentOps gives multi-agent systems end-to-end traces and cost visibility without requiring you to rewrite your agent logic.

What You Get

• Two-line initialization (agentops.init()) that auto-instruments supported LLM and agent frameworks
• Automatic tracking of LLM calls, token usage, cost estimation, latency, and errors
• Session replays and end-to-end traces for multi-agent workflows
• Prebuilt integrations for OpenAI, Anthropic, CrewAI, LangChain, AutoGen/AG2, Mistral, Ollama, and Groq
• A hosted dashboard for analytics, plus an MIT-licensed open-source codebase

Common Use Cases

• Debugging why a multi-agent workflow produced an unexpected result by replaying its full session trace
• Tracking and estimating LLM token costs across models and agent runs
• Monitoring agent latency, errors, and reliability once deployed to production

Under The Hood

Architecture - The SDK is organized around a client that initializes a session and a set of instrumentation modules (agentops/instrumentation) that monkey-patch or hook supported providers to emit spans. A config module handles API keys and endpoints, an enums/exceptions layer defines the domain model, and a legacy module preserves backward-compatible decorators. Captured events are batched and exported to the AgentOps backend as OpenTelemetry-style traces.

Tech Stack - Written in Python and packaged with modern tooling (pyproject.toml, uv.lock). It builds on OpenTelemetry semantics for tracing and provides per-framework instrumentation adapters for OpenAI, Anthropic, CrewAI, LangChain, AG2, Mistral, Ollama, and Groq.

Code Quality - The repository has an extensive tests directory covering unit, integration, smoke, and benchmark suites plus fixtures and a conftest, indicating a mature testing culture. Instrumentation is cleanly separated per integration, and the project maintains a CONTRIBUTING guide and frequent releases (100+).

API Design - Developer experience is a core selling point: agentops.init() plus optional decorators is enough to get full tracing, so most users add observability with two lines and no changes to agent logic. Framework auto-instrumentation removes per-call boilerplate, and configuration is handled through environment variables or init arguments.