dd-trace-py

ddtrace is Datadog’s official Python client library for Application Performance Monitoring. It instruments Python applications automatically or manually to produce distributed traces, continuous profiles, and runtime metrics that stream into Datadog’s APM product. A single ddtrace-run wrapper or patch_all() call auto-instruments dozens of popular frameworks and libraries — web servers, ORMs, HTTP clients, task queues, and databases — without touching application code.

Beyond core tracing, the library is the entry point for Datadog’s wider Python observability suite: Continuous Profiler, Error Tracking, Dynamic Instrumentation, Test Optimization (CI Visibility), Application Security Management (ASM/IAST), and LLM Observability all ship as products layered on the same tracer and Core API event bus. Each product can be enabled independently via environment variables or remote configuration, so teams adopt only what they need.

Because it targets a single vendor’s backend, ddtrace is scoped specifically to Datadog rather than being a generic OpenTelemetry-style SDK — though it does ship an OpenTelemetry API bridge for teams standardizing on OTel instrumentation while still shipping data to Datadog.

What You Get

• Automatic instrumentation for 100+ frameworks and libraries (Flask, Django, FastAPI, Celery, SQLAlchemy, requests, aiohttp, and more) via ddtrace-run or patch_all()
• A manual tracing API (tracer.trace(), decorators, context managers) for custom spans alongside auto-instrumentation
• Continuous Profiler for CPU, memory, and lock-contention profiling with near-zero overhead
• Error Tracking and Exception Replay that capture stack traces and local variables at the point of failure
• Dynamic Instrumentation for adding logs, metrics, and traces to running production code without a redeploy
• Test Optimization (CI Visibility) integration via a pytest plugin for flaky-test detection and test-impact analysis
• Application Security Management (ASM) and Interactive Application Security Testing (IAST) for runtime vulnerability and attack detection
• LLM Observability for tracing calls to OpenAI, Anthropic, and other LLM providers

Common Use Cases

• Instrumenting a Flask, Django, or FastAPI service to get automatic distributed traces across HTTP, database, and cache calls
• Correlating logs, traces, and profiles for a single request to speed up incident triage
• Adding custom spans around business-critical code paths that auto-instrumentation doesn’t cover
• Attaching Dynamic Instrumentation log points to a production service to debug an issue without redeploying
• Running Test Optimization in CI to detect flaky tests and skip unaffected test suites
• Tracing LLM API calls (OpenAI, Anthropic, Bedrock) for cost, latency, and prompt/response visibility

Under The Hood

Architecture The library centers on a global Tracer (ddtrace/_trace/tracer.py) that owns a SpanAggregator/SpanProcessor chain, a pluggable BaseContextProvider for propagating trace context across threads and async tasks, and a HTTPWriter/AgentWriterInterface that batches and ships spans to the Datadog Agent. Layered above tracing sits the Core API (ddtrace/internal/core), an explicit event-bus and ExecutionContext tree that deliberately decouples per-library “Integrations” (in ddtrace/contrib/internal/, 100+ modules for frameworks like Flask, Celery, and SQLAlchemy) from “Product” code (Tracing, AppSec, Profiling, Debugging) — integrations dispatch generic context events and never import product internals, and products subscribe to those events without importing the instrumented library. Products themselves are registered as pluggable ddtrace.products entry points (apm-tracing-rc, llmobs, appsec, iast, dynamic-instrumentation, etc.), so each can be independently enabled, remote-configured, and shipped in the same wheel without hard-coupling to the core tracer.

Tech Stack Core language is Python (3.9-3.14), with performance-critical paths written in Rust, C++, C, and Cython and built via a CMake + setuptools-rust toolchain (pyproject.toml, setup.py) — the profiler’s native collectors and the IAST taint-tracking engine live here. Runtime dependencies are deliberately minimal: envier for typed environment-variable config, wrapt for function wrapping/monkey-patching used by every integration, bytecode for bytecode rewriting (Dynamic Instrumentation), and an opentelemetry-api bridge for OTel interop. A large contrib/internal tree provides optional integrations for the wider Python web/data ecosystem rather than bundling those frameworks as hard dependencies.

Code Quality The test suite is extensive — 900+ test_*.py files spanning unit, integration, and per-integration snapshot tests, run through riot (a custom tool defined in riotfile.py that matrix-tests each integration against multiple versions of the library it instruments) and orchestrated in an extensive GitLab CI pipeline. Configuration is typed via envier, most of the codebase carries type hints validated against a large mypy.ini, and flake8/codespell enforce style and spelling. Error handling favors explicit, product-scoped exception types and defensive guards around instrumentation hooks so a failure in a single integration doesn’t crash the host application.

API Design The public surface is intentionally small for a library with this much internal complexity: most users only ever call ddtrace-run or patch_all() plus tracer.trace()/@tracer.wrap() for manual spans, with everything else (profiling, ASM, LLM Observability) opt-in via environment variables rather than additional API surface. Internal modules are prefixed with underscores (_trace, _monkey, _logger) to make the public/private boundary explicit, and product code is intentionally isolated behind the Core API so that enabling a product like Dynamic Instrumentation never requires learning a second SDK.