newrelic-python-agent

The newrelic package is New Relic’s official Python agent, an auto-instrumentation library that hooks into over 100 popular web frameworks, database drivers, task queues, and async runtimes to capture transaction traces, database query timing, external service calls, and errors without requiring code changes at most integration points. It ships as both an importable API (newrelic.agent) for manual instrumentation and a newrelic-admin CLI wrapper that can launch any WSGI/ASGI server or Python process with instrumentation already attached via run-program.

Under the hood the agent maintains a background reporting layer that batches transaction, error, and custom-event data and streams it to New Relic’s data collector over HTTPS or gRPC (for infinite tracing), while dynamically patching target libraries through an import-hook system so instrumentation activates lazily the first time a monitored module is imported.

What You Get

• Zero-config auto-instrumentation for 100+ frameworks (Django, Flask, FastAPI, Celery, gRPC, asyncio) via newrelic.ini and newrelic-admin run-program
• A manual instrumentation API (newrelic.agent) with decorators and context managers for custom transactions, function traces, and database/external traces
• Distributed tracing and infinite tracing support via gRPC/protobuf for full-fidelity trace collection
• Structured logging integration (NewRelicContextFormatter) that enriches log records with trace and entity metadata
• Custom event and metric recording APIs for application-specific telemetry alongside auto-captured data

Common Use Cases

• Diagnosing slow database queries and external API calls in a production Django/Flask/FastAPI service
• Tracking Celery/background-job task duration and failure rates
• Correlating application logs with distributed traces during an incident
• Monitoring async Python services (asyncio, aiohttp, ASGI apps) end to end
• Capturing custom business metrics alongside infrastructure-level APM data

Under The Hood

Architecture The agent is organized in layered modules: newrelic/config.py handles settings loading (newrelic.ini, env vars) and orchestrates registration of instrumentation via newrelic.api.import_hook, which lazily monkey-patches target modules the first time they’re imported, avoiding eager imports of every supported framework. newrelic/api/ exposes the public tracing surface (transaction.py, function_trace.py, database_trace.py, etc.) built on a core Transaction/Trace object model in newrelic/core/ (database_node.py, external_node.py, function_node.py) that assembles a tree of trace nodes per request. newrelic/core/data_collector.py and agent_protocol.py batch and transmit this data to New Relic’s backend over HTTPS, while agent_streaming.py and the infinite_tracing_*.proto/_pb2.py files implement a gRPC-based streaming path for infinite tracing. newrelic/hooks/ (121 files) contains per-library instrumentation split by category (adapter_* for WSGI/ASGI servers, database_*, framework_*, datastore_*), each hooking into a specific library’s public API surface, keeping vendor-specific code isolated from the core tracing engine. If the core Transaction/Trace abstraction changed, essentially every hook module and the collector’s data model would need updating, since they all construct and thread these objects together.

Tech Stack The project targets Python 3.9-3.14 (including PyPy), built via setuptools + setuptools_scm for version derivation from git tags, with an optional C extension (newrelic/core/_thread_utilization.c) compiled for thread-utilization sampling. Optional extras add grpcio/protobuf for infinite tracing and certifi for certificate bundling; the core agent has no required third-party runtime dependencies, deliberately vendoring some pure-Python helpers under newrelic/packages/ to avoid dependency conflicts with instrumented applications. Tooling includes ruff for linting/formatting, tox to run many framework-specific test suites against library version matrices, codecov for coverage reporting, trivy for dependency security scanning, and mega-linter plus pre-commit hooks for contributor consistency, all orchestrated through GitHub Actions workflows.

Code Quality Tests are organized per integration target (tests/datastore_postgresql, tests/framework_django, tests/adapter_gunicorn, and many more, totaling well over a hundred directories), run through tox against a matrix of library versions, giving strong regression coverage for each supported framework/version combination; a cross_agent suite additionally checks compliance against New Relic’s cross-language agent spec fixtures. Error handling favors explicit exception types and defensive logging rather than silent swallowing, since the agent must never crash a host application, instrumentation code is wrapped so a hook failure logs and disables that instrumentation point instead of propagating. Naming is consistent snake_case throughout with clear module-per-concern boundaries; type hints are sparse given the codebase’s long history and broad Python-version support, but ruff, mega-linter, and pre-commit enforce consistent style, and CI runs the full tox matrix plus coverage upload and security scanning on every change.

API Design The public API balances two integration styles well: near-zero-touch auto-instrumentation via newrelic-admin run-program for teams who want default coverage, and a granular manual API (function_trace, background_task, decorators and context managers) for teams needing custom spans, both sharing the same underlying Transaction/Trace primitives so mixing the two doesn’t create parallel systems. Getting started requires only a generated newrelic.ini and either an env var or a single initialize() call, and external documentation is thorough with per-framework integration notes. The main friction is the sheer surface area, over a hundred hook modules and many config options, which is inherent to being a single agent supporting a huge ecosystem rather than a design flaw.