OpenTelemetry Instrumentation for kafka-python

opentelemetry-instrumentation-kafka-python wraps the widely-used synchronous kafka-python (and kafka-python-ng) client so every message produced or consumed automatically generates an OpenTelemetry span, without changes at each call site. It registers via the standard opentelemetry_instrumentor entry point and exposes produce and consume hooks so applications can attach custom span attributes, part of the broader OpenTelemetry Python contrib instrumentation suite maintained in the opentelemetry-python-contrib monorepo.

What You Get

• One-line KafkaInstrumentor().instrument() call that patches KafkaProducer and KafkaConsumer to emit spans automatically
• Produce and consume hooks (produce_hook, consume_hook) for attaching custom span attributes at send/receive time
• Standard OpenTelemetry semantic-convention attributes for Kafka messaging spans, compatible with any OTel-configured exporter/backend
• Automatic entry-point registration (opentelemetry_instrumentor) so the instrumentor can be discovered by the OpenTelemetry auto-instrumentation agent

Common Use Cases

• Tracing message flow through synchronous Kafka producers and consumers in a threaded or WSGI-style application
• Correlating Kafka produce/consume spans with upstream HTTP or RPC spans in a distributed trace
• Attaching custom business attributes (e.g. order ID, tenant ID) to Kafka spans via the produce/consume hooks
• Feeding Kafka messaging telemetry into an existing OpenTelemetry Collector / observability backend with zero manual span code

Under The Hood

Architecture - The instrumentor lives at instrumentation/opentelemetry-instrumentation-kafka-python/src/opentelemetry/instrumentation/kafka/__init__.py and follows the standard OpenTelemetry Python instrumentor pattern: a KafkaInstrumentor class (subclassing BaseInstrumentor) monkey-patches KafkaProducer.send and the consumer’s iteration/poll methods to wrap calls in spans, with utils.py housing shared span-attribute-building logic reused across both the sync and async (aiokafka) instrumentors. Tech Stack - Pure Python (requires-python >=3.10), built with hatchling, depending on opentelemetry-api, opentelemetry-instrumentation, and opentelemetry-semantic-conventions (pinned to the monorepo’s synchronized release version), with kafka-python or kafka-python-ng as optional instruments-any extras so the base package installs without pulling in either client variant. Code Quality - Tests live under tests/, covering both kafka-python and kafka-python-ng variants via separate requirements files (test-requirements.txt, test-requirements-ng.txt), following the same conventions used across the ~70 other instrumentation packages in the opentelemetry-python-contrib monorepo, with shared CI, linting, and release tooling. API Design - The API mirrors every other OpenTelemetry Python instrumentor: instrument()/uninstrument() lifecycle methods, an optional tracer_provider kwarg, and hook callables with a fixed (span, args, kwargs) (or (span, record, args, kwargs) for consume) signature — this consistency across the whole instrumentation family lowers the learning curve for anyone who has used another OTel Python instrumentor before.