confluent-kafka-python

confluent-kafka is Confluent’s official Python client for Apache Kafka, wrapping the battle-tested C library librdkafka through a compiled CPython extension (cimpl) so producing and consuming messages runs at native C speed rather than pure-Python speed. It ships high-level Producer, Consumer, and AdminClient classes, a preview ShareConsumer for KIP-932 queue-style consumption, and both synchronous and asynchronous Schema Registry clients with Avro, Protobuf, and JSON Schema (de)serializers that handle schema evolution automatically.

Beyond the synchronous core, the library includes a native AsyncIO producer (AIOProducer) and consumer for services built on an event loop, such as FastAPI or aiohttp, full transactional support for exactly-once semantics, and typed stubs (a bundled .pyi file plus a py.typed marker) for editor and mypy integration. It works against any Kafka-compatible broker — open-source Apache Kafka, Confluent Platform, or Confluent Cloud — and runtime dependencies stay minimal since the heavy lifting happens in the compiled extension rather than pure-Python code.

What You Get

• High-level Producer, Consumer, and AdminClient APIs backed by the compiled librdkafka C extension
• A native AsyncIO producer and consumer (AIOProducer/AIOConsumer) for event-loop-based applications
• Synchronous and asynchronous Schema Registry clients with Avro, Protobuf, and JSON Schema (de)serializers
• A preview ShareConsumer implementing KIP-932 queue-style, per-record-acknowledged consumption
• Bundled type stubs (cimpl.pyi) and a py.typed marker for static type checking
• Optional extras for AWS IAM OAUTHBEARER authentication and Data Contract rules (including client-side field level encryption)

Common Use Cases

• Producing and consuming high-throughput event streams from Python backend services
• Exactly-once stream processing pipelines built on the transactional producer API
• Serializing and deserializing Avro, Protobuf, or JSON messages against Confluent Schema Registry with automatic schema evolution
• Building non-blocking producers and consumers inside FastAPI or aiohttp services via AIOProducer
• Administering topics, ACLs, and consumer-group/cluster metadata through AdminClient

Under The Hood

Architecture The public Python surface in src/confluent_kafka/__init__.py re-exports classes straight from cimpl, a CPython extension built in setup.py from C sources under src/confluent_kafka/src/ (confluent_kafka.c, Producer.c, Consumer.c, Admin.c, Metadata.c, ShareConsumer.c, AdminTypes.c) that link against librdkafka. Higher-level Python modules layer on top of that thin core: serializing_producer.py/deserializing_consumer.py add (de)serialization hooks, admin/__init__.py (1,380 lines) implements the full administrative surface, and aio/ contains a self-contained async subsystem (_AIOProducer.py, _AIOConsumer.py, plus a producer/ package with _message_batch.py, _producer_batch_processor.py, _kafka_batch_executor.py, and _buffer_timeout_manager.py) that batches and dispatches work onto the event loop without blocking it. The schema_registry/ package mirrors this sync/async split via parallel _sync and _async subpackages sharing a common serde.py. Changing the core C extension’s ABI would ripple through every higher Python layer, but the layering itself (thin native core, Python convenience wrappers on top) is clean and consistently applied across admin, aio, and schema_registry.

Tech Stack CPython 3.8+ with an Extension module (confluent_kafka.cimpl) built via setuptools against the system librdkafka/rdkafka library, declared in setup.py. Runtime Python dependencies are intentionally minimal — requirements/requirements.txt pulls in only typing-extensions for Python <3.11 — because the performance-critical path lives in C, not pure Python. Optional extras (declared in pyproject.toml’s dynamic dependencies) add Avro/Protobuf/JSON Schema Registry support, Data Contract rules with CSFLE, boto3-based AWS IAM OAUTHBEARER authentication, and a ducktape-based system/soak test harness. Build/lint tooling includes black, isort, and mypy (configured in pyproject.toml with targeted per-module overrides for generated protobuf code), and CI runs on Semaphore (semaphore.yml) alongside GitHub Actions workflows for PyPI publishing.

Code Quality The test suite is extensive: 116+ test_*.py files spread across tests/integration (producer, consumer, admin, share_consumer, schema_registry), tests/schema_registry (with parallel _sync/_async suites), tests/avro, tests/oauthbearer, tests/soak, and a tests/ducktape system-test harness (excluded from pytest collection due to a pinned-dependency conflict, per an explicit TODO in pyproject.toml). Test files carry detailed docstring-style comments explaining test strategy (e.g. the wakeable-poll/flush producer tests document their methodology inline). Error handling is explicit and typed: error.py defines a typed _KafkaClientError wrapping the native KafkaError/KafkaException, with Optional[...] type hints throughout. The package ships its own type stubs (cimpl.pyi, 843 lines) and a py.typed marker, and mypy is configured project-wide with narrow, justified overrides for generated protobuf modules — a sign of deliberate type discipline rather than blanket suppression.

API Design The public API favors familiar, minimal-boilerplate entry points: Producer({...}).produce(topic, value, callback=...) and Consumer({...}).poll() mirror librdkafka’s own idioms, so users already familiar with Kafka’s C/Java clients feel at home immediately. The async surface is deliberately symmetric with the sync one — the README states the async Schema Registry interface “mirrors the synchronous client exactly,” differing only by an await and an Async-prefixed class name, which keeps the learning curve low for teams adopting AIOProducer/AIOConsumer after already using the sync clients. Config is passed as plain dicts matching librdkafka’s own configuration property names, which is powerful but pushes documentation-lookup work onto users for less-common settings; the tradeoff favors experienced Kafka operators over newcomers.