kafka-python-ng

kafka-python-ng is a Python client for the Apache Kafka distributed stream-processing system, designed to function much like the official Java client while adding pythonic interfaces such as consumer iterators. It is a community fork of the widely used kafka-python, created to keep the project alive with releases for newer Python and Kafka versions.

The library provides high-level KafkaConsumer and KafkaProducer classes, a KafkaAdminClient, and lower-level protocol access. It supports coordinated consumer groups on Kafka 0.9+ brokers and remains backwards-compatible down to broker version 0.8.

What You Get

• KafkaConsumer high-level consumer with pythonic iterator interface and consumer-group coordination
• KafkaProducer for publishing messages with configurable batching, compression, and delivery guarantees
• KafkaAdminClient for creating and managing topics, partitions, ACLs, and configs
• Broad broker compatibility from Kafka 0.8 through 2.6, with feature negotiation
• Pluggable serializers, SASL/SCRAM/OAuth authentication, and compression codecs

Common Use Cases

• Consuming a Kafka topic as a Python iterator for stream processing
• Publishing events to Kafka from a Python service or data pipeline
• Programmatically creating and configuring topics with the admin client
• Building consumer groups that dynamically balance partitions across workers

Under The Hood

Architecture The client is organized around an async I/O core (client_async.py, conn.py, cluster.py) that manages broker connections and metadata, with high-level consumer/, producer/, and admin/ packages built on top; a coordinator/ package implements consumer-group membership and partition assignment, while protocol/, record/, and serializer/ handle wire encoding and message formats. Tech Stack Pure Python packaged via setup.py, with optional native extras for compression (lz4, snappy, zstandard) and pluggable SASL/SCRAM/OAuth auth under sasl/, scram.py, and oauth/; broker feature negotiation lets one codebase support Kafka 0.8 through 2.6. Code Quality The repository carries a substantial test/ suite including unit and integration tests (test_admin_integration.py, test_assignors.py, protocol/record tests) with fixtures and Coveralls coverage tracking, inherited and extended from the mature kafka-python lineage of 2,200+ commits and 220+ contributors. API Design The public surface deliberately mirrors the official Java client’s naming and semantics, easing migration for teams familiar with Kafka, while pythonic touches like iterating directly over a KafkaConsumer reduce boilerplate; the breadth of tunable configuration is powerful but contributes a moderate learning curve.