rust-rdkafka

rust-rdkafka is a Rust binding around librdkafka (the C library that most language Kafka clients wrap), exposing idiomatic async producer and consumer APIs for building Kafka-based applications in Rust. The workspace is split into the rdkafka crate (the safe, ergonomic Rust API most users depend on directly) and rdkafka-sys (the low-level unsafe FFI bindings to librdkafka), which is why rdkafka — not rdkafka-sys — is the canonical dependency for application code.

The crate integrates with both Tokio and async-std via its futures-based consumer/producer implementations, and exposes admin client functionality (topic/partition management), consumer group operations, and detailed statistics callbacks from the underlying librdkafka instance. It’s the de facto standard Kafka client for the Rust ecosystem, used in data pipelines, stream processing services, and event-driven microservices that need production-grade Kafka connectivity from Rust.

What You Get

• Async producer and consumer implementations that integrate with Tokio or async-std via futures-channel/futures-util
• A StreamConsumer providing a Stream of incoming Kafka messages for idiomatic async consumption loops
• An admin client for managing topics, partitions, and consumer groups programmatically
• Serde integration for serializing/deserializing message payloads to/from JSON and other formats
• Detailed statistics callbacks surfaced directly from librdkafka for monitoring producer/consumer internals
• A mocking module for testing Kafka-dependent code without a live broker

Common Use Cases

• Building high-throughput Kafka producers for event/log ingestion pipelines in Rust services
• Implementing async consumer groups that process Kafka topics as part of a stream-processing pipeline
• Managing Kafka topics and partitions programmatically via the admin client, e.g. as part of deployment automation
• Building event-driven microservices where Rust services communicate via Kafka topics
• Monitoring Kafka client health and throughput using the exposed librdkafka statistics callbacks

Under The Hood

Architecture - The repository is a Cargo workspace with two members: rdkafka (the crate most users depend on, ~10,400 lines across src/) and rdkafka-sys (raw unsafe FFI bindings to the vendored/system librdkafka C library). Within rdkafka, client.rs and config.rs manage the underlying librdkafka client handle and configuration, while consumer/ and producer/ submodules build safe async wrappers (StreamConsumer, FutureProducer, BaseProducer) on top of librdkafka’s callback-driven C API, translating C callbacks into Rust futures via futures-channel. Tech Stack - Built on Rust 2021 edition (MSRV 1.74), depending on rdkafka-sys for the librdkafka FFI layer, futures-channel/futures-util for async plumbing, optional tokio or async-std runtime integration, and serde/serde_json for payload serialization; libc bridges the C ABI boundary. Code Quality - The tests/ directory contains 21 integration test files exercising producer, consumer, admin, and mocking behavior, typically run against a real or Dockerized Kafka broker in CI; the safe/unsafe boundary is carefully isolated in rdkafka-sys so the majority of rdkafka’s own code is safe Rust. API Design - The crate offers multiple producer/consumer flavors (BaseProducer/FutureProducer, BaseConsumer/StreamConsumer) so users can pick a synchronous-callback or async-Stream style depending on their runtime, at the cost of a larger API surface than a single-flavor client would have.