RSKafka

RSKafka is a deliberately minimal Rust client for Apache Kafka. Rather than being a general-purpose Kafka implementation, it is heavily optimized for simplicity — both in its implementation and its operational behavior — targeting workloads that use Kafka as a distributed write-ahead log.

It intentionally omits offset tracking, consumer groups, transactions, built-in buffering, aggregation, and linger timeouts, giving you independent write streams per partition and a small, predictable surface. It speaks the Kafka protocol directly over async Tokio, with optional TLS, SASL, and compression support. It was originally used by InfluxDB 3.0.

What You Get

• A minimal async Kafka client speaking the Kafka wire protocol directly over Tokio
• Independent per-partition write streams for produce and consume
• Optional TLS (rustls), SASL (SCRAM/OAuth), and compression (gzip, lz4, snappy, zstd) features
• A small, predictable API with no consumer groups, transactions, or hidden buffering

Common Use Cases

• Using Kafka as a distributed write-ahead log with application-managed offsets
• Lightweight produce/consume in Rust services that don’t need consumer-group semantics
• Embedding a small-footprint Kafka client where librdkafka’s size or complexity is unwanted

Under The Hood

Architecture — The crate implements the Kafka wire protocol from scratch: protocol/ encodes/decodes Kafka requests and responses, connection.rs/connection/ and messenger.rs manage broker connections and multiplex in-flight requests, and the client/ module exposes the public ClientBuilder, partition clients, and topic operations. Supporting modules handle record encoding, backoff, throttle, and validation. There is deliberately no consumer-group or offset-storage layer.

Tech Stack — Rust on async Tokio, using bytes, crc32c, integer-encoding, parking_lot, and thiserror; optional features pull in rustls/tokio-rustls for TLS, rsasl for SASL, and flate2/lz4/snap/zstd for compression. Dev tests use criterion, proptest, and cross-check against rdkafka.

Code Quality — A focused, well-factored codebase with protocol-level property tests (proptest), benchmarks (criterion), and integration tests that validate behavior against the reference rdkafka client. Maintenance is now infrequent but the surface is small and stable.

API Design — The API is intentionally lean: build a client, get a partition client, and produce/consume records. Because it omits consumer groups and offset management, callers take on that responsibility, which is the main conceptual cost; for its target write-ahead-log use case the surface is minimal and predictable.