pulsar-rs

pulsar-rs (the pulsar crate) is a pure Rust client for Apache Pulsar that does not depend on the C++ Pulsar library. It exposes an async/await API compatible with both Tokio and async-std, covering producers, consumers, and readers over pulsar:// and pulsar+ssl:// connections.

The client handles the operational realities of talking to a Pulsar cluster: DNS-based broker discovery, TLS, automatic reconnection with exponential backoff, multi-topic consumers by regex or list, message batching, and optional compression with LZ4, zlib, zstd, or Snappy. Executor choice and most heavyweight features are gated behind Cargo features so you only pull in what you use.

What You Get

• Async producers, consumers, and readers built on futures and async/await
• Executor abstraction supporting both Tokio and async-std
• URL-based pulsar:// / pulsar+ssl:// connections with DNS lookup and TLS
• Multi-topic consumers selected by regex or explicit list
• Automatic reconnection with exponential backoff and message batching
• Optional compression (LZ4, zlib, zstd, Snappy) and tracing-based telemetry via Cargo features

Common Use Cases

• Producing messages to Apache Pulsar topics from a Rust service
• Consuming and acknowledging messages from one or many Pulsar topics
• Building event-driven Rust systems on Pulsar with TLS and OAuth/OIDC authentication

Under The Hood

Architecture — The crate is organized around a Pulsar client (client.rs) that owns a connection_manager.rs and service_discovery.rs layer for locating brokers and maintaining pooled connection.rs links. producer.rs, consumer/, and reader.rs implement the three main message-flow abstractions, while executor.rs abstracts over Tokio vs async-std, retry_op.rs implements exponential-backoff reconnection, routing_policy.rs handles partitioned-topic routing, and compression.rs/message.rs deal with the wire format. The Pulsar binary protocol is encoded with prost (Protobuf) and parsed with nom.

Tech Stack — Rust (edition 2021) built on futures, async-channel, async-trait, bytes, prost/prost-derive, nom, crc, and murmur3 for partition routing. TLS (rustls/native-tls), compression (lz4, flate2, snap, zstd), OAuth/OIDC auth (oauth2, openidconnect), and the async-std executor are all optional feature-gated dependencies.

Code Quality — A mature, well-adopted client (78 contributors, 23 releases, high fork ratio) with runnable examples (producer, consumer, reader, batching, round_trip) and inline tests across the source tree. The module boundaries between connection management, discovery, and the producer/consumer APIs are clean, and the pure-Rust implementation avoids FFI complexity.

API Design — The API mirrors Pulsar’s own concepts (client → producer/consumer/reader) and uses builder patterns for configuration, making it approachable for anyone familiar with Pulsar. Executor and feature selection require some upfront Cargo configuration, but the examples get a working producer/consumer going with little boilerplate.