Sarama

Sarama is a Go client library for Apache Kafka, originally built at Shopify and now maintained by IBM. It implements the Kafka wire protocol directly in Go, giving applications synchronous and asynchronous producers, consumer-group coordination with pluggable partition-assignment strategies, and a ClusterAdmin API for managing topics, ACLs, configs, and partition reassignments — all without shelling out to a JVM client or cgo bindings.

Beyond the core client, Sarama ships a mocks subpackage for testing producer/consumer code without a live broker, an internal/toxiproxy-backed fault-injection test harness, and a tools/ directory of command-line utilities (console consumer/producer, performance tooling, TLS helpers) that double as usage references. It supports SASL (PLAIN, SCRAM, GSSAPI/Kerberos, OAUTHBEARER), TLS, idempotent and transactional (exactly-once) producers, and follows a stated compatibility guarantee of the two latest Kafka and Go releases plus a two-month grace window.

What You Get

• SyncProducer and AsyncProducer implementations with configurable acknowledgment levels, batching, compression, and retry/backoff behavior
• A ConsumerGroup API that handles partition assignment, rebalancing (including cooperative-sticky), and offset commit/checkpointing automatically
• A ClusterAdmin client covering topic/ACL/config management and partition reassignment, used to build custom Kafka tooling
• Idempotent and transactional producer support for exactly-once delivery semantics
• A mocks subpackage that fakes producers and consumers so application code can be unit-tested without a running Kafka cluster
• Working example applications (consumer groups, exactly-once transactions, SASL/SCRAM auth, HTTP server integration) under examples/

Common Use Cases

• Publishing and subscribing to domain events between Go microservices over Kafka topics
• Streaming high-volume logs, metrics, or telemetry into Kafka for downstream processing
• Building custom Kafka CLI tools and cluster-management utilities on top of the admin/broker APIs
• Unit-testing Kafka-integrated application code using the mocks package instead of a live broker

Under The Hood

Architecture Sarama is organized as a flat, protocol-driven package: each Kafka API request/response pair gets its own file (e.g. acl_create_request.go/acl_create_response.go, add_partitions_to_txn_request.go), encoded/decoded through a shared packet-encoding layer, while broker.go owns the actual TCP connection, correlation-ID tracking, and in-flight request bookkeeping per broker. Higher-level concerns are layered on top of that protocol layer: client.go maintains cluster metadata and broker lookups, async_producer.go and consumer.go build producer/consumer semantics (batching, retry buffering via internal/queue, partition offset tracking) on top of Client/Broker, and consumer_group.go adds group coordination (joins, rebalances, heartbeats) as its own layer above the plain consumer. This separation means the wire protocol, connection management, and higher-level producer/consumer/group semantics can evolve mostly independently, though all of it funnels through the shared Broker.send request path.

Tech Stack Sarama is written in modern Go (go.mod targets Go 1.25) with a deliberately small dependency footprint: klauspost/compress and pierrec/lz4 for Kafka’s compression codecs, rcrowley/go-metrics for per-broker/request metrics, eapache/go-resiliency for circuit-breaker behavior in the producer, and jcmturner/gokrb5 for GSSAPI/Kerberos SASL auth. It has no runtime dependency on cgo or the JVM. CI (.github/workflows/ci.yml, fvt*.yml) runs unit tests plus functional-verification tests against real Kafka brokers, alongside CodeQL, OpenSSF Scorecard, dependency-review, and a Go API-diff check (apidiff.yml) to catch breaking changes.

Code Quality The project pairs nearly every source file with a matching _test.go file (151 test files against 158 non-test files) and uses stretchr/testify extensively for assertions across dozens of files. Errors are modeled as explicit sentinel values (ErrOutOfBrokers, ErrClosedClient, ErrSessionPartitionCountChanged, etc.) rather than opaque strings, several implementing Unwrap() for Go’s error-wrapping conventions. .golangci.yml and a pre-commit config enforce linting, and the fuzz.yml workflow runs Go fuzz testing against the protocol decoding paths. Naming is consistent and protocol-request/response types are generated in a uniform pattern across the codebase.

What Makes It Unique Sarama reimplements the Kafka wire protocol natively in Go rather than wrapping the official Java client or librdkafka via cgo, which avoids cgo’s cross-compilation and deployment friction while giving Go programs a client whose internals (connection pooling, request pipelining, retry buffering) are visible and tunable in Go itself. Its explicit compatibility guarantee (two latest Kafka + Go releases, two-month grace period) and long production history at Shopify and now IBM make it a de facto standard for Kafka access from Go, with a mocks package and fault-injection test harness that are more built-out than most protocol-client libraries offer.