kafka-go

kafka-go is a pure-Go client library for Apache Kafka created by Segment to replace the two dominant options in the Go ecosystem: sarama, which exposes low-level protocol concepts and predates Go contexts, and confluent-kafka-go, a cgo wrapper around librdkafka that drags a C dependency into every build. It provides both a low-level Conn type that mirrors the raw Kafka wire protocol and higher-level Reader/Writer types that feel like idiomatic Go — context-aware, safe for concurrent use, and modeled on patterns from the standard library.

The package supports consumer groups with automatic and explicit offset commits, TLS and the full set of SASL mechanisms (PLAIN, SCRAM, AWS MSK IAM), pluggable partition balancers for producers, gzip/snappy/lz4/zstd compression, and the full complement of Kafka admin APIs (topic/partition management, ACLs, quotas, transactions) exposed through a typed Client. It is tested against a real, dockerized Kafka broker in CI across multiple broker versions rather than relying solely on mocks.

What You Get

• A low-level Conn type for direct control over topics, partitions, and the raw Kafka protocol
• High-level Reader and Writer types with context support, automatic reconnection, and offset management
• Consumer group support with both automatic and explicit (FetchMessage/CommitMessages) offset commits
• Pluggable producer balancers (round-robin, least-bytes, hash, CRC32) for controlling message-to-partition distribution
• TLS and SASL authentication including PLAIN, SCRAM, and AWS MSK IAM mechanisms
• Full admin API coverage: topic/partition creation and reassignment, ACLs, client quotas, and transactional producers

Common Use Cases

• Building event-driven microservices that produce or consume Kafka topics from Go
• Implementing consumer-group based stream processing with reliable offset commits
• Connecting to managed Kafka (e.g. AWS MSK) using SASL/IAM authentication without a cgo toolchain
• Writing lightweight producers/consumers for log or metrics pipelines that need Kafka without the sarama API’s protocol-level complexity

Under The Hood

Architecture kafka-go is organized in clear layers: Conn (in conn.go) wraps a raw net.Conn and speaks the Kafka wire protocol directly, exposing primitives like DialLeader, WriteMessages, and ReadBatch; the Reader (reader.go) and Writer (writer.go) types build on top of that to provide the common produce/consume workflow with automatic reconnection, offset tracking, and consumer-group coordination (consumergroup.go); and a Transport/RoundTripper abstraction (transport.go) manages a broker connection pool and cluster topology discovery, routing each typed request (defined per-API under the protocol/ package) to the correct broker. A typed Client (client.go) sits on top of Transport to expose the full Kafka admin surface — topic/partition management, ACLs, quotas, and transactions — through one consistent request/response pattern.

Tech Stack The library is pure Go with no cgo dependency, targeting Go 1.23+ per go.mod. Its runtime dependencies are minimal and purpose-specific: klauspost/compress and pierrec/lz4 for compression codecs, xdg-go/scram for SASL SCRAM authentication, and golang.org/x/net for protocol-adjacent networking; stretchr/testify is used only in tests. CI (CircleCI) runs golangci-lint and executes the test suite against dockerized Kafka/Zookeeper brokers across multiple Kafka versions rather than relying solely on protocol mocks.

Code Quality Test coverage is extensive — roughly half of the root-level Go files are _test.go counterparts to their implementation, and integration tests exercise real broker behavior (with an explicit skip flag for known-flaky nettest cases). Error handling favors typed values: the Error type enumerates every Kafka protocol error code so callers can use errors.Is against well-known conditions, and internal errors are wrapped with %w for context. golangci-lint is enforced in CI, and package boundaries (protocol/, sasl/, compress codecs) keep concerns separated rather than piling logic into one file.

API Design The public API mirrors familiar standard-library idioms — context-first method signatures, an io-like read/write model for Conn, and a declarative ReaderConfig/WriterConfig struct pattern for setup — which keeps the barrier to a working producer or consumer low. Consumer groups only require setting a GroupID field to switch from single-partition reading to full group-managed consumption, and pluggable Balancer and SASL Mechanism interfaces let callers extend routing and auth behavior without forking the library.