confluent-kafka-go

confluent-kafka-go is Confluent’s official Golang client for Apache Kafka, providing high-level Producer and Consumer APIs backed by librdkafka, the battle-tested C library that also powers Confluent’s Python and .NET clients. It bundles prebuilt librdkafka binaries for macOS, glibc and musl Linux, and Windows, so most applications get Kafka connectivity without a separate C toolchain.

Beyond core producer/consumer functionality, the library ships an Admin API for topic and ACL management, a mock cluster for local integration testing without a running broker, and a Schema Registry client with Avro, Protobuf, and JSON Schema serializers/deserializers — including client-side field-level encryption and a FIPS 140-3 compliant mode for Schema Registry TLS communication.

What You Get

• High-level Producer and Consumer APIs backed by librdkafka’s C implementation for near-native performance
• Prebuilt, statically-linked librdkafka binaries for macOS, glibc/musl Linux, and Windows — no separate C toolchain required
• An Admin API for managing topics, ACLs, consumer groups, and cluster configuration
• A Schema Registry client with Avro, Protobuf, and JSON Schema serializers, including client-side field-level encryption
• A mock cluster for integration testing without a live Kafka broker

Common Use Cases

• Building event-driven microservices that produce and consume Kafka topics in Go
• Streaming data pipelines that need Confluent Cloud or Confluent Platform connectivity
• Applications requiring Schema Registry-backed Avro/Protobuf message validation
• Administrative tooling for managing Kafka topics, ACLs, and consumer group offsets

Under The Hood

Architecture The library is a cgo wrapper: the kafka package binds directly to librdkafka through import "C", with glue code in glue_rdkafka.h/select_rdkafka.h converting between Go and C representations of messages, headers, and topic-partition lists. A shared Handle interface (handle.go) unifies state common to Producer and Consumer, including OAuth/SASL token refresh. The Producer exposes a non-blocking, event-driven Produce() with delivery reports on a channel, while the Consumer follows librdkafka’s native poll model via Poll(). Platform-specific build-tag files (build_darwin_amd64.go, build_glibc_linux_amd64.go, build_musl_linux_arm64.go, build_windows.go, build_dynamic.go) select static vs. dynamic librdkafka linking at compile time, giving zero-config cross-platform builds. The Schema Registry client lives in a fully separate schemaregistry/ package with its own REST client, serde codecs, and rules engine, so it can be adopted independently of the core Kafka client.

Tech Stack Go 1.25+ with cgo bindings to bundled, prebuilt librdkafka static binaries. Schema Registry support pulls in Avro codecs (hamba/avro, gogen-avro, Confluent’s own avro-go fork), Protobuf tooling (google.golang.org/protobuf, jhump/protoreflect), JSON Schema validation (santhosh-tekuri/jsonschema, invopop/jsonschema), and CEL (google/cel-go) for schema migration rules. Client-side field-level encryption is built on Google’s Tink library with pluggable KMS backends for AWS KMS, Azure Key Vault, and HashiCorp Vault, backed by the respective AWS SDK v2, Azure SDK for Go, and Vault API clients. CI runs on Semaphore across macOS/Linux (glibc, musl) and Windows, exercising both static and dynamic librdkafka linking.

Code Quality Over 40 _test.go files span the kafka and schemaregistry packages, combining unit tests, benchmark tests (go test -bench), and integration tests that run against a real Kafka cluster (testconf.json). Errors from librdkafka are surfaced through a dedicated Error type carrying retriable/fatal/txnRequiresAbort flags, with error codes auto-generated from librdkafka via go generate. Contributor docs require gofmt, go vet, and golint before merging, and every source file carries Apache License headers plus exported Go doc comments.

API Design The API favors idiomatic Go patterns — ConfigMap-based configuration, a poll-based Consumer, and a non-blocking async Producer — that map cleanly onto librdkafka’s native model rather than reinventing Kafka’s wire protocol in pure Go, giving it day-one parity with Confluent’s Python/.NET/C++ clients for features like transactions, exactly-once semantics, and OAuth/SASL. The tradeoff is that cgo linking (-tags musl, -tags dynamic) leaks into consumers’ build configuration. Distinctive additions beyond a typical Kafka client are the FIPS 140-3 compliant mode for Schema Registry TLS and a Schema Registry serde layer with cloud-KMS-backed field-level encryption, both uncommon in general-purpose Kafka client libraries.