franz-go

franz-go is a pure Go client for Apache Kafka that aims to support every client-facing Kafka feature since v0.8.0, from idempotent and transactional producing through exactly-once semantics, regex topic subscriptions, cooperative-sticky and rack-aware group balancing, and the newer share-group (queue) consumer model. It talks to any Kafka-compatible broker — Kafka, Redpanda, Confluent Platform, Amazon MSK, and Microsoft Event Hubs — and ships as pure Go with no cgo or C client bindings.

The project is split into small, focused modules: the core kgo client, the auto-generated kmsg protocol package, a kadm high-level admin client, a sasl package covering every SASL mechanism, a schema-registry client (sr), and observability plugins for Prometheus, OpenTelemetry, zap, logrus, and more. A companion kfake package provides an in-memory fake Kafka broker for testing consumer and producer code without a real cluster.

What You Get

• A single kgo.Client that can both produce and consume, with direct (no group) or consumer-group based consumption
• Full idempotent and transactional producer support with exactly-once semantics (EOS)
• A high-level kadm admin package for cluster administration (topics, ACLs, configs, group and transaction management) without hand-building raw requests
• A kfake in-memory fake Kafka broker for writing fast, dependency-free tests against real client behavior
• Pluggable observability via dedicated modules for Prometheus, OpenTelemetry, zap, zerolog, logrus, and go-metrics
• A schema registry client (sr) with a convenience Serde type for encoding/decoding Avro, Protobuf, or JSON records

Common Use Cases

• Building a Go microservice that produces and consumes Kafka records with exactly-once guarantees
• Migrating a service off a cgo-based Kafka client (e.g. confluent-kafka-go/librdkafka) to remove the C dependency
• Writing an admin/ops tool that manages topics, ACLs, or consumer group offsets via kadm
• Testing Kafka-dependent application code in CI using kfake instead of spinning up a real broker
• Wiring Kafka client metrics and structured logs into an existing observability stack via the plugin packages

Under The Hood

Architecture The client is organized around kgo.Client (pkg/kgo/client.go, roughly 5,800 lines) which composes dedicated subsystems: broker.go manages per-broker connections and request pipelining, consumer.go/consumer_direct.go/consumer_group.go implement direct and group-coordinated consumption with cooperative-sticky and rack-aware balancers (group_balancer.go), producer.go drives idempotent/transactional producing, and the auto-generated kmsg package (a separate submodule) supplies every wire-protocol request/response type so kgo never hand-rolls protocol parsing. Configuration is expressed as functional options (config.go’s Opt interface) rather than a struct literal, keeping the public API additive as new options are introduced. Changing a core abstraction like the broker connection pool or the group coordinator would ripple through sink/source management and the sasl and kadm packages that build on top of kgo’s exported Request function, since those packages depend on kgo’s client and protocol types rather than duplicating them.

Tech Stack franz-go targets a recent Go toolchain and deliberately keeps its dependency surface minimal: a compression library and an lz4 implementation for codec support, plus its own kmsg submodule for protocol types - no cgo, no C Kafka client bindings. The repository is a multi-module workspace: the root module (kgo, kerr, kbin, kversion, sasl, sr, kfake, kadm) and each plugin (kotel, kprom, kzap, kzerolog, klogr, klogrus, kvictoria, kphuslog, kgmetrics) are released as independent Go modules, so consumers only pull in the observability integration they actually use. CI runs lint and tests across configurations including 32-bit architectures and a real Kerberos setup, and a dedicated generate/ directory contains the code generator that produces kmsg’s request/response types from protocol definitions.

Code Quality The project has an unusually thorough test suite spanning many dozens of _test.go files across the repo, including targeted regression tests for specific race conditions and protocol-version-specific behavior around the next-generation rebalance protocol. A per-package style guide documents house conventions, including a specific idiom for context keys (pointer-to-string instead of an empty struct, to avoid key collisions across packages) and a requirement that internal comments explain “why” rather than “what.” Errors are handled explicitly and typed via a dedicated errors package that translates Kafka’s numeric error codes into Go errors. CI enforces formatting and linting, and a validation test in the generator checks that generated protocol code matches its definitions, guarding against protocol drift.

API Design franz-go’s most distinctive design choice is treating protocol completeness as a first-class goal - the documentation tracks a very long list of Kafka KIPs individually with explicit supported/hidden/rationale status, including KIPs intentionally left unimplemented with a stated reason, which is unusual transparency for a client library. The in-memory fake-broker package is a notable developer-experience feature: it lets consumers of the library test real produce/consume/admin code paths without Docker or a live cluster. The functional-options configuration API and callback-based record iteration reduce boilerplate compared to iterator-only APIs, while the separate plugin packages let teams opt into exactly the observability integration they need rather than pulling in a fixed set of dependencies.