memberlist

memberlist is a Go library from HashiCorp that gives distributed systems a reusable way to track cluster membership and detect failed nodes, without relying on a central coordinator. It implements the SWIM gossip protocol, extended with HashiCorp’s own Lifeguard improvements, so that membership state propagates across a cluster through peer-to-peer UDP gossip and periodic TCP push/pull synchronization rather than through a single point of failure.

It is eventually consistent: every node maintains its own local view of the cluster and that view converges quickly as gossip rounds are exchanged, with convergence speed and bandwidth usage tunable through configuration. Node failures are detected through direct and indirect probing across multiple network paths, so memberlist can partially tolerate network partitions and asymmetric connectivity while still reaching agreement about which nodes are alive, suspect, or dead. It underlies HashiCorp’s own Serf, Consul, and Nomad, and is embedded directly into applications rather than run as a standalone service.

What You Get

• A Delegate interface for hooking custom user data, metadata, and application state into the gossip layer without forking the protocol implementation
• A pluggable Transport interface (with a built-in NetTransport over UDP/TCP) so the wire layer can be swapped for testing or custom networking
• Tunable protocol knobs (RetransmitMult, SuspicionMult, IndirectChecks, probe/gossip intervals) to trade convergence speed against bandwidth for a given cluster size
• Built-in optional gossip encryption (via a Keyring) and packet/stream labeling to distinguish clusters sharing a network segment
• Lifeguard-derived awareness scoring that adapts local timeouts under CPU starvation or network delay so a slow node doesn’t get wrongly declared dead
• A Config.Default*() family (DefaultLocalConfig, DefaultLANConfig, DefaultWANConfig) providing safe starting points tuned for different network topologies

Common Use Cases

• Building a distributed coordination or orchestration system that needs to know which nodes are currently alive without a central registry
• Implementing service discovery for a custom platform where nodes need to find and track each other directly
• Adding gossip-based state dissemination to an existing distributed application via the Delegate.GetBroadcasts/NotifyMsg hooks
• Replacing a heartbeat-over-a-database membership scheme with a protocol that scales sub-linearly with cluster size and tolerates partial network partitions

Under The Hood

Architecture The library is organized around a single Memberlist struct (memberlist.go) that owns the node’s local view of the cluster (nodes/nodeMap), a message queue split into high/low priority lists, and a NodeAwareTransport abstraction it drives to send and receive gossip traffic. Execution is event-driven: background goroutines started from schedule() run independent tickers for probing (state.go), gossiping (net.go), and push/pull anti-entropy, while incoming packets and streams are read off transport-provided channels and dispatched by message type (net.go’s ingest*/handleX functions cover ping, indirect-ping, alive, suspect, dead, and compound message types). State transitions for a node (alive → suspect → dead) live in state.go and are driven by both local probe failures and gossiped messages from peers, giving the system a clearly separated failure-detection layer (state.go/suspicion.go/awareness.go) from the wire/transport layer (net.go/net_transport.go/transport.go) and the pluggable extension layer (delegate.go). Changing the core nodeState representation would ripple through probing, gossip encoding, and the public Node/Members() API, so that struct functions as the system’s central abstraction.

Tech Stack The module targets Go 1.25 and has a deliberately small dependency surface: hashicorp/go-msgpack/v2 for wire encoding, hashicorp/go-metrics (with a compatibility shim for the older armon/go-metrics) for instrumentation, hashicorp/go-multierror for aggregating shutdown/probe errors, hashicorp/go-sockaddr for address resolution, miekg/dns for DNS-based address parsing, google/btree internally, and stretchr/testify for assertions in tests. There is no external service dependency — the library owns its own UDP/TCP transport (net_transport.go) by default, though that transport is fully replaceable via the Transport interface. Build tags (armonmetrics/hashicorpmetrics) let consumers choose which metrics backend receives emitted stats.

Code Quality Testing is extensive relative to the implementation: memberlist_test.go, net_test.go, and state_test.go alone total well over 100KB of test code alongside dedicated suites for awareness, broadcast, keyring, label, queue, security, suspicion, transport, and utility code, plus a mock_transport.go purpose-built for deterministic network simulation in tests. CI (.github/workflows/check.yml) runs golangci-lint (errcheck, govet, ineffassign, modernize, staticcheck, unparam, unused) and a full unit test + coverage pass across both the previous and current stable Go toolchains on every PR, and a make tidy step keeps go.mod/go.sum consistent. Errors are returned as typed Go errors and aggregated with go-multierror rather than swallowed, and a dedicated SECURITY.md documents the library’s threat model explicitly, including the fact that gossip traffic is unauthenticated and unencrypted unless a key is configured.

What Makes It Unique Rather than a textbook SWIM implementation, memberlist layers in HashiCorp’s own Lifeguard extensions — a local awareness score (awareness.go) that dynamically stretches or shrinks probe timeouts based on how promptly the local node itself has been responding, so that a node under transient CPU or network pressure doesn’t misclassify healthy peers as dead. It also adds nack messages to indirect probes, TCP fallback pings, and protocol-versioned messages so clusters can roll out protocol changes without a hard cutover, none of which are part of the original academic SWIM protocol.