Otter

Otter is an in-memory caching library for Go built to combine high hit rates with excellent throughput under concurrent load. It uses an adaptive W-TinyLFU admission policy — the same family of algorithm behind Java’s Caffeine cache — to decide which entries are worth keeping, backed by a segmented window/probation/protected structure and a lock-free frequency sketch so eviction decisions stay cheap even under heavy contention.

Beyond raw cache mechanics, Otter offers a fully optional feature set: size- or weight-based eviction, time-based expiration (after access or write), automatic loading with request coalescing (singleflight-style deduplication so concurrent misses for the same key trigger only one load), asynchronous refresh-ahead of stale entries, deletion listeners, access statistics, and cache persistence to/from disk. Because Otter generates a specialized cache node type per feature combination at build time instead of relying on interfaces or reflection, caches that don’t use a given feature pay effectively no memory or performance cost for it.

What You Get

• Adaptive W-TinyLFU eviction that adjusts to a workload’s access pattern for consistently high hit rates across synthetic and real-world traces
• Size-based or weight-based capacity bounds, with time-based expiration measured from last access or last write
• Automatic entry loading with built-in request coalescing, so concurrent Get calls for a missing key trigger a single loader invocation
• Asynchronous refresh-ahead that serves the stale value immediately while reloading in the background on the first stale request
• Pluggable statistics recording (hit/miss/eviction counters) and deletion listeners for observability and cache-aside integration
• Built-in persistence to save a cache’s contents to a file and restore it on startup

Common Use Cases

• Dropping in a bounded, high-throughput in-process cache in front of a database or remote API call
• Replacing a hand-rolled sync.Map or basic LRU cache where hit-rate quality under skewed access patterns matters
• Read-through caching where a Loader function fetches on miss and concurrent requests for the same key must not stampede the backing store
• Time-windowed caching (e.g. session data, rate-limit counters) using access- or write-based expiration instead of manual TTL bookkeeping
• Services that need cache statistics and deletion notifications wired into existing metrics/observability pipelines

Under The Hood

Architecture Otter’s public Cache[K, V] (cache.go) wraps an unexported cache implementation (cache_impl.go) that composes a hashmap (internal/hashmap), a probabilistic frequency sketch (sketch.go), and a segmented window/probation/protected eviction policy (policy.go) built on custom deques (internal/deque). Entries are represented by node types generated ahead of time by a code generator (cmd/generator, emitting into internal/generated/node) rather than through interfaces or reflection, so a cache configured without expiration or weighing carries none of that struct overhead or branch cost. Maintenance work (admission, eviction, promotion between window/probation/protected segments) is batched and drained through a lock-free buffer (internal/xsync) so it stays off the hot read path, while loader.go and singleflight.go layer request-coalescing and refresh-ahead semantics on top of the core map. The result is a deliberately layered design — public API, policy/eviction core, generated storage, and asynchronous maintenance — that keeps the common Get/Set path lean while pushing complexity into isolated, well-tested internal packages.

Tech Stack Otter is pure Go with no runtime dependencies (only stretchr/testify for tests), targeting Go 1.24+ to use recent standard library additions like iter iterators and runtime.AddCleanup. The eviction node types are produced via go generate through a purpose-built generator rather than hand-written per feature combination. CI runs a dedicated GitHub Actions matrix (test.yml, test-32-bit.yml, lint.yml, docs.yml) that includes 32-bit architecture testing — notable for a library doing manual atomic/bit-packed sketch operations — plus golangci-lint enforcement. Documentation is a separate mkdocs site (its own go.mod under docs/) published to GitHub Pages, and benchmarks (throughput, hit-ratio simulation, memory consumption, client comparisons) live in their own isolated Go module under benchmarks/.

Code Quality The repository carries 31 _test.go files exercising the cache core, options validation, loading, refresh, persistence, deletion, expiry/refresh calculators, the frequency sketch, the eviction policy, and the clock abstraction, with a coverage badge tracked in CI. Errors are explicit and typed: a sentinel ErrNotFound constant for loader misses, and a dedicated panicError type that captures and re-raises panics from user-supplied callbacks (loaders, listeners) with their original stack trace rather than swallowing them. The public generic API (Cache[K, V]) carries doc comments on every exported type and method, and golangci-lint runs in CI alongside the test matrix — this reads as a codebase written with production reliability, not just benchmark performance, in mind.

API Design Configuration is a single declarative Options[K, V] struct with pluggable strategy fields (ExpiryCalculator, RefreshCalculator, StatsRecorder, Weigher, Loader) rather than a large method-chaining builder, so a working cache with expiration, refresh, and stats takes only a few lines, as shown in the project’s own README example. What sets Otter apart technically is combining an adaptive W-TinyLFU admission policy — normally seen in JVM caching libraries like Caffeine — with ahead-of-time generated cache node types per enabled feature, giving Go developers both the hit-rate quality of a research-grade eviction algorithm and near-zero overhead for whichever optional features (expiry, weight, refresh, stats) they don’t turn on.