CacheControl

CacheControl is a port of the caching algorithms from httplib2, packaged for use with the popular Python requests library. It wraps a requests session so that responses carrying cache-control, etag, expires, or vary headers are automatically stored and revalidated, giving your HTTP calls a proper caching layer without changing how you write requests.

Unlike httplib2’s built-in caching, CacheControl is thread-safe and pluggable: it ships with in-memory, file-based, and Redis storage backends, and lets you customize heuristics for servers that do not send explicit cache headers. It is maintained under the Python Software Foundation and is widely used across the packaging ecosystem, including pip.

What You Get

• A drop-in wrapper that adds caching to any existing requests Session
• RFC-compliant handling of cache-control, etag, expires, and vary headers
• Pluggable storage backends: in-memory dict, file cache, and Redis
• Customizable heuristics for origins that omit explicit cache headers
• Thread-safe caching suitable for concurrent request workloads

Common Use Cases

• Speeding up repeated API calls to endpoints that expose cache headers
• Reducing bandwidth and latency for CLI tools and scrapers built on requests
• Caching package metadata downloads (as used by pip and packaging tooling)
• Persisting cached responses to disk or Redis across process runs

Under The Hood

Architecture - CacheControl is built around a CacheControlAdapter that plugs into requests’ transport layer, delegating decisions to a CacheController (cachecontrol/controller.py) which parses HTTP caching headers via RFC 3986/2616 logic to decide whether to serve a cached response, revalidate with etag/last-modified, or fetch fresh. Serialized responses flow through a Serializer and a BaseCache backend abstraction (dict, file, or Redis).

Tech Stack - Pure Python (>=3.10) with two runtime dependencies: requests (>=2.16) and msgpack for compact serialization. It uses the uv build backend, is fully type-annotated (ships py.typed), and is developed with pytest, mypy, and ruff.

Code Quality - The codebase is small, focused, and well-typed, with an extensive test suite in tests/ covering etags, expires heuristics, vary headers, redirects, chunked responses, and both file and Redis storage backends. Modules are cleanly separated by concern (controller, serialize, cache, heuristics, adapter).

API Design - The public surface is minimal and ergonomic: wrapping a session is a single call, cached_sess = CacheControl(requests.session()), after which all requests are cached transparently. Backends and heuristics are supplied via clear constructor arguments, keeping the common case trivial while remaining extensible.