requests-futures

requests-futures is a small add-on for Python’s requests library that makes HTTP calls asynchronous using the standard library’s concurrent.futures. It subclasses requests.Session as FuturesSession, so every verb method (get, post, put, patch, delete, head, options) keeps the exact same signature but returns a Future instead of a Response — callers keep working, submit more requests, and call .result() only when they actually need the response.

By default, requests are submitted to a ThreadPoolExecutor sized by max_workers, with the underlying HTTPAdapter’s connection pool automatically resized to match so worker threads aren’t throttled by urllib3’s default pool size. A ProcessPoolExecutor can be supplied instead for CPU-heavy response handling, in which case the library verifies the whole call is picklable up front and raises a clear RuntimeError rather than letting a pickling failure surface deep inside a worker process. Used as a context manager, exiting the with block cancels every queued-but-not-yet-started request in one shot.

What You Get

• FuturesSession, a requests.Session subclass whose HTTP verb methods return concurrent.futures.Future objects
• Support for both ThreadPoolExecutor (default) and a caller-supplied ProcessPoolExecutor as the execution backend
• Automatic HTTPAdapter connection-pool resizing so pool_maxsize scales with max_workers
• hooks-based response post-processing, plus the deprecated background_callback path for older code
• Context-manager close() semantics that cancel every queued-but-unstarted request and wait for in-flight ones to finish

Common Use Cases

• Concurrently fetching many URLs from scripts and crawlers without blocking on each one sequentially
• Layering non-blocking HTTP calls onto an existing requests.Session-based codebase with a minimal import change
• Offloading CPU-heavy response parsing to worker processes via a supplied ProcessPoolExecutor
• Issuing batches of API calls that need clean, immediate cancellation of unstarted requests on early exit

Under The Hood

Architecture FuturesSession subclasses requests.Session and overrides request() to submit the actual HTTP call — a partial binding of Session.request, or an externally supplied session’s own request method — to an Executor, returning a Future instead of blocking. A module-level wrap() function exists purely because bound methods aren’t picklable: it’s what actually gets submitted when the deprecated background_callback is used with a ProcessPoolExecutor. A separate _configure_adapters() helper reconfigures the mounted HTTPAdapters in place, rather than replacing them, so pool sizing changes without discarding a caller’s retry policy or custom adapter subclass. Pending-future bookkeeping (a set guarded by a lock) exists only for the “own executor + no supplied session” ownership case, used by close() to cancel outstanding work and block until in-flight requests finish; the other two ownership combinations shut the whole executor down or leave it untouched. The whole abstraction is deliberately thin — one class in one module.

Tech Stack Python 3.10+, built directly on requests (Session, HTTPAdapter, DEFAULT_POOLSIZE, DEFAULT_RETRIES, Retry) and the standard library’s concurrent.futures (ThreadPoolExecutor, ProcessPoolExecutor, wait) — no other runtime dependencies. Packaging is setuptools via pyproject.toml with a dynamic version read from requests_futures.version. Documentation is built with Sphinx (sphinx_rtd_theme, sphinx-copybutton) and published to Read the Docs.

Code Quality Tests live in a single, extensive tests/test_requests_futures.py using unittest.TestCase plus pytest fixtures, exercising both ThreadPoolExecutor and ProcessPoolExecutor paths, background_callback deprecation warnings, adapter reconfiguration, and each close()-ownership combination against a real pytest-httpbin server rather than mocks. Docstrings are unusually thorough — every public method documents its parameters, return type, and the exact RuntimeError conditions a caller can hit. Formatting and import order are enforced via black and isort configuration in pyproject.toml, with CI running through GitHub Actions.

API Design The entire public surface mirrors requests.Session — get/post/put/patch/delete/head/options — so existing requests code needs only an import change plus a .result() call at the point it actually needs the response. Extras (background_callback, hooks, executor=, session=, adapter_kwargs=) are all optional keyword arguments layered on top rather than a new API shape, and failure cases — unpicklable calls under ProcessPoolExecutor, scheduling after close() — fail fast with pointed messages instead of deferring to opaque errors from concurrent.futures internals.