execnet

execnet provides a minimal, well-tested API for spawning and communicating with Python interpreters across process, version, platform, and network boundaries. It creates gateways to local subprocesses, SSH hosts, or sockets, then bootstraps a remote Python interpreter without requiring execnet (or anything else) to be pre-installed on the remote side, since the bootstrap code is shipped over the wire itself.

Once a gateway is open, execnet exposes a simple channel-based send/receive and callback/queue API for exchanging Python’s built-in types, using its own lightweight serialization instead of pickle. It underlies pytest-xdist’s ability to distribute test runs across multiple worker processes and hosts, which is the primary way most users encounter it today even though the library itself is general-purpose.

What You Get

• Gateway abstractions for spawning Python interpreters over subprocess, SSH, or raw sockets
• Zero-install remote bootstrapping so there is no need to install execnet or your code on the remote host first
• A channel-based send/receive API plus callback/queue mechanisms for exchanging data between local and remote processes
• A built-in lightweight serializer for Python’s basic types, avoiding the security and compatibility concerns of pickle
• Cross-platform interoperability between Windows and Unix-like systems
• Grouped process creation and robust termination handling for managing multiple gateways at once

Common Use Cases

• Powering pytest-xdist’s distribution of test runs across multiple worker processes and remote hosts
• Writing scripts that administer or run commands across multiple remote machines from a single Python controller
• Building hybrid multi-process applications that need to bridge different Python versions or platforms
• Ad-hoc distributed task execution where installing a full RPC or task-queue framework would be overkill

Under The Hood

Architecture - The library centers on src/execnet/gateway.py and gateway_base.py, which implement the core process and channel protocol shared by all gateway types, with gateway_io.py and gateway_socket.py providing the actual subprocess and socket transport and gateway_bootstrap.py handling the zero-install remote bootstrap sequence, shipping a small bootstrap script that pulls in the rest of execnet’s code over the wire. multi.py layers grouped gateway management on top for handling many workers at once, and rsync.py and rsync_remote.py provide a directory-sync utility built on the same channel mechanism. xspec.py parses the gateway spec strings used to describe how to create a gateway, such as ssh equals host with a python version qualifier. Tech Stack - Pure Python with no required third-party dependencies, built with a standard pyproject.toml src-layout packaging setup; it ships a py.typed marker for static-typing support and targets interoperability across CPython versions and Windows/Unix platforms specifically because it must bootstrap arbitrary remote interpreters. Code Quality - The testing directory contains 11 test files (matching 11 test_*.py files across the repo) covering gateway creation, channel communication, rsync, and cross-platform behavior; CI badges in the README and a documented release process (RELEASING.rst) plus a long-maintained CHANGELOG.rst indicate an established, if slow-moving, maintenance process under the pytest-dev organization. API Design - The core workflow, execnet.makegateway(spec) then gateway.remote_exec(code_or_module) returning a Channel for send/receive, is small and consistent across subprocess, SSH, and socket gateway types, though the gateway spec string syntax and low-level channel API require reading the docs rather than being immediately guessable.