pytest-xdist

pytest-xdist is the standard pytest plugin for parallelizing test execution. Running pytest -n auto spawns one worker process per available CPU core and distributes the collected tests across them, turning a slow single-threaded suite into one that finishes in a fraction of the time. Beyond simple load distribution, it supports several distribution modes (by test, by file, by test class/scope, or manual grouping) so suites with shared fixtures or stateful tests can still be split safely.

Beyond local multi-core speedups, pytest-xdist also supports distributing tests across multiple remote hosts over SSH or a custom rsync setup, and includes a loop-on-failure mode that reruns only the tests that failed on every file save. It has been part of the pytest ecosystem for over a decade and is maintained under the pytest-dev GitHub organization alongside pytest itself.

What You Get

• The -n auto / -n <N> CLI flags to run tests across N worker processes automatically sized to available CPUs
• Multiple distribution algorithms (load, loadscope, loadfile, loadgroup, worksteal, each) for balancing work while respecting fixture scope
• Remote/distributed test execution across multiple machines via SSH, rsync, or a custom setup
• A loop-on-failure mode (--looponfail) that reruns only failed tests automatically on file changes
• Boxed/each mode to run the same test session against multiple Python interpreters or environments in one invocation

Common Use Cases

• Speeding up large CI test suites by running them across all available CPU cores instead of serially
• Splitting a slow integration/e2e suite across several worker processes while keeping tests that share a fixture scope on the same worker via loadscope
• Running a test suite against multiple Python versions or environments in a single command using multi-interpreter distribution
• Fast local development feedback loops using --looponfail to rerun only failing tests as files change

Under The Hood

Architecture - pytest-xdist implements a controller/worker model on top of execnet: the main pytest process (the DSession in dsession.py) collects tests, then spawns worker subprocesses (remote.py) that each import pytest and execute an assigned slice of the test IDs, streaming results back over an execnet gateway. workermanage.py tracks worker lifecycle (spawn, restart on crash, shutdown), while the scheduler/ package (load.py, loadscope.py, loadfile.py, loadgroup.py, worksteal.py, each.py) implements pluggable algorithms for deciding which test IDs go to which worker, so parallelism can be traded off against fixture-scope safety. looponfail.py layers a file-watching rerun loop on top of the same worker infrastructure. Tech Stack - Pure Python, built entirely as a pytest plugin registered via pytest11 entry points (xdist.plugin, xdist.looponfail); its only runtime dependencies are execnet>=2.1 for inter-process communication and pytest>=7.0, with filelock, psutil, and setproctitle as optional extras used for coordination and process naming. Packaging uses setuptools with setuptools-scm for version derivation from git tags. Code Quality - The testing/ directory contains a substantial suite (test_dsession.py, test_workermanage.py, test_remote.py, test_looponfail.py, test_newhooks.py, test_plugin.py) exercising the controller/worker protocol, scheduling logic, and CLI option parsing; the project uses type hints throughout (from __future__ import annotations, TYPE_CHECKING imports) and enforces black formatting. Error handling favors explicit warnings.warn for recoverable misconfiguration (e.g. invalid PYTEST_XDIST_AUTO_NUM_WORKERS values) over silent failure. API Design - The plugin’s public surface is a small set of well-documented CLI flags (-n, --dist, --tx, --looponfail) rather than an importable API, which keeps day-to-day usage to a one-line change in CI config; the custom pytest hooks it exposes (newhooks.py) are intended for advanced users writing their own distribution logic and are clearly separated from the common-path flags.