pyo3-asyncio

pyo3-asyncio is a Rust crate built on top of PyO3 that bridges Rust’s async/await model with Python’s asyncio event loop. Because Rust futures and Python coroutines are driven by two independent event loops, the crate provides explicit conversions in both directions: into_future turns a Python awaitable into a Rust Future, while runtime-specific future_into_py functions turn a Rust Future into a Python awaitable that asyncio code can await normally.

Rather than caching a single global reference to the Python event loop, pyo3-asyncio tracks the active loop and contextvars context explicitly through a TaskLocals struct passed at each conversion site, a design change from earlier versions made specifically to keep the crate correct inside libraries that don’t control how or how many event loops the host application creates. It ships adapters for both tokio and async-std, attribute macros that wire up interpreter initialization and main-thread handoff, and a custom pytest-style test harness for exercising async Rust/Python interop in CI.

What You Get

• Future/Coroutine Conversion - Converts Python awaitables into Rust Futures via into_future and Rust Futures into Python awaitables via future_into_py, in both directions.
• Dual Runtime Support - Ships first-class tokio and async_std modules with matching APIs so the same conversion pattern works with either Rust async runtime.
• Explicit TaskLocals Tracking - Uses a TaskLocals struct to carry the Python event loop and copied contextvars context through conversions instead of a global reference, so it works correctly inside libraries with multiple event loops.
• Attribute Macros for Entry Points - Provides #[pyo3_asyncio::tokio::main] and #[pyo3_asyncio::async_std::main] proc-macro attributes that initialize the Python interpreter and run an async main on Python’s main thread without manual boilerplate.
• Pytest-Style Rust Test Harness - Ships a custom testing module (backed by inventory and clap) that registers Rust tests against a live Python asyncio loop, replacing Cargo’s default test harness which can’t share the main thread with Python.
• Unstable Async Streams - Behind the unstable-streams feature flag, exposes conversions between Rust Streams and Python async generators via async-channel.

Common Use Cases

• Wrapping an async Python library from a Rust CLI - A Rust application that depends on a Python library exposing only async functions uses into_future to await those coroutines directly inside a Tokio runtime.
• Writing native PyO3 extension modules with async functions - A developer building a cdylib Python extension in Rust exposes #[pyfunction]s that return awaitables via future_into_py, letting Python callers await Rust-side async work like disk or network I/O.
• Bridging two event loops in a hybrid application - An application that needs Python’s asyncio for signal handling and CTRL-C behavior on the main thread while running the bulk of its logic on a background Tokio runtime uses this crate’s TaskLocals scoping to pass data between the two.
• Testing native PyO3 async bindings end-to-end - A maintainer of a PyO3 extension uses the crate’s testing module and #[pyo3_asyncio::tokio::test] macro to run integration tests that spin up both a Tokio runtime and a real asyncio loop per test case.

Under The Hood

Architecture The crate splits into a runtime-agnostic core (generic.rs, defining Runtime, ContextExt, SpawnLocalExt traits and the future/stream conversion machinery) plus two thin adapters, tokio.rs and async_std.rs, that implement those traits over Tokio’s Runtime/task API and async-std’s task API respectively. A companion proc-macro crate (pyo3-asyncio-macros) generates the #[tokio::main]/#[async_std::main]-style entry points and registers #[tokio::test]-style cases with the testing module, which itself uses inventory to collect tests at link time and clap to filter them at runtime. Core conversions in lib.rs route through the TaskLocals struct rather than a cached global loop reference, a deliberate correctness tradeoff documented at length in the crate’s own module docs. Because every runtime adapter and every macro-generated entry point is built directly against the generic::Runtime trait and TaskLocals, changes to either would ripple through the whole crate.

Tech Stack A Rust crate (2021 edition) built on pyo3 0.20 for the CPython FFI layer, with futures 0.3 and pin-project-lite providing the future/stream plumbing and once_cell for lazily-initialized statics. tokio (rt, rt-multi-thread, time features) and async-std (unstable feature) are both optional, feature-gated runtime backends, with async-channel gating the unstable-streams feature and clap/inventory gating the testing feature. The companion pyo3-asyncio-macros workspace member implements the attribute macros. Tests run via a custom harness (harness = false in Cargo.toml) since Python’s asyncio requires control of the main thread. CI runs rustfmt and black formatting checks plus cargo clippy, and the crate publishes to crates.io with docs on docs.rs.

Code Quality No unsafe code appears anywhere in the main crate’s src/, and the macros crate forbids it outright (#![forbid(unsafe_code, future_incompatible, rust_2018_idioms)]). Testing is unconventional but deliberate: nine [[test]] entries in Cargo.toml run through the crate’s own inventory+clap-based harness rather than Cargo’s default, since each test needs a live Python interpreter and asyncio loop alongside a Rust runtime. There’s no separate unit-test module for pure-Rust logic; coverage instead comes from these integration-style tests exercising every runtime/asyncio-implementation combination. Error handling is minimal but explicit, with a small err.rs defining a RustPanic type that’s propagated through PyO3’s PyErr. Naming is consistent throughout (snake_case, a _with_locals suffix convention for the explicit-locals variants of each conversion function), and #![warn(missing_docs)] is set on the main crate.

API Design The headline design decision — passing TaskLocals explicitly instead of relying on a single cached event-loop reference — directly targets a real problem the crate outgrew in its own v0.13: a global reference breaks down for library code that doesn’t control how many event loops the host application creates. The per-runtime modules mirror each other closely (tokio::future_into_py / async_std::future_into_py, tokio::into_future / async_std::into_future), so switching runtimes is largely a matter of swapping the module path. Attribute macros remove most of the interpreter-initialization boilerplate for application entry points. The tradeoff is a real learning curve: the README’s own primer walks through why two event loops exist, why contextvars needs special handling, and when to reach for the locals-aware variants — necessary reading before the API feels natural. The project has also seen no commits since May 2024, and newer PyO3 releases have started adding native async support that may eventually make crates like this one unnecessary.