How to call Rust async method from Python?

Question

I want to use a Rust async method in Python. I'm trying to use PyO3 or rust-cpython.

For example, for sync Rust functions, I can use,

#[pyfunction]
fn myfunc(a: String) -> PyResult<String> {
   let mut contents = String::new();
   contents = a.to_string() + " appended";
   Ok((contents))
}

#[pymodule]
fn MyModule(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(urlshot))?;
    Ok(())
}

For async methods, how can I do it? For example, I want to call the following method in Python,

async fn hello_world() {
    println!("hello, world!");
}

To be able to await an async function, you must be inside an async function (defined with `async def` in Python). Who is calling **that** function in the use case you envision? You won't be able to mix&match Python asyncio and Rust tokio (or others), but if your Python async function is itself awaited from async Rust, awaiting a Rust async fn should at least be possible. — user4815162342, Jun 28 '20 at 16:45

Hotte Shen · Accepted Answer · 2020-06-28T16:12:43.497

Since there was no easy way of solving this issue (at least, I hadn't found), I converted my async method to sync one. And called it on Python side as,

async fn my_method(s: &str) -> Result<String, Error> {
    // do something
}

#[pyfunction]
fn my_sync_method(s: String) -> PyResult<String> {
    let mut rt = tokio::runtime::Runtime::new().unwrap();
    let mut contents = String::new();
    rt.block_on(async {
        result = format!("{}", my_sync_method(&s).await.unwrap()).to_string();
    });
   Ok((result))
}

#[pymodule]
fn MyModule(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(my_sync_method))?;
    Ok(())
}

Edited

In the Cargo.toml file, I added the following dependencies,


[dependencies.pyo3]
git = "https://github.com/PyO3/pyo3"
features = ["extension-module"]

After running cargo build --release, target/release/libMyModule.so binary file is generated. Rename it as MyModule.so and it now can be imported from Python.

import MyModule
result = MyModule.my_sync_method("hello")

Using setuptools-rust, I could bundle it as an ordinary Python package.

All of the above code and commands are tested on newly-released Linux Mint 20. On MacOS, the binary file will be libMyModule.dylib.

Would help others if you would post your Python code also which calls these functions! — ruohola, Jun 28 '20 at 14:45
@ruohola I added some more instructions. If you wanted a fully working sample, let me know. — Hotte Shen, Jun 28 '20 at 16:14
To future readers: if you take the dependencies above, you should write `branch = "main"` under the link to the pyo3 repository. Looks like they've changed the branch name, and rust still defaults to assuming the branch is master. — Kraigolas, Apr 10 '21 at 20:01

whilrun · Answer 2 · 2020-06-28T10:35:05.250

4

If you want to use Python to control Rust's async function, I don't think it will work (Or at least it is very complicated, as you need to connect two different future mechanism). For async functions, Rust compiler will maintain a state machine to manage the coroutines run correctly under await's control. This is an internal state of Rust applications and Python cannot touch it. Similarly Python interpreter also has a state machine that cannot be touched by Rust.

I do found this topic about how to export an async function using FFI. The main idea is to wrap the async in a BoxFuture and let C control the timing of returning it to Rust. However, you cannot use BoxFuture in PyO3 since its pyfunction macro cannot convert a function returns BoxFuture to a Python callback. You may try to create a library using FFI and use python's cffi module to load it.

edited Jun 28 '20 at 10:35

answered Jun 28 '20 at 10:20

whilrun

979
5
19

Thank you. I also found [Python Async/Await Interface](https://docs.rs/pyo3/0.2.5/pyo3/class/async/index.html) in PyO3 documentation. Could you explain what this class is for? – Hotte Shen Jun 28 '20 at 12:45
@HotteShen This interface is to coopearate with Python's state machine. If you read the Python's document you will find that these functions need to return an `awaitable` object to Python. You know that Python uses duck type, so as long as you can satisfy these structure's requirement Python can use your extension function with await. But it does not mean you can just return a Rust's async function, since their implementations of future are different. – whilrun Jun 28 '20 at 17:16

How to call Rust async method from Python?

2 Answers2

Edited

Linked