Async/Await series

[withoutboats]

Making a thread to track this entire blog series. I wrote five posts last month which can be found on my blog. Here’s the newest one, posted today:

Async/Await VI: 6 weeks of great progress

[smmalis37]

Will the new Pin APIs allow us to make custom self-referential structs? Or are they meant only for compiler-generated generators?

[withoutboats]

You can create them but only using raw pointers & unsafe code. This makes it safe to operate on them, but its still very unsafe to define them.

[jpernst]

Sometime in the near future I’ll be exploring the utility of the Pin API for general self-ref structs via rental. It may make the situation slightly more ergonomic in some cases, but in the end, creating a self-ref struct is still quite unsafe and will require the use of helper crates to avoid the unsafety.

[alkis]

Awesome post, much like the other five. Thanks for taking the time to write this down. Not only it increases visibility to the larger community but it also creates a trail we can get back to in the future if one wants to investigate how Rust’s async APIs shaped up and why.

One question: where do the design discussions happen? How can one participate?

[Mr_Byte]

The link to the RFC for immovable types links directly to the RFC repo, instead of the RFC itself.

[matthieum]

One little note: !Unpin.

I think I understand the motivation for the particular choice of semantics (it’s not possible to auto derive !Pin) however I always find it difficult to wrap my head around double negatives (not-not-pin, so it’s pin, right?).

I cannot help but hope that an affirmative name can be found in time. Maybe… Movable?

[cramertj]

(not-not-pin, so it’s pin, right?).

It might be helpful to think of Unpin as “this type can be unpinned” rather than “this type cannot be pinned.” !Unpin then means “this type cannot be unpinned” rather than “this type cannot not be pinned”, which reads like nonsense

[aturon]

That said, I agree that Unpin is an awkward name.

[seanmonstar]

I’m very excited that we can soon await while having borrowed some arguments!

fn poll(self: Pin<Self>, ctx: &mut task::Context)

Is it possible to not modify Future to need to take Pin<Self>? It feels like a leaked abstraction, that if I’m implementing Future, I need know something about pins, when its expected that almost everything will not need to be pinned. Particularly, nothing else is going to need to take Pin<Self>, so it sticks out to me.

Could the pinning not be dealt with at the await keyword/macro, wrapping the expression in a pinned type that implements Future for it?

[Nemo157]

Could the pinning not be dealt with at the await keyword/macro, wrapping the expression in a pinned type that implements Future for it?

Note that not all immovable futures are defined via async/await. Just today I manually implemented a StableStream that uses an internal buffer written to via DMA from a radio peripheral. It’s so nice to be able to avoid the standard static global/externally allocated buffer implementation that’s necessary to ensure safety.

[borealis]

One challenge of the current async situation is that any function annotated with #[async] returns impl Future which is an unnameable type. This is somewhat viral in nature: any other Future which uses it also has an unnameable type. And the only escape hatch, is to put it into a Box<Future>. So libraries can’t use #[async] without forcing users box up futures returned from the library. The end result seems to be that #[async] provides nice benefits for a library author, but by using it, the library provides futures which are less useful to client - a strong incentive not to use it. Authors of client code who use #[async] are also in an awkward position - on the one hand, libraries that avoid #[async] are more useful, on the other hand, if the client author wants to dig into the library’s code they have to read futures written in a somewhat different style than they may be accustomed to.

I thought I read something a while ago about plans to make impl Future return types nameable and I was wondering if / how that fit into the current await plans?

[Ixrec]

If I understand all the impl Trait-related RFCs correctly, the current answer to “making impl Trait types nameable” is https://github.com/rust-lang/rfcs/pull/2071 modulo a fresh syntax bikeshed when this is all closer to stabilization

[withoutboats]

Is it possible to not modify Future to need to take Pin<Self>? It feels like a leaked abstraction, that if I’m implementing Future, I need know something about pins, when its expected that almost everything will not need to be pinned. Particularly, nothing else is going to need to take Pin<Self>, so it sticks out to me.

It is not possible if you want to be able to treat the return type of an async fn as impl Future, because that means it has to be pollable, but polling it advances its state, creating self-references. And we certainly want the return type to be a future, so we can call combinators on it without heap allocating it.

The key point is that Pin is isomorphic to &mut if the type implements Unpin. So for manual futures the change is fairly insignificant (this is discussed more in the post).

[seanmonstar]

It is not possible if you want to be able to treat the return type of an async fn as impl Future

I meant that the wrapped value would still implement Future, just that the details that it needs to be pinned would be specific to that one future, not to all manual implementations of Future. Such that an async function state machine becomes:

struct __AsyncGenerated {
    inner: Pin<__State>,
}

impl Future for __AsyncGenerated {
    fn poll(&mut self) {
        // the state is pinned, and we deal with the pin in here, because we have to
    }
}

The key point is that Pin is isomorphic to &mut if the type implements Unpin. So for manual futures the change is fairly insignificant

Oh sure, but my thoughts were more that seeing self: Pin<Self> at all, even if I’m not writing a self-referential future, seems like leaking the Pin/Unpin concept.

I find it similar to if poll required self: Arc<Self>, or something. I’d be left wondering… “huh? why?”

[seanmonstar]

I’ve since reasoned through why my suggestion of wrapping a Pin cannot work: you can’t move around the wrapper that has a pin, you need to pin at the outermost future, and forward the pin through.

I still think it’s leaky to have to receive Pin<Self> universally in every single manual Future implementation, and so hope to still find a way to not have to do so.

[twmb]

This work is very nifty, but all of it combined with the futures-await crate is still a bit over my head. I’m looking forward to a how-to to show up in https://aturon.github.io/apr/async-in-rust/chapter.html.

[withoutboats]

I think another way to state what you’ve realized is that the Pin type and the PinBox type both indirect through pointers. You can’t just have a Pin newtype - moving that moves the inner value. There’s no way to have the functions return a pointer unless they heap allocate every time, which is untenable.

We’ve explored a lot of different API designs here - at least a dozen - and I’m pretty confident that you can’t get less “leaky” than this API while maintaining the performance properties we care about.

[Nemo157]

If I understand all the impl Trait-related RFCs correctly, the current answer to “making impl Trait types nameable” is https://github.com/rust-lang/rfcs/pull/2071 modulo a fresh syntax bikeshed when this is all closer to stabilization

Unfortunately the current #[async] macro does not support this, there is nowhere to set the name that you want to use instead of impl Future. Hopefully this can become possible in the future though.

[kirillkh]

Please forgive my ignorance, but I wonder how this compares to Kotlin’s coroutines. Has anyone looked at those from Rust community? I successfully used Kotlin coroutines in production and can attest first-hand that they make writing nonblocking code so incredibly easy it feels like magic.