tl;dr; I’ve implemented an initial version of #[async] and await! and it’s going great, I’d love to help keep this discussion moving forward to get this into nightly compilers for others to easily provide feedback for.
Hello everyone! It seems the discussion here has sort of stalled but I wanted to see if I could help breathe some new life into it. I’ve been thinking recently how async/await syntax would potentially massively help the usage of the futures crate, and the stackless coroutines/generators that we’ve been talking about here are absolutely perfect for this! In that sense I was curious what was needed for an MVP here. Something that doesn’t do 100%, may still have some unresolved questions as it’s unstable, but is solid enough to experiment with.
One of the first things I noticed is that in @vadimcn’s excellent summary of the contentious points on his RFC most of those don’t actually apply to an async/await MVP! Recall that the points here are about couroutines, not async/await, but to review each in summary:
- asynchronous streams - while relevant for a full implementation, we can likely punt on this for a “let’s get our feet wet” scenario. This should definitely be considered before stabilization, but probably doesn’t impact the core async/await system much.
- the self-borrowing problem - ok this is a big issue, but for now my thinking is that an “MVP quality async/await implementation” punts on this entirely. The compiler errors on any borrows active across yield points.
- traits - doesn’t actually matter for async/await! No matter the trait construction for coroutines/generators, async/await will have a translation for it both ways.
- declaration syntax - also doesn’t matter for async/await! Users would only interact with async/await, not the underlying coroutine/generator (in theory)
- top level generator functions - like above, highly relevant for coroutines but less so for async/await if async/await just uses coroutines as a building block
- coroutines vs generators - as you can imagine, doesn’t matter too much for async/await! The important part is compiler-generated state machines which is what’s happening here.
So given all the reasons that RFC 1823 was closed it turns out that most of them don’t end up applying to an MVP of async/await! This got me thinking how difficult it would be to prototype an implementation of this RFC just to see how far we can get with an async/await MVP. The intention here was to just toy around with async/await, see how much it can benefit, and use it as data to evaluate how to best move forward with the coroutines/generators story.
It turns out that @Zoxc was already way ahead of me and has already implemented most of generators! @Zoxc’s branch I learned recently already implements a ton of generators. The implementation is not the one described in RFC 1823, but is similar in some respects. The takeaway that I had from this though was that @Zoxc’s done some awesome leg work in implementing the bare fundamentals of generators (e.g. MIR translation, type checking, etc). Much of this work should be usable in any implementation of couroutines/generators, and I was quite eager to start developing async/await almost immediately on top of this!
I worked with @Zoxc to help identify some remaining ICEs on his branch and have resulted in an initial version of an async/await implementation for Rust. This implementation only works on @Zoxc’s branch (e.g. requires generators) and uses the proc_macro feature to implement #[async] and uses other unstable features like conservative_impl_trait and use_extern_macros for some zero-cost and ergonomic goodness. You can find more info about this in the README.
Ok, so that’s a lot to digest! Where to go next? From what I’ve learned so far I’ve concluded:
- Generators/coroutines work perfectly for async/await, and async/await in a “usable” form is possible even today with that implemented!
- @Zoxc’s branch looks to be a pretty solid implementation of generators. I haven’t done a lot of correctness testing yet, but I also don’t have any known bugs. I’ve ported a good chunk of sccache to async/await and it’s still compiling LLVM correctly though!
- The attainability of a “production ready” implementation of async/await seems to be much nearer than originally thought. I’m thinking that this may change the calculus around decision making here.
So in general I’d like to help spur along discussion here and see if we can reach a consensus on how to move forward. My focus here is mainly on getting an initial implementation of generators/coroutines landed in the compiler on nightly. Now this is a pretty major feature, and landing something in the compiler ends up having a lot of inertia, so we need to be sure to tread carefully. I’m hoping, though, that we can get all stakeholders on board.
My thinking is that we can land, likely as a form of @Zoxc’s branch, an implementation of generators which is not set in stone but has the major design decisions taken care of. For example the type checking and methodology of defining a generator/coroutine would probably want to be mostly hammered down and agreed on, but the precise syntax and/or extra syntax sugar can probably wait for later. Another example of this is that APIs as they relate to libstd can probably mostly be glossed over at this stage. I’d naively assume at least that most compiler implementations would be easily translateable to “similar traits” or similar protocols for invocation, so whether we call a method foo or bar probably isn’t so relevant for an initial implementation.
So in order to move forward, here’s what I think we should do:
- Identify key features in @vadimcn’s RFC 1823 that block landing an initial implementation.
- Identify key differences between @Zoxc’s branch and @vadimcn’s RFC, working towards a resolution here
- Work with the compiler team to ensure it’s ok to move forward here
- Start hacking away at async/await caveats!
What do others think? Am I trying to push this way too quickly? Are we still far away from consensus to put something in the compiler? Curious to hear others’ opinions!