Pre-Pre-RFC: async methods & bounding async fns

[scottmcm]

I went back to the RFC to look at the interior/exterior type discussion: https://github.com/rust-lang/rfcs/blob/master/text/2394-async_await.md#the-return-type-t-instead-of-impl-futureoutput--t

Lifetime Elision: Now that we’re talking about 'in, what about just changing the lifetime elision rules for async functions? I don’t think that makes async any harder to learn, since it’s still needs to capture those lifetimes anyway.

Polymorphic Return: Rather than “a non-factor for us”, this thread reads to me like it is a factor, just that the polymorphism is on additional trait qualifications rather than the carrier type. (Personally, I really don’t like the look of async(Copy).) And the RFC does mention three different uses of a polymorphic return today, just arguing that they won’t be needed eventually.

Learnability: The RFC says there are arguments “in favor of both”.

Also, with trait aliases, couldn’t something like trait Async<T = ()> = Future<Output = T>; resolve a bunch of the problems? Like async fn foo() -> impl Async to return (), and async fn foo() -> impl Async<i32> to actually return something.

I also liked the questions about combining async with try, though I don’t have answers here. It feels like there are situations for both -> io::Result<impl Async<i32>> and -> impl Async<io::Result<i32>…

(If this has all been discussed before, please let me know; I don’t want to re-open things if I’m not adding anything. I didn’t follow all these discussions that closely.)

[vadimcn]

@withoutboats: Given all these conceptual difficulties that async fn is running into, perhaps we should revisit the idea that adding keywords will make this stuff ergonomic?

If we introduced only the bare minimum needed to support coroutines into the language and outsourced all desugaring to macros, as I’d proposed back in the day, we wouldn’t need to design a single perfect keyword that covers all use cases. The minimalistic design was much more flexible and composable.

IMO, we should introduce async functions to the users as regular functions, whose return type impl’s Future trait, and then add sugary macros like #[async] on top of that. That way noone will be confused about what’s actually going on underneath.

[Nemo157]

@vadimcn the exposed API in your proposal seems basically equivalent to the current generators API (which makes sense now that I glance at the summary again and see that generators are the same thing). Are you aware of the current #[async] macro limitations and have real ideas for fixing those that don’t require compiler support (especially the error messages)?

I also fail to see how having a macro instead of builtin syntax will lead to

That way noone will be confused about what’s actually going on underneath.

rather, someone will just be confused about what the #[async] macro is doing instead of what the compiler’s builtin is doing.

As far as I’m aware the current plan is to continue building on the same foundation of generators, by making this a builtin instead of a proc-macro there is just much more information about what the user is really doing, which can be used to better guide the transformation.

[withoutboats]

Async/await has not run into any unanticipated questions, all of these issues were known when we made the decision to focus on it. In contrast, generators have a huge number of unresolved questions, because they are more foundational, and support many more use cases. Two prominent examples are how to handle both self-referential and non-self-referential generators, and what to do about resume arguments.

Specifically because this feature is so much more narrow, the questions are also narrower, and our solutions can be more opinionated. That’s why its been able to make so much progress so fast (your impression that we’re running into difficulties is very different from mine!)

Additionally, even if async were written #[async], we’d have all of these same issues to resolve, in addition to the issues that fully general generators introduce.

[Nemo157]

@Nemo157 I’m especially interested in hearing your thoughts about this because I think in your embedded use case you’re using a single threaded executor. Are you taking advantage of it with non-Send futures?

@withoutboats I’ve stopped working on my prototype for now, there’s many more pieces around IO that need to be in place before I think I could really get what I wanted working. I at least got it to a point that I think I can see it working really nicely in the future, once those pieces are resolved. (The big two off the top of my head: no_std support for futures-io (probably just getting io::Error support via the portability initiative) and transitioning most protocol implementations from using tokio directly to being generic over futures-io traits).

I was using a single-threaded executor with no interrupts, and with no ability to spawn new futures. Everything was running under a single root future, and any concurrency had to be handled further down the tree via select/join (I had plans for scoped, pre-allocated executors as well, but those would essentially be a somewhat dynamic form of join).

I don’t think I had anywhere that would have been able to take advantage of non-Send futures. I was not consciously doing it anyway. It may be that some of the lower-level hardware abstractions I was using was non-Send, but from what I remember those were generally Send + !Sync.

Either way, I think the stuff I was doing is more likely to be using fn -> impl Future for trait definitions anyway to more tightly control the lifetimes, so missing out on the async method sugar for the cases where it could have been used isn’t a massive deal. (As long as I get the sugar for actually defining the monstrous state machines, I’m happy).

[vadimcn]

@vadimcn the exposed API in your proposal seems basically equivalent to the current generators API

Of course it is! This is where the current implementation of generators had started from.

Are you aware of the current #[async] macro limitations and have real ideas for fixing those that don’t require compiler support (especially the error messages)?

Some, but not that much, actualy. If there had been an extensive discussion of error messages, I must have missed it.

The async/await RFC just says that “we will want dedicated syntax for async functions, because it is more ergonomic & the use case is compelling and significant enough to justify it”.

Originally, my position was that #[async] on top level functions is not even desirable, as hiding the “outer” return type and construction of generator closure from users only muddies the waters and impedes learnability.

rather, someone will just be confused about what the #[async] macro is doing instead of what the compiler’s builtin is doing.

If we’d started explaining at the other end, i.e. with the Future trait, and then said “BTW, here’s a macro that generates the boring bits for you”, I think it would go over much smoother.

Async/await has not run into any unanticipated questions,

Depends on what you call a problem. If there’s no problem, then why are we talking about extending syntax with stuff like async(Send), and how are we going to have both async fn and fn ... -> impl Future in parallel, - because the former is not as expressive as regular Rust syntax?

Two prominent examples are how to handle both self-referential and non-self-referential generators, and what to do about resume arguments.

ISTM, that Unpin is just as applicable to generalized coroutines as it is to async fn's. And final design for resume arguments could have been postponed just as it’s been now.

Additionally, even if async were written #[async] , we’d have all of these same issues to resolve, in addition to the issues that fully general generators introduce.

Yes, and we’d have solved (or postponed them) in the same way.

---

I am wondering (uneasily), what's going to happen when you finally get to design other applications of coroutines such as iterators. Are we in for another bout of keyword addition, because `async` was not general enough? Are we going to have to solve the self-referentiality problem all over again?

I think that first version of async IO in Rust should be shipped with just macros as surface syntax, implementation bits staying completely in the unstable land. And I think there’s a fair chance that it’ll grow on people and they’ll realize that dedicated async/await syntax isn’t even needed in Rust.

[gbutler]

EDIT: I was having a brain-fart here. C# uses the Task whether or not the funcion is async. The async keyword simply allows one to “await” inside the function on another Task or Async function.

C# has async functions. In C#, the following is used:

public Task<int> Foo ( ... ) { ... }

For a function that returns a Task that will return an Int, but, that is itself NOT an async function that can “await” inside it (it returns the new Task immediately). Whereas:

public async Task<int> Foo ( ... ) { ... }

Is a function that may execute, using “await” an internal Task and return the resumable Task

In C#, the following:

public async int Foo ( ... ) { ... }

Is pretty much equivalent to:

public Task<int> Foo ( ... ) { ... }

The only real difference is with the former you may use the “await” keyword before the function call, whereas, for the latter you can call the away method (or similar) on the return value.

I think this pretty well correlates with using the inner return type with the async keyword and using the outer return type when not using the async keyword.

Not definitive, but, a pretty good “Prior Art” point that is going to match up with a lot of people’s expectations and experience.

[Nemo157]

@gbutler except C# uses the outer return type, and you can await the return value from either an async function, or any non-async function returning some awaitable type, the difference is that you can only use the await keyword when within an async function.

async Task<int> Foo() {
    return 5;
}

Task<int> Bar() {
    return Foo();
}

async Task<int> Baz() {
    return (await Foo()) + (await Baz());
}

EDIT: You can see some other languages that also use the outer return type in my old pre-rfc. There is one language that appears to have planned to use the inner type, Scala via SIP-22, but that’s listed as dormant so I assume it’s not implemented yet. Kotlin also uses the inner type, but it also implicitly awaits suspend functions in suspend context and disallows calling them outside it, so that makes sense while not contradicting the other languages.

[MajorBreakfast]

Here’s an article about C#'s async/await. I don’t know C#, but it definitely looks like the outer return type approach. The Rust async/await RFC mentioned this as well.

@Nemo157 Your pre-RFC is from before I started following async Rust. Thanks for posting the link!

A word of caution: Prior art is great to draw inspiration from, but we need to consider the differences when doing so (traits, abstract return types, lifetimes, laziness, etc.). The main reasons for the solution we choose should not depend on prior art, but on Rust’s needs.

[withoutboats]

Second blog post, tackling an issue we haven’t talked about yet at all: object safety.

https://gist.github.com/withoutboats/6fc5ea032166330022164dd3c12f7826

(posted as a gist for now because gitlab won’t deploy my blog update today)

[CAD97]

How does the method erased-serde use for object-safe versions of serde’s traits compare to what you describe here?

[withoutboats]

How does the method erased-serde use for object-safe versions of serde’s traits compare to what you describe here?

They’re related! They both depend on an impl of this form existing in order to bridge everything:

impl<T> $TRAIT for Box<T> where T: ?Sized + $TRAIT

However, there are some important differences. erased-serde is trying to make generics object-safe, whereas we are trying to unify the associated types of a trait. They’re different problems, and so they require different solutions. Its not the case that we could use an ErasedFuture trait to solve this problem, because when I do this, it needs to work:

fn foo<T: ?Sized + Trait>(trait: &T) {
    let f: impl Future = trait.async_method(); // This must return an impl Future
}

foo::<dyn Trait>(trait_object)

[MajorBreakfast]

I still believe that the outer return type approach would solve all our troubles. I’ve created a blog post that explains the situation how I see it. Enjoy reading!

Link to blog post

[illustrious-you]

C# does use the outer return type approach. This turns out rather useful for optimization, because C# draws a type-level distinction between reference types (class) and value types (struct). Often, a value type doesn’t require a heap allocation, and thus one can optimize an async method by returning a ValueTask.

C# differs from Rust, however, in that an async function allows duck-typing of its result. An async function can return any type implementing a visible GetAwaiter method (citation). An interface isn’t used because that would box the ValueTask. In Rust terms, returning a .Net interface is roughly equivalent to dyn Trait and duck typing is as close as .Net gets to impl Trait.

As you pointed out in your blog, the crux of the problem remains. If additional trait bounds are occasionally necessary, how should they be specified?

If the problem is limited to Send, I’d argue that the trait bound should be automatic. It’s perhaps unfortunate that it’s applied to inherent methods and free functions, but I’d argue it would be surprising if certain futures could be sent and others not. (As I understand it, Send is what allows the future to run on a multi-threaded executor–I’d expect that quality to apply to any future.)

[lordan]

That is, if we write:

fn foo() -> async i32 { .. }

Then you should also write:

fn foo() -> unsafe i32 { .. }
fn foo() -> const i32 { .. }

Coming back to this after reading @withoutboats’ blog, I don’t think these are quite equivalent:

Both unsafe and const restrict/extend how you can call the function itself, in addition to what you can do within the function body. Once it’s returned, you can do with the returned value whatever you want.

Contrary to that, async doesn’t force you to call the function in any special way, but it changes how you treat the return value (in addition to, again, what you can do inside the function body).

Inasmuch it IMHO wouldn’t be inconsistent to mark the function itself unsafe/const, but allow async on the return value.

Edit: One could argue that Rust already has unsafe (return-) values in the language, in the form of pointers:

fn foo() -> *const i32

Here, calling the function does not require unsafe, but using the return value does.

regarding async(Send)

Coming from a C++ background, this could be read a bit like a conditional async - like C++‘s noexcept(...) or the newer explicit(...) syntax (e.g., "this function is async if it’s implemented on a structure that’s Send "). Not that Id’ suggest Rust should ever go even near this direction…

[DDOtten]

I think the best approach would be to initially only add async blocks and to wait a bit before adding the async fn sugar. async blocks are the important part that make things possible and we can afford to wait a bit with async fn. This way we can get some experience with using async in rust before making a final decision about async fn. This was suggested in the reddit thread and was received well by people commenting there.

[KasMA1990]

When we don’t give programmers meaningful choices, explicitness is often a hindrance - this is where people start talking about boilerplate, when you have verbosity without control. (The classic example of boilerplate to me is Java’s public static void main - reveals a lot without ceding control.)

I would be careful to push this argument too much. I think it has merit, but I think it’s important that even when you don’t cede control, explicitness can still be the right choice. With regards to Java’s main, typing public static void main can seem like a bunch of boilerplate, but within the confines of Java, that is pretty much how I would expect main to look. I.e. it should be public, it should be static, and it should have a return type (though whether void is the right choice is debatable at least). In other words, no, Java doesn’t cede control in these aspects, but they conform quite well to my mental model of how Java works. This makes it quite easy to wrap my head around, whereas if main had been given even more special status than it already has (so you only had to type e.g. void main or main), it would require that I add a bigger special case to my mental model.

Quite a few years passed before I even learned what main really did; up till then, I wouldn’t worry about it, because it always came pre-populated when I created a new project in whatever IDE I was using. I just knew it was the entry point for my code to run from. I think the same argument applies for the outer return type here. Whenever I type async fn in my IDE, I’m going to get all the boilerplate handled for me, and then it becomes really easy to read the code later when my mental model catches up to what is actually going on.

Of course, not everyone uses an IDE, but the underlying point for me is that I want to be able to see that my code fulfills the “async contract”, in the same way that main in Java fulfills the “entry point contract”. The most common way of enforcing such contracts in Rust is of course to have users implements traits, but that would be too much boilerplate. But I would still expect any function that returns impl Future to be awaitable, because that’s the contract. But then what does async do? If it just declares that I can use await within a given block, that’s fine since it’s just a simple addition to my mental model. But if it also starts interacting with function declaration syntax, I need to start special casing how functions work in my mental model. This might be nice at first when learning, but I’m personally more likely to just ignore the details in the beginning, and appreciate having them later, than the other way around (based on prior experience).

[MajorBreakfast]

@KasMA1990 I agree. The benefit of the async fn compared to fn + async block should be that we save one indentation level, one pair of {} and get lifetime elision. Everything else should be the same. No “special casing how functions work”. Exactly!

[DDOtten]

Another reason i like the impl Future<Ouput = T> aproach is that it allows us to us to name the return type for free using existential type. Without this we would have to come up with additional syntax for what seems like a subset of the larger problem to me.

We also need extra syntax for specifying the additional trait bounds of the return type and I think that we will keep running in to situations like this.

[withoutboats]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.