# Help test async/await/generators/coroutines!

**URL:** <https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835>\
**Category:** announcements\
**Created:** [August 30, 2017, 12:57am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835 "2017-08-30T00:57:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\
**Post date:** [August 30, 2017, 12:57am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/1 "2017-08-30T00:57:26Z")

</div>

Good evening/morning Rustlers!

The saga of generators that started [long](https://github.com/rust-lang/rfcs/pull/1823) long [ago](https://github.com/rust-lang/rfcs/pull/1832) recently made progress as [we concluded](https://github.com/rust-lang/rfcs/pull/2033) that we should land an implementation in the compiler and start experimenting (but not stabilizing). Even _more_ recently the heroic efforts of @Zoxc and many others has culminated in an [epic PR to implement coroutines in rustc](https://github.com/rust-lang/rust/pull/43076). Now that we’ve made it this far, we need _your_ help to go farther!

The purpose of the [eRFC for coroutines](https://github.com/rust-lang/rfcs/pull/2033) was to provide the ability for the Rust community to test out generators on the nightly channel of the compiler. This feedback and testing will be the driving force for stabilization for this language feature and it’s absolutely essential!

One of the main driving factors for implementing coroutines has been [async/await syntax for the `futures` crate](https://github.com/alexcrichton/futures-await), and so we’re also very eager to have you test that out as well! You’ll not only exercise the `generators` language feature but also `conservative_impl_trait` and the fledgling `proc_macro` “macros 2.0” ecosystem. We’re very eager to get feedback on _all_ of these features as we approach stabilization!

If you’d like to get started here’s some helpful links:

- [Nightly documentation for generators](https://doc.rust-lang.org/nightly/unstable-book/language-features/generators.html)
- [async/await introduction](https://github.com/alexcrichton/futures-await)
- [Generator examples in the form of tests](https://github.com/rust-lang/rust/tree/master/src/test/run-pass/generator)

I’m quite eager to hear everyone’s feedback on all these features and goodies, please give them a spin and let us know how it goes! The next step for generators will be an RFC for stabilization, but we’re still aways away from that. With your help, though, we can try to approach it more quickly!

---

<div class="post-metadata">

**Author:** ![vadimcn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/vadimcn/32/2789_2.png) [@vadimcn](https://internals.rust-lang.org/u/vadimcn)\
**Post date:** [August 30, 2017, 1:14am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/2 "2017-08-30T01:14:53Z")

</div>

We salute you @Zoxc! You actually did it!

---

<div class="post-metadata">

**Author:** ![rozaliev](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@rozaliev](https://internals.rust-lang.org/u/rozaliev)\
**Post date:** [August 30, 2017, 1:29am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/3 "2017-08-30T01:29:36Z")

</div>

❤ Great job guys!

I’m personally very interested in blocking style APIs, but given that self-referential types are yet to come I’ll have to come up with a different design. I wonder if anyone has a good idea on how to approach it.

The problem with blocking style API is that we basically want this:

```rust
impl TcpStream {
    fn read(&self, buf: &mut [u8]) -> Async<io::Result<usize>> { ... } 
}

```

But it’s not yet possible to make it work. Any ideas?

---

<div class="post-metadata">

**Author:** ![rpjohnst](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/rpjohnst/32/9524_2.png) [@rpjohnst](https://internals.rust-lang.org/u/rpjohnst)\
**Post date:** [August 30, 2017, 2:47am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/4 "2017-08-30T02:47:32Z")

</div>

What do you mean by “it’s not yet possible to make it work”? That looks to me like exactly what `futures-rs` enables, with this new development making the method body nicer.

---

<div class="post-metadata">

**Author:** ![bestouff](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bestouff/32/2595_2.png) [@bestouff](https://internals.rust-lang.org/u/bestouff)\
**Post date:** [August 30, 2017, 3:18am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/5 "2017-08-30T03:18:32Z")

</div>

I guess it’s because of the references in the arguments - impossible for now.

---

<div class="post-metadata">

**Author:** ![rozaliev](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@rozaliev](https://internals.rust-lang.org/u/rozaliev)\
**Post date:** [August 30, 2017, 3:19am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/6 "2017-08-30T03:19:14Z")

</div>

Not exactly, while current iteration of futures-await helps with using futures and allows you to write less spaghetti code, you are still limited. For example:

```rust
fn handle_conn(mut conn: Conn) -> impl Async<()> {
    let mut buf = [0; 1024];

    loop {
        let n = await!(conn.read(&mut buf)).unwrap();
        if n == 0 {
            return;
        }
        await!(conn.write_all(&buf[0..n]));
    }
}

```

(Where `Async<T> is Generator<Yield=T, Return=()>`) It is not possible to make this example compile. The problem here is with &mut buf and &mut Conn. After the generator is transformed into a state machine it will have both Conn and &mut Conn as its struct fields, the same for buf. We don’t have self-referential types yet, so it won’t compile.

The ideal API for me is smth like this: [https://github.com/rozaliev/async-await-rs/blob/master/src/apps/echo.rs](https://github.com/rozaliev/async-await-rs/blob/master/src/apps/echo.rs)

---

<div class="post-metadata">

**Author:** ![rpjohnst](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/rpjohnst/32/9524_2.png) [@rpjohnst](https://internals.rust-lang.org/u/rpjohnst)\
**Post date:** [August 30, 2017, 3:54am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/7 "2017-08-30T03:54:39Z")

</div>

Ah, yes, you’re completely right. This is a pretty long-standing issue, with lots of additional use cases. My favorite solution is described in @Zoxc’s [immovable types RFC](https://github.com/rust-lang/rfcs/pull/1858). That would enable generators that take pointers to their local state and keep them across suspension points, while still allowing them to be moved around by value (as in `-> impl Future`) before they are first run.

---

<div class="post-metadata">

**Author:** ![vadimcn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/vadimcn/32/2789_2.png) [@vadimcn](https://internals.rust-lang.org/u/vadimcn)\
**Post date:** [August 30, 2017, 4:43am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/8 "2017-08-30T04:43:51Z")

</div>

> [@rpjohnst](#):
>
> My favorite solution is described in @Zoxc’s immovable types RFC.

Another solution might be to use nonblocking IO, as I tried to demonstrate [here](https://github.com/vadimcn/rfcs/blob/coroutines2/text/0000-coroutines.md#asynchronous-io).  
(reader's fd must be put into non-blocking mode before calling copy(), of course).

---

<div class="post-metadata">

**Author:** ![ishitatsuyuki](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ishitatsuyuki/32/2163_2.png) [@ishitatsuyuki](https://internals.rust-lang.org/u/ishitatsuyuki)\
**Post date:** [August 30, 2017, 6:27am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/9 "2017-08-30T06:27:24Z")

</div>

My experience with `impl Trait` so far is that the compile time is insane, possibly due to a bug not being resolved for a while. My API client is still taking 5 minutes to compile on nightly.

---

<div class="post-metadata">

**Author:** ![mehcode](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mehcode/32/2144_2.png) [@mehcode](https://internals.rust-lang.org/u/mehcode)\
**Post date:** [August 30, 2017, 8:53am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/10 "2017-08-30T08:53:49Z")

</div>

❤ Amazing job guys.

* * *

I’ve implemented `#[async]` / `await!` in [Shio](https://github.com/mehcode/shio-rs)’s examples and saying it’s significantly better feels like the understatement of the year. Compare the postgres example on the [master branch](https://github.com/mehcode/shio-rs/blob/master/examples/postgres/src/main.rs#L42) with the [await branch](https://github.com/mehcode/shio-rs/blob/await/examples/postgres/src/main.rs#L46).

---

<div class="post-metadata">

**Author:** ![bugaevc](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bugaevc/32/13651_2.png) [@bugaevc](https://internals.rust-lang.org/u/bugaevc)\
**Post date:** [August 30, 2017, 10:09am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/11 "2017-08-30T10:09:52Z")

</div>

`<good-part>`

This is really _awesome_. I wasn’t expecting this to be implemented for another year or so 😄 Hopefully this will make Rust/Tokio much more feasible for real-life production usage.

My favorite _recursive_ Fibonacci sequence example [works fine](https://play.rust-lang.org/?gist=196bd8e2d8994aee305f1e44d33fb12b&version=nightly) (yay!)

`</good-part> <bad-part>`

But there seems to be some problem with generator objects size calculation when the generator invokes itself. Naturally, a generator cannot store a copy of itself on the stack across yield points (that would make its size infinite), but it should work fine to store itself unboxed _between_ yield points (this works) and to store itself boxed _across_ yield points (this causes `rustc` to overflow its stack).

And when I try to cast the boxed-self to a boxed trait object, `rustc` reports a weird lifetime error.

[Playground link](https://play.rust-lang.org/?gist=eeafbf668c4d107e5143fbb74fda9fff&version=nightly)

`</bad-part>`

---

<div class="post-metadata">

**Author:** ![dwrensha](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/dwrensha/32/371_2.png) [@dwrensha](https://internals.rust-lang.org/u/dwrensha)\
**Post date:** [August 30, 2017, 11:44am UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/12 "2017-08-30T11:44:22Z")

</div>

> But there seems to be some problem with generator objects size calculation when the generator invokes itself.

Sounds related to [impl trait: rustc stack overflow · Issue #35706 · rust-lang/rust · GitHub](https://github.com/rust-lang/rust/issues/35706)

---

<div class="post-metadata">

**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\
**Post date:** [August 30, 2017, 1:55pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/13 "2017-08-30T13:55:40Z")

</div>

> [@ishitatsuyuki](#):
>
> My API client is still taking 5 minutes to compile on nightly.

Oh dear sounds bad! Do you know if there's a bug filed for this against rust-lang/rust?

> [@bugavec](#):
>
> And when I try to cast the boxed-self to a boxed trait object, rustc reports a weird lifetime error.

Thanks for the report! I'll make sure we track the bug @dwrensha cc'd

---

<div class="post-metadata">

**Author:** ![ishitatsuyuki](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ishitatsuyuki/32/2163_2.png) [@ishitatsuyuki](https://internals.rust-lang.org/u/ishitatsuyuki)\
**Post date:** [August 30, 2017, 2:09pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/14 "2017-08-30T14:09:20Z")

</div>

> [@alexcrichton](#):
>
> Oh dear sounds bad! Do you know if there’s a bug filed for this against rust-lang/rust?

AFAIK #43787 is suspected, but I haven't profiled to determine the cause.

An interesting thing about this is that the code compiles in 20 seconds for the lib, but insanely long for any binary that uses it (tests, CLI bin).

---

<div class="post-metadata">

**Author:** ![sfackler](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sfackler/32/9_2.png) [@sfackler](https://internals.rust-lang.org/u/sfackler)\
**Post date:** [August 30, 2017, 4:03pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/15 "2017-08-30T16:03:28Z")

</div>

Probably [https://github.com/rust-lang/rust/issues/38528](https://github.com/rust-lang/rust/issues/38528).

---

<div class="post-metadata">

**Author:** ![mbrubeck](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mbrubeck/32/174_2.png) [@mbrubeck](https://internals.rust-lang.org/u/mbrubeck)\
**Post date:** [August 30, 2017, 8:40pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/16 "2017-08-30T20:40:42Z")

</div>

I think using generators to implement iterators will be a huge benefit. For example [this playground](https://play.rust-lang.org/?gist=7f8d2960361237403fe4d19161c00731&version=nightly) shows an example of an iterator that is very simple to implement with a generator, but otherwise somewhat complex and/or unsafe.

The eRFC [discusses](https://github.com/rust-lang/rfcs/blob/edbb8219138b426749724cdda7bae63ef73e1c0b/text/2033-experimental-coroutines.md#open-questions---coroutines) using wrapper types to convert generators to iterators. (That’s also the approach used in my playground above.) It mentions that wrapper types may be necessary because a generic `impl<G: Generator> Iterator for G` could cause coherence errors. But as an alternative to wrapper types, would it be possible for the compiler to automatically generate both `Iterator` and `Generator` implementations for each generator literal?

---

<div class="post-metadata">

**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\
**Post date:** [August 30, 2017, 9:11pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/17 "2017-08-30T21:11:43Z")

</div>

One could likewise hope for `impl<I: Iterator> Generator for I` – could specialization allow us to implement a mutual relationship like this?

---

<div class="post-metadata">

**Author:** ![leonardo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/leonardo/32/1344_2.png) [@leonardo](https://internals.rust-lang.org/u/leonardo)\
**Post date:** [August 30, 2017, 9:14pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/18 "2017-08-30T21:14:24Z")

</div>

From my preliminary tests I’ve seen that a generator-converted-to-iterator in some cases is much faster (and leads to cleaner code) than Iterator::scan().

---

<div class="post-metadata">

**Author:** ![leonardo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/leonardo/32/1344_2.png) [@leonardo](https://internals.rust-lang.org/u/leonardo)\
**Post date:** [August 30, 2017, 9:24pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/19 "2017-08-30T21:24:02Z")

</div>

> [@mbrubeck](#):
>
> I think using generators to implement iterators will be a huge benefit.

But I think performance of such iterators-from-generators still needs to be assessed.

---

<div class="post-metadata">

**Author:** ![leonardo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/leonardo/32/1344_2.png) [@leonardo](https://internals.rust-lang.org/u/leonardo)\
**Post date:** [August 30, 2017, 11:51pm UTC](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835/20 "2017-08-30T23:51:06Z")

</div>

> [@bugaevc](#):
>
> My favorite recursive Fibonacci sequence example works fine (yay!)

Can you use that on this function too?

```
// pairwise(1 2 3) => (1, 1) (1, 2) (1, 3) (2, 2) (2, 3) (3, 3)
fn pairwise<'a, T>(items: &'a [T]) -> impl Iterator<Item=(&'a T, &'a T)> + 'a {
    items
    .iter()
    .enumerate()
    .flat_map(move |(i, x1)| items[i ..]
                             .iter()
                             .map(move |x2| (x1, x2)))
}

```

[Next page](https://internals.rust-lang.org/t/help-test-async-await-generators-coroutines/5835.md?page=2)