Post: streams concurrency

[yoshuawuyts]

https://blog.yoshuawuyts.com/streams-concurrency/

Hi all; here's (hopefully) my last blog post of the year, in which we're talking about streams concurrency. This has been in my drafts for what feels like 6 months now, and I'm glad to finally have published it!

This captures some of the thinking that has gone into async-std's design for parallel stream processing, some of the directions we're currently exploring, and how that might end up back into the language eventually (which is why I'm sharing this on internals ) . A lot of this is still quite experimental, but I wanted to share this early on as we're still figuring out patterns for Async Rust.

I hope some of you will enjoy this post. Happy holidays!

[mbrubeck]

Great post! Found a minor typo in a code example: num % 1 should be num % 2.

[Centril]

I'd like us to think big on this; following the tradition of making seemingly complex concepts surprisingly accessible.

I would like to encourage modesty when it comes to extending the core constructs of the language to suit what is a domain-specific feature. In particular, the line for par stream.await? in listener.incoming() { packs in it both a modifier on for-loops par, which seemingly extends .await in a context-sensitive way, and moreover, .await? embeds side-effects into patterns themselves (which are a pure and total fragment of the language), in a seemingly backwards way (pattern matching is like expressions but run in reverse).

[yoshuawuyts]

@centril oh, I guess I should've clarified better — but this builds on ideas outlined in https://boats.gitlab.io/blog/post/for-await-i.

I know this hasn't made it past any stage of the RFC process yet, but I was under the impression that the desire for async iteration syntax, and being able to use ? in expressions was not something new.

The purpose of the last section was to showcase a possible avenue of exploration beyond async iteration — because I think we'll be there some day, and think it's fun to think about what could come after.

[Centril]

@yoshuawuyts

but this builds on ideas outlined in https://boats.gitlab.io/blog/post/for-await-i.

See the associated discussion, where similar concerns were raised, and where elaborations of them are available.

For await loops

Writing a couple of blog posts about the syntax for “for await” loops - loops that process streams (instead of iterators) by awaiting each element. This thread is for discussing them. We’re trying to experiment with different discussion forms that keep the conversation more manageable for everyone, so I’d like to try out an experiment here. For clarifying questions and small points of feedback, please feel free to write a post in this thread. But for larger arguments and full alternative propos…

@yoshuawuyts

I know this hasn't made it past any stage of the RFC process yet, but I was under the impression that the desire for async iteration syntax, and being able to use ? in expressions was not something new.

I know that there is desire for such things, but there is also opposition to making ? or try { .. } into a pattern (which is what I assume you meant) because it has deep implications (some of which are undesirable).

Last time the language team discussed for-await loops, I think the rough idea was that while let was sufficient for the sequential thing. Minutes can be found in:

Login - Dropbox

[Matthias247]

select! is an abstraction which is somewhat similar to match but operates on streams and futures directly. But it's not without its shortcoming. select! introduces a two new keywords: default and complete . It requires all futures and streams to be manually fused. And has also required changes in futures that have created deviations from the stdlib, which is a cost in itself.

Please note that the current version of select!

• is also more powerful than a stream combinator. E.g. it allows to use Futures and Streams sometimes in a select!, and afterwards directly again.
• and maybe even more importantly: It allows to use an imperative coding style, which can be a lot more familiar to a certain set of people (for example: me)

Therefore the complexity (which the users of select! barely encounter) is justified. I share the concern that the dependency on FusedFuture which is only defined in a non-stable futures-rs version is not ideal. But I think this can be solved without giving up on a powerful select!. I wrote a proposal on futures-rs for this a while ago - without feedback so far.

And because there's no precedent for including a macro as complex as select! in the stdlib

Actually I think select! is not that complicated in what it does. println! might be on the same level. The question would be more along whether people feel it's mature enough to stabilize it. I would be very reluctant to claim it is, since we are still just starting to work with async/await. I am even not sure for the underlying FusedFuture - which would be required to move to core in order to be able to provide select!able futures in crates which do not depend on a specific futures-rs version. That was one of the reasons I wrote the linked proposal above, which would also remove the necessity for that.