[Pre-RFC] Parallel fail fast path for error handling / stage pipelining & currying

This is meant for purely thread-level parallelization with a fairly small userland scheduler (if possible). If there will be global executor API this will automatically fit to GE API too.

Please Note: This is not a syntactical suggestion directly. Simply it would be nice to have it. It doesn't matter which approach we take to put in the language. Half-baked.

Currently, we don't have a real parallel way of error handling in code. I want to fail fast for the side effects which are marked with a dedicated syntax. This syntax later can be extended to enable parallel job pipelining and currying for pipelines.

Let's go one by one:

Error handling

We are currently short-circuiting with ? in a sequential way like this, can be written better with ? but for loops still will be there:

fn not_failing_fast_metadata() -> Result<String, io::Error> {
    if let Ok(entries) = fs::read_dir("/root") {
        for entry in entries {
            if let Ok(entry) = entry {
                if let Ok(metadata) = entry.metadata() {
                    println!("{:?}: {:?}", entry.path(), metadata.permissions());
                } else {
                    dbg!("Broken at metadata");
            } else {
                dbg!("Broken at entries");
    } else {
        dbg!("Broken at read_dir");

In a parallel way there can be a method like:

fn fail_fast_metadata() -> Result<String, io::Error> {

Where this code is seen, it creates an STM like underlying scratchpad, then operations are combined with a pool of threads and finally when last metadata() call successfully finishes we get a vector of metadata.

Parallel Pipelining I can't put this up in any category but neither iterators nor Rayon supplies these. The main idea is that enabling parallel pipelining without extra API constructs. BYO executor is not solution to this problem since this API can be extended for plenty of crossbar specific domains and more...

Infallible case would be:

(1..1000) |> map(parallelism=10, |_| { }) |> transform(parallelism=50, |_| {})

This case just enables any non-Result type to be parallelized at the compilation level with probably simple threads with their workloads. Parallel Pipe |> operator can check the Send + Sync during the compile-time across all pipeline before yielding it down.

Ignored Fallible case would be:

fs::read_dir("/") |#> map(parallelism=10, |_| { }) |#> transform(parallelism=50, |_| {})

This case just enables mixed or fully Result type based stages to be parallelized at the compilation level with probably simple threads with their workloads Fallible Ignoring Parallel Pipe |#> operator can check the Send + Sync during the compile time across all pipeline before yielding it down.

That being said, we can have 100 directories from first stage, during the mapping we can come across 20 errors and we continue with 80 object then still we have a transformation can still fail but deliberately continue working with 50 objects.

Short-Circuited Fallible case would be:

fs::read_dir("/") |?> map(parallelism=10, |_| { }) |?> transform(parallelism=50, |_| {})

Above implementation, similar approach but here we are looking for strict guarantees across our operations. Any failure between |?> boundaries will make our code return early and have zero side effects.



  1. Because other languages have it with filtering errors out at the matching side like Erlang. This enables pipelining and error handling far more structured and neat.

  2. Big collections (3M+), where traversal is needed, staggers from hitting the mem boundaries with copied Arcs. This is bad because while you are doing nothing, rayon copies Arcs everywhere, and you have explicitly constrained the throughput to make it work. The API I am suggesting eliminates stage congestion and enables varying throughput for varying stages. This approach is coming from Scala and JVM languages.

  3. Finally "one error and do rollback approach" is how GHC parallelizes (or suggest parallelization with Haskell threads)

Perhaps it's just me, but I don't understand the need for parallel error handling in the first place.

If there are different threads, each thread can handle any errors it encounters independently from other threads. And within a thread the control flow is by definition serial, so any form of parallel error management can just be done in terms of a parallel iterator library like Rayon. So what am I missing that's new here?


Your proposal describes a very large set of ideas without describing any of them in detail.

If you’re planning on landing an RFC for this, or anything else, you would need to aim a lot lower splitting up your large and vague dreams into smaller chunks that one can better overlook and understand one at a time.

More concretely, you are proposing, as far as I understand, error-handling primitives / APIs, (implicitly) assuming they should go into the standard library directly and finally even get short and cryptic special syntax involving even new syntactic tokens. I’d suggest you to try to split up those different aspects as well.

Especially since you mention and compare to ?, that operator was (as far as I know) and definitely would be (if it didn’t exist yet) introduced in several steps over a longer period of time: First you would have to create (and perhaps iterate) the API and perhaps macros for it in a crate. The ? operator, too, was a macro called try! for the longest time. Then that might find its way into the standard library if it was deemed a valuable idea to do so. I don’t really know the exact criteria for getting into the standard library, but most things in the rust ecosystem live happily in crates and stay there forever.

The last step, special syntax is a rare thing; I’m sorry to say so, but I’m seeing a very high chance that a syntax as you propose would never make it into the language in the foreseeable future.

Now for the practical advice how to continue: If your syntax ideas are important to you, I’d encourage you to look into rust macros and find a compromise that doesn’t need the rust language itself to change. Macros are pretty flexible. However even better would be a solution without macros, since they can get confusing if overused, so consider whether that is a feasible option, as well. The right abstraction can spare you boilerplate and stay within standard Rust syntax at the same time.

The last, but most important points: Inventing nice-looking expressions is good and well, but not very productive if you don’t also flesh out what they mean, what problems they are supposed to solve, etc... the devil lies in these details. As the previous commenter said, existing crates might offer a lot already and if it doesn’t cover you with everything you need, you need to figure out what you’re missing and perhaps do your own crate, who knows.

As this forum is about the compiler, language and standard library while your starting point should stay within crates independent of all of these aspect, you can better get help or feedback with finding existing solutions or making new ones in other places like for example in the forum over on users.rust-lang.org.


This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.