This is meant for purely thread-level parallelization with a fairly small userland scheduler (if possible). If there will be global executor API this will automatically fit to GE API too.
Please Note: This is not a syntactical suggestion directly. Simply it would be nice to have it. It doesn't matter which approach we take to put in the language. Half-baked.
Currently, we don't have a real parallel way of error handling in code. I want to fail fast for the side effects which are marked with a dedicated syntax. This syntax later can be extended to enable parallel job pipelining and currying for pipelines.
Let's go one by one:
Error handling
We are currently short-circuiting with ? in a sequential way like this, can be written better with ? but for loops still will be there:
fn not_failing_fast_metadata() -> Result<String, io::Error> {
if let Ok(entries) = fs::read_dir("/root") {
for entry in entries {
if let Ok(entry) = entry {
if let Ok(metadata) = entry.metadata() {
println!("{:?}: {:?}", entry.path(), metadata.permissions());
} else {
dbg!("Broken at metadata");
break;
}
} else {
dbg!("Broken at entries");
break;
}
}
} else {
dbg!("Broken at read_dir");
}
}
In a parallel way there can be a method like:
fn fail_fast_metadata() -> Result<String, io::Error> {
fs::read_dir("/root")
.entry
.metadata().#?;
}
Where this code is seen, it creates an STM like underlying scratchpad, then operations are combined with a pool of threads and finally when last metadata() call successfully finishes we get a vector of metadata.
Parallel Pipelining I can't put this up in any category but neither iterators nor Rayon supplies these. The main idea is that enabling parallel pipelining without extra API constructs. BYO executor is not solution to this problem since this API can be extended for plenty of crossbar specific domains and more...
Infallible case would be:
(1..1000) |> map(parallelism=10, |_| { }) |> transform(parallelism=50, |_| {})
This case just enables any non-Result type to be parallelized at the compilation level with probably simple threads with their workloads.
Parallel Pipe |> operator can check the Send + Sync during the compile-time across all pipeline before yielding it down.
Ignored Fallible case would be:
fs::read_dir("/") |#> map(parallelism=10, |_| { }) |#> transform(parallelism=50, |_| {})
This case just enables mixed or fully Result type based stages to be parallelized at the compilation level with probably simple threads with their workloads
Fallible Ignoring Parallel Pipe |#> operator can check the Send + Sync during the compile time across all pipeline before yielding it down.
That being said, we can have 100 directories from first stage, during the mapping we can come across 20 errors and we continue with 80 object then still we have a transformation can still fail but deliberately continue working with 50 objects.
Short-Circuited Fallible case would be:
fs::read_dir("/") |?> map(parallelism=10, |_| { }) |?> transform(parallelism=50, |_| {})
Above implementation, similar approach but here we are looking for strict guarantees across our operations. Any failure between |?> boundaries will make our code return early and have zero side effects.
EDIT:
Why?
-
Because other languages have it with filtering errors out at the matching side like Erlang. This enables pipelining and error handling far more structured and neat.
-
Big collections (3M+), where traversal is needed, staggers from hitting the mem boundaries with copied Arcs. This is bad because while you are doing nothing, rayon copies Arcs everywhere, and you have explicitly constrained the throughput to make it work. The API I am suggesting eliminates stage congestion and enables varying throughput for varying stages. This approach is coming from Scala and JVM languages.
-
Finally "one error and do rollback approach" is how GHC parallelizes (or suggest parallelization with Haskell threads)