Motivation
Although Generator was initially introduced with async story in mind, it can be also used for enhancing core for loop functionality. Integration of Generator with for loops looks quite natural:
for value in generator {
// ..
}
Here we iterate over yielded values and discard return value of the generator.
Introduction of this feature will also prevent a lot of confusion for people coming from Python, as syntax of for loops and generators are quite close, thus it will be natural for them to try to use Rust Generator with for loops. Of course the big difference between Generator and Iterator is ability to return result in addition to yielding values. This can be again quite naturally incorporated as resulting value of for loop expression, so it will be possible to write:
let result = for value in generator {
// ..
};
Which can make some use-cases much more pleasant, for example in data processing:
let metadata = for record in parse_file(f) {
// process record
}?;
Here parse_file returns Generator<Yield=Record, Return=io::Result<Metadata>>. On first encountered error in data processing it will return io::Error which will be bubbled by ?. If everything went without problems all records will be processed and user can get some additional data about processed file (number of records, time of creation, authors, description, etc.).
let leftover_data = for chunk in data.array_chunks(4) {
// process chunk which has type &'a [T; 4]
};
Here we use imaginary array_chunks(&self, const N: usize) -> Generator<Yield=&'a [T; N], Return=&'a [T]> method for slices which yields chunks in the form of array references and returns leftover data in the form of slice &'a [T], so it could be e.g. asserted to be empty
As will be showed later Generator is a natural generalization of Iterator trait, which can be perceived as Generator<Yield=T, Return=()>;. Also it will solve problem of breaking with value from for loops without introducing (arguably) ad-hoc else blocks.
Implementation
Currently this code:
for value in values {
// body
}
Desugars into:
let mut iter = IntoIterator::into_iter(values);
loop {
match iter.next() {
Some(value) => {
// body
},
None => break,
}
}
To allow Generators to be used with for loops we’ll have to change it, so this code:
let result = for value in values {
// body
};
Will be desugared into:
let generator = IntoGenerator::into_generator(values);
let result = loop {
match generator.resume() {
GeneratorState::Yielded(value) => {
// body
},
GeneratorState::Complete(result) => break result,
}
};
As it can be seen this desugaring allows breaking with values from for loop body with appropriate type restrictions (borrowing Rusky’s examples):
// iterator returns (); loop returns ()
let _: () = for x in iter { }
// iterator returns (); loop breaks with ()
let _: () = for x in iter { ... break ... }
// ERROR: iterator's type R=() and break's type S don't unify
let _: () = for x in iter { ... break s ... }
// generator returns R; loop returns R
let _: R = for x in gen { }
// generator returns R; loop can break with R
let _: R = for x in gen { ... break r ... }
// ERROR: generator's type R and break's type S don't unify
let _: R = for x in gen { ... break s ... }
To allow convenient value breaking for iterators we can design several helper methods which will explicitly convert iterator into generator, for example:
fn generator_default<T>(self, value: T) -> impl Generator<Yield= Self::Item, Return=T> { .. }
Which can be used as follows:
let result = for value in iterator.generator_default(0u32) {
match value {
Foo => foo(),
Bar => break 10u32,
}
}
To save backward compatibility with Iterator based for loops we’ll need to add the following support code to the standard library:
impl<T: Iterator> Generator for T {
type Yield = T::Item;
type Return = ();
fn resume(&mut self) -> GeneratorState<Self::Yield, Self::Return> {
match self.next() {
Some(val) => GeneratorState::Yielded(val),
None => GeneratorState::Complete(()),
}
}
}
trait IntoGenerator {
type Yield;
type Return;
type IntoGen: Generator<Yield=Self::Yield, Return=Self::Return>;
fn into_generator(self) -> Self::IntoGen;
}
// this line does not currently work
impl<T: Generator> !IntoIterator for T {}
impl<G: Generator> IntoGenerator for G {
type Yield = G::Yield;
type Return = G::Return;
type IntoGen = G;
fn into_generator(self) -> Self::IntoGen {
self
}
}
impl<T: IntoIterator> IntoGenerator for T {
type Yield = T::Item;
type Return = ();
type IntoGen = <Self as IntoIterator>::IntoIter;
fn into_generator(self) -> Self::IntoGen {
self.into_iter()
}
}
Unfortunately this code relies on implementation of mutually exclusive traits. (otherwise known as negative trait bounds)
Alternatives
else block
Previously discussed approach for adding breaking with value to for loops was to use the following syntax:
let r: R = for val in iter {
// .. do stuff, which can break with type `R`
} else {
// this block must always evaluate to type `R`, this value
// will be used if `for` loop body didn't break with a value
}
(see also then proposal)
The big drawback of this approach is that we replace currently used simple desugaring of for loops with a significantly more involved construct understanding by compiler, which will have a negative effect on the compiler complexity. Also arguably this solution is quite ad-hoc in nature and can be surprising for new users.
It does not integrate generators with for loops, but should be considered as an alternative due to the poor compatibility with this proposal. Also it can be used in conjunction with the next alternative.
Converting generators to iterators by discarding result value
The easiest approach for generators integration with for loops will be to use blank implementation of Iterator for T: Generator with discarded result value. It automatically provides integration with for loops.
impl<T> Iterator for Generator<Yield=T, Return=()> {
type Item = T;
fn next(&mut self) -> Option<T> {
match self.resume() {
GeneratorState::Yielded(value) => Some(value),
GeneratorState::Complete(_) => None
}
}
}
The big drawback of this approach is that users lose return value unconditionally, otherwise they’ll have to use explicit loop which will be a less ergonomic and pleasant experience.
Rationale
While two listed alternatives combined can be a working solution, they both provide ad-hoc solutions to narrow problems and combined provide less compact and expressive overall result. On the other hand the proposed solution naturally solves two problems: improving Generator ergonomics through for loop integration and breaking with value from for loops, without losing desugaring simplicity and without extending for loops with new blocks, although at the expense of support code amount and by introducing additional IntoGenerator trait.
Unresolved questions
- We cover only
Iterator -> Generatorconversions, but it could be usefull (especially for interacting withIteratorbased code) to doGenerator -> Iteratorconversion as well with discarding return result. Probably the best approach will be introduction of explicit helper conversion method forGenerator. - Is there alternatives for using negative trait bounds? And if yes, should we use them instead of waiting? For example, specialization with lattice rules could be such alternative, and it looks like a lot closer to being implemented.
- What about possible extension of
GeneratorwithArgassociated type forresume(arg: Self::Arg)? Should we doforloop integration only forArg = ()or should we add something likecontinue valexpression? (Personally I am not sure if we needArgat all) - Can we somehow make
Iterator<Item=T>just an alias forGenerator<Yield=T, Return=()>and change accordingly all associated traits without breaking backward compatibility?