[Pre-RFC]: Generator integration with for loops

[newpavlov]

We should support else blocks (or whatever we want to call them. I prefer then) and allow them to take an argument. This argument is the final result that the generator yielded.

One of motivations to for this proposal was to remove the need for else/then blocks. I can’t see how you example is better than:

let y: Foo = for x in wow {
    ...
};
let foo: u32 = {
    // compute `foo` based on `y`
};

Well, except one thing, it creates explicit scope for y which could make handling lifetime containing results better, but with NLL, I think, its importance significantly diminishes.

The Generator trait should also be extended to take a resume argument. If this happens then we need a way to pass the argument when iterating the for loop. For this we can use continue

You probably meant the same as an Arg argument? I.e. Generator signature will look like this:

pub trait Generator {
    type Yield;
    type Return;
    type Arg;
    fn resume(&mut self, arg: Self::Arg) -> GeneratorState<Self::Yield, Self::Return>;
}

One problem with you example is that we can’t decide what value to use as an argument on the first iteration. We could leave resume argumentless and add method resume_with_arg, or use Option<Self::Arg>, but it will make generators significantly more complex. And overall I think requiring to use continue in every path of the for-loop body will make for loops harder to write, modify and read.

Also it will make desugaring of for loops significantly more complex, as continue with the value currently does not work anywhere, in contrast with break expressions.

Could you please provide examples where argument based iterators and continue with value could be useful?

[canndrew]

One of motivations to for this proposal was to remove the need for else/then blocks. I can’t see how you example is better than:

My example only runs if the for loop doesn’t break. Your idea doesn’t negate the need for else clauses, it only makes them even more useful since sometimes people will want to handle the final result of the generator separately.

One problem with you example is that we can’t decide what value to use as an argument on the first iteration.

Generators should have separate start and resume methods. That’s the way I’m imagining it at least.

And overall I think requiring to use continue in every path of the for-loop body will make for loops harder to write, modify and read.

Well the feature is only there if you want to use it. If you’re resuming with () then you don’t need to use continues. And if you do want to resume with a value then continue seems like the most natural place to put it.

Also it will make desugaring of for loops significantly more complex, as continue with the value currently does not work anywhere, in contrast with break expressions.

for loops don’t necessarily have to desugar to a loop. They can be their own thing entirely.

Could you please provide examples where argument based iterators and continue with value could be useful?

Anywhere where yield returning a value is useful. eg. you want to write an async implementation where coroutines yield the set of events they’re waiting on and get resumed with the event they were woken up for.

[newpavlov]

it only makes them even more useful since sometimes people will want to handle the final result of the generator separately.

Ah, so you want to be able to distinguish between break result from loop body and from generator result.

Generators should have separate start and resume methods. That’s the way I’m imagining it at least.

And what should in general happen if resume method is called before start?

for loops don’t necessarily have to desugar to a loop. They can be their own thing entirely.

I think changing relatively simple desugaring to more involved understanding of for loops by the compiler is a big drawback of your proposal.

While I am personally not convinced that we need complexities which you proposal introduces, your ideas seems more or less compatible with the current proposal, so they probably can be introduced later as separate RFC(s).

[HadrienG]

I like the idea of improving the interoperability between iterators and generators a lot. I also agree with your point that this would be best done by making Generator<Yield=T, _> and Iterator<T> as equivalent as possible in the eye of the type system.

I would strengthen the point that Generator -> Iterator conversion and Iterator -> Generator conversion are both very important. We have a rather large body of Iterator-based code (standard Iterator methods, itertools, rayon, etc…), which should not need to be rewritten in order to be able to consume generators. Similarly, iterators should be usable in any place where generators are expected in order to ease adoption of generators in existing iterator-based codebases.

I am less sure about the proposed for loop changes. It would make for loops more special with respect to “raw” loop statements, since break value; would be much more usable with the later than with the former. That is because in the raw loop case, the user controls the type of the loop return value, whereas in the for loop case, the underlying iterator/generator type controls it. I’m not sure how much of a problem that would be in practice, though, and the proposed use cases do look pretty nice.

[canndrew]

And what should in general happen if resume method is called before start?

Same thing that happens if resume is called after the generator has finished - it panics. (Though I’d prefer a Generator API based on move semantics rather than using &mut).

[rpjohnst]

Move semantics for generators are insufficient. We want them to be able to self-borrow across suspension points, which means the trait needs to work with immovable types. Further, event loops are a major consumer of relatively large, heap-allocated generators that definitely shouldn’t be moved in and out of whatever collection they’re a part of, because that would mean a lot of unnecessary (de)allocation.

Even a partially move-based trait (say, fn start(A) -> Self and fn resume(&mut self, R) -> Y) isn’t great on this front. The trait could be implemented for G: Generator directly, forbidding self-borrows in start; or it could be implemented for Box<G>, forcing an extra allocation on the client.

Separate start and resume methods could technically have their ordering enforced via zero-sized token types, but that’s probably overkill. I do like the idea as a solution to resume arguments, though.

[canndrew]

Move semantics for generators are insufficient. We want them to be able to self-borrow across suspension points, which means the trait needs to work with immovable types.

Damn.

Separate start and resume methods could technically have their ordering enforced via zero-sized token types, but that’s probably overkill.

That’s an interesting idea. Without dependent types though there’s nothing to link the token to a specific instance of the generator. So it wouldn’t be fool-proof. Might still be worth protecting against inappropriate resumes though:

trait Generator {
    type Yield;
    type Resume;
    type Return;

    fn start(&mut self) -> GeneratorResult<Self>;
    fn resume(&mut self, arg: Self::Resume, token: ResumeToken<Self>) -> GeneratorResult<Self>;
}

enum GeneratorResult<G> {
    Yield(Self::Yield, ResumeToken<G>),
    Return(Self::Return),
}

struct ResumeToken<G>(PhantomData<G>);

for-loop integration would make juggling the ResumeTokens much less of a chore of course.

[rpjohnst]

Without dependent types though there’s nothing to link the token to a specific instance of the generator.

Now that you mention it, that reminds me of generative types. We already will probably need generative lifetimes to make self-referential types sound, so maybe we could apply the same thing to the resume tokens.

That is a lot of machinery to bake this low in the stack, though. Not sure if it’s worth it.

[idanarye]

The Generator trait should also be extended to take a resume argument. If this happens then we need a way to pass the argument when iterating the for loop. For this we can use continue

let gen: impl Generator<Resume=u32>;
for x in gen {
    ...
    continue 123;
}

If the generator resumes with anything other than () then the continue is mandatory at the end of every path in the for-loop body. If the generator does resume with () then we can continue to use continue with no argument, or to omit it entirely and the end of the body. This makes the usage of continue resemble the usage of break and return with regard to ().

What’s wrong with the way regular way Rust returns a value from a block?

let gen: impl Generator<Resume=u32>;
for x in gen {
    ...
    123
}

[troiganto]

Something the motivation section somewhat glazes over is why it would be preferrable to make for-loops based on generators instead of implementing IntoIterator for a subset of generators. (i.e. why to prefer the conversion iterators->generators over generators->iterators) To me, the latter seems more backwards-compatible, easier to implement, and easier to gather community support for.

Maybe the answer is obvious to someone more intimate with the issue than me, but I think an eventual RFC should also cover this question.

[jnicklas]

Why not simply impl Iterator for Generator where Return=()? That seems like a far simpler solution, and it maps well to working with functions which yield some type and return (). Kind of like this:

impl<T> Iterator for Generator<Yield=T, Return=()> {
    type Item = T;
    
    fn next(&mut self) -> Option<T> {
        match self.resume() {
            GeneratorState::Yielded(value) => Some(value),
            GeneratorState::Complete(_) => None
        }
    }
}

Playground link: https://play.rust-lang.org/?gist=c594c65f258ca5a97e5d79acb4d0e176&version=stable

[jdahlstrom]

Why not simply impl Iterator for Generator where Return=()? That seems like a far simpler solution, and it maps well to working with functions which yield some type and return (). Kind of like this:

Simpler, yes, but less expressive. You couldn’t extend the for loop syntax to support generators with non-() return type (or would have to ignore the return value), or those that take resume arguments.

[newpavlov]

I’ve updated pre-RFC text with alternatives section, in which I’ve tried to answer question from @troiganto and @jnicklas.

[jdahlstrom]

The big drawback of this approach is that we replace currently used simple desugaring of for loops with a significantly more involved construct understanding by compiler, which will have a negative effect on the compiler complexity. Also arguably this solution is quite ad-hoc in nature and can be surprising for new users.

I’d add that the biggest drawback with using else specifically (yeah, I realize this is a bikeshedding issue) is definitely the fact that in python which has the exact feature it has never been intuitive to anyone what the else branch does (mostly people expect it to be evaluated if the loop never executes). And the construct’s rare enough that every time you encounter it you have to go and check what the semantics were again.

[idanarye]

Instead of adding an else clause, why not use an enum to determine if the for has finished or if it was terminated with a break? Something like this: https://play.rust-lang.org/?gist=f2a1eb4cd59ebba46485c7d3278a9bff&version=nightly

[nikomatsakis]

(Disclaimer: I’ve not read the full thread yet, just the first post.)

I find this idea simultaneously very elegant – using Generators return value as the result of a for loop is quite clever – and simultaneously quite confusing. This example certainly caused me to do a double take:

let number_of_records = for record in parse_file(f) {
    // process record
}?;

Where is this number_of_records value coming from? It looks like it’s somehow being computed by the loop, but it’s not, not really, it’s being computed by the generator returned by parse_file I guess. It feels quite non-obvious.

(In contrast, when you break out of a loop with a value, the flow of the value feels more direct – the same code that lets you reach uses of that variable gives you its value.)

I think my overall feeling is that this might be an interesting direction, but it’s a step I would not want to take until we’ve gained a lot more experience with generators. It is an interesting idea to have around.

(For the record, I remain pretty opposed to things like else blocks being attached to for loops and so forth. It just feels like too much to me, and I’ve been persuaded time and time again that nobody knows what such constructs ought to mean.)

[newpavlov]

The main point which I wanted to demonstrate with that example is an alternative to this pattern, which I don’t like much:

for record in parse_file(f) {
    let record = record?;
    // process record
}

With generators we can decouple error reporting channel and “business” data channel.

number_of_records here is just to demonstrate unwrapped Ok value. I’ll try to add a bit more explanation. But I get your point that for unaware user it can be non-obvious from where he receives result value, I guess it can be only solved by becoming accustomed to generator based for loops.

[nikomatsakis]

With generators we can decouple error reporting channel and “business” data channel.

Ah, I kinda’ missed that at first, I admit. I was wondering what that ? at the end of the for loop was all about, kinda took it for a typo. =)

I’m not wild about the let x = x? pattern either.

@withoutboats had another proposal of ? patterns, though more for iterators (but I guess it applies):

for r? in ... {
}

[rpjohnst]

I’m wary of a pattern syntax that matches the unwrapping/projection syntax rather than the constructing syntax like everything else (except ref, which is confusing and something we’re moving away from for precisely this reason), but I do like the idea of integrating ? into for loops somehow, since it can’t be just applied to the iterator like with other patterns.

[withoutboats]

With generators we can decouple error reporting channel and “business” data channel.

This is an interesting perspective, but wouldn’t it be better if the solution worked with Iterators of Results, which are already not uncommon today?

It would also ideally work with Streams; today in the futures-await craite, #[async] for elem in stream implicitly ?s the value, which I think most people would agree is probably wrong. Changing this would make using #[async] for with streams essentially the same as an Iterator of Results.