Autoclosures (via swift)

[withoutboats]

Earlier, @Gankro wrote some tweets.

Alexis Beingessner (Gankro)

Alexis Beingessner (Gankro)

He linked to this blog post in particular, which describes the “autoclosure” attribute. This attribute causes a function to treat its arguments as if they were wrapped in a closure, delaying their evaluation until they are actually used in the function call.

I don’t know Swift super well, but I think the story on this is a bit more complicated in Rust. We care more about performance transparency than Swift does (in my impression), and ownership semantics definitely make wrapping arbitrary expressions in a closure a bit trickier.

I think this has some very valid use cases though (making assert a function isn’t one of them because it takes a fmt string). In particular, it seems like it could allow Vec::push and Box::new to transparently use emplacement.

I’m concerned about the way placer syntax will impact people learning Rust. In particular, I’m concerned about a potention “to_string vs into” situation, in which people posting code for feedback will be told that they should be using vec <- obj instead of vec.push(obj), even when the performance implications are not significant. This throws a lot of complicated information at the user as they try to understand the concept of emplacement, the motivation for using it, the specifics of Rust’s placer syntax and semantics, and so on. It would be ideal if the method forms had the same internal semantics as placer syntax.

So this thread is just to raise these questions:

• What are some obvious use cases for autoclosure in Rust?
• What are the challenges to implementing this attribute i.r.t. Rust’s unique semantics and compilation model?
• Does the use case discussed above actually work? Can the closure passed to e.g. vec::push be translated to the same semantics as <-?

[eefriedman]

Autoclosure wouldn’t work nearly as well in Rust as it does in Swift. The primary issue is probably the way capturing local variables works. In Rust, a move closure moves all the captured variables into the closure; a non-move closure can’t move out of captured variables. In general, you have to introduce additional local variables to get the right semantics out of a closure.

Given that closures in Rust aren’t really transparent, autoclosures would be messy at best. Imagine people complaining that debug_assert([...]); doesn’t work, so they have to use let b = if cfg!(debug) { [...] } else { false }; debug_assert(b);.

---

If obj is a local variable, vec <- obj; and vec.push(obj); will compile to exactly the same thing. I assume that isn’t actually your concern.

vec <- f() and vec.push(f()) don’t actually have the same semantics if f() panics, so we can’t transform one into the other.

[leonardo]

D language has his “autoclosure” feature since many years, it’s named lazy function arguments:

https://dlang.org/lazy-evaluation.html

You can see an usage example here:

http://rosettacode.org/wiki/Man_or_boy_test#Straightforward_Version

import core.stdc.stdio: printf;
 
int a(int k, const lazy int x1, const lazy int x2, const lazy int x3,
      const lazy int x4, const lazy int x5) pure {
    int b() {
        k--;
        return a(k, b(), x1, x2, x3, x4);
    }
    return k <= 0 ? x4 + x5 : b();
}
 
void main() {
    printf("%d\n", a(10, 1, -1, -1, 1, 0));
}

In theory it’s a nice feature, I have used it few times, but in practice I rarely use it. And in D assert() is a built-in.

[withoutboats]

If obj is a local variable, vec <- obj; and vec.push(obj); will compile to exactly the same thing. I assume that isn’t actually your concern.

Actually, this is exactly my concern! Someone will find out that vec.push(foo()) is “less efficient” than vec <- foo(), and think “OK, push is slower than <-.” Then they’ll advise a new user not to do vec.push(var) in a reddit thread, and then someone else will reply that they’re actually the same. This is an incredibly confusing digression for a new user, and we’ve already seen similar things happen with to_string vs into.

[aturon]

FWIW, this is a feature that Scala has long has as well, and is often used there to make certain pieces of “lightweight syntax” via closures feel even more first class. The syntax in Scala is even more flexible than in Swift – when combined with currying, you can get something that actually lets you define your own while loop with the same end-user syntax as the normal form.

At a high level, there’s no reason the feature can’t work in Rust – after all, you’re just signaling to the compiler that it should inject an implicit || { ... } around the argument at call sites. But the question is whether, when you do that, things work out – e.g., does the ownership system work well enough to cover interesting use cases?

FWIW, the ownership analysis for closures is set up to match the normal way of reading Rust code – you look at the closure body for all uses of variables from its environment, which will tell you whether the body needs that variable by reference, by mutuable reference, or with full ownership. See http://huonw.github.io/blog/2015/05/finding-closure-in-rust/ for more details. Offhand, it seems like this could work reasonably well with “autoclosure” syntax.

Some implications for libraries: this would eliminate the need for distinctions like and versus and_then, or various other places where we have a variant method whose sole purpose is allowing you to delay a computation. In these cases, we’re already using closures in the more complex cases, and I suspect the simpler cases would work fine with auto-closures.

The interaction with placement is quite interesting, although IIRC there are problems using a closure at all. Paging @eddyb.

[arielb1]

The problem with closures is that you can’t return through them. This makes try! not work.

[aturon]

The problem with closures is that you can’t return through them. This makes try! not work.

Note that this is already a problem for the various places we use closures for “lightweight syntax” – like Option::map or Option::and_then. I don’t see autoclosures as particularly exacerbating the problem, but rather offering a way to make existing methods more ergonomic (or to drop variants, in some cases).

[arielb1]

I am afraid that, without any “visible” closures, the errors you will get from an accidental return capture could be quite confusing.

[aturon]

I am afraid that, without any “visible” closures, the errors you will get from an accidental return capture could be quite confusing.

Hm, that’s a good point. Presumably, though, both Scala and Swift face similar problems already. It might be worth experimenting.

Longer-term, we’ve occasionally talked about the potential for a “non-escaping” notion of closure for which return and break actually refer to the surrounding environment. Ruby has a notion similar to this one. Obviously there are tons of tradeoffs and design to be done to make that work, but just wanted to mention it as a possibility for moving toward a world in which closures for things like Option::map are more trivial/“transparent” and ergonomic.

[leonardo]

Longer-term, we’ve occasionally talked about the potential for a “non-escaping” notion of closure for which return and break actually refer to the surrounding environment. Ruby has a notion similar to this one.

This is OK for Ruby, but for a medium-integrity language as Rust, that sound too much magic/invisible.

[aturon]

This is OK for Ruby, but for a medium-integrity language as Rust, that sound too much magic/invisible.

It would most likely need to use a different closure syntax, or provide some other way of explicit signaling, for back-compat reasons at least.

[withoutboats]

If we were to have transparent closures, I hope they would implement the same Fn traits as ‘opaque’ closures; the baggage in Ruby around procs vs blocks vs methods is a lot of complexity that seems inessential to me.

This idea is shaping up to require a lot of innovation. In addition to automatic enclosure and transparent closure, @aturon’s and_then example (which is a good one) also requires that autoclosures transform expressions into closures taking any number of arguments (because and_then takes a function of 1 argument). This is a lot of sugar, and though it could be quite useful if designed carefully, we should be wary of creating a system with too many special cases.

[aturon]

I had some follow-up conversation with @Gankro on twitter, where he said that the more interesting addition, to him, was not autoclosures but rather simplified closure syntax: array.sort { $0 < $1 } rather than array.sort(|(x, y)| x < y).

Once again this is similar to an existing feature in Scala, where you can use underscores to implicitly introduce parameters for a closure.

Also, there’s the idea that when a function takes a closure as its last argument, you can write the closing ) before that argument, and then write the closure “externally” using braces, or something along those lines. I think Rust, long ago, had a feature like that.

It’s definitely worth revisiting these bits of sugar to cut down on the noise needed for writing and using closures. I think some of these simpler, purely-call-side bits of sugar would be totally unproblematic for Rust semantically; just a question of syntax design and motivation.

[yigal100]

Re implicit closure parameters - this feature exists in several languages (again, there’s a variation on it in D) and is nothing but trouble. For one, it doesn’t buy us all that much - it’s a trivial macro implementation. It also reasies issues of hygene.

Rust already has plenty of syntax complexity and there are still core features in planning ( HKT, impl trait, &move references, etc…) that would require even more syntax which are way more importnt. Let’s not waste the complexity budget on trivial stuff.

[sgrif]

One point of note is that our closures are syntactically pretty lightweight compared to many languages. For the types of cases this would cover, the comparison is || expr not || { expr }

[sgrif]

I have frequently wished for implicit argument reference though (if I had to pick, I would chose _ as the sigil, but drop the complexity that Scala has around it and just have it work for single argument closures)

[withoutboats]

I have frequently wished for implicit argument reference though (if I had to pick, I would chose _ as the sigil, but drop the complexity that Scala has around it and just have it work for single argument closures)

In my experience, the majority of cases a single argument function would be useful are covered by just passing the function without a closure (e.g. string.chars().map(char::is_whitespace)). That’s not your experience? Is it that you are frequently composing multiple functions in the closure or something else?

[wycats]

I have experienced something like this feature in Scala (where some expressions might actually be closures depending on the type of the function you’re calling).

I’m concerned that, while unambiguous to the compiler, it would be difficult for a reader of a function call to know which parameters are closures and which are eagerly-evaluated arguments. As a one-time user of Scala, I found this pretty confusing and would not want to inflict it upon myself in Rust.

That said, something like the _ syntax is still explicit to readers that they’re seeing a closure, and I think it’s high-value enough to justify a shorthand. I’d be in favor of doing it

I don’t love _ for the same reason I didn’t like it in Scala; we already use _ for various things, and “what does _ mean” was definitely a confusion I had a lot when reading and writing Scala code as a beginner and intermediate programmer. Other single-char options, like ? or $, are probably worth exploring if we agree that the shorthand notation is valuable.

[sgrif]

Yes, we do have a pretty decent alternative and that is it. However, that’s still often more verbose than |c| c.is_whitespace(), and the goal is to make that as lightweight as possible.

[wycats]

Yes, we do have a pretty decent alternative and that is it. However, that’s still often more verbose than |c| c.is_whitespace(), and the goal is to make that as lightweight as possible.

I occasionally miss the pretzel trick from Ruby (chars.any?(&:whitespace?), which expands to chars.any(:whitespace?.to_proc) which expands to chars.any? { |c| c.whitespace? }, which is equivalent to chars.any(|c| c.is_whitespace()) in Rust).

It might seem like a bunch of contortions to save a few chars, but it added sufficient clarity to code that it was included in Rails pretty early on and eventually made its way into Ruby proper. And that’s without the extra benefits of any-position _, which would make the shorthand further expressive.