Idea: "strict" closure syntax

Implicit environment capture by closures is often a confusing feature for beginners, especially with the move keyword added to the mix. And even for proficient users it results in various papercuts. For example:

let a: Arc<T1> = ...;
let b: T2 = ...;
let c: T3 = ...;
// Annoying cloning
let a_clone = a.clone();
// I don't want to move b, but I have to use move on closure...
let b_ref = &b;
// Was c moved or copied? Dunno... Need to check T3 docs.
// Also use of c could be buried deep in the closure body.
use_closure(move || {
    // use a_clone, b_ref, c
});

Usually, proposals to fix the papercuts focus on some kind of auto-cloning (e.g. the recent Claim proposal). I think we could reduce some of the pain associated with closures and improve their clarity by introducing and promoting a "strict" closure syntax. The example above could look like this:

use_closure(closure!(
    // variables captured from the environment
    clone a; b; copy c;
    // closure's body 
    || {
        // use a: Arc<T1>, b: &T2, and c: T3
        // this closure body can only use explicitly captured values 
    }
))

The macro syntax is just a placeholder to demonstrate the idea.

Explicit captures can use the following modifiers:

  • none: use shared borrow of value
  • mut: use mutable borrow of value
  • move: move value into closure state
  • copy: same as move, but enforces that value implements Copy
  • clone: clone value into closure state

For brevity we can list several variables with one modifier:

let f = closure!(
    a, b, c; move d, e;
    |k: u32| -> u32 { ... }
);

To promote the "strict" syntax the compiler (or Clippy) may raise warnings for closures which use the current syntax and implicitly capture environment.

9 Likes

the closure! macro seems like something that could be implemented as a stable polyfill, by generating a struct that implements Fn and friends.

2 Likes

Yes, but the main idea is to eventually replace (or strongly discourage) the current closure syntax with implicit capture and leave it only for non-capturing closures. Plus, a non-macro syntax may be more pleasant, though I don't have a concrete proposal for it right now.

2 Likes

rust closure syntax was designed to be ergonomic instead of verbose. you can see this with how it allows argument types to elided. i doubt this design decision will be reverted.

5 Likes

Most closures do not capture environment. In other words, we usually use the closure syntax to create anonymous functions. The "ergonomic" syntax works quite well in such cases. This is why I propose to deprecate the "ergonomic" syntax only for closures which capture environment. I think that ergonomic hit of such deprecation impact will be relatively minor, but it will make teaching and reading such closures significantly easier.

Also, explicit environment capture will discourage environment capture with "syntactic salt", which may be usefull for guiding users toward more functional solutions.

Here how a hypothetical non-macro "strict" closure syntax could look like:

let my_fn = |k: u32| [{a b c} move {d e} clone f] { ... };

I think it's concise enough and, since usually closures capture only one-two variables, additional overhead of explicit captures should be small enough in most cases.

1 Like

citation needed.

perhaps you usually use closures this way, but many apis are nearly useless without capturing closures. most notably catch_unwind and thread::spawn.

I see a lot of problems with this:

  1. ambiguous without unbounded token lookahead. (remember, |x| [x] is already valid syntax). this is not only bad for compilers, but also for any human reading rust source code.
  2. requires all capturing closures have a block body to prevent the above ambiguity
  3. clone is not a keyword, so it is a valid name for an identifier
  4. adds several syntactic constructs that are not used anywhere else in the language. this would possibly require adding another fragment specifier so that declarative macros can construct this type of closure.
  5. unlike other closure modifiers like move and async, it is in the body of the closure instead of prefixing it.

Let's be realistic, in the real world the variables won't be called with one-letter names a, b, c, and there can be more captures, so your proposal is much more verbose than you make it look. Also, the explicit capture needs to be introduced not just for closures, but also for async {} blocks. In fact, the latter are a much worse ergonomic offender, since their autocapture rules are somewhy more strict and different from closures' ones. Finally, remember that async closures don't exist yet, you need to return an async {} from a closure. This means you must nest your explicit captures.

Here is an only slightly cherry-picked realistic example, from the Zed editor source code (crates/assistant/src/assistant_panel.rs::open_context):

cx.spawn(closure!(
    move saved_context, lsp_adapter_delegate;
    clone path, self.languages, self.telemetry, self.slash_commands;
    workspace;
    |this, mut cx| async_block!(
        mut cx;
        move this, saved_context, lsp_adapter_delegate;
        clone path, self.languages, self.telemetry, self.slash_commands;
        workspace;
        async move {
            /* body */
        }
    )
))

I don't think anyone would want to write or read this. Frankly, it's worse than the status quo, even more verbose and hard to get right. And if I modify the body, I need to carefully modify twice the explicit captures as well?

Actually, your syntax doesn't allow to capture fields, e.g. self.slash_commands. There's no (and probably shouldn't be) a rule for implicit introduction of bindings based on expressions in the capture list. This means that those fields must be manually cloned above, with explicit new bindings, and then those bindings need to be explicitly moved, twice.

Frankly, I don't know why you would even include mandatory capture lists in your proposal. It seems that the current autocapture rules are perfectly fine for the vast majority of captures, and one only need explicit capture mode listed for a few specific variables which aren't captured in the desired way. That would eliminate much of the boilerplate in your proposal without requiring any changes to existing code.


I see the following issues with the explicit capture approach in general:

  • The duplication of captures for nested closures and async blocks, as noted above. This will also get worse if Rust adds more types of blocks, like generators.
  • Capturing expressions, including very trivial ones like field accesses, requires a syntax for explicit introduction of new bindings. It's more verbose and becomes even less of an ergonomic win over the current approach (nested blocks with explicit bindings). But if the syntax is restricted only to simple binding names, like for string and macro interpolation, then much of the benefit is lost, since it's quite common to capture specific fields in methods, like in the example above.
  • The simple form of "move this, clone that" doesn't have support for more complex operations than mere Clone::clone call. I'd very much want to see support for custom allocators stabilized, but such collections would need to take the allocator as explicit parameter for cloning (at least in certain approaches). There are also cases like DynClone, where you want to clone a value of T from an erased trait object Box<dyn Foo>: you add a separate trait DynClone, Foo: DynClone, and that trait has methods to clone trait objects. Granted, those are a bit of niche cases, but Rust has historically been great at providing first-class support for usually neglected niche cases, like embedded and low-level development.
5 Likes

I think something in this space makes sense, C++'s capture rules allows (effectively):

  • explicitly requesting no (default) captures, or
  • setting copy or reference capture by default, as well as
  • setting copy or reference capture explicitly per name, and
  • creating arbitrarily evaluated local bindings

Rust has only the second, and can fake the fourth with a wrapping block quite well, but the first is useful to get better errors in generic callback contexts, and it's a bit clumsy to do the third with a wrapping block. It's a bit lame to have worse ergonomics that c++!

4 Likes

It doesn't have special syntax to request moves, though, and C++ moves are already a mongrel. You can do [x = std::move(x)], but I wouldn't call that better than whatever we have or propose in Rust. It also punts on all the issues I listed above. Not that C++ cares much about such edge cases anyway, allocator is built-in in the language and used pervasively in the stdlib, while exceptions are basically assumed in the design. Good luck if you don't want them.

I don't mean in the sense of move vs copy, but in the general sense of "capture type" (copy ~= move anyway, since that's a type-level decision); nor do I mean that C++ is better in general. Just that there's at least room for being better in every way than C++ :stuck_out_tongue:

I have made an alternative proposal.

In my experience, in the vast majority of cases, I want implicit captures, as it's obvious what needs to be captured how and is significantly less verbose. I don't think that there are no cases where explicit captures would be beneficial (though I'm not sure whether it's significant enough to warrant a language feature) so I'm not opposed to the idea in cases where it's nicer, but I do think implicit captures are the right default.

3 Likes

I'd like to revisit this in view of @afetisov's alternate proposal (above) having (it seems to me) foundered on lack of clarity regarding when captures and their side effects occur. Contra most of the other posters, I wouldn't have a problem requiring all captures to be stated up front, and I think in the complex cases where there's currently a lot of boilerplate to write, explicit capturing might actually make things easier to write.

Here's @afetisov's motivating example from the other thread, again.
fn do_stuff(serv: Server, foo: Arc<Foo>, bar: Arc<Bar>) {
    serv.set_handler_fn_mut({
            let foo = foo.clone();
            let bar = bar.clone();
            |req| {
                let foo = foo.clone();
                let bar = bar.clone();
                async move {
                    stuff_1(&foo, &bar);
                    handle_1(req, foo, bar);
                }
            }
        })
        .set_handler_fn_mut({
            let foo = foo.clone();
            let bar = bar.clone();
            |req| {
                let foo = foo.clone();
                let bar = bar.clone();
                async move {
                    stuff_2(&req);
                    handle_2(req, foo, bar);
                }
            }
        });
}

What if you could write it like this?

fn do_stuff(serv: Server, foo: Arc<Foo>, bar: Arc<Bar>) {
    serv.set_handler_fn_mut(async |req, /*move*/ foo = foo.clone(), /*move*/ bar = bar.clone()| {
        stuff_1(&foo, &bar);
        handle_1(req, foo, bar);
    })
    .set_handler_fn_mut(async |req, /*move*/ foo = foo.clone(), /*move*/ bar = bar.clone()| {
        stuff_2(&req);
        handle_2(req, foo, bar);
    });
}

Up to syntax, this is the same concept as @newpavlov's original suggestion. But by refining the syntax a little, I think it becomes genuinely competitive with both what we have and what @afetisov proposed.

I'm calling the new notation, foo = foo.clone() "capture arguments." They should not be thought of as optional arguments to the closure itself; the closure is callable as async fn (req: Request) -> () or similar, exactly as it was in the original, as-you'd-write-it-today form. Rather, they should be thought of as arguments to the closure's constructor, like you'd have if you implemented the closure using an explicit struct.

The commented-out "move" keywords are for illustration; the idea is that you can write "move" on specific capture arguments if you need to, but in this case the compiler should be able to infer that move semantics are appropriate.

2 Likes

That isn't quite the same as the baseline, even if we ignore that async fn closures aren't a thing yet either. Namely, the original gets roughly impl 'static + Fn(Req) -> impl 'static + Future<Output=()>, whereas yours can only provide FnOnce. You could get a repeatable closure by adding let foo = foo.clone() into the async closure body, but then your future borrows the closure instead of being independent.

To get the entirely same semantics, you do need to maintain the clone in both layers, e.g. perhaps:

fn do_stuff(serv: Server, foo: Arc<Foo>, bar: Arc<Bar>) {
    serv
        .set_handler_fn_mut(
            move(foo = foo.clone(), bar = bar.clone()) |req| {
                async move(foo = foo.clone(), bar = bar.clone()) {
                    stuff_1(&foo, &bar);
                    handle_1(req, foo, bar);
                }
            },
        )
        .set_handler_fn_mut(
            move(foo = foo.clone(), bar = bar.clone()) |req| {
                async move(foo = foo.clone(), bar = bar.clone()) {
                    stuff_2(&foo, &bar);
                    handle_2(req, foo, bar);
                }
            },
        )
}

which I think just barely edges in under the default "small" heuristics that rustfmt uses to avoid making the move clause multiline. Comparing this to a version with fully implicit autoclone:

fn do_stuff(serv: Server, foo: Arc<Foo>, bar: Arc<Bar>) {
    serv
        .set_handler_fn_mut(|req| async {
            stuff_1(&foo, &bar);
            handle_1(req, foo, bar);
        })
        .set_handler_fn_mut(|req| async {
            stuff_2(&foo, &bar);
            handle_2(req, foo, bar);
        })
}

it should be fairly evident why people would like to have autoclone. Unless you're writing a lot of async you likely don't have as strong of a desire for autoclone, but when you are doing async work, you essentially always want the autoclone behavior, since independently 'static tasks are necessary in order to spawn the tasks if you want to actually run them.

I want to be sure I understand why the second clone is needed: The closure object holds a reference to foo and bar. If you don't clone these references inside the body of the closure function, then ...

    |req| {
       stuff_1(&foo, &bar);
       handle_1(req, foo, bar)
    }

... the closure object's references are moved into the call to handle_1, invalidating the object. Yes?

I'd personally be inclined to write the clones at the point where they are actually needed; continuing with my notation

fn do_stuff(serv: Server, foo: Arc<Foo>, bar: Arc<Bar>) {
    serv.set_handler_fn_mut(async |req, foo = foo.clone(), bar = bar.clone()| {
        stuff_1(&foo, &bar);
        handle_1(req, foo.clone(), bar.clone());
    })
    ...
}

I can see how that would get tedious if the closure needed to call several functions in a row that take Arc<T> (or anything else you need to clone) by value, but in that case let foo = foo.clone() at the top of the closure body wouldn't help either! And if they all take &Arc<T> or even &mut Arc<T>, you don't need the inner clones at all, unless I'm missing something still.

I can imagine a need for a construct that specifies "capture these variables into the closure object by cloning, and then clone them again when the closure function is invoked" but this doesn't seem to be it, and without a more compelling example it feels too special-purpose to put into the language.

This is tangential, but is there any semantic difference between async |args...| { ... } and |args...| { async { ... } } other than its not actually being possible to write the former yet? I have not been following developments in async notation closely.

That's exactly correct.

That would give you a repeatable closure, but

pub fn example(obj: Arc<Obj>) -> impl Send + Sync {
    move || async {
        obj.clone()
    }
}
error: lifetime may not live long enough
  --> src/lib.rs:9:13
   |
9  |       move || async {
   |  _____-------_^
   | |     |     |
   | |     |     return type of closure `{async block@src/lib.rs:9:13: 11:6}` contains a lifetime `'2`
   | |     lifetime `'1` represents this closure's body
10 | |         obj.clone()
11 | |     }
   | |_____^ returning this value requires that `'1` must outlive `'2`
   |
   = note: closure implements `Fn`, so references to captured variables can't escape the closure

it isn't allowed for the output of a closure (the async block future) to borrow from the closure captures.

The difference is a resolution for exactly the above issue; it's the difference between:

// |args| async { … }
trait Fn<Args> {
    type Output;
    fn call(&self, args: Args) -> Self::Output;
}

// async |args| { … }
trait AsyncFn<Args> {
    type Output;
    async fn call(&self, args: Args) -> Self::Output;
}

// pseudo desugar:
trait AsyncFn<Args> {
    type Output;
    type IntoFuture<'a>: Future<Output = Self::Output>;
    fn call<'a>(&'a self, args: Args) -> Self::IntoFuture<'a>;
}

Basically, in the medium term |args| async { … } is "always" an error, and you want to write |args| async move { … } instead. And even once there's some solution for lending closures, async in practice still would strongly prefer the created task to be 'static to be able to spawn it[1].


  1. It is theoretically possible to scope polling of callbacks inside polling the object owning them by using FuturesUnordered or similar, but it's far from nice to do so while still being able to push, and nesting executors isn't great for performance either. ↩︎

I had to muck around in the playground a bit to understand that this is not because the cloned Arc itself is returned -- any return value, including (), triggers the same error. Moving the move keyword to the other side of the closure introducer ( |args| async move, like you said) does make it compile.

But, crucially, this doesn't compile either:

pub fn example(obj: Arc<Obj>) -> impl Send + Sync {
    move || async {
        let obj = obj.clone();
        obj  // or `do_something_with(obj)`, no matter what it returns
    }
}

with the same error message. Only putting the move on the other side of the closure introducer makes it compile, and, if I understand what that does correctly, it produces a FnOnce closure, which wouldn't work for the do_stuff example, would it?


I understand much better now why people are frustrated with the present state of async closures. In particular I understand better why writing the "natural thing" clashes with the borrow checker and the async transformation, and I feel that it is bad that the ordering of modifier keywords vs. closure introducer changes what the expression means, and also that...

// |args| async { … }
trait Fn<Args> {
    type Output;
    fn call(&self, args: Args) -> Self::Output;
}

// async |args| { … }
trait AsyncFn<Args> {
    type Output;
    async fn call(&self, args: Args) -> Self::Output;
}

... at any call site, one of these is correct and the other is wrong, so why isn't the compiler inferring it, like it does for closure return types?

But I still think that the right way to provide manual control over how captures are stored in the closure object and how they are supplied to the closure body, looks more like @newpavlov's original suggestion in this thread than @afetisov's postfix move operator. It's going to be easier to define and easier to teach if we do it that way, and it will give people a clearer mental model of what is actually happening. Lexical scope is nice and all but it clearly only works 90% of the time, and for the cases where it falls down, I think we should be looking at notation that's closer to what you'd write if you explicitly defined the closure object out-of-line (i.e. impl FnOnce/Fn/FnMut/AsyncFn/... for ExampleClosure { ... }) just because it's more transparent and less magical.

2 Likes

Having be catch up for a while. Since today we have async closure already, and according to the above discussion the difference between move || async and || async move seems to be very tricky I expect to find it explained in the official documents (namely The Rust Book ). But I didn't find it yet. So please put this content in.

Undoing shadowing after a move could ease the pain a bit. Then one can clone without having to come up with awkward names or having to introduce additional blocks.

I realize that it would only help a little, but maybe it's a low-hanging fruit.