`with` clauses

[tema2]

I've seen this idea here.

It's superb idea, so I start mentioning of post ending questions:

• Dyn value construction:

• What happens if the Box<dyn Trait> is allowed to escape the with scope, and the methods are invoked?

Also, things we don't want to allow (at least at MVP): allowing to move out from binding declared in with clause.

The obvious solution: is to forbid moving out from this binding, but I would rather:

Better solution: Allow only pointer types to be passed in such clauses, this brings in their lifetimes.

Pro: dyn Trait constructions has to deal only with pointers, which can easily be stored and doesn't require any attention.

Pro: Brings in a lifetime, so dyn Trait value gets implicit obligation. This prevents leaks.

Con: how to properly type these obligations? => complications.

Solution is simple: only allow calling with bounded impls in with expression context.

Feature motivation: passing implicit references to Executor in async, to Allocator in no_std, etc... and also this fits to pass a reference to an allocator for allocating trait objects impl trait return types in traits.

[CAD97]

Super quick idea overview:

impl Visitor for MyNode 
with(cx: &mut Context)
{
    fn visit(&self) {
        cx.counter += 1;
    }
}

fn process_item() {
    let cx = Context { counter: 0 };
    let v = vec![MyNode, MyNode, MyNode];
    with(cx: &mut cx) {
        v.visit();
    }
    assert_eq!(cx.counter, 3);
}

Then the supposed issue is

fn foo<T: Visitor + 'static>(v: T) {
    let x: Box<dyn Visitor> = Box::new(v);
}

---

For this to be a problem, we fist have to even just allow

fn foo<T: Visitor + 'static>(v: T) {
    v.visit();
}

with(cx: &mut Context { .. }) {
    foo(MyNode)
}

which is not simple itself! MyNode implements Visit "when in a with context", so there's two ways I can see this realistically semantically done:

• this is effectively sugar for thread locals, or a similar mechanism, and it fails at runtime if the with context isn't actually present, or
• this is effectively sugar for anonymous ad hoc newtyping, or a similar mechanism, and lifetimes are inherited that way.

For an even simpler illustration of the root issue:

let mut cx = Context::new();
let visitor: impl Visitor;
with(cx: &mut cx) {
    visitor = MyNode; // MyNode: Visitor
}
visitor.visit(); // MyNode: !Visitor

I think my latter provision here makes more sense. Utilizing an imaginary structural types to illustrate, with context could semantically behave as (and derive borrow rules from) roughly the following:

// with clause
impl Visitor for MyNode 
with(cx: &mut Context);

// explicit structural newtype
impl Visitor for
WithContext<
    MyNode,
    { cx: &mut Context, .. },
>;

// with block
with(cx: &mut cx) {
    foo(MyNode)
}

// explicit structural newtype
{
    __with_context = AddStructure![
        __with_context
        & { cx: &mut cx }
    ];
    foo(WithContext {
        0: MyNode,
        1: SelectContext![
            __with_context
            | { &mut cx }
        ],
    });
}

That is, the requested context is carried along as a tuple, copying/moving/reborowing the context as required when coerced from a concrete type with the context into a impl Trait that is no longer asking for context.

Yes, this means that boxing it would not be Box<MyNode>, it'd be Box<WithContext<MyNode, { cx: &mut Context }>>. And it carries the lifetime (and isn't 'static).

Personally I don't really like with/implicits, but I do see the value they can provide for easily adding in some more context to existing trait signatures. I think if Rust were to have them, it'd be as I've described here as essentially sugar over newtypes that carry the context, and easily/quickly constructing them just-in-time as-needed.

Here's another, more closure-like syntax (inspired also by Java anonymous classes) to try to help illustrate how I think it could work:

foo(impl Visitor {
    visit: |&self| {
        cx.counter += 1;
    },
})

Also of note is that this wouldn't provide with context to "static methods" (that is, ones without self arguments). Also, you couldn't pass two arguments that were with-dependent on the same with context, because they can't both borrow the same context, it has to be passed in.

The issue is that the called code doesn't know about the context, so it can't pass it.

Even before considering dyn, a with proposal needs to consider and explain exactly how the with context is passed around through code that knows about it (has the with bound) and is packaged for code that doesn't know about the with context (just uses genetics without a with bound).

What does it "desugar" as? The answer to that question makes the answers to the rest of the questions much easier to answer.

[ibraheemdev]

Is this a more explicit version of Haskell/Scala-like implicits?

[tema2]

@CAD97

impl Visitor for MyNode 
with(cx: &mut Context)
{
    fn visit(&self) {
        cx.counter += 1;
    }
}

fn process_item() {
    let cx = Context { counter: 0 };
    let v = vec![MyNode, MyNode, MyNode];
    with(cx: &mut cx) {
        v.visit();
    }
    assert_eq!(cx.counter, 3);
}

Then the supposed issue is

fn foo<T: Visitor + 'static>(v: T) {
    let x: Box<dyn Visitor> = Box::new(v);
}

In the problematic case you mentioned we bound T: Visitor by 'static => means that foo only takes instances of T with attached lifetimes being at least 'static: so, T: Visitor can be static if both 1) implementor type has no references in it 2) with context where our foo has been called provide &'static mut Context reference.

Actually, I would regard mechanism as capturing lifetimes of with-expression binding.

@CAD97

let mut cx = Context::new();
let visitor: impl Visitor;
with(cx: &mut cx) {
    visitor = MyNode; // MyNode: Visitor
}
visitor.visit(); // MyNode: !Visitor

I don't see why this is a problem. It is not that type is changing, it's more of it's methods cannot be called because the implicit argument (which is taken from env.) is absent.

So people calling Visitor methods outside of with block get an error like
"enviroment to call this method is not setup\present; help: put method call inside of with block"

@CAD97

What does it "desugar" as? The answer to that question makes the answers to the rest of the questions much easier to answer.

Isn't it rather implementation concern?

[CAD97]

@tema2

I don't see why this is a problem. It is not that type is changing

Notice that it's been put in a binding of type impl Visitor.

@tema2

Isn't [desugaring] rather implementation concern?

We don't need a specific desugar, but explaining in code exactly what the expected semantics are would go a long way to clarifying, well, what exactly the feature does.

Questions like:

• How is context passed between stack frames? Extra implicit arguments? Thread locals?
• What does it mean to "concretize" an impl ... with? When you pass an impl to a normal non-with bound, how does that capture the context? What about passing it down further?
• What requirements are put on with contexts? How does it know to reborrow (mutable) references, but move custom types?
• How does with context interact with closures?

These are not implementation questions; these are important questions about the design that have a large impact on how it can be used. An example desugaring makes answering these (and more) questions about semantics a lot easier.

[tema2]

@CAD97

@tema2

I don't see why this is a problem. It is not that type is changing

Notice that it's been put in a binding of type impl Visitor .

I noticed, as I said we disable methods of Visitor outside of with context, it render impl Visitor type useless... but it's what user has requsted, there should probably be a warning about that.

I see what you're talking about and indeed usability of such impl Visitor type depends on what type it is actually resolved to. Here I would suggest 3 possible solutions:

• Force a trait to also declare with => limiting;
• Invent a syntax for impl Trait with (...) => probably not worthy;
• Forbid casting with bound implementors into abstract types (both an impl and dyn) => unfortunate.

As a compromise, I would forbid casting with bound types into both impl and dyn Trait unless that Trait is declaring matching with context.

Other cases can be added later.

@CAD97

• How is context passed between stack frames? Extra implicit arguments? Thread locals?
• What does it mean to "concretize" an impl ... with ? When you pass an impl to a normal non- with bound, how does that capture the context? What about passing it down further?
• What requirements are put on with contexts? How does it know to reborrow (mutable) references, but move custom types?
• How does with context interact with closures?

• It's captured by implementors;

• Via newtype; we simply append all with block sourcing bindings to an anonymous tuple-struct which is unique for each combination of type and context in with bound impl; These anonymous types always implement Deref,DerefMut and (unexistent -_- ) DerefMove;

• I don't like the idea of allowing to move values via with clauses: it is pointless: the moved value can only be consumed once, effectively resulting in passing an implicit argument taken by first called method of an impl => no other methods would be possible to call.

• They capture it.

So here we go:

impl Visitor for MyNode 
with(cx: &mut Context)
{
    fn visit(&self) {
        cx.counter += 1;
    }
}

//becomes

struct $<'a>(MyNode,&'a mut Context)

//here go `Deref`s impl's...

impl<'a> Visitor for $<'a>
{
    fn visit((ref self1, ref mut  cx): &mut self) { //self1 is renamed self
        let mut a = |arg| cx.counter += 1; //here we produce `impl FnMut(Arg) + 'a`
        a(get_arg());
    }
}

@CAD97

When you pass an impl to a normal non- with bound, how does that capture the context? What about passing it down further?

With what I've got, we have no problems with calling any methods in desugared with bound types. Passing context down is hard.... and undesirable, given that we can always pass that context to free standing fn explicitly, we may not want to do it implicitly. As a last resort, we can allow with clauses on freestanding functions later.

[camelid]

The with clauses idea reminds me of this academic paper: Programming with Implicit Values, Functions, and Control (or, Implicit Functions: Dynamic Binding with Lexical Scoping) - Microsoft Research

[SkiFire13]

This desugaring doesn't feel right:

• What are get_arg() and Arg here?
• How did the &self parameter become &mut self? This is unsound in presence of code that calls visit while expecting it to take an &self.

// `Foo`'s definition is valid today, so it has to compile
trait Foo {
    fn foo(&self);
    fn bar(&self, f: impl Fn());
    fn baz(&self) {
        // This is possible only because both `bar` and `foo` take `&self`.
        self.bar(|| self.foo())
    }
}

impl Foo for MyNode
with(ob: &mut Option<Box<i32>>)
{
    fn foo(&self) {
        *ob = None; // Invalidates any reference to the inside of `ob`
    }
    fn bar(&self, f: impl Fn()) {
        // Let's assume that the box is valid now
        let box_ref: &Box<i32> = ob.as_ref().unwrap();
        f(); // This calls `self.foo()` which invalidates `box_ref`
        println!("{}", box_ref); // Oops, now we're printing invalid data
    }
}

fn main() {
    with(cx: &mut Some(Box::new(0)) {
        MyNode.baz();
    }
}

[tema2]

@SkiFire13

What are get_arg() and Arg here?

Dummies for closure. Sorry.

@SkiFire13

How did the &self parameter become &mut self ? This is unsound in presence of code that calls visit while expecting it to take an &self .

I commented about rename; actually we pack immutable reference to self and mutable one to ctx in one implicit instance of an anonymous struct => to acess ctx we need mutable reference to our instance and since it is only created in surrounding with context passing it as such is okay. Therefore a mutable reference. Then, we do destructure: obtain a copy of stored shared refernce &self and a mutable reference to ctx.

@SkiFire13

// `Foo`'s definition is valid today, so it has to compile
trait Foo {
    fn foo(&self);
    fn bar(&self, f: impl Fn());
    fn baz(&self) {
        // This is possible only because both `bar` and `foo` take `&self`.
        self.bar(|| self.foo())
    }
}
...

This example is interesting. The problem is that fn foo from impl is and may stay impl Fn while in fact it isn't.

Probably the best thing we can do is to disallow using mutable references from with clauses in method that take &self. But it is limiting...

[PoignardAzur]

I'm usually in favor of adding implicit arguments to Rust, but I don't think this proposal is a good idea.

The main proposed use-case (adding arguments to a trait implementation that aren't declared in the trait definition) seems like an anti-pattern to me.

When I use a type that implements a trait, I want to be able to look at the trait declaration and treat it as a summary of everything the trait is able to do. If you want an escape hatch from that, there's already cells.

(Also, I'm not sure exactly what the proposed semantics would be, but it sounds like they'd add potential for post-monomorphization errors?)

[H2CO3]

@CAD97

foo(impl Visitor {
    visit: |&self| {
        cx.counter += 1;
    },
})

Arrgh, that makes me have C# delegate PTSD.

---

Anyway, @tema2, your proposal doesn't even describe what this is supposed to be, let alone what problem it solves or why we need this.

[ibraheemdev]

If I understand correctly, this is an incredibly powerful idea:

fn main() {
    let logger = EnvLogger::new();
    with(logger: &logger) {
        run();
    }
}

fn run() {
    // uses implict logger!
    log::info("starting to do thing...");
    do_thing();
    log::info("done thing");
}

fn do_thing() {
    // ...
    log::info("...");
    // ...
}

#[test]
fn test_run() {
    let logger = TestLogger::new();
    with(logger: &logger) {
        run();
    }
}

crate log {
    fn info(s: &str) with(logger: &dyn Logger) {
        logger.info(s);
    }
}

But also comes with drawbacks that have been discussed over the years:

Implicit Parameters

This isn't an RFC. I just wanted to bring this up for discussion and see what other people think about this idea. I have been programing a bit more in Scala recently, and there is a feature in scala called Implicit Parameters. I'm not 100% knowledgeable about all the implementation details in Scala but here is a quick pseudo-rust example: fn caller() { let implicit name = "world"; hello(); // NOTE: no argument passed } fn hello(name : implicit &str) { println!("Hello {}!", name);…

Proposal: Auto-propagated function parameters

[mathstuf]

@ibraheemdev

fn main() {
    let logger = EnvLogger::new();
    with(logger: &logger) {
        run();
    }
}

fn run() {
    // uses implict logger!
    log::info("starting to do thing...");
    do_thing();
    log::info("done thing");
}

What kind of error is expected for:

fn main() {
    run();
}

How, based on the run signature, is the compiler to know that context is missing (the proposal here has a with statement bubbled up, but is it expected to work as you've given?). But this is a new kind of function pointer. How does the "I need a logger context" get added to the returned iterator type here:

fn logged_iter<I, T>(iter: I) -> impl Iterator<Item = T>
where
    I: Iterator<Item = T>,
    T: Debug,
with(logger: &mut Logger)
{
    iter.map(|it| log::info("iterating on {:?}", it); it)
}

How does the with get attached to that impl Iterator so that the caller knows it needs to supply the proper context? If the passed-in iterator has its own with attachment, it needs propagated to the result as well.

I don't see how this isn't more complicated than just adding a parameter to pass the context around as necessary. It's both more explicit and does not require language-level changes to make happen.

[ibraheemdev]

What kind of error is expected

error[E0277]: the trait bound `with(logger: &dyn Logger)` is not satisfied
 --> src/main.rs:5:11
  |
5 |     run();
  |
note: required by a bound in `info`
 --> src/main.rs:2:11
  |
2 | fn info(s: &str) with(logger: &dyn Logger)
  |                            ^^^^^^^ required by this bound in `info`
= help: consider introducing `logger` with a `with` statement

How, based on the run signature, is the compiler to know that context is missing

From the text:

Within this block, you can invoke fuctions that have a with(x: T) clause (presuming that value is of type T ); you can also invoke code which calls such functions transitively. The values are propagated from the with block to the functions you invoke.

How does the with get attached to that impl Iterator so that the caller knows it needs to supply the proper context?

I don't know much about how rustc type checking works, but I assume the closure type generated would have a where with(logger: &dyn Logger) bound that would be propagated to the iter::Map and the existential impl Iterator as well.

After matching an impl, we are left with a "residual" of implicit parameters. When we monomorphize a function applied to some particular types, we will check the where clauses declared on the function against those types and collect the residual parameters. These are added to the function and supplied by the caller (which must have them in scope).

I don't see how this isn't more complicated than just adding a parameter to pass the context around as necessary. It's both more explicit and does not require language-level changes to make happen.

Having literally every function in every program taking a &dyn Logger is far from ideal, and globals introduce a few limitations that implicits don't. Not that I am 100% advocating for with to be added to Rust, but there are things that an implicit variable makes a lot of sense for (logger, allocator, async runtime, etc.).

[ibraheemdev]

Would with(x: impl Trait) be possible?

[mathstuf]

@ibraheemdev

Having literally every function in every program taking a &dyn Logger is far from ideal, and globals introduce a few limitations that implicits don't.

How is having to add with(…) bounds everywhere better than adding a parameter? Or are these dependencies going to be invisibly added/required at various places? One of the things I really like about Rust is that I can write the signature, fill it in with todo!() and then come back later (but still at least cargo check my way through the "higher level" bits). Having the signature mutate based on the body of the function is certainly new AFAIK (sure, impl Trait "changes" with the body, but the guarantees the caller can rely on do not).

I'll note that having these things mutate based on the body is also a semver hazard: how can I, as a maintainer, make sure that my edits aren't adding or removing with guarantees?

[tema2]

@PoignardAzur

(Also, I'm not sure exactly what the proposed semantics would be, but it sounds like they'd add potential for post-monomorphization errors?)

At least, not yet: I see an implementation as desugaring to anonymous type, as I have descibed before.

@PoignardAzur

The main proposed use-case (adding arguments to a trait implementation that aren't declared in the trait definition) seems like an anti-pattern to me.

While it can turn out to be a bad practice, I think that allowing impls to require an implicit references to caller's scope has it's own use cases. Which are worthy to go for.

@mathstuf

What kind of error is expected for:

fn main() {
    run();
}

While initially I wanted to left with clauses on free standing fns for future extensions, Now I really don't think that this proposal is worth without it.

As of I see a lot of disconnect here =>
I just redescribe the feature again, considering everything that have been written here before.

with clauses:

As an expresion:

Syntax:

with((binding: &(mut) Type,)*) {
    ...
}

Semantics:

The body of with expresion is a block expresion with additional properties:

• inside with expr. block with bound impls and fns can be used, if with expr. header contain bindings of required types.

As a bound:

• Syntax is analoguos to one of where clauses, except it brace encloses contens;
• May appear along with where clause on impl,fn and trait items;

Semantics

Pulls in an annotated scope immutable binding of a reference type.
When introduced on:

• impl => these references are stored in an implicit newtype that contains Self value and the reference. This newtype becomes Self type of all methods in the impl, it implements Deref and DerefMut traits with the Output type being equal to old Self type; it provides capability to move out form itself.
  Special case is when with bound impl block itself declares a with bound method. In this case Self type of a method captures both an with bound refrence from method and a refrence to the new Self type of the impl;
• fn => here, these references are passed as an implicit last arguments;
• trait => in this position, with bounds serve as declaration that all and every impl of the trait must adhere the same with bounds;

Passing references around and borrow checking:

All implicit structs crated by desugaring have their lifetime as the least of captured references and original Self type attached lifetimes. (known rule, may be understood but is here for clarity)

Closures capture references from with expr. if needed.

And as of we want to avoid cases like this we have to forbid usage of mutable with sourcing references, in methods taking &self.

[tema2]

Not by move, only by references, the concrete type is elided by the same rule and in the same scopes as plain impl Trait

[ibraheemdev]

@mathstuf

How is having to add with(…) bounds everywhere better than adding a parameter? Or are these dependencies going to be invisibly added/required at various places?

Well yes that's the point. Lower-level functions that access the parameters directly have with bounds that will be fulfilled higher up in the program, like the log example I gave. Every function adding with(...) bounds would defeat the it's purpose.

[tema2]

@mathstuf

Or are these dependencies going to be invisibly added/required at various places?

This is actually makes sense, and is possible. The only thing to get it is to alter semantics from: "explicit introduction, explicit involvment" to "expl. introduction, implicit propagation, explicit requirements, explicit use".

And we got painless with clauses.

However, consider this example:

struct Context;

mod A {
   fn snd() with (&mut Context) {
      
   }
   pub fn fst() {
      snd()
   }
}

mod B {
   fn works() {
      ...
      with(...) {
         A::fst()
      }
   }
}

Depending on contents of B::works (what reference goes to with expr.) we can have an error about not providing implicits to private function.

Also what should error messages look like?

Another concern about this approach is how to implement it?