Implicit Parameters

This isn't an RFC. I just wanted to bring this up for discussion and see what other people think about this idea.

I have been programing a bit more in Scala recently, and there is a feature in scala called Implicit Parameters.

I'm not 100% knowledgeable about all the implementation details in Scala but here is a quick pseudo-rust example:

fn caller() {
    let implicit name = "world";
    hello(); // NOTE: no argument passed
}

fn hello(name : implicit &str) {
    println!("Hello {}!", name);
}

In the example above, the hello function takes an implicit argument called name. When the caller function calls hello, the compiler would need to wire in the implicit name variable into the implicit name argument.

In the web frameworks I'm using in scala, this feature is mainly used as a sort of dependency injection. Objects like database connections, rest clients, and actor systems are passed through functions implicitly.


There has been recent talk about parameterizing types and functions with Allocators and it got me thinking if a form of Implicit Parameters would be worth exploring. I'm imagining a flavor of implicit Parameters that resolve their values in 3 steps.

  1. Check to see if the implicit parameter was set explicitly: Box::new(..., allocator)
  2. Check to see if the implicit parameter is set in the local scope: let implicit allocator = ...;
  3. Implicitly add the parameter to the current functions paramaters:
    • fn my_fn() implicitly becomes fn my_fn(allocator: implicit Allocator)

I imagine that standard library functions, such as Box::new() would start requiring an implicit Allocator argument. The user can optionally pass in a local allocator, but if they choose not to, then their function will simply require that it's caller pass it one. This will continue up untill either a local allocator is defined, or until the caller is main() which will have the global allocator defined as allocator. allocator would be a special parameter set by rust, other implicit parameters would result in a compiler error if never defined.

Examples:

fn String::from_str(f: &str, allocator: implicit Allocator) -> String {
  ... // This function takes an allocator as a param, but that param is
}     // implicitly defined by the calling code.

fn library_code() -> String { // this header is implicitly converted to
   String::from_str("Hello")  // library_code(allocator: implicit Allocator)
}                             // because no implicit param named allocator was found

fn user_code() {    // this header is implicitly converted to
    library_code(); // user_code(allocator: implicit Allocator)
}                   // because no implicit param named allocator was found

fn main() {    // allocator would be a special implicit param, added by rust in main
  user_code(); // So user_code will be passed the global allocator implicitly
} // The param is completely implicit from main > user_code > library_code > std

fn user_code2() {                // This fn doesn't take an implicit allocator
   let implicit allocator = ...; // because one was defined in it's local scope
   library_code();               // So library code will use the local allocator
}

So for most situations, this wouldn't change the explicit interfaces. You would just call String::from(...) and an allocator will be implicitly passed in down from main. But for situations that use local allocators, this would enable the control of the allocations. Even in libraries, and libraries of libraries. As library code that performs allocation would now be deferring to the caller to pass in an Allocator automatically.


Some of the benefits I can think of:

  1. Might be useful as a sort of dependency injection.
  2. People/Situations that don't care about allocators still wouldn't need to worry about them.
  3. People/Situations that do care about allocators would have control over library allocations, even if the library wasn't designed with custom allocators in mind.

Some major drawbacks I can think of:

  1. It complicates the language
  2. Unforeseen/unaddressed complications: such as usage with traits
  3. Would probably require a new edition
  4. Might not even be possible with a new edition since it would change the standard library, older editions would need to wire in the global allocator.
  5. Implicit parameters would become explicit across FFI or older editions, might be verbose.
  6. This feature might have negative effects on performance
  7. Implicit changes to users API
  8. Might be overly burdensome on the compiler: needing to resolve implicit arguments.
  9. Might encourage/enable bad design and harder to read code.
4 Likes

I think the biggest drawback is the cognitive overhead for the programmer. They see where an implicit argument is declared and where it is used, but they don't see to which functions it is passed. It's almost like the argument becomes invisible and reappears later. There's also the risk to accidentally introduce breaking changes by adding a function call somewhere that transitively requires an implicit argument.

16 Likes

One drawback you didn't mention is that to make allocator parameters work as a zero-cost abstraction, they need to be generic and monomorphized. And that would make essentially every function that could transitively allocate into a generic function that needs monomorphization. I guess now you've solved library compile times, in that every bin compile now is effectively a clean build that monomorphizes the entire world :stuck_out_tongue:

8 Likes

This goes against lessons learnt from rust - explicit over implicit and that local reasoning is key.

Implicits in scala are too powerful and therefore overused and misused and therefore cause lots of difficulty especially for new users. Even after the redesign and simplification for the next version they are still too complicated IMO.

Instead of baking a global implicit variant of DI into the language proper I'd rather see the language have the needed facilities to allow for an ergonomic (and less subtle) di framework in a library. The big ticket items that would be useful for this are variadic genetics/tuples and compile time introspection. That would clean up the current generation of di solutions that seem to require macros & annotations. I worked with this style in Java spring and it is also overused and feels to me very much like a code smell compared to cleaner API based alternatives. The best middle ground that I found in my experience between tedious boiler plate and spooky action in the distance is something something like simple injector in C#: It doesn't polute my code base with annotations everywhere and it also provides an in-language explicit registration API that allows the programmer to reason about the code. This latter bit is crucial IMO. I don't want to guess how the framework or the compiler figured what to inject where based on magic, I rather have it listed in one simple to follow place in my code. No XML, no special resolution rules, just API calls and user discretion where/how to use it.

12 Likes

The non-locality effects of this would be devastating, at both type and value level. There is just no good reason I can think of for which I would want my function signatures to change silently based on their implementation.

In fact, functions in Rust are designed to have explicit, mandatory type signatures and no type inference exactly for this reason, so that the inner workings of a function are guaranteed never to influence its type signature after the fact.

Generics also have trait bounds (which is often contrasted with C++ templates): you can only rely on what you declare, in order to avoid bad surprises and accidental changes.

The non-locality also means that the compiler has to do more work: analyzing a function body is no longer restricted to that function body, but it needs to transitively descend into all callers in order to find out any implicit parameters.

All in all, I think any feature like this would conflict strongly with how Rust was designed to be used, and I would not welcome it at all. If you want to pass around database connections, make them an explicit parameter and refactor your code into a flat architecture so that it's less annoying to pass them around. I've done this kind of transformation before, it's certainly possible and makes so much more sense to future readers of the code.

13 Likes

Yes. Definitely. However the usecases for this are real. The more traditional solution to this is ThreadLocal variables. These generally work but have a couple of issues:

  1. In Rust they are not as cheap as they could be due to various issues. This precludes their use for the allocator.
  2. They don't work in an Async context where code is not tied to a thread.

I am not sure there is a way to avoid special casing the allocator.

The performance problems can be improved. However it cannot be made free or even particularly close, but that's not necessarily a problem. The prefered, default, and normal way to pass data should always be via explicit parameter.

It would however be good to have a solution that worked for both sync and async code. If there were some sort of 'Context' trait that were defined the provided a map-like interface, a default implementation could be provided for sync code, and alternative implementations could be provided by the various Async Runtimes.

There is work on designing and prototyping what task-local variables would look like, FWIW. Part of the reason that async passes around a Context rather than the Waker directly is so that we can add extra context such as task-local keying in the future.

But even right now, runtimes could implement task-local storage, piggybacking off of existing global executor state, without any extra language support.

To remove the spooky action at a distance, it could be required that implicit parameters must be either:

  • passed explicitly
  • declared locally
  • be themselves declared as implicit parameters of the current function

Adding an implicit parameter, like normal parameters would still be a breaking change, and would still need to modify the API of the the whole call-stack, but at least would not need to modify the implementation of any (recursively) callers. It would also be very easy to automate such change with an IDE.

fn String::from_str(f: &str, allocator: implicit &mut dyn Allocator) -> String {
  ... // This function takes an allocator as a param, but that param is
}     // implicitly defined by the calling code.

fn library_code(allocator: implicit &mut dyn Allocator) -> String { // the interface must be changed
   String::from_str("Hello")  // no change here. The allocator is passed implicitely
}

fn user_code(allocator: implicit &mut dyn Allocator) {  // the interface must be changed
    library_code(); // once again, no change in the implimentation
}

fn user_code2() { // no implicit allocator
   let implicit mut allocator = ...; // implicit allocator defined in it's local scope
   library_code();               // So library code will use the local allocator
}

// EDIT: added this function
fn user_code3(allocator: implicit & mut dyn Allocator) {
  user_code(); // the allocator from the argument is passed implicitely
  user_code_2(); // the allocator *isn’t* passed implicitely, it’s not the same allocator that will be used inside user_code_2
}

fn main() { // main cannot have implicit parameters
  let implicit mut allocator = Default::default(); // must be instantiated explicitly
  user_code3();
}

The main issue with this approach is that since Rust doesn’t have overloading, we can’t add a new implicit parameter to a method without it being a breaking change. So Box::new(value: T) cannot takes an a new implicit parameter without breaking change, so a new function would need to be created Box::new_with_allocator(value: T, allocator: implicit &mut Allocator).

But changing an argument from explicit to implicit isn’t a breaking change. So foo(x: Bar) can become foo(x: implicit Bar) since you can always pass the argument explicitly. If implicit needs an edition change, previous version could still call it without changes, as-if the implicit argument was just a regular explicit argument.

3 Likes

Even if technically feasible, how is that conceptually coherent?

1 Like

It is true that it solves that problem at least. However this deviates from current idioms and best practices of Rust. In particular the ? and the await operators were designed on purpose so all call sites need to be marked explicitly. And there is no need to avoid being explicit here anyway, since all the functions on the call chain would need to be refactored anyway when adding a new implicit parameter. So might as well update the call sites too for a more idiomatic result.

That leaves us with nothing more than a C++ like default parameters feature (in addition to the issue of function overloading already discussed).

1 Like

I'm pretty much in full agreement that the feature as presented has significantly more downsides than upsides. It's a tradeoff between complexity or fragmentation, and I agree that as is, is not the right tradeoff.


That said I'm still wondering if features adjacent to this would still be useful.

Imagine a flag that substituted the std library functions with ones described using an implicit allocator. The feature wouldn't be general, there would only ever be one implicit and it's called allocator. This might at least enable those working with custom allocators the ability to use crates that otherwise don't support them. At that point the feature should probably be called "Force Allocator Argument" or something along those lines.


Of course this might not be the right direction in any regards, but I'm glad it at least got a discussion :slight_smile:

2 Likes

For Allocation there is the work going on with "custom allocators" or "custom storage". And of course default type bounds the stdlib will get this for free without needing implicit arguments.

2 Likes

I believe the scoped_tls solves most problems mentioned here, though in somewhat verbose way.

You can change the global allocator. Global allocators - The Edition Guide

Similar to scoped_tls is illicit - Rust, which uses types rather than identifiers for its environment lookup and offers a proc macro that emulates implicit arguments (albeit with a runtime panic if called in a context without the type available).

EDIT: I should also say that I think approaching the implicit variable problem only from the angle of providing singletons is useful, as this is the way that most conventional thread locals I’ve seen get used. It seems to me like the full generality of scalas feature is part of what makes it scary.

Implicits in scala are too powerful and therefore overused and misused and therefore cause lots of difficulty especially for new users. Even after the redesign and simplification for the next version they are still too complicated IMO.

I've often seen this claim in discussions of adding implicits to Rust, but do you have a source?

I can easily imagine how implicits could lead to problems in principle, but I'm curious how much that's the case in real-life, actually written code.

1 Like

I think it'd be acceptable if it worked only within a single function's scope, one level down.

i.e. if you had:

fn a(implicit) { b(); }
fn b() { c(); }
fn c(implicit) {}

then implicit argument for a wouldn't be available in c, because b doesn't explicitly pass it through.

However, before Rust gets implicit arguments, I think it should decide whether it wants optional/default arguments, because this looks like an optional argument with a customized default value.

4 Likes

Just some notion I picked up from people that used to program Scala. They said implicit parameters 'ruined' Scala for them. Sure there is nuance in how they are implemented, but in general they go too strongly against the principle that code should be optimized for reading, and easy local reasoning. There might be some places where they are a true benefit, but history has shown, at least for Scala and the people I've talked to this balance is very off, and it's misused much more often than not.

5 Likes

The only way I could see implicit parameters work would be if their presence was made known somehow, e.g.:

fn takes_implicit_params(a: i32, *) -> i32 { ... } 

Still not even remotely a fan, though.

That could be described as an "explicit implicit". As I said earlier in this thread, such a thing might actually be implementable, but once you think about the notion of an explicit implicit, you notice that the very concept makes no sense.