Generic Variables: simple implicity context passing

language design
21

In the case of free functions, its always valid, since any generic variables that it could use will be on the type signature, either directly (e.g <var X: usize>) or indirectly (the function takes a generic T and T has a generic variable). This means that once the function has been monomorphised, the compiler knows exactly which variables to pass in as arguments for each monomorphisation.

For methods/associated functions, it's a bit trickier. The rule would have to be "you can't use generic variables defined on an impl block in method bodies inside that impl block, unless the type of the impl block is directly mentioned in the signature of the method.". If that wasn't the case, you could create an associated function that uses generic variables defined on the type in its body, but otherwise takes no arguments. Then you could cast it to a function pointer, and now their is no indication on the type signature that is uses generic variables, or which generic variables.

Consider this example:

fn get_num() -> usize {
    2
}

struct Foo<'a, var X: 'a usize>;

impl<'a, var X: 'a usize> Foo<'a, X> {
    fn get_generic() -> usize {
        X
    }
}

fn assign_fn_ptr() {
    var X: usize = 10;
    let mut ptr: fn() -> usize = get_num;
    ptr = Foo::<X>::get_generic;
}

Here, the free function get_num takes no arguments, no generic arguments, and returns a constant, 2. The associated function get_generic on Foo copies the generic variable X defined on Foo and returns it. Both get_num and get_generic have the same type signature, so they can be assigned to the same variable. This works fine for get_num because it doesn't capture anything. But for get_generic, because it captures the generic variable X on Foo, X must be passed as an argument for it to function. But the compiler can't distinguish between get_num and get_generic when they are cast to function pointers. For get_generic to be valid, it would need to have Foo appear somewhere in it's type signature, either as a regular argument, as self, or in the return type. This could also include appearing as a generic parameter on another type, for instance returning a Result<Self, Error> also counts as 'appearing in the type signature', since it appears as a generic parameter on Result. What wouldn't count, would be being used as a concrete field on another type. That could work in theory, but would create fragile code which depends on the compiler 'seeing inside' private types/fields to do type checking.

With this rule in place we can still use generic variables on traits like Default, which return a Self and Drop which takes an &mut self, but it precludes their use trait methods where Self doesn't appear as part of the type signature.

1 Like
22

If we're going to require Sync though, you might as well just use a global variable. The feature might still have some use cases, but they will probably be paying for the cost of cross-thread synchronisation when they don't need to.

I agree the bar for adding auto-traits is high, but I think if Rust users can be convinced of the usefulness of this feature, then it could be done. But that seems to be the problem that keeps reoccurring, explaining the semantics of 'types that capture runtime variables, but you can express which variables are captured in the type signature.', is surprisingly difficult and often leads to misunderstandings. This also makes it hard to onboard people into the discussion and garner support.

23

So I think I've gotten a better handle on this based on some of the more recent examples. This really feels like adding dynamic scoping to Rust. This has come up before and I really don't like the type being the key upon which everything hinges. It makes it very difficult to implement modifiers or other "higher-order" things to these variables. Consider having a "render backend" global variable. I can imagine that low-level libraries would want to run tests with a given context with and without some feature (say, anti-aliasing) at some level in the stack. So one would want to do something like var Renderer = outer_Renderer.disable_fxaa(); at some level in the stack. Perhaps testing with off-screen rendering as well with otherwise-equivalent context settings. Also consider some logging library. Scopes would want to enable or disable features (var LogSink = outer_LogSink.enable(Level::Trace);). Or log to multiple locations (var LogSink = outer_LogSink.as_well_as(my_pipe);).

Basically, there's a reason we give names to explicit arguments rather than relying solely on the type: one might want to work with more than one instance of any given type in the argument list.

1 Like
24

I strongly disagree with this ā€“ the feature is a lot more useful than global variables even if you require Sync. For example, being able to store an allocator in a type means that objects of that type can allocate using the allocator and then know they're using the same allocator when deallocating, even if the allocator itself isn't stored in values of the type (and this is obviously useful if you have, say, a very large number of Boxes that all share the same non-zero-sized allocator, as it avoids needing to store a copy of the allocator in every Box).

One of the main motivating examples for me was related to an arbitrary-precision integer library I was writing, which could store numbers in any base, but didn't want to support mixing bases. A BigInt<10> could be added to another BigInt<10>, but not to a BigInt<2>. Unfortunately, const generics wouldn't work for this because the programs using the library wouldn't know the base until runtime. Here, the base is just an integer being interpreted as an integer, so is obviously Sync, but it's still useful to store it in the type (both so that you don't need to store the base on every integer, and so that the implementation of Add would be able to in the typechecker prove that the two numbers being added had the same base).

25

I am not sure I understand what the problem is. What do you mean by "modifiers" and "higher-order things"? The examples you gave seem like they could be solved with interior mutability or just passing a regular argument.

It sounds like you're asking for something functional in nature, being able to express exactly how a variable is mutated or which iteration of a variable you are using at each point the program, but that's outside the scope of my proposal.

26

The problem here is that you're trying to fit this new feature in the current lifetime system, but they are different things, albeit very similar.

I think a better solution would be to have this new kind of generics be infectious like lifetimes, but not be lifetimes themselves.

If I convert a fn foo<var X: usize>() { ...} to a fn() then the information about the generic variable X is lost in the same way it's lost with a trait object.

Remember that free functions and methods/associated functions are exactly the same behind the scenes.

27

Interior mutability means adding Builder-like RAII objects to undo anything done within the scope (cf. I/O Stream-State Saver Library to avoid leaking C++ stream state changes within a scope except asking T for its state in a restorable way without T: Clone is basically futile). Why not support "I'd like to take whatever is there, apply these changes, then they can be popped back like any other Rust scoping"?

28

I love the blogpost about contexts and with passing by tmandry. You framed your pdf as your view on how to fit it into Rust, right? The surface of those generic variables is really big while I don't see that much value - yeah, you get to use the idea of context without storing the pointer to the allocator. But you can as well have ZST allocator that had with in it's signature to get a provided allocator, it will be 100% equivalent at the runtime while require a lot, a lot less disrupting changes.

In other words, all the motivating example I've seen for this feature can be reimplemented with the machinery of the post by tmandry you linked.

29

What disruptive changes? The new keyword? The new auto-trait I suggested doesn't have to exist if we force all generic variables to be Sync, so we don't need to add the implicit bound for backwards compatibility. And yes, if apply the with clause idea to types, then both proposals function the same, but then why do you see my proposal as more disruptive if it's only the syntax that differ?

30

If you don't need simulations access to each version of the state, you can just use interior mutability to make and then pop the changes.

If you're okay with cloning, you could just create a new var in the inner scope with a cloned and mutated state.

If neither of those suffice, I am not sure what you are asking for exactly.

31

But you can't do that, not safely. Just like you can't cast fn foo<const N: usize>() to fn(), those type signatures are different.

If the function has generic variable annotations then you can't cast it to an fn pointer. Since a generic variable in that position works like a parameter, you can't cast it away like you could fn foo<T>() using let _: fn() = foo::<u32>;. In this regard it is more like a fn(usize) then a fn<T>().

32

I meant to instantiate the foo function with some generic parameter, and then cast it to a fn().

Instantiating a fn foo<const N: usize>() with a compile time known N is fine, the compiler just creates a new copy of the function with that value.

But how would instantiating a fn foo<var N: usize>() behave?

33

Using the "variable-capturing types should work just like lifetimes" theory, I assume that you wouldn't be able to cast a function item that captures a variable (either directly as a generic parameter, or via a generic parameter of a type) into a function pointer for the same reason that you can't cast a fn foo<T>() to a fn() if T is not 'static.

One complication is that In current Rust, you can actually cast away a lifetime like this. But doing so is unsound despite being possible in safe code. Presumably, a fix would mean that function items which have a lifetime among or inside their generic parameters wouldn't be castable into function pointers, and you could use the same fix for function items that have a variable among or inside their function pointers.

34

You just can't do that. It doesn't make any sense. It's like casting a fn foo(x: &usize) to a fn() by doing this let _: fn() = foo(15);. At best, because its a runtime value, you could cast it to a closure, but not a plain function pointer.

In short, it would be a compilation error.

35

I believe you're misreading that issue. In that case the function does not look like fn foo<T>() { ... } or fn foo<'a>() { ... }, instead the lifetime is part of the return type (i.e. fn foo<'a>() -> &'a i32) while being ignored for the purpose of the 'static check. That's what allows the unsoundness, because this can be passed to an associated type, making it !'static even though the function itself is (unsoundly) considered 'static.

In the case of casting to fn() however there's no such issue, as the final type does not contain the lifetime and hence is surely 'static. In other words, I don't believe any solution for that issue will break the following code:

fn foo<T>() {
    todo!()
}

fn bar<'a>() -> fn() {
    foo::<&'a ()>
}

However this does break if instead of a lifetime you had a generic var. Hence I don't think you can treat a generic var like a lifetime in this context, because the generic var must always be considered part of the signature, even after instantiating the generic function, while a lifetime does not need this restriction.

You're then arguing that a generic var is equivalent to a parameter on a function rather than equivalent to a lifetime. That was the original point that I wanted to show:

36

It allows you to create a value with a lifetime outside that lifetime. For example, you can do something like this:

use core::fmt::Debug;

fn foo<T: Default + Debug>() {
    dbg!(T::default()); // creates &'a mut str
}

fn bar<T: Default + Debug>(_: T) -> fn() {
    foo::<T>
}
fn main() {
    let a = &mut *String::from("abc");
    let f = bar(&mut *a); // T is &'a mut str
    a.make_ascii_uppercase(); // ends lifetime 'a
    (f)()
}

In this case, foo creates and debug-prints a &'a mut str even though the lifetime 'a has already ended. You aren't supposed to be able to use values outside their lifetimes in Rust. (I am not sure whether this can be converted into a segfault or the like using only safe Rust, but it wouldn't surprise me.)

37

This is actually not a problem, lifetimes are erased at runtime so a function cannot change its behaviour depending on them. So even if at the type level the function might indirectly name a lifetime, without values (i.e. inputs or outputs) directly tied to that lifetime the function will be valid anyway.