Feature idea: Edition dependent names / “replacing” standard library items

Regarding prior discussion

I’ve seen this before somewhere, but I can’t remember where. It was described in terms of: We could re-claim namespace in std in order to “delete” deprecated items for one edition, then in a later re-use the same name for something new. Please anyone who knows, provide a link to previous discussion(s).

I don’t remember any particular syntax being proposed in previous discussion. I do remember very long discussions about how editions can’t “delete” anything and why-would-you-need-this and how-does-this-look-in-the-documentation and whatnot. This is besides the point because I’m having a different example in mind. Maybe the point about documentation still applies.

Let’s not think too much about deprecating anything, or about reclaiming names from deprecated items in order to use them for something entirely unrelated. Sure that may be possible with the proposal, too, but can be discussed separately when it’s needed.

Context

I was reminded today of the fact that Vec::retain has the wrong type signature and it’s a known issue for over 6 years now. It will remain an issue for the next decades if we do nothing about it.

current signature

impl<T> Vec<T> {
    pub fn retain<F>(&mut self, mut f: F)
    where
        F: FnMut(&T) -> bool,
    { … }
}

correct signature

impl<T> Vec<T> {
    pub fn retain<F>(&mut self, mut f: F)
    where
        F: FnMut(&mut T) -> bool,
    { … }
}

Changing this signature is unfortunately a breaking change. There doesn’t seem to be any good way around it either, otherwise we’d have a fix for it by now. Indeed, the best proposal right now seems to be to just add another method™ with the proposed name retain_mut. But it would be dissatisfying to have two different methods for doing the same thing; deprecating retain for retain_mut seems weird, too; retain is a much nicer name. Etc… So let’s try discussing an edition-based approach.

I’m not imagining that this could happen in edition2021, but my hope is that we can rectify the situation within 3 years time when an edition2024 would (/might) come.

Changing name resolution based on edition isn’t new either. The 2021 edition changed the meaning of .into_iter() on an array-typed variable. Admitted, that’s method resolution, technically, but I think it goes into the same direction: avoid breakage of a small-but-technically-breaking change with the help of editions.

Feature Idea

Allow definition of multiple items with the same name but applicable to different editions. Syntax example for what’s written in the standard library

// in the standard library
impl<T> Vec<T> {
    pub fn edition2021#retain<F>(&mut self, mut f: F)
    where
        F: FnMut(&T) -> bool,
    { … }
    pub fn retain<F>(&mut self, mut f: F)
    where
        F: FnMut(&mut T) -> bool,
    { … }
}

The definition of such items is only allowed in the standard library. One of the set of items with the same name (but different prefix) must be prefix-free.

However using them works in user crates, too, as follows

// some crate; the edition of this code is edition2021
fn main() {
    let mut foo = vec![1, 2, 3, 4];
    foo.retain(|&i| {
        i % 2 == 0
    });
    let mut bar = vec![1, 2, 3, 4];
    bar.edition2024#retain(|&mut i| {
        i % 2 == 0
    });
}
// another crate; the edition of this code is edition2024
fn main() {
    let mut foo1 = vec![1, 2, 3, 4];
    foo1.edition2015#retain(|&i| {
        i % 2 == 0
    });
    let mut foo2 = vec![1, 2, 3, 4];
    foo2.edition2018#retain(|&i| {
        i % 2 == 0
    });
    let mut foo3 = vec![1, 2, 3, 4];
    foo3.edition2021#retain(|&i| {
        i % 2 == 0
    });
    let mut bar1 = vec![1, 2, 3, 4];
    bar1.retain(|&mut i| {
        i % 2 == 0
    });
    let mut bar2 = vec![1, 2, 3, 4];
    bar2.edition2024#retain(|&mut i| {
        i % 2 == 0
    });
}

The idea here is that typically you wouldn’t want to use these edition… prefixes. However when upgrading to a new edition, any mention of an item like retain that changes meaning in the upcoming edition would get a editionDDDD#-prefix by cargo fix --edition.

This proposed syntax uses the newly reserved foo#bar identifier syntax from edition 2021. The already reserved prefixes of the form editionDDDD (where DDDD is the date of an existing edition) get a meaning with this feature proposal. One could also use crater runs to determine if any pre-2021 edition crates would break from reserving all the editionDDDD prefixes (where DDDD is a 4-digit number) to get support throughout all rust editions.

More details

If the standard library defines

editionN#foo { definition1 }
editionM#foo { definition2 }
foo { definition3 }

(where N < M) then for an edition X, writing editionX#foo resolves to

  • definition1, if XN
  • definition2, if N < XM
  • definition3, if M < X

Writing foo in a crate that’s on edition X is the same as writing editionX#foo.

Writing editionX#foo only works if X is an edition that already exists and is stabilized; however, X may be higher than the edition that the crate writing editionX#foo is on if the crate is not on the latest stable edition yet.

We could also allow edition_next#foo for an upcoming edition (it might be worth using a shorter prefix than “edition_next”). This way, a stably usable new retain implementation could be offered via edition_next#retain once this feature idea has been implemented and stabilized. Once an upcoming edition like edition2024 is released, cargo fix --edition would replace any usage of edition_next#foo on edition 2021 with edition2024#foo, then the edition could be upgraded, then the post-upgrade cargo fix --edition would remove the prefix. I guess, we’d also want something like edition_next_after_2021#foo for using the newly stable foo in an older edition than 2021. Presumably, once 2024 ist released, this one could be replaced by edition2024#foo in older code as-well by ordinary cargo fix.


The standard library will make sure to define all the edition2024#foo items before edition 2024 is released/stabilized.

Macros

In order not to break macros that use paths like std::vec::Vec::retain, when this path is resolved, the edition of the code that the token retain comes from counts.


When upgrading code containing macros to a new edition, it’s not clear how to detect names in macros being used for such items. Maybe macro_rules macros might just get edition-warnings pointing out any identifier retain appearing with the definition; or successful edition upgrading of crates with macros must rely on sufficient test cases being present so that there’s an actual some usage in a path to an edition-dependent item that can be traced back to the ident token from the macro. Automatic fixing for procedural macros seems off-limits, but sufficiently-tested proc-macros could generate edition-upgrade warnings as well (that would need to be manually fixed).

use  and  pub use

When re-exporting a module that contains edition dependent names, e.g. path::to:m::edition2021#foo and path::to:m::edition2024#foo both exist, and you’re doing any of

use path::to:m;
use path::to:m as n;
pub use path::to:m;
pub use path::to:m as n;

then all the versions of foo enter the new scope, you can use m::edition2021#foo or n::edition2021#foo, etc. Similar for glob-imports,

use path::to::m::*;
fn main() {
    edition2021#foo();
    edition2024#foo();
}

If you import the item itself, then only a single version is imported, so you can do

use path::to::m::foo;
fn main() {
    foo(); // works
    // edition2024#foo(); doesn’t work!
}
// this code is on edition `2021`
use path::to::m::foo;
// the above is equivalent to
use path::to::m::edition2021#foo as foo;
// or
use path::to::m::{edition2021#foo as foo};
// the latter two versions will keep working in future editions

And cargo fix --edition would add the edition-prefixes together with the corresponding as … to the use statements, accordingly. I haven’t thought through the implications of how to do this in macros.

std-internal re-exports of items could do something like

mod m {
    fn editionN#foo() {}
    fn editionM#foo() {}
    fn foo() {}
}
pub use m::{editionN#foo, editionM#foo, foo};
// or equivalently
pub use m::{editionN#foo as editionN#foo, editionM#foo as editionM#foo, foo as foo};

but for user crates, edition-prefixes on the RHS of the as (or without an as) aren’t allowed. I haven’t thought through the backwards-compatibility implications of this rule for macros.

Possible extension

We could also allow “removing” items entirely for some editions. The definition in the standard library could look like

from_edition2018_to_edition2021#foo { definition1 }
from_edition2027#foo { definition2 }

this way, foo (without prefix) wouldn’t be defined at all in editions 2015 and 2024. Similarly, explicit edition2015#foo or edition2027#foo wouldn’t work either. The previous example

editionN#foo { definition1 }
editionM#foo { definition2 }
foo { definition3 }

would be equivalent to

from_edition2015_to_editionN#foo { definition1 }
from_editionA_to_editionM#foo { definition2 }
from_editionB#foo { definition3 }

where A is the edition right after N, and B is the edition right after M. In general, editionX#foo-definitions translate to from_editionC_to_editionX#foo definitions that are as large as possible without producing overlap. If overlap cannot be avoided even with maximally short translations, then the set of definitions is rejected (compilation error).


There’s no to_editionDDDD#foo, because that’s essentially what editionDDDD#foo-definitions are supposed to mean. The extension that items could be removed also drops the requirement that one of the items must be prefix-free.

All of the above in “Possible extension” so far only addresses the definition site, i.e. the standard library.


Onto the user’s side. Names that fall into the “gaps” don’t exist at all. E.g. by marking an old deprecated trait method as follows

trait Foo {
    fn edition2021#foobar(&self) {/* with default impl */}
}
// note how this only works when we have dropped
// the requirement that one of the items must be prefix-free

when third-party crates offer an extension-trait

trait FooExt: Foo {
    fn foobar(&self);
}

then foo.foobar() calls can successfully resolve to FooExt::foobar in code on edition 2024 or later. This way, not only the standard library but also external crates can benefit from “reclaimed namespace” like that. (On 2021-or-older-edition code, the call must be disambiguated FooExt::foobar(&foo).)

I want to repeat that this idea of “reclaiming namespace” is only part of a possible extension of the main proposal and not the main intended use-case that I’d like to discuss for now.

Possible changes/extensions that came up in the discussion so far



22 Likes

Seems like a sensible enough idea for future-proofing std, although I very much don't like the syntax. Why do we need new syntax for this at all? Couldn't it be a forever-unstable #[attribute] instead?

4 Likes

I think the desire is to be able to call the wrong edition's version in some cases.

On the defining end it could very well be #[attribute] based. I found the syntax I used nicer and used it for demonstration purposes, furthermore rustdoc could use a similar syntax for documenting the different versions. But the definitions are std-internal, so the syntax could be anything.

On the calling end, prefixes allow lightweight syntax for calling into the other-edition-versions of an item, as @mcy mentioned above.

When you're programming on the latest available edition, you should never need to use any of these prefixes though; e. g. with retain, one should never need the old FnMut(&T) version when writing new code.

1 Like

What if the the identifiers are defined in a crate but the path is assembled in another crate? For example if the macro matched on $($segment:ident)::+. Seems unlikely though.

In general, anything edition related can only be 'best-effort' when macros are involved. Proc-macro have the ability to assign arbitrary edition spans to their output tokens (via Span::resolved_at), defeating any attempts at detecting the 'real' edition of a token. I think using the span of the token is the most sensible choice, and is most likely to do the correct thing.

4 Likes

Isn't it simpler to have something like this? Rust already has a number of places that can have a #[cfg] and this mechanism is well understood.

#[cfg(edition(2015)]
fn test() {
    // something
}

#[cfg(edition(2018)]
fn test() {
    // something else
}

#[cfg(edition(2021)]
fn test() {
    // other thing
}

Or am I missing something?

Edit: but in the case of Vec::retain it seems better to just create a new method with the right signature, and (maybe?) deprecate the old method. This doesn't need to happen in an edition boundary.

2 Likes

A thought that's probably not at all easy, but might work: closure coercions.

For example, .map(Vec::len) might not work even though .map(|v| v.len()) would, because of things like autoref.

So it makes me wonder if we could find a way to change the signature to where F: FnMut(&mut T) -> bool, but still allow it to be called with an F: FnMut(&T) -> bool since if the function wants to pass a mutable reference, not just a shared one, that's ok since it can coerce to the less-restrictive type anyway.

Wasn't that suggested as part of the initial thread? Though I completely agree that for this specific case it would be a better fix.

cfg is conditional compilation. Editions aren’t something that’s consistent throughout the dependency tree, so you can’t do conditional compilation based on it. The example

fn edition2015#test() {
    // something
}

fn edition2018#test() {
    // something else
}

fn test() {
    // other thing
}

would be defining three methods, and all of them are callable from every edition, just the way that the prefix-free name foo is resolved is different depending on the edition of the user’s code.

If that’s a good idea, why hasn’t it been done within the last 6 years?

Other types (e.g. HashMap) do have a .retain method that offers &mut V access (to the values, not the keys), so creating a new method makes naming inconsistend. The point is: “retain” is a really good name and &T also works for many use-cases already. What should the new method be called? Something like retain_mut next to retain seems redundant. Resolving redundancy by deprecating seems harsh.

This proposal is basically a twist on the proposal of adding a new method: the new method just has the same name. With a short version of a prefix for what’s described as edition_next#foo in the original post above, (maybe something like x#foo?), we can make a “change” to retain without needing to wait for an edition-boundary; users can immediately use vec![1, 2, 3].x#retain(|x: &mut T| /* … */) to use the new method. Once the next edition comes and the user upgrades their code, the automatic migration process will be able to eliminate the x# prefix.

10 Likes

Note that we can do both. In case this proposal gets implemented (and stabilized) before closure coercions work, we can offer the new retain method right away (immediately usable with a prefix, prefix-free from the next edition onwards). If at any later point in time, we have a closure coercions feature that’s in a state such that the old version of retain can be changed to using &mut T arguments in a backwards-compatible manner, we can re-unify the two versions of retain.

In this proposal, there’s no reason why you wouldn’t be allowed to put a (redundant and effectless) editionDDDD# prefix before any item (even if it isn’t an edition-dependently named item from the standard library). There’d be a warning for those for sure and probably an automatic fix to remove redundant prefixes, but turning retain back into a simply non-edition-dependent function wouldn’t be a breaking change.

The edition20XX# syntax is quite tedious and takes quite a lot of horizontal space. I believe it should be abbreviated into e21#.

Abbreviations are a reasonable extension or change to the proposal.

Another idea I’m having is that the prefix names could be independent of edition names, so we could have something like v1#retain and v2#retain (and in general v𝘯 for some number 𝘯). The standard library would still use some syntax (e.g. attributes) to define which editions ranges correspond to v1-default and which correspond to v2-default.

1 Like

I want to push-back against this a bit -- this makes running cargo fix required to switch to the new edition. The original epoch RFC says:

Hard constraints: warning-free code on edition N must compile on edition N+1 and have the same behavior.

source

I don't think either of these would be nice:

  • require updating all retain to v1#retain on edition upgrade
  • emitting a warning for retain old edition, and requiring to change to v1#retain on old edition (you might not want to switch editions at all)

It seems that, when we want to change X to Y, we generally need at least three editions: original edition where X is totally fine and warning-free, edition where both X and Y are available (under different names), and where X starts to warn mid-edition, edition where X is not available and Y is renamed to X.

It can be argued that the original RFC's hard constraint is too hard, and that atomic upgrades via cargo fix are OK, but that would probably needs to be a separate RFC. Personally, I would be worry a lot about making cargo fix a required part of edition upgrade process -- due to macros, 100% correct automatic code transformation in Rust is impossible, and 95% solutions have a drawback that they might work OK for relatively small, relatively well-maintained crates.io crates, but might hit various edge cases in the dark corners of various private monorepos.

5 Likes

The introduction of a warning for something that cargo fix changes was kind-of implicit in my original post. But I’m not sure if the warning must be enabled by default. Looking at the upcoming edition 2021, it seems to me -Wrust-2021-compatibility warnings are not enabled by default, and similarly for -Wrust-2018-compatibility warnings on edition 2015.

$ rustc +nightly -W help | rg 'keyword-idents|anonymous-parameters|tyvar-behind-raw-pointer|absolute-paths-not-starting-with-crate'
                     absolute-paths-not-starting-with-crate  allow    fully qualified paths that start with a module name instead of `crate`, `self`, or an extern crate name
                                             keyword-idents  allow    detects edition keywords being used as an identifier
                                       anonymous-parameters  warn     detects anonymous parameters
                                   tyvar-behind-raw-pointer  warn     raw pointer to an inference variable
    rust-2018-compatibility  keyword-idents, anonymous-parameters, tyvar-behind-raw-pointer, absolute-paths-not-starting-with-crate
$ rustc +nightly -W help | rg 'ellipsis-inclusive-range-patterns|bare-trait-objects|rust-2021-incompatible-closure-captures|rust-2021-incompatible-or-patterns|rust-2021-prelude-collisions|rust-2021-prefixes-incompatible-syntax|array-into-iter|non-fmt-panics'
                    rust-2021-incompatible-closure-captures  allow    detects closures affected by Rust 2021 changes
                         rust-2021-incompatible-or-patterns  allow    detects usage of old versions of or-patterns
                     rust-2021-prefixes-incompatible-syntax  allow    identifiers that will be parsed as a prefix in Rust 2021
                               rust-2021-prelude-collisions  allow    detects the usage of trait methods which are ambiguous with traits added to the prelude in future editions
                                            array-into-iter  warn     detects calling `into_iter` on arrays in Rust 2015 and 2018
                                         bare-trait-objects  warn     suggest using `dyn Trait` for trait objects
                          ellipsis-inclusive-range-patterns  warn     `...` range patterns are deprecated
                                             non-fmt-panics  warn     detect single-argument panic!() invocations in which the argument is not a format string
           rust-2018-idioms  bare-trait-objects, unused-extern-crates, ellipsis-inclusive-range-patterns, elided-lifetimes-in-paths, explicit-outlives-requirements
    rust-2021-compatibility  ellipsis-inclusive-range-patterns, bare-trait-objects, rust-2021-incompatible-closure-captures, rust-2021-incompatible-or-patterns, rust-2021-prelude-collisions, rust-2021-prefixes-incompatible-syntax, array-into-iter, non-fmt-panics

Seems like the current situation is not that cargo fix is required, but that enabling -Wrust-XXXX-compatibility is required. Something that cargo fix --edition does, too.


Even with a warn-by-default warning that fires when using X on the original edition, you could introduce Y as a new name for the old X. (I.e. Y becomes a nicer alternative to v1#X.) Using Y on the old edition wouldn’t warn; tooling could support automatically replacing X with Y.

Edit: This might be comparable with panic! macro changing meaning and the introduction of panic_any! as an alternative to keep the old meaning. Note that the lint about panic is warn-by-default.

I suppose the decision between whether warning by default or not warning by default is better could perhaps be best done on a case-by-case basis, so that some attribute on the standard library definition decides the warning behavior.

2 Likes

FWIW, this does remind me of inline namespaces in C++, which address a similar concern. However, this proposal looks more flexible.

Would these names be unstable to define, and then stable to use, unstable to define and unstable (or impossible ) to use, or stable to define or use?

I can think of a few cases where it may be beneficial to define edition-dependant names in user crates - for example, if you had a trait that closely mirrors a stdlib trait.

Another question would be if it's possible to compile the body of functions defined with edition-dependant names as-though it were that edition (but excluding macro definitions). If so, that may be a useful feature if the names would be stable-to-define.

1 Like

That's interesting! I would probably expect them to be enabled by default before we switch the edition, so that people can update code incrementally, instead of doing one big-bang update when the edition comes out.

Unstable to define (i.e. only an compiler-internal ability); stable to use, at least as far as this proposal goes.

An extension to user-code definition seems like a more reasonable discussion to have in case we’ve actually added an edition-dependent trait to the standard library. I guess, it does make sense to collect more candiate std-items besides Vec::retain and VecDeque::retain that can benefit from the proposed feature, if anyone can think of any right now, please mention them; and we could discuss the potential benefit of user-crates mirroring std traits once there is a concrete trait that we’d want to change – however… I don’t think that using this feature for changes that warrant “user crates mirroring stdlib” cases would be a good idea in the first place. If there is a trait Foo that we want to replace, there’s going to be a need to still name the old trait. So you should properly give it a new name. But that feels to me like a case of doing renaming (with deprecation of the old name), and then re-claiming deprecated space.

My main example being retain is in line with this feature being intended to be used for small, yet technically breaking changes. We could even use it to e.g. replace a function with a more generic one if there are concerns that a significant number of use-cases would suffer breakage due to worse type inference. I can’t imagine need for user crates to track std for those kinds of changes. By the way, even if “mirroring stdlib” becomes something that’s wanted, user crates can also make major version bumps. They can even minimize ecosystem-splitting on a major version bump with the semver-trick.

I don’t understand this question fully; you don’t explain what you’re planning to achieve with this. There isn’t any mechanism to do more-fine-grained-than-crate-level edition changes in Rust for anything else yet either, so I’d find such a thing confusing. If you want to avoid re-typing editionDDDD#-prefixes, then use statements can help only needing to do that once. Anyways, you’re somewhat guarding this question on “names being stable-to-define”, so maybe I don’t even need to answer.

1 Like

I think that that’s a valid thing to think about; however as things stand, as quoted from rustc output above, take note that AFAICT the two rust-2018-compatibility lints absolute-paths-not-starting-with-crate and keyword-idents were never warn-by-default. Your code containing a local variable called async would simply break without warning when you’re upgrading edition from 2015 to 2018. Hence similarly, I don’t really expect that the four rust-2021-* lints will ever be warn-by-default.

Also, incremental updating is possible with explicitly activated -Wrust-DDDD-compatibility lints. I do agree that updating incrementally is a viable approach for crates to take, so even with disabled warnings-by-default, we should make sure that the necessary changes to get warning-free code when -Wrust-DDDD-compatibility is enabled don’t make the code look too terrible.