[pre-rfc] Stable features for breaking changes

[nikomatsakis]

I think the previous paragraph also applies to the idea of ecosystem splits caused by the use of new features – that is, it IS an important problem, but probably one that we have to address in other ways.

OK, my blizzard of comments stops with this one, but I just wanted to add: it might be worth linking this proposal to others intended to target “fear of upgrading” in all forms. Unclear.

(e.g., I think there is an interaction with the cargo schema RFC.)

[seanmonstar]

“Stability” means you should never dread upgrading the compiler. It means that that code written for different versions of the Rust compiler and standard library should work seamlessly together

[…]

Ideally, it also means that we can produce versions of Rust that, for example, essentially remove (or change) deprecated features.

I feel like these two statements are at odds with each other. The first part, never dreading to upgrade the compiler, should be true for projects that have laid dormant for months. Imagine building a binary that fulfills its purpose, doesn’t have any big crashes, and so you just put the binary in production or wherever and forget about it for a year. Then, you decide you need to add some feature. Should you worry that upgrading the compiler after so long will mean features have been removed? Hopefully not!

Coming from a server development position, I can say that in many places that I’ve worked at (even at Mozilla, as forward reaching with technology as we are), it’s very common to not keep up with the latest version of a language. We actually specifically do not want to run the latest release in production. We want all horrible bugs and vulnerabilities to be patched out of new features before we ship that stuff. So, it’s actually quite common for us to upgrade huge versions at a time, often times from one LTS to the next. It’d be sad if someone in our position felt we couldn’t upgrade because then we’d need to do all this busywork fixing up our code for the newest compiler.

Calling these Epochs doesn’t do much for me. You’ve just changed the name of the thing we call “versions”. Instead of Rust v2.0, we just have Rust v2017. If Rust v2018 has features removed that existed in 2017, it’s essentially 2.0 (or 3.0 or whatever) and not backwards-compatible.

JavaScript has seemingly moved away from single digit version numbers to years as well. We had ES5, and ES6, but now we’re talking about ES2017 and ES2018. It doesn’t really matter. What matters is that JavaScript I write now, in 2017, will still do exactly what I told it to with a browser that implements ES2025. That’s backwards compatibility.

In Epoch 1, we deprecate Iterator as a type and suggest people migrate to dyn Iterator.

I care a lot about backwards compatibility of the libraries I write, perhaps more than many. While I appreciate the desire to get people to use the newer syntax, I’d dislike the compiler actively nagging me to use something that would make hyper no longer compile for users on older compiler versions. In this case, fixing the nagging requires me to completely remove support for the versions of the compiler that don’t understand dyn Trait. I think that to make such a change, I’d need a way to tell the compiler “OK sure, on #[cfg(has_dyn_trait)], I’ll use dyn Trait, but if not, please just use the older and keep working.”

If it were possible for the compiler to process cfg attributes before trying to parse the syntax gated by them, then this scheme could work. I can make use of the rustc_version crate to conditionally make use of new libstd APIs, but not to make use of new syntax features.

[tom.prince]

There is some talk of how python handled incompatible changes via from __future import ... to allow opting into new features. That feature has been used for more than just the 2->3 transition (by my count, 3 of the 7 __future__ imports are for things that became default before 3.x).

One of the nice things about having granular control (and why a bunch of people were unhappy with the 2->3 transition) is that you can make the fixes for each change independently, and then test and deploy your code with just those changes. One of the big troubles with the 2->3 transition is that you had to fix your code for all of the changes at once.

I think that epochs make sense as well, having an ever-growing feature list doesn’t make sense, so every so often collecting all the stable features and defaulting them to on in a new epoch would be a good thing.

[lukaramu]

One thing to note about the Haskell is that GHC (the de facto standard Haskell dialect’s compiler) has, in addition to adding language-level features behind {#- LANGUAGE -#} pragmas, also made major non-standard breaking changes in Haskell’s standard library in the past; e.g. implementing the [Applicative Monad Proposal][1] and the [Burning Bridges Proposal][2] two years ago (see also [GHC 7.10’s release notes][3]). For the former proposal, the compiler issued warnings for code that would be broken by it starting with [GHC 7.8][4] (which was about a year lead time). For the latter proposal, this wasn’t done, as it landed relatively late in their release cycle. This was not uncontroversial (see some advocacy against immediately including it [here][5]). In the end, they decided against intrucing a {-# LANGUAGE #-} pragma for it. I haven’t used Haskell in a some time and don’t know in which ways this affected their ecosystem, but I think it’s something worth investigating. [1]: https://wiki.haskell.org/Functor-Applicative-Monad_Proposal [2]: https://wiki.haskell.org/Foldable_Traversable_In_Prelude [3]: https://downloads.haskell.org/~ghc/7.10.3/docs/html/users_guide/release-7-10-1.html [4]: https://downloads.haskell.org/~ghc/7.8.4/docs/html/users_guide/release-7-8-1.html [5]: https://ghc.haskell.org/trac/ghc/wiki/BurningBridgesSlowly

[nikomatsakis]

I care a lot about backwards compatibility of the libraries I write, perhaps more than many. While I appreciate the desire to get people to use the newer syntax, I’d dislike the compiler actively nagging me to use something that would make hyper no longer compile for users on older compiler versions.

It seems to me that this solved by having the ability to #![allow()] the deprecation warning. One shortcoming in our current system (something @wycats has pointed out again and again…) is that you want a more targeted way – i.e., the ability to allow certain deprecations but not all.

JavaScript has seemingly moved away from single digit version numbers to years as well. We had ES5, and ES6, but now we’re talking about ES2017 and ES2018. It doesn’t really matter. What matters is that JavaScript I write now, in 2017, will still do exactly what I told it to with a browser that implements ES2025. That’s backwards compatibility.

Just to be clear, we are proposing to maintain the same standard of compatibility. From the rest of your message, it sounds as if you might think otherwise, which probably suggests that we need to work on how we explain the proposal!

This is indeed precisely why we pursued the “Epoch” naming instead of saying Rust 2.0. Calling something Rust 2.0 suggests that upgrading to it is a “major version bump” and hence means you may face incompatibility. But the idea is that when you upgrade to the latest Rust release, you are not required to upgrade your code to the latest epoch. The only reason you would ever have to upgrade is to take advantage of new features (e.g., the catch keyword).

In the same way, when you upgrade your browser, you are not limited to nice JavaScript that uses let, modules, and all the latest goodies. You can still run the old stuff.

(Also, just to be clear, even the JS committee makes breaking changes from time to time, but only if they are very confident they can get away with it – e.g., because different browsers implement different behavior, and hence people are not relying on it. And yes, they test that this is true.)

[nikomatsakis]

One of the nice things about having granular control (and why a bunch of people were unhappy with the 2->3 transition) is that you can make the fixes for each change independently, and then test and deploy your code with just those changes. One of the big troubles with the 2->3 transition is that you had to fix your code for all of the changes at once.

This is an interesting point. I do hope though that we can make the upgrade fully automated. Indeed, in each of the examples that I gave, it would be possible to do a fully automatic – and 100% semantically faithful – transition, although it may do things you might not have done had you transitioned by hand.

This is fairly clear for the first few examples. But it’s also true for the match ergonomics example. In retrospect I didn’t fully appreciate this while writing the post and hence I didn’t emphasize it. But naturally if the compiler is unsure whether the destructor should run earlier or later, it could just conservatively insert a mem::drop into the match (i.e., to select the current semantics):

match opt_v {
    Some(v) => {
        println!("{:?}", v);
        mem::drop(v); // forces `v` to be moved, even under the newer proposals
    }
    _ => ... 
}

You could imagine the transition tool leaving behind markers when it makes conservative choices of this kind, that you could go and remove at your leisure. For example maybe it would generate:

match opt_v {
    Some(v) => {
        println!("{:?}", v);

        // rustfix: The following line could be removed,
        // if it is not important for the destructor of `v`
        // to execute early.
        mem::drop(v); 
    }
    _ => ... 
}

[seanmonstar]

It seems to me that this solved by having the ability to #![allow()] the deprecation warning. One shortcoming in our current system […] is that you want a more targeted way – i.e., the ability to allow certain deprecations but not all.

Being able to put an attribute on a specific expression is probably enough. I suppose some expressions could be exercising multiple deprecated things at once, but if you really wanted to, you could probably separate the expression into multiple.

But the idea is that when you upgrade to the latest Rust release, you are not required to upgrade your code to the latest epoch.

If that is the case, that sounds good. I feel that part of the error of understanding was on me, as part of my understanding came from noticing other people’s pull quotes, and that some of those quotes in isolation made me feel that way. If I’m not the only one that felt this way, then maybe that using a new epoch or feature set is opt-in should be made to standout more, with bold and fireworks.

[stjepang]

After reading this discussion I’m still left confused. Here’s how I understand the epochs proposal, so please correct me if I’m wrong.

There are two kinds of language upgrades:

1. New features are added. The existing parts of the language are unchanged. This is what I expect from upgrades to C++ and Java. If I have code that works with -std=c++11, I expect upgrading to -std=c++14 will be effortless. The only thing that might go wrong is name clashing, that’s it. This is just like our regular minor upgrades, e.g. from Rust 1.14 to Rust 1.15.

• Existing language semantics are changed. Proposed changes to match and traits fall into this category. This is like the Python 2 to Python 3 upgrade. Just like we would subtly change the semantics of match, Python subtly changed the semantics of / operator, for example. This kind of upgrade is like going from Rust 1.x to Rust 2.0.

I think that the Epoch idea is indeed very similar to what C/C++/Java does. I agree that people do sometimes avoid upgrading there, but I’m not sure if there is anything that could be done to alleviate that.

Epochs clearly fall into the category #2. They change the existing semantics, and I do not expect that from C++/Java upgrades (except perhaps a few small and reasonable fixes nobody will ever notice anyway).

Upgrading an epoch sounds like going from Python 2 to Python 3, except it is less frustrating because the most recent compiler version understands all previous epochs and can intermix Rust code written in multiple epochs.

Another concern I have is - how do we teach this to new users? How do I explain to a friend how match works? Do I say “hey, if you’re using epoch 2017, you need this ref to borrow the object rather than take ownership, but if you’re using epoch 2018 then ref is implicit”? What if they copy some code from StackOverflow and it doesn’t compile?

Epochs don’t just add stuff. They change existing stuff. C++ and Java don’t change existing stuff. Python changed existing stuff once and made a big mistake. We’re about to change existing stuff but much more gracefully than Python did…

Is this the right way to look at it?

[withoutboats]

1. New features are added. The existing parts of the language are unchanged. This is what I expect from upgrades to C++ and Java. If I have code that works with -std=c++11, I expect upgrading to -std=c++14 will be effortless. The only thing that might go wrong is name clashing, that’s it. This is just like our regular minor upgrades, e.g. from Rust 1.14 to Rust 1.15.
2. Existing language semantics are changed. Proposed changes to match and traits fall into this category. This is like the Python 2 to Python 3 upgrade. Just like we would subtly change the semantics of match, Python subtly changed the semantics of / operator, for example. This kind of upgrade is like going from Rust 1.x to Rust 2.0.

This is a common sentiment, but I don’t agree that its true.

First, C++ does make breaking changes & they’re not just name clashes. Here’s a list of breaking changes in C++11, and its not just the introduction of keywords but many subtle semantic changes.

The issue with Python 3 is, in my perception, not a subtle semantic change, but a blatant one - the core string type just completely changes its semantics in ways that are pervasive and difficult to upgrade across. This is different from match changes, where the breaking change is when exactly destructors run, and we can support a trivial ‘explication’ which gets you back to the form you had. (You refer to C++ making “a few small and reasonable fixes nobody will ever notice anyway”, but that’s exactly what the breakages in the match proposal are IMO).

Whatever we call them, I think we can’t support making changes as deep as the changes to the string type. But all of the changes we’ve seriously considered are, in my opinion, in line with the kinds of changes that are made in upgrades to the C standard.

However, our current compatibility guarantee is much firmer than what C++ version upgrades provide; we can’t even introduce new non-contextual reserved words. This makes sense - C++ upgrades every 3 years & we upgrade every 6 weeks. But its quite difficult when there’s this misalignment in perception, where our 1.X upgrades are treated as equivalent to C++ upgrades, and what we’ve been talking about as “breaking changes” are treated as equivalent to Python 3, when they’re really equivalent to C++11.

I think we need an approach which allows us to make changes of the sort C++ does on the same time frame that C++ does. And, like C++, we need to support compiling code before this switch. I think this epoch proposal is exactly the right approach to solving this problem.

Hopefully, we’ll be able to have fewer epochs than C++ has standards because our rolling release model allows us to make non-breaking updates any time, so there isn’t this pressure to ‘release the new epoch’ the way there is in C++.

[withoutboats]

I also think doing this granularly has significant downsides. Most notably, for every ‘epochal’ feature we support, if they can be mixed and match, we get an expontial growth in ‘rust versions’ that exist, and everyone will be in a different one. While making a granular transition is valuable, I think we can have a more targeted approach to that.

Its important that ‘during an epoch transition,’ the first version of the new epoch is a subset of the last version of the old epoch. That is, there is a subset of Rust that will compile in both epochs (and not a hobbled subset, like “if you never use Strings or integer division”). As we build toward a new epoch, we should release granular lints to help you identify what won’t compile in the next epoch, so that you can perform that granular transition. But these should just be lints, and you should still be ‘in the previous epoch’ until you make a switch, which turns on all of the hard errors across the board.

[tom.prince]

Most notably, for every ‘epochal’ feature we support, if they can be mixed and match, we get an expontial growth in ‘rust versions’ that exist, and everyone will be in a different one. While making a granular transition is valuable, I think we can have a more targeted approach to that.

One way to (somewhat) address the combinatorial explosion is to mix granular features and epochs, and only allow features to be turned on if you are using the epoch where they were introduced.

For example, consider the following sequence of changes.

• The epoch from the release of 1.0 to now is epoch 2015.
• Three new incompatible features are released A, B, C.
• Sometime later this year, epoch 2017 is defined, which makes A and B the default (and not able to be disabled in that epoch).
• More new features get added: D, E.
• Next year, epoch 2018 is defined, which makes C and E (in addition to A and B) the default.

My suggestions is that the following would be legal feature combinations:

• Any combination of A, B and C could be used in epoch 2015.
• Any combination of C, D and E could be used in epoch 2017 (and A and B would always be enabled).
• E could optionally be used in epoch 2018 (and all of A-D would always be enabled).

And probably features like C and E that don’t get made default in the next epoch shouldn’t exist either.

[oli-obk]

But we do not need to maintain the old code around either! Instead, it is entirely plausible to just ship multiple binaries/rustcs, and the driver would invoke the pipeline for requested “epoch”/“version”/etc. Sure, this approach would increase size of the binaries, but if balanced well with keeping some old code paths around, it can make the horrors of maintaining old versions a non-problem (i.e. whenever it starts becoming a problem, fork out a new “pipeline”).

This won’t work. You want a crate and all its dependencies to compile with rust 1.x, but the epochs may differ between all crates involved, as long as the API stays compatible. A function fn foo(x: Trait) in Epoch 2 will be fn foo<T: Trait>(x: T) in Epoch 1. These are 100% compatible, but multiple compilers won’t produce compatible binary artifacts.

Making the binary artifacts stable is not an option, because we’ll want to add new features to MIR or other parts that are exposed. The standard library can remove a function in a new epoch, but it will still need to be available in older epochs. So the binary standard library needs to include everything, even things that have name collisions (resolved by the chosen epoch).

[matklad]

I think it is theoretically possible to make something like this work, if we don’t relay on binary artifacts.

The prerequisite is that compiler can upgrade source code from epoch n to epoch n + 1 without a single fail (the code might not be pretty, but it must be equivalent). Then we can compose older compilers to upgrade source code (and source code is stable anyway) to the latest version, and then feed it to the compiler without old code paths Totally not sure that this is practical

EDIT: the stdlib problems can be solved by rewriting deprecated usages into equivalent non-deprecated.

[leonardo]

Where things get a bit tricky is are things like bits of public API. In that case, there could be Epoch 1 crates that (e.g.) have a public type named catch that is not typeable in Epoch 2. This could be circumvented by introducing some form of “escape” that allows identifiers with more creative names (e.g., I think that scala uses backticks for this purpose).

And C# uses “@keword”.

[stjepang]

However, our current compatibility guarantee is much firmer than what C++ version upgrades provide; we can’t even introduce new non-contextual reserved words. This makes sense - C++ upgrades every 3 years & we upgrade every 6 weeks. But its quite difficult when there’s this misalignment in perception, where our 1.X upgrades are treated as equivalent to C++ upgrades, and what we’ve been talking about as “breaking changes” are treated as equivalent to Python 3, when they’re really equivalent to C++11.

This is reassuring and great to hear. Thanks for clearing that up!

Whatever we do, it’s very important that we clearly communicate what this all means to people unfamiliar with Rust. They will ask how similar epochs are to upgrades in C++, Java, Python, Swift, Go, D. And some of these languages are notorious for breaking compatibility.

[nikomatsakis]

Another concern I have is - how do we teach this to new users? How do I explain to a friend how match works? Do I say “hey, if you’re using epoch 2017, you need this ref to borrow the object rather than take ownership, but if you’re using epoch 2018 then ref is implicit”? What if they copy some code from StackOverflow and it doesn’t compile?

This is definitely a key concern, and it’s one of the reason that I favor something like epochs – that is, a kind of “big release” that groups together a number of changes. This makes it easier to categorize code samples and things as “older code”. It also means you can label stackoverflow answers with a simple epoch number to give context and so forth.

I think another big factor here of course is error messages. That is, if the problem is the code not compiling, we can presumably recognize patterns from older epochs and give a very helpful error (much like we used to do when removing obsolete syntax etc).

[colin_kiegel]

Can we have epoch code names (like android/ubuntu)?

This feature will become available in the bronze epoch.

[Ixrec]

We could name the epochs after different crustaceans, then have different logos (also like Android), so there’d be a Crayfish Ferris and a Lobster Ferris and a Fiddler Crab Ferris and a…anyway.

Some serious points:

I believe a large portion of the perceived instability of Rust simply comes from the amount of Rust code that’s still stuck on unstable, and the unavoidable fact that no one’s had Rust code in production for several years yet. I’m not sure there’s a clear action item there, other than maybe trying to reduce the number of features in “unstable limbo”, or trying not to design and implement features that we aren’t going to be pushing to stabilize in the relatively near future (or are needed to make progress on stabilizing other stuff).

I completely agree that languages like C++ do make plenty of minor breaking changes in practice, and that “epochs” as described here are pretty much exactly what every other serious language seems to be doing. I’m totally on board with the idea of doing something similar for Rust, especially if the “breaking changes” are as minor and/or desirable as the examples given so far.

But I did have one concern about the impl/bare/dyn Trait syntax example (which I would love to see actually happen btw). For features which have a significant impact on the public APIs that libraries present to their clients, I think it’s desirable for there to be one syntax that libraries can use such that none of their clients, on any past or future epoch, get warnings, errors or bugs. In this particular example, I think achieving that is as simple as not deprecating the “impl Trait” syntax, which seems fine to me since the plan is merely to make the “impl” keyword redundant, not to repurpose it after we’ve redefined bare Trait.

[milesrout]

I’m not sure how much experience people here have writing Haskell, but a problem that has been growing and growing and growing lately is this at the start of every file:

{-# LANGUAGE GADTs #-}
{-# LANGUAGE RecursiveDo #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE ViewPatterns #-}
{-# LANGUAGE PatternSynonyms #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE ExistentialQuantification #-}
{-# LANGUAGE PartialTypeSignatures #-}
{-# LANGUAGE DeriveGeneric, DeriveFoldable, DeriveTraversable #-}
{-# LANGUAGE FunctionalDependencies, MultiParamTypeClasses #-}
{-# LANGUAGE TemplateHaskell #-}

Now that’s not to say that language extensions and #[feature] are a bad idea. But they do post the risk of ending up, in a couple of years, with every non-trivial project starting with a list of a dozen features and changes etc. etc.

Has Perl-style use v5.20 been considered?

[glaebhoerl]

Has Perl-style use v5.20 been considered?

This is the epochs proposal.

Whatever we call them, I think we can’t support making changes as deep as the changes to the string type. But all of the changes we’ve seriously considered are, in my opinion, in line with the kinds of changes that are made in upgrades to the C standard.

However, our current compatibility guarantee is much firmer than what C++ version upgrades provide; we can’t even introduce new non-contextual reserved words. This makes sense - C++ upgrades every 3 years & we upgrade every 6 weeks. But its quite difficult when there’s this misalignment in perception, where our 1.X upgrades are treated as equivalent to C++ upgrades, and what we’ve been talking about as “breaking changes” are treated as equivalent to Python 3, when they’re really equivalent to C++11.

I think this point is quite valid for things like adding new keywords (catch etc.), and probably the change to match semantics as well, but… less so for changing the meaning of &Trait (or banning it or whatever). Not sure what difference this makes to your argument, just wanted to point it out.