How hard would it be to allow mixing stable and nightly crates?

[earthengine]

I know this will not be easy. However, it would be a grate beneficial: it allows us to “reuse” code that have already compiled under nightly, and just compile your own code in stable.

A concrete example of this is the fn_trait feature. This was not being stabilized for a long while and stopping us from writing our own function objects in stable. However, if someone created a crate with customized function objects, I didn’t see a reason why this would not be compatible with the stable crates.

Other “desugaring” features also should not affect the output compatibility, for example use Try Trait in a crate should not make the output of the crate being incompatible to the stable. I think this is also the case for async/await.

As NLL is in the MIR level, I also believe this should also work.

I am not sure about the “unsized rvalues” feature, but if it is just use alloca as an FFI to do the job, I believe this should work as well. So technically there is a long list of hightly demanded features to be sit in this category.

For the difficulties that I could think of:

• Some features may affect the intermediate formats and so will not be compatible with the stable. This can be handled by having a special flag on the feature gates, so if a specific feature was triggered, the crate will be marked as “incompatible” with the stable.

• The cargo.toml file will need to be extended to allow specifying a dependency to be compiled in nightly, in nightly and compatible with stable, or have to be in stable.

• The user will need to have the nightly toolchain being installed to compile their stable crate, if it depends on any traits that need to be built under nightly. We can add a global flag on cargo.toml to indicate we need the nightly tool chain, and allow the dependencies to specify they have to be built under stable.

• The user should be able to compile without installing the nightly toolchain. Instead, they can use cargo.toml to indicate they want to download a pre-compiled intermediate result for the stable version. Of cause, they can also compile with the nightly toolchain, but this will subject to breaking changes on nightly anyways.

A plan

1. We start with adding a flag to the crate output indicating the compatibility level. It can be a minimum version of the stable Rust version.

2. We review or adjust the current MIR design, to see if we can define a intermediate format that preserve all crate config options (most importantly, crate features) available, but not the targets, and functions remains generic for monomorphization.

3. We add a new feature gate to allow nightly rustc to generate “compatible” output targeting a stable Rust version.

4. We allow cargo to specify how it should compile each dependency, and check the compatibility flag on the output to verify. When publishing to crates.io the intermediate result should be uploaded with the source code. The user then should be able to use the uploaded result instead of compiling their own, as an option. Of cause they can choose to compile but they should know this may fail even when they have the nightly toolchain installed, because there would be breaking changes.

5. We are now ready to make the mixing of compiles happen: just let the stable toolchain to combine all compatible crate outputs to generate the result.

6. However we still need to review the features/libraries to make sure they are actually compatible with stable Rust, and if so, modify to make sure they use the flag to allow stable compatible output.

Some of the steps above are really complicated (5 for example), but I believe it is conservative enough to avoid many unexpected problems. Furthermore, once the first steps has been done the last step can be working in a per-request mode: we only review and check compatibility for features that the users requested.

Other benifits

If we follow the steps above, we will be able to use intermediate results from crates.io, which will accelerate the first build of new crate imports.

Furthermore, being able to reuse intermediate results on stable versions technically freezes a specification of the format - because it is connected to a stable version. This will motivate third party tool developers, and encourage unofficial documentations.

[Ixrec]

I don't think it would be hard to do this, but hardness isn't the issue. The problem is that if we actually did this, then any breaking change to an unstable nightly API might also break a lot of stable code, which basically means stable is no longer stable.

Discussion on this goes at least back to when Rust originally adopted "stable", "beta" and "nightly" release channels, so I'll just quote a section from Stability as a Deliverable | Rust Blog

Why not allow opting in to instability in the stable release?

There are three problems with allowing unstable features on the stable release.

First, as the web has shown numerous times, merely advertising instability doesn’t work. Once features are in wide use it is very hard to change them – and once features are available at all, it is very hard to prevent them from being used. Mechanisms like “vendor prefixes” on the web that were meant to support experimentation instead led to de facto standardization.

Second, unstable features are by definition work in progress. But the beta/stable snapshots freeze the feature at scheduled points in time, while library authors will want to work with the latest version of the feature.

Finally, we simply cannot deliver stability for Rust unless we enforce it. Our promise is that, if you are using the stable release of Rust, you will never dread upgrading to the next release. If libraries could opt in to instability, then we could only keep this promise if all library authors guaranteed the same thing by supporting all three release channels simultaneously.

It’s not realistic or necessary for the entire ecosystem to flawlessly deal with these problems. Instead, we will enforce that stable means stable: the stable channel provides only stable features.

[withoutboats]

I don’t understand the request. Crates that compile on stable also compile on nightly: you can mix stable and nightly crates by using the nightly compiler.

[earthengine]

@Ixrec

I don’t think it would be hard to do this, but hardness isn’t the issue.

Great, so we can continue to talk. First let me address this:

@withoutboats

I don’t understand the request. Crates that compile on stable also compile on nightly: you can mix stable and nightly crates by using the nightly compiler.

There are a few ways to mix stable and nightly crates today already:

1. As what you said, we can compile with nightly compiler to have a mixed result. This makes your code start to depend on the nightly and will break some day.

2. We can also compile with stable compiler and have all crates request no unstable features. The crates are compatible with the stable compiler in the source code level. But you are not able to access crates that uses unstable features. (Technically this is not a mix as we only have stable code; however by using crate level features we can limit the nightly feature under a specific config, so if we do not request a specific crate feature, we can compile the crate in stable even when the crate do contains nightly only code)

3. We can export all public interface through FFI from a crate that request unstable features, compile as a static library, and import it as FFI in a crate that do not request features. In this way, the static library is compatible with the linker in the binary level.

Option 1 is not what I am talking about. Option 2 is too restrictive and it is what I want to improve. Option 3 is like what I actually wanted, but it is too complicated to the end user, and will have problems in terms what I will discuss later.

Now let me address this:

@Ixrec

The problem is that if we actually did this, then any breaking change to an unstable nightly API might also break a lot of stable code, which basically means stable is no longer stable.

This is the point I missed in the original post. However, it is interesting to think what it actually means. Let me make the following as a principle:

Any breaking change on the nightly API should not prevent the mixed result of stable and nightly crates remains valid.

According to this, I conclude that we should not require the user to install the nightly toolchain at all to mix the stable and nightly crate, because the nightly compiler would make the code that was compiled yesterday invalid.

This also means, whatever the format it is, the (partially) compiled result have to be available without compiling in the user's computer. I would say it seems to request that, a partially compiled result have to be uploaded to crates.io by the user.

Now we can see why binary compatibility is not feasible: this simply means we have to prepare the pre-compiled binary for ALL targets that Rust supports, which the majority of developers would not have the right tools to do.

So, we have to draw a line in between the extreme ends: binary compatibility and source compatibility . For many reasons I will recommend to put the bar right before "monomorphization", because

• The higher the bar we set, the more features we lost the ability to mix
• Monomorphization is the point that the size of result would explode, putting the bar here helps reduce the size of intermediate results to be uploaded to crates.io.
• Another place that the result explodes is "macro expansion", this bar is far too high that it will rule out too many features. (All features I mentioned in the main post are ruled out, sadly)

I am now going to update the top post, hopefully this will give every a clear view of what I am actually thinking and what the benefit it has.

[scottmcm]

@earthengine

For many reasons I will recommend to put the bar right before “monomorphization”

That seems equivalent to "make MIR a stable interface", which isn't going to happen anytime soon.

Also, this is after #[cfg], so doesn't solve the "for all targets" problem.

[kornel]

I think it’d be better to support “transpilers” that transform new Rust syntax to old Rust syntax, same as Babel does for ES6/2017 to older JavaScript.

It would be less capable (you couldn’t get NLL, unless you reinvent the whole thing and emit a soup of unsafe pointers?), but all syntax-sugar and library features could be backported this way, and it’d be entirely stable, because it’d ship its own compiler.

[earthengine]

@scottmcm

That seems equivalent to “make MIR a stable interface”, which isn’t going to happen anytime soon.

Sad to hear

@scottmcm

Also, this is after #[cfg], so doesn’t solve the “for all targets” problem.

So it would take more effects to move #[cfg] down, or raise the bar on top of just it.

@kornel

I think it’d be better to support “transpilers” that transform new Rust syntax to old Rust syntax, same as Babel does for ES6/2017 to older JavaScript.

Great idea. But I think both solutions should be able to live together - my proposal is to allow reusing pre-compiled results, yours intended to "transpile + recompile". Both will have its pros and cons.

@kornel

It would be less capable (you couldn’t get NLL, unless you reinvent the whole thing and emit a soup of unsafe pointers?)

I don't think so. Technically we can transpile from C can into any stable version of Rust, just need to put everything in a giant unsafe block, if you are already sure the code is safe (with your proposal, it is the job of the transpiler to ensure the input is safe). (that said, for NLL, we simply ignore lifetimes to get things done)

[gbutler]

I would vote against this if I had a vote. Stable means stable. If you want to use unstable features, use Beta or Nightly. No mixing. Never. Ever. Please.

[bujiraso]

I will side with @gbutler and @Ixrec here and say that I worry about unstable features slipping outside of their chartered territory.

Perhaps the point has been missed that this would introduce a whole bunch of unstable features’ bugs into Rust Stable by virtue of “running new code paths always means running into new bugs”. Even if they do not install the toolchain they will still receive the bugs.

For the stable-side Rustaceans, perhaps they will even end up troubleshooting errors about features they know nothing about and thereby cannot solve. Imagine if they misuse a nightly Crate’s API in an unhandled manner – what will that error message look like to the stable user if it failed on a new feature’s limitations?

Such errors might even constitute actual Rust compatibility breakages or defects, too, and not simply API misusages. If there were serious “stability-mixing” defects that started popping up, that would be a whole new category of problematic bugs, where each one may end up simply being closed as “WONTFIX” once Nightly drifts beyond reproducibility.

Since, for very experienced or advanced Rustaceans, stability mixing is already possible to do on Beta and Nightly, I think this may end up hurting newbies more than helping the odd earnest Rustacean who, for various reasons, cannot run Beta or Nightly. I don’t see this as a major use-case, either.

This is a good question worth asking, of course, but I don’t think the rewards outweigh the risks from my vantage.

[withoutboats]

@earthengine

As what you said, we can compile with nightly compiler to have a mixed result. This makes your code start to depend on the nightly and will break some day.

If you don't use any !features, your code will not break, because it is written in stable Rust being compiled with the nightly compiler. You should use the nightly compiler if you want to depend on crates that use nightly features.

[earthengine]

@withoutboats

If you don’t use any !feature s, your code will not break

My assumsion is there is no such garantee. "Nightly" means active development, it is very possible that a unforseen issue breaks codes that still compiles yesterday, then get fixed tomorrow. Again, there is no garantee that the code didn't request any features will not be affected.

For the stable compiler, this would not happen as common as the above, because the frequency of update is much slower.

[johnthagen]

@earthengine

For the stable compiler, this would not happen as common as the above, because the frequency of update is much slower.

Could you just pin to a specific nightly, pin your dependencies in Cargo.toml, and only jump every 6 weeks to emulate the slow frequency of updates?

[Nemo157]

@johnthagen

Could you just pin to a specific nightly, pin your dependencies in Cargo.toml , and only jump every 6 weeks to emulate the slow frequency of updates?

Exactly. Even if you could somehow compile some dependencies with a nightly compiler and your own code with a stable compiler you would still have to be doing this to avoid accidentally breaking your dependencies (and finding a new nightly compiler that's capable of successfully compiling all your nightly dependencies whenever you want to update them). Since you've already got this high overhead on switching compilers for the nightly dependencies adding in the extra constraint "and it doesn't break my stable code" doesn't really affect updating your compiler (Rust's testing is soo good for this, I have almost never seen miscompilation issues when updating the nightly I use).

[ahmedcharles]

Imagine a rustup toolchain which was called ‘unstable’ and which downloaded ‘stable’, but set the environment variable that results in it allowing unstable features to be used. That’s the only difference between it and a nightly compiler and since it’s the same binary as stable, the concerns about bugs are limited, if not removed entirely.