Mechanism for beta testing unstable features

tools and infrastructure
1

First, some context. I recently landed an unstable feature in cargo: -Zpatch-in-config (tracking issue) which allows [patch] sections to appear in .cargo/config files. This particular feature would significantly improve developer ergonomics in a larger build system I'm working with that "wraps" cargo in the context of a much larger build process.

Now that this feature is in the nightly compiler, the next step is to work towards stabilization, much of which consists on getting feedback on how well the feature works. However, this is where I run into an issue — while I have the opportunity to test the feature at a fairly large scale by using it in said build system, I cannot currently do so in a reasonable way. I must either switch the build system to use a nightly version of cargo/rustc, or use RUSTC_BOOTSTRAP=1 on stable/beta, both of which enable all unstable features. In the context of this build system, that's not okay, as it would enable all the developers that use said build system to start using any unstable features they'd like without any additional safe-guards.

I think this gets at a broader point that currently it's very difficult to get large-scale testing of an unstable feature, because any users that may be in a position to test at a large scale are likely unable or unwilling to deploy a nightly compiler or open the flood-gates of all unstable features with RUSTC_BOOTSTRAP=1.

What I'd like to see is a mechanism to enable just a particular set of unstable features, and no others, so that users can beta-test specific features without having to test all the other features at the same time. This is probably primarily useful in the context of custom build systems that wish to test-deploy a change across a larger package repository, though there may also be other use-cases I haven't thought of.

As for the exact mechanism, I'd propose a new environment variable RUST_ONLY_ALLOW_FEATURES that takes a comma-separated list of unstable features to allow. For example:

env RUST_ONLY_ALLOW_FEATURES="cargo:patch-in-config,rustc:sanitizer,rustc:doc-cfg" cargo

would permit cargo -Zpatch-in-config, cargo rustc -Zsanitizer, #[doc(cfg(..))] and no other unstable features. I propose an environment variable because:

  • It makes it difficult for crate authors to depend on this kind of selective enabling, which discourages its use, just like for RUSTC_BOOTSTRAP
  • It enables the feature to span both cargo and rustc
  • It aligns the feature closely with RUSTC_BOOTSTRAP as a mechanism that should be used sparingly

Anecdotally I'll add that this particular environment variable is likely to be used in conjunction with RUSTC_BOOTSTRAP=1 so that larger organizations can help beta-test a particular feature (or set of features) without subjecting themselves to the instability of nightly.

This feature is tangentially related to some previous discussions on internals.r-l.o:

though takes a slightly different approach. Rather than constructing another mechanism for opting into unstable features, it paves the path for using the existing mechanism, RUSTC_BOOTSTRAP=1, and making it safer to use by restricting what setting RUSTC_BOOTSTRAP actually enables.

This kind of change would almost certainly need to go through an RFC, but I figured I'd post it here for some initial feedback.

3 Likes
2

Mostly copied from the Zulip discussion - I think limiting features on nightly is useful, although it doesn't quite fit into the RFC process. However I'm very hesitant to continue making RUSTC_BOOTSTRAP a "feature" instead of just a historical accident - I don't think we should be encouraging anyone to use it. So I think whatever way of opting into this should only work on nightly toolchains, not on beta or stable.

3

I understand the reticence, though at the same time I also think that means most unstable features simply will not get wide-spread testing. As I mention in the original post, those who are in a position to do wide-spread testing of a feature usually cannot reasonably deploy a nightly toolchain, as that also gets into the game of nightly regressions and other nightly-only bits like stabilizations.

One option that has been raised multiple times before elsewhere is a fourth channel: "nightly-stable", which is the current stable compiler with unstable features enabled. That, paired with the proposed feature to limit which unstable features are available, would also get us to the same outcome.

2 Likes
4

That's kind of my point though - unstable features are no more stable than anything else on nightly, they can be removed at any point. Letting you use them on beta gives a false sense of security that they're supported when they aren't, they could change or be removed in any new release.

5

I'm not sure I follow — I'm not proposing that unstable features should be usable on beta. Instead, I'm looking for a way to test-deploy (internally in an org for example) a particular unstable change to get feedback for stabilization without also bringing in all the other things that using nightly changes.

6

I don't understand the distinction. That particular unstable change is also unstable and could break in any release.

7

Yes, but there is a world of difference between "opt into everything that's different about nightly" and "opt into one specific nightly feature and leave everything else as-is on stable". The former is unlikely to fly in a big org, the latter can be argued for for a particularly important feature, as part of internal beta-testing programs, dry-runs, controlled deploys, etc.

8

Right, I understand why you want this feature from the perspective of an end-user. My point is that this is still breaking Rust's stability guarantee and I don't see why we would support it from the perspective of the language.

I feel like we're going in circles, I'll let someone else respond.

9

Allow making `RUSTC_BOOTSTRAP` conditional on the crate name by jyn514 · Pull Request #77802 · rust-lang/rust · GitHub enables limiting the "bootstrappiness" of rustc to only be enabled for specific crates. Which is tangential to the request here.

What I have seen in the past is a separate step in CI that ripgreps for uses of feature and compares it against a whitelist. Not great, but works today. Every time I've used the verboten flag, I have always left a comment of which features were the underlying cause and relied on human pushback to avoid adding more. That is not always enough, of course.

10

Allow making RUSTC_BOOTSTRAP conditional on the crate name by jyn514 · Pull Request #77802 · rust-lang/rust · GitHub enables limiting the "bootstrappiness" of rustc to only be enabled for specific crates. Which is tangential to the request here.

In my mind the difference between that change and this one is that is meant to prevent people from using RUSTC_BOOTSTRAP in build scripts, and I wanted to give some sort of way to still build crates without needing a new toolchain version: https://github.com/rust-lang/cargo/pull/9181. @jonhoo is asking for a change that has people use RUSTC_BOOTSTRAP more, in cases where they wouldn't have previously. I wouldn't have opened #77802 if I wasn't planning to ban the use in build scripts later.

11

Ah, I understand where you're coming from now. Thank you for clarifying.

I don't think this feature in particular breaks Rust's stability guarantees. Rather, it allows users to limit the breakage possible by running nightly or setting RUSTC_BOOTSTRAP. I could even imagine going as far as to say that RUSTC_BOOTSTRAP=1 should never be used without an explicit allow-list of features.

As for why having something like this (or rather, some way to test an unstable feature in isolation) is "worth it" for the language, I think it comes down to enabling more users to provide input for stabilization decisions. Currently, it is difficult to get solid data on uses of features (I'm specifically thinking of cargo features here admittedly) from larger users since they simply have no path for testing a single unstable feature at scale. Given the options of use nightly, set RUSTC_BOOTSTRAP, and apply internal-only patches to the compiler, all of them are bad, and only the last two stand some chance of being deemed acceptable for internal testing. This in turn means that many such features get no feedback from those who may be the feature's biggest users during stabilization, which is in turn bad for the language as a whole. It may lead to features never being stabilized (not enough demonstrated used) or being stabilized too early when there are bugs or design issues that could have been caught by additional testing.

While that works decently for #![feature()] features, I think it gets tricky for things like -Z flags and unstable flags in configurations (e.g., [unstable] in .cargo/config). And those problems grow when when operating on a large, diverse code base where understanding what might end up being compiled (e.g., due to code generation) or how cargo/rustc end up being invoked (e.g., due to custom build logic) is non-trivial. I feel like it'd be better to have a standardized, supported way of only allowing certain features that doesn't require such hacks.

I think the RUSTC_BOOTSTRAP point is somewhat orthogonal to the proposal. The proposal is simply to have a way to only allow a particular set of unstable features, which has uses for users who simply use nightly as well. As @ekuber points out above, some nightly-only projects may check in CI that only a particular set of features are used, even though all are "available" since they are running nightly.

That said, you are right that if something like this proposal existed, more users may choose to use RUSTC_BOOTSTRAP since it feels "safer" with such a safe-guard in place. Though I'm guessing here. I do think that, as discussed in the other linked threads in the original post, it would be good to have a way to test unstable features in some way that doesn't also opt you into nightly. I agree with you that RUSTC_BOOTSTRAP shouldn't be that, but something should (see, for example, the stable-nightly idea).

12

I think that we should do everything we can to raise more roadbumps for breaking out of the stability guarantee on stable. But I also think we should permit people to test out "beta quality" features while staying on stable.

The nightly toolchain is surprisingly stable if you don't use unstable features. (We can thank CI and Bors for that!) But it's still less hardened than stable; stable gets backports in the period between cutting beta and publishing stable. There is currently no way to opt into an experimental feature while maintaining the quality control of the stable toolchain.

This is a good thing for maintaining Rust's stability guarantees, as it means that people who want the stable quality control cannot opt into features that might change in the future. But it also inherently means that these same users (mostly: companies, individuals aren't as fearful of nightly) can't preview experimental features. (This is @jonhoo's point.)

This is what a "fortnightly" (stable-nightly but wrong in a different way) would be. It would be exactly equivalent to the stable toolchain, but with some mechanism enabled to allow people to opt in to experimental features. And, ideally for @jonhoo's use case, on a granular level, like $env:RUSTC_PLEASE_ALLOW_UNSTABLE_EXPERIMENTAL_FEATURE=list,of,feature_flags and $env:CARGO_PLEASE_ALLOW_UNSTABLE_EXPERIMENTAL_FEATURE=list,of,feature,flags, and ideally (to @jyn514's point) this should only be settable at the root build system point when first calling into cargo/rustc, and not in build scripts.

This way we allow enterprise users who can't (or don't want to) use non-stable-quality toolchains still preview not-yet-stable functionality with an explicit, well controlled opt in.

There is, of course, the danger of OSSification of features available this way. But I think we can counteract this, by putting up the right amount of warning signs, and by even potentially only allowing the use of explicitly decided "let's let people test this implementation" features.

4 Likes
13

I haven't decided if I like this idea or hate this idea, but if you wanted to add another guard against fortnightly being treated like it had a stability guarantee, you could intentionally break it on a regular basis by refusing to compile when a stabilized feature is in the please-allow list. "Now is the time to review your list and maybe go back to stable."

2 Likes
14

The ability to constrain which features can be used already exists in rustc, including the ability to disable all unstable features:

rustc -Zallow-features= - <<END
#![feature(const_fn)]
fn main() {}
END
error[E0725]: the feature `const_fn` is not in the list of allowed features
 --> <anon>:1:12
  |
1 | #![feature(const_fn)]
  |            ^^^^^^^^

error: aborting due to previous error

This can cause issues with crates that do naïve channel detection and enable nightly features on the nightly channel, e.g. Provide way to disable `nightly` cfg · Issue #175 · alexcrichton/proc-macro2 · GitHub.

I don't know of anything similar for limiting the allowed cargo-features or -Z command line flags.

2 Likes
15

I think that would be a good idea, yeah - have a counterpart to incomplete_features which are explicitly allowed on stable/beta, but with a warning that the feature is unsupported and might break. I think my main concern is that RUSTC_BOOTSTRAP allows the user to pick any feature, included half broken and deprecated ones, and use them on stable as if they're supported. If we limited that to "features we plan to stabilize soon" I would feel a lot better about it.

Note that this also means we can get rid of the silly environment variable and just make this part of the compiler directly. It sounds like @jonhoo would also appreciate a Cargo equivalent of -Z allow-features, which seems fine, but independent of the stable/beta issue.

16

@CAD97 Yes, that's a great summary, thank you!

@quinedot I like that idea too!

@Nemo157 Ooooh, I had no idea. Will go implement that for cargo straight-away! I think that pretty much settles this thread — such a mechanism already exists in rustc, and it just needs to also be implemented for cargo (PR incoming). I think the question of how to test unstable features in isolation (on something stable-like, as @CAD97 described so well) remains, but maybe that should get its own thread?

@jyn514 Mmm, I like that, though I think I'd still want them to not be available on "normal" stable. In many ways, I'm in complete agreement with you that stable should not allow nightly features under any circumstances, and would strongly favor a separate version of stable that explicitly allows certain unstable features. And yeah, cargo -Zallow-features would indeed settle my main point for this thread.

With ^ all said, should we close this thread and open a new one to discuss stable-nightly?

17

Though actually, thinking about it for an additional second, -Zallowed-features has the downside that it must be passed explicitly to each tool, and cannot easily be set from the larger build system. In my particular instance, the build system allows each package to define its own custom build steps (not to be confused with a Rust crate's build.rs), which can themselves call cargo and rustc should they so desire. But since those are written by users, the build system cannot trivially inject -Zallowed-features into those calls. I suppose I could wrap rustc and cargo with a shell script that prefixes the argument list with -Zallowed-features, but that's not exactly pretty. Whereas an environment variable would allow the build system to set the allowed features once and forget about it.

Now, there's the argument that a user could always unset the given environment variable, but that would still require explicit maliciousness as opposed to simple carelessness ("huh, neat, I can just use #![feature()]").

18

You could use RUSTFLAGS="-Zallowed-features=..." with cargo.

19

In isolation, yes, but unfortunately that doesn't work when many packages also need to pass particular rustflags to rustc through build.rustflags (linker flags mostly).

20

I've moved the discussion of a non-nightly-non-bootstrap mechanism for testing unstable features to The case for a new relese channel: testing. I also implemented -Zallow-features for cargo (with forwarding to rustc) in Add -Zallow-features to match rustc's -Z by jonhoo · Pull Request #9283 · rust-lang/cargo · GitHub.

1 Like