# Mechanism for beta testing unstable features **URL:** \ **Category:** tools and infrastructure\ **Created:** [March 16, 2021, 9:12pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280 "2021-03-16T21:12:54Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 16, 2021, 9:12pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/1 "2021-03-16T21:12:55Z")
First, some context. I recently landed an unstable feature in cargo: `-Zpatch-in-config` ([tracking issue](https://github.com/rust-lang/cargo/issues/9269)) which allows `[patch]` sections to appear in `.cargo/config` files. This particular feature would significantly improve developer ergonomics in a larger build system I'm working with that "wraps" cargo in the context of a much larger build process. Now that this feature is in the nightly compiler, the next step is to work towards stabilization, much of which consists on getting feedback on how well the feature works. However, this is where I run into an issue — while I have the opportunity to test the feature at a fairly large scale by using it in said build system, I cannot currently do so in a reasonable way. I must either switch the build system to use a nightly version of cargo/rustc, or use `RUSTC_BOOTSTRAP=1` on stable/beta, both of which enable _all_ unstable features. In the context of this build system, that's not okay, as it would enable all the developers that use said build system to start using any unstable features they'd like without any additional safe-guards. I think this gets at a broader point that currently it's very difficult to get large-scale testing of an unstable feature, because any users that may be in a position to test at a large scale are likely unable or unwilling to deploy a nightly compiler or open the flood-gates of all unstable features with `RUSTC_BOOTSTRAP=1`. What I'd like to see is a mechanism to enable _just_ a particular set of unstable features, and no others, so that users can beta-test specific features without having to test all the other features at the same time. This is probably primarily useful in the context of custom build systems that wish to test-deploy a change across a larger package repository, though there may also be other use-cases I haven't thought of. As for the exact mechanism, I'd propose a new environment variable `RUST_ONLY_ALLOW_FEATURES` that takes a comma-separated list of unstable features to allow. For example: ```rust env RUST_ONLY_ALLOW_FEATURES="cargo:patch-in-config,rustc:sanitizer,rustc:doc-cfg" cargo ``` would permit [`cargo -Zpatch-in-config`](https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#patch-in-config), [`cargo rustc -Zsanitizer`](https://doc.rust-lang.org/beta/unstable-book/compiler-flags/sanitizer.html), [`#[doc(cfg(..))]`](https://doc.rust-lang.org/beta/unstable-book/language-features/doc-cfg.html) and **no other unstable features**. I propose an environment variable because: - It makes it difficult for crate authors to depend on this kind of selective enabling, which discourages its use, just like for `RUSTC_BOOTSTRAP` - It enables the feature to span both `cargo` and `rustc` - It aligns the feature closely with `RUSTC_BOOTSTRAP` as a mechanism that should be used sparingly Anecdotally I'll add that this particular environment variable is likely to be used in conjunction with `RUSTC_BOOTSTRAP=1` so that larger organizations can help beta-test a particular feature (or set of features) without subjecting themselves to the instability of `nightly`. This feature is tangentially related to some previous discussions on internals.r-l.o: - [Getting more testing of unstable features](https://internals.rust-lang.org/t/getting-more-testing-of-unstable-features/4954/) - [Allow unstable features on beta?](https://internals.rust-lang.org/t/allow-unstable-features-on-beta/2986/) though takes a slightly different approach. Rather than constructing another mechanism for opting _into_ unstable features, it paves the path for using the existing mechanism, `RUSTC_BOOTSTRAP=1`, and making it safer to use by restricting what setting `RUSTC_BOOTSTRAP` actually enables. This kind of change would almost certainly need to go through an RFC, but I figured I'd post it here for some initial feedback. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 16, 2021, 9:45pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/2 "2021-03-16T21:45:48Z")
Mostly copied from [the Zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/unstable-on-beta) - I think limiting features on nightly is useful, although it doesn't quite fit into the RFC process. However I'm very hesitant to continue making RUSTC\_BOOTSTRAP a "feature" instead of just a historical accident - I don't think we should be encouraging anyone to use it. So I think whatever way of opting into this should only work on nightly toolchains, not on beta or stable. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 16, 2021, 9:53pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/3 "2021-03-16T21:53:56Z")
> [@jyn514](#): > > However I'm very hesitant to continue making RUSTC\_BOOTSTRAP a "feature" instead of just a historical accident - I don't think we should be encouraging anyone to use it. So I think whatever way of opting into this should only work on nightly toolchains, not on beta or stable. I understand the reticence, though at the same time I also think that means most unstable features simply will not get wide-spread testing. As I mention in the original post, those who are in a position to do wide-spread testing of a feature usually cannot reasonably deploy a nightly toolchain, as that _also_ gets into the game of nightly regressions and other nightly-only bits like stabilizations. One option that has been raised multiple times before elsewhere is a fourth channel: "nightly-stable", which is the current stable compiler with unstable features enabled. That, paired with the proposed feature to limit which unstable features are available, would also get us to the same outcome. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 16, 2021, 9:56pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/4 "2021-03-16T21:56:57Z")
> [@jonhoo](#): > > As I mention in the original post, those who are in a position to do wide-spread testing of a feature usually cannot reasonably deploy a nightly toolchain, as that _also_ gets into the game of nightly regressions and other nightly-only bits like stabilizations. That's kind of my point though - unstable features are no more stable than anything else on nightly, they can be removed at any point. Letting you use them on beta gives a false sense of security that they're supported when they aren't, they could change or be removed in any new release. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 16, 2021, 10:01pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/5 "2021-03-16T22:01:14Z")
> [@jyn514](#): > > Letting you use them on beta gives a false sense of security that they're supported when they aren't, they could change or be removed in any new release. I'm not sure I follow — I'm not proposing that unstable features should be usable on beta. Instead, I'm looking for a way to test-deploy (internally in an org for example) a _particular_ unstable change to get feedback for stabilization without _also_ bringing in all the other things that using nightly changes. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 16, 2021, 10:11pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/6 "2021-03-16T22:11:30Z")
I don't understand the distinction. That particular unstable change is also unstable and could break in any release. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 16, 2021, 10:15pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/7 "2021-03-16T22:15:13Z")
Yes, but there is a world of difference between "opt into everything that's different about nightly" and "opt into one specific nightly feature and leave everything else as-is on stable". The former is unlikely to fly in a big org, the latter can be argued for for a particularly important feature, as part of internal beta-testing programs, dry-runs, controlled deploys, etc. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 16, 2021, 10:16pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/8 "2021-03-16T22:16:25Z")
Right, I understand why you want this feature from the perspective of an end-user. My point is that this is still breaking Rust's stability guarantee and I don't see why we would support it from the perspective of the language. I feel like we're going in circles, I'll let someone else respond. ---
**Author:** ![ekuber](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ekuber/32/1646_2.png) [@ekuber](https://internals.rust-lang.org/u/ekuber)\ **Post date:** [March 16, 2021, 10:22pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/9 "2021-03-16T22:22:11Z")
[Allow making `RUSTC_BOOTSTRAP` conditional on the crate name by jyn514 · Pull Request #77802 · rust-lang/rust · GitHub](https://github.com/rust-lang/rust/pull/77802) enables limiting the "bootstrappiness" of rustc to only be enabled for specific crates. Which is tangential to the request here. What I have seen in the past is a separate step in CI that ripgreps for uses of `feature` and compares it against a whitelist. Not great, but works _today_. Every time I've used the verboten flag, I have always left a comment of which features were the underlying cause and relied on human pushback to avoid adding more. That is not always enough, of course. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 16, 2021, 10:25pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/10 "2021-03-16T22:25:40Z")
> [Allow making `RUSTC_BOOTSTRAP` conditional on the crate name by jyn514 · Pull Request #77802 · rust-lang/rust · GitHub](https://github.com/rust-lang/rust/pull/77802) enables limiting the "bootstrappiness" of rustc to only be enabled for specific crates. Which is tangential to the request here. In my mind the difference between that change and this one is that is meant to _prevent_ people from using RUSTC\_BOOTSTRAP in build scripts, and I wanted to give some sort of way to still build crates without needing a new toolchain version: [https://github.com/rust-lang/cargo/pull/9181](https://github.com/rust-lang/cargo/pull/9181). @jonhoo is asking for a change that has people use RUSTC\_BOOTSTRAP _more_, in cases where they wouldn't have previously. I wouldn't have opened #77802 if I wasn't planning to ban the use in build scripts later. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 16, 2021, 10:56pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/11 "2021-03-16T22:56:35Z")
> [@jyn514](#): > > My point is that this is still breaking Rust's stability guarantee and I don't see why we would support it from the perspective of the language. Ah, I understand where you're coming from now. Thank you for clarifying. I don't think this feature in particular breaks Rust's stability guarantees. Rather, it allows users to limit the breakage possible by running nightly or setting `RUSTC_BOOTSTRAP`. I could even imagine going as far as to say that `RUSTC_BOOTSTRAP=1` should never be used without an explicit allow-list of features. As for why having _something_ like this (or rather, some way to test an unstable feature in isolation) is "worth it" for the language, I think it comes down to enabling more users to provide input for stabilization decisions. Currently, it is difficult to get solid data on uses of features (I'm specifically thinking of cargo features here admittedly) from larger users since they simply have no path for testing a single unstable feature at scale. Given the options of use nightly, set `RUSTC_BOOTSTRAP`, and apply internal-only patches to the compiler, all of them are bad, and only the last two stand _some_ chance of being deemed acceptable for internal testing. This in turn means that many such features get no feedback from those who may be the feature's biggest users during stabilization, which is in turn bad for the language as a whole. It may lead to features never being stabilized (not enough demonstrated used) or being stabilized too early when there are bugs or design issues that _could_ have been caught by additional testing. > [@ekuber](#): > > What I have seen in the past is a separate step in CI that ripgreps for uses of `feature` and compares it against a whitelist. Not great, but works _today_ . Every time I've used the verboten flag, I have always left a comment of which features were the underlying cause and relied on human pushback to avoid adding more. That is not always enough, of course. While that works decently for `#![feature()]` features, I think it gets tricky for things like `-Z` flags and unstable flags in configurations (e.g., `[unstable]` in `.cargo/config`). And those problems grow when when operating on a large, diverse code base where understanding what _might_ end up being compiled (e.g., due to code generation) or how cargo/rustc end up being invoked (e.g., due to custom build logic) is non-trivial. I feel like it'd be better to have a standardized, supported way of only allowing certain features that doesn't require such hacks. > [@jyn514](#): > > @jonhoo is asking for a change that has people use RUSTC\_BOOTSTRAP _more_ , in cases where they wouldn't have previously. I think the `RUSTC_BOOTSTRAP` point is somewhat orthogonal to the proposal. The proposal is simply to have a way to only allow a particular set of unstable features, which has uses for users who simply use nightly as well. As @ekuber points out above, some nightly-only projects may check in CI that only a particular set of features are used, even though all are "available" since they are running nightly. That said, you are right that if something like this proposal existed, more users may _choose_ to use `RUSTC_BOOTSTRAP` since it feels "safer" with such a safe-guard in place. Though I'm guessing here. I do think that, as discussed in the other linked threads in the original post, it would be good to have a way to test unstable features in some way that doesn't _also_ opt you into nightly. I agree with you that `RUSTC_BOOTSTRAP` _shouldn't_ be that, but _something_ should (see, for example, the `stable-nightly` idea). ---
**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\ **Post date:** [March 17, 2021, 3:29am UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/12 "2021-03-17T03:29:07Z")
I think that we should do everything we can to raise more roadbumps for breaking out of the stability guarantee on stable. But I also think we should permit people to test out "beta quality" features while staying on stable. The nightly toolchain is surprisingly stable if you don't use unstable features. (We can thank CI and Bors for that!) But it's still less hardened than stable; stable gets backports in the period between cutting beta and publishing stable. There is currently _no_ way to opt into an experimental feature while maintaining the quality control of the stable toolchain. This is a good thing for maintaining Rust's stability guarantees, as it means that people who want the stable quality control _cannot_ opt into features that might change in the future. But it also inherently means that these same users (mostly: companies, individuals aren't as fearful of nightly) can't preview experimental features. (This is @jonhoo's point.) This is what a "fortnightly" (stable-nightly but wrong in a different way) would be. It would be exactly equivalent to the stable toolchain, but with _some_ mechanism enabled to allow people to opt in to experimental features. And, ideally for @jonhoo's use case, on a granular level, like `$env:RUSTC_PLEASE_ALLOW_UNSTABLE_EXPERIMENTAL_FEATURE=list,of,feature_flags` and `$env:CARGO_PLEASE_ALLOW_UNSTABLE_EXPERIMENTAL_FEATURE=list,of,feature,flags`, and ideally (to @jyn514's point) this should only be settable at the _root_ build system point when first calling into cargo/rustc, and not in build scripts. This way we allow enterprise users who can't (or don't want to) use non-stable-quality toolchains still preview not-yet-stable _functionality_ with an explicit, well controlled opt in. There is, of course, the danger of OSSification of features available this way. But I think we can counteract this, by putting up the right amount of warning signs, and by even potentially only allowing the use of explicitly decided "let's let people test this implementation" features. ---
**Author:** ![quinedot](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/quinedot/32/7294_2.png) [@quinedot](https://internals.rust-lang.org/u/quinedot)\ **Post date:** [March 17, 2021, 6:48am UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/13 "2021-03-17T06:48:30Z")
I haven't decided if I like this idea or hate this idea, but if you wanted to add another guard against `fortnightly` being treated like it had a stability guarantee, you could intentionally break it on a regular basis by refusing to compile when a stabilized feature is in the please-allow list. "Now is the time to review your list and maybe go back to `stable`." ---
**Author:** ![Nemo157](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nemo157/32/11585_2.png) [@Nemo157](https://internals.rust-lang.org/u/Nemo157)\ **Post date:** [March 17, 2021, 7:31am UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/14 "2021-03-17T07:31:40Z")
The ability to constrain which features can be used already exists in `rustc`, including the ability to disable all unstable features: ```rust rustc -Zallow-features= - < :1:12 | 1 | #![feature(const_fn)] | ^^^^^^^^ error: aborting due to previous error ``` This can cause issues with crates that do naïve channel detection and enable nightly features on the nightly channel, e.g. [Provide way to disable `nightly` cfg · Issue #175 · alexcrichton/proc-macro2 · GitHub](https://github.com/alexcrichton/proc-macro2/issues/175). I don't know of anything similar for limiting the allowed cargo-features or `-Z` command line flags. ---
**Author:** ![jyn514](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jyn514/32/13245_2.png) [@jyn514](https://internals.rust-lang.org/u/jyn514)\ **Post date:** [March 17, 2021, 4:26pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/15 "2021-03-17T16:26:06Z")
> [@CAD97](#): > > There is, of course, the danger of OSSification of features available this way. But I think we can counteract this, by putting up the right amount of warning signs, and by even potentially only allowing the use of explicitly decided "let's let people test this implementation" features. I think that would be a good idea, yeah - have a counterpart to `incomplete_features` which are explicitly allowed on stable/beta, but with a warning that the feature is unsupported and might break. I think my main concern is that RUSTC\_BOOTSTRAP allows the user to pick _any_ feature, included half broken and deprecated ones, and use them on stable as if they're supported. If we limited that to "features we plan to stabilize soon" I would feel a lot better about it. Note that this also means we can get rid of the silly environment variable and just make this part of the compiler directly. It sounds like @jonhoo would also appreciate a Cargo equivalent of `-Z allow-features`, which seems fine, but independent of the stable/beta issue. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 17, 2021, 6:03pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/16 "2021-03-17T18:03:16Z")
@CAD97 Yes, that's a great summary, thank you! @quinedot I like that idea too! @Nemo157 Ooooh, I had no idea. Will go implement that for cargo straight-away! I think that pretty much settles this thread — such a mechanism already exists in rustc, and it just needs to also be implemented for cargo (PR incoming). I think the question of how to test unstable features in isolation (on something stable-like, as @CAD97 described so well) remains, but maybe that should get its own thread? @jyn514 Mmm, I like that, though I think I'd still want them to not be available on "normal" stable. In many ways, I'm in complete agreement with you that `stable` should not allow nightly features under any circumstances, and would strongly favor a separate version of `stable` that explicitly allows certain unstable features. And yeah, `cargo -Zallow-features` would indeed settle my main point for this thread. With ^ all said, should we close this thread and open a new one to discuss `stable-nightly`? ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 17, 2021, 6:08pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/17 "2021-03-17T18:08:54Z")
Though actually, thinking about it for an additional second, `-Zallowed-features` has the downside that it must be passed explicitly to each tool, and cannot easily be set from the larger build system. In my particular instance, the build system allows each package to define its own custom build steps (not to be confused with a Rust crate's `build.rs`), which can themselves call `cargo` and `rustc` should they so desire. But since those are written by users, the build system cannot trivially inject `-Zallowed-features` into those calls. I suppose I could wrap `rustc` and `cargo` with a shell script that prefixes the argument list with `-Zallowed-features`, but that's not exactly pretty. Whereas an environment variable would allow the build system to set the allowed features once and forget about it. Now, there's the argument that a user could always _unset_ the given environment variable, but that would still require explicit maliciousness as opposed to simple carelessness ("huh, neat, I can just use `#![feature()]`"). ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [March 17, 2021, 6:39pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/18 "2021-03-17T18:39:15Z")
You could use `RUSTFLAGS="-Zallowed-features=..."` with cargo. ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [March 17, 2021, 8:29pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/19 "2021-03-17T20:29:01Z")
In isolation, yes, but unfortunately that doesn't work when many packages also need to pass particular rustflags to `rustc` through `build.rustflags` (linker flags mostly). ---
**Author:** ![jonhoo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jonhoo/32/3367_2.png) [@jonhoo](https://internals.rust-lang.org/u/jonhoo)\ **Post date:** [April 2, 2021, 11:49pm UTC](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280/20 "2021-04-02T23:49:58Z")
I've moved the discussion of a non-nightly-non-bootstrap mechanism for testing unstable features to [The case for a new relese channel: testing](https://internals.rust-lang.org/t/the-case-for-a-new-relese-channel-testing/14412). I also implemented `-Zallow-features` for cargo (with forwarding to `rustc`) in [Add -Zallow-features to match rustc's -Z by jonhoo · Pull Request #9283 · rust-lang/cargo · GitHub](https://github.com/rust-lang/cargo/pull/9283). [Next page](https://internals.rust-lang.org/t/mechanism-for-beta-testing-unstable-features/14280.md?page=2)