If the testing channel is exactly the same as beta, except that testing additionally allows the -Zallow-features argument, then I guess nobody would use beta anymore, since testing is strictly superior from a user's perspective. That makes me wonder if it wouldn't be better to just allow the -Zallow-features argument on beta. You already gave an answer to this:
Could you explain why this is bad? I understand that the beta branch shouldn't be modified when turning it into stable, but just disabling an argument doesn't seem like a big deal to me. The other thing you mention is the opting out of stability guarantees, but again this doesn't seem like a problem to me. Users who care about stability can simply choose not to use the -Zallow-features argument, so nothing changes for them. They can't get unstable features breaking their code by accident.
In my reading of the original post, the difference between 'nightly' and 'testing' is supposed to be that 'testing' would have a higher stability of implementation, due to receiving beta backports.
Some feature flags gate functionality that is half-implemented and full of compiler crashes; allowing those would make a mockery of any implementation stability guarantee. But others gate functionality that works perfectly, and just hasn't garnered consensus on whether it should exist or what final name/syntax it should have. The latter category includes lots of standard library feature flags, for example (but also some core language features).
In theory, 'testing' could allow a relatively long list of features with a stable implementation, rather than a short list of high-priority things to test. I don't know if it should. But it could, while still being meaningfully different from nightly.
There is also a psychological aspect. Perhaps your large corporation is willing to use nightly, but based on posts in this forum, it seems clear that many are not, at least in part just because "nightly" sounds like an unstable/buggy implementation.
So, it's not just -Zallow-features that I'm proposing to allow on testing. Rather, testing will allow all "greenlit" unstable features, but will require that you opt into a particular subset as specified by -Zallow-features. If we were to use beta for this, it would basically mean allowing unstable features on beta, which is not what beta is for — beta is specifically for testing the upcoming stable release. And as I mentioned in the original post, I think it'd be unfortunate if lots of users moved from nightly to beta, since they'd then be using older versions of unstable features, while also muddling the feedback for beta with things that are really about unstable features.
You could argue the same thing for stable: why not just allow unstable features on stable with a -Z flag? And the reason is that we really want to ensure that users can rely on the stability guarantees of Rust on the stable channel — if it's trivial to opt into unstable features on stable, then inevitably people will recommend that (think StackOverflow answers), and we'll end up with a host of reports of "Rust is broken/isn't stable" from those who don't realize quite the implication of #![feature] and friends. This carries over to beta: the beta is going to be the next stable release, and the more we can get everyone to treat it that way, the more likely it is that users will a) test it the way they test stable and b) provide feedback specifically about regressions from stable-to-beta. If beta also has additional capabilities, it will be treated differently than stable, which in turn means the testing won't be as accurate.
What's the difference? Beta (and stable) already support unstable features (by using RUSTC_BOOTSTRAP). The proposed -Zallow-features argument is just another, more granular way to enable unstable features. When stable is branched off of beta, this argument can simply be disabled, so RUSTC_BOOTSTRAP is again the only way to opt into unstable features on stable. This is very unlikely to cause regressions, unless people use -Zallow-features (which is not possible on stable). But it should be made clear that by using this argument, you opt out of Rust's stability guarantees. I'm thinking of a warning such as:
Thank you for trying out unstable features. These might still contain bugs, so please report any bugs you encounter on <https://github.com/rust-lang/rust/issues>. Also note that unstable features might have breaking changes in a future release.
This also applies to a testing channel. There are two conflicting desires: The Rust team wants a tight feedback loop on new features, to detect bugs quickly, so it's best for them if many people use nightly. However, users want a mature compiler with very few bugs, so it's better for them to use stable. This is solved by making it appealing for some users to use nightly, by allowing unstable features there.
When unstable features become available on testing (or beta), two things will happen:
Some nightly users will move to testing, because it has all the features they need while being more mature
Some stable users will move to testing, because it has some features they would like to use, will still being relatively mature.
Effect 2 is the desired one by the Rust team, but effect 1 is undesired. So it seems like we have to strike a good balance about which unstable features to allow, to minimize effect 1 and maximize effect 2. On the other hand, effect 1 might actually be a good thing if it leads to increased user satisfaction, leading to broader adoption of Rust in the industry.
This is a good point: With a testing channel, it is easier to categorize bug reports correctly.
Your last paragraph makes a lot of sense, thank you for explaining!
To be clear, -Zallow-features does not enable anything. It is itself an unstable feature, and will only work on nightly or if RUSTC_BOOSTRAP is already enabled. Rather, -Zallow-features is an allow-list that means only the listed unstable features can be used even if unstable features are generally allowed (by using nightly or R_B). You can already use -Zallow-features on nightly, and you can use it on both stable and beta with R_B.
The proposal is that testing would require-Zallow-features, so that you cannot use testing just to have all unstable features on a more stable release. This is to encourage the use of testingonly for targeted testing of a particular feature (or small set of features), as opposed to "I need a bunch of unstable features", which is better suited for nightly.
For your case 1, I don't think users will jump to testing "because it has all the features they need while being more mature". It is going to be more cumbersome to use (they must specify -Zallow-features), it will only have selected features, and only for a limited amount of time (see @josh's suggestion), and if changes are made to those features, the fixes won't make it to testing for a while. The idea is that most users who need unstable features should be encouraged to stay on nightly. The testing channel is only for users who aren't willing to adopt nightly (for lack of stability) but want to help test a particular unstable feature (likely because they really need it).
For your case 2, I don't think the goal is for stable users to move to testing in the general sense. Rather, testing should be for if there's a particular feature we'd like users who are generally only willing to use stable to test out. Chances are users would only try out testing for short periods of time (~the 8 week timeline for a feature test that @josh suggested).
Actually I would like to argue just that. I too have noticed the two purposes of nightly that you mention in the original post, and as a user I would very much like to have "nightly, but 6 weeks old". For discussion purposes let's say this is the unstable channel. Of course stability can vary a lot depending on the feature in question, but I don't see why the decision for what features to enable isn't put in the hands of users, especially if features are categorized by how stable the rust team feels they are.
I suppose this option is best for users that just want the best Rust compiler and aren't actively involved in the community or testing features (but will incidentally serve that purpose if something starts crashing). Is it really so bad to offer this as an option?
Regarding "muddying the waters" of feedback, I don't think this is a significant concern; it is easy enough to ask that people reproduce any bugs originally discovered on unstable channel on nightly to see if they haven't been squashed in the meantime.
Given that I've seen a not insignificant number of issues opened that existed only because people were using an out of date distro Rust compiler rather than rustup managed stable, I wish I shared your optimism.
The other thing is that feature flags and vendor prefixes really aren't that big of a salt when the reward is cool new toys. They'll get stuck in a build script (make kind, not cargo kind), or otherwise forgotten until suddenly everything breaks. (Hopefully loudly, if you're lucky.)
The unstable channel is nightly. It's honestly shocking how reliable nightly is and how few regressions you'll run into, no matter what your update cadence is. rustup even prevents you from updating to a nightly without the tooling you have installed, and find the most recent that does have it. There's no good reason to not just use the nightly channel if you need to (slash want to) use unstable features, unless you're a big conglomerate that runs the risk of accidentally allowing people to rely on unstable features you didn't intend to make available.
Perhaps not. But again, anecdotal evidence shows that some people, including people making decisions for corporations, avoid nightly because of concerns about implementation quality.
If there is no good reason for them to do so, then we have something of a messaging failure.
Or perhaps a messaging success? I've gotten an impression that users being steered away from unstable features because they're tied to a scary-sounding release channel is... not necessarily seen as a bad thing by the community, given the strong desire to encourage users to use stable.
But that does come at the cost of testing prior to stabilization.
It's been a while since I opened this now, and it seems like there is a very rough consensus that something like this might be a good idea. Would a good next step be to write an RFC (@josh) to structure the proposal a bit more? Are there specific concerns that the people in this thread would like to see addressed as part of the RFC that haven't been articulated clearly so far?
First of all, please note that while I've been giving lots of feedback here, I'm not on the team that would make this decision. With my lang hat on, I'd like to see ways to get more testing of specific features; I don't know if this is the best way to do so, but it seems like a reasonable proposal to me. However, I think this would need to be, at a minimum, a decision made by release and possibly compiler and infra. Part of the issue here would be "whose decision is this".
Before writing an RFC, I would suggest reaching out to people on the release team, compiler team, and infra team, to discuss what team or teams would need to review and approve a decision like this. Once you have a clear story about who would review and approve such an RFC, then it may make sense to sketch an initial draft RFC that spells out the proposal, benefits, drawbacks, alternatives, and similar.
While it's true that I could totally have an internal "fork" of the compiler with that particular patch applied, I think that's something that should be avoided if possible. It both makes "internal" Rust different from "external" Rust, increases the chance of divergence over time (the patch has to be maintained), and runs the risk of developers adding more patches as time goes on since the mechanism is already in place. I'd much prefer to have a standard way to do this that others in similar positions can also take advantage of.
