[Idea] Extending the RFC process: Experimental features

[japaric]

TL;DR Experimental features are features that land in tree before going through the RFC process with the purpose of having end users evaluate them and producing a better RFC document. These experimental features will be maintained as feature branches, to not block the development of the master branch, for a predetermined amount of time.

Overview

Introduce the concept of "experimental features" into the RFC process. These features go through an experimentation process, where they get implemented so end users can evalute them, before going through the existing RFC process (at that point they become unstable features).

Motivation

Sometimes is hard to come up with a convincing motivation for a RFC because it's hard to predict how a feature will be used in practice or whether the use cases one comes up with will have positive / significant impact in the ecosystem.

Other times, it's hard to evaluate the correctness of a RFC design because one can't foresee all the interactions of the proposed feature with all the other existing ones.

This document proposes a mechanism for implementing features without going through the RFC process with the goal of answering the above questions through end user feedback.

Detailed design

Request for Experiment

Experimental features start with a Request For Experiment (RFE) document, a lighter version of the RFC document.

The RFE document should consist of:

• A motivation. More like a justification for the feature going through experimentation rather than through the normal RFC process.

• An initial design.

• Questions to answer (during the experimentation process)

The important part at this stage is collecting relevant questions about the feature. These will probably be along the lines of:

• How does this new feature interacts with feature X, in practice?

• Does this new feature leads to Rustic programming patterns or, instead, leads to footguns?

• How does this feature improves the API of existing libraries?

• Does this feature prevent the adoption of a new compiler backend (e.g. Cretonne) or third party implementation of rustc / the Rust language?

Bikeshedding the syntax at this stage is discouraged (modulo discussion about possible ambiguous grammar) as that should be left up to the RFC process.

Eligibility

What kind of features should go through an experimentation phase first?

If a feature can be implemented and evaluated out of tree (i.e. without changes to the compiler), then it should be evaluated out of tree and go through the normal RFC process.

If writing a RFC produces several unresolved questions that don't seem like they can be answered purely through discussion, then the proposed feature is a good candidate for experimentation.

Experimentation phase

Upon accepting the RFC, the Rust team will decide on a experimentation period for the feature. This period could be measured in Rust cycles. For example: "the experiment will run for three 6-week cycles".

Then an implementation will be sent as a PR to the rust-lang/rust repo. However, instead of merging the PR into the master branch (once it passes the test suite), it will become a feature branch. This way, the experimental feature won't block the development of the master branch.

The Rust infrastructure will produce binary artifacts once the PR is merged. Users will then be able to use rustup to try out the feature branch:

$ rustup default dev-$feature

$ rustc -V
rustc 1.17.0-dev (..)

An issue will be open in the rust-lang/rfcs repo to discuss the experimental feature. The discussion in this issue should try to answer the questions set in the RFE. This issue should also keep track of user experiences, both good and bad, with the feature.

The feature branch will receive bug fixes and be rebased on top of master when relevant during the duration of the experimentation phase. Every update to the feature branch will produce a new binary release.

Previous releases of the feature branch will be available via rustup:

$ rustup default dev-$feature-$sha

End of the experiment

Once the experiment is over, two things can happen:

If the feature didn't recieve the interest of the community, the associated feature branch will be deleted and the associated rust-lang/rfcs issue will be closed.

Otherwise, the associated rust-lang/rfcs issue will be closed and a proper RFC, that collects the insights from the experimentation phase, will be proposed. The feature branch will become frozen while the RFC is being discussed. If the RFC is accepted, the feature branch will be updated to match the RFC text, then merged into master and go through the existing stabilization process. If the RFC is closed, then the feature branch will be deleted.

Alternatives

Instead of maintaining a feature branch, we could merge the experimental feature into master and maintain it just like the other features. In that case, we may want to have more clear separation between experimental features and unstable features. Maybe #[feature(experimental($feature_name))].

Unresolved questions

• Should we try to prevent users from publishing crates that use experimental features?

• Should we keep the branches of failed experiments instead of deleting them?

• Should we not impose a deadline for the experimentation phase?

• What other aspects should considered for evaluating the eligibility of a feature for experimentation?

---

Potential candidates for experimentation:

-Z linker-flavor

That PR allows experimentation with LLD as a external linker.

What wants to be evaluated here is the benefits of embedding LLD into rustc to use it as the default linker.

LLD being a fast multiarch linker are clear benefits but having to explicitly specify startup objects, library search paths and the dynamic linker to link "Hello, world" is a clear downside.

Experimentation could let us find a solution for the downside and benchmark LLD vs GNU's ld (is the speedup worthwhile?).

Custom Dynamically Sized Types

Letting users define custom DST seems like it could lead to more ergonomic linear algebra libraries while also reducing the size of their code bases (by removing duplicate logic through Deref).

Experimentation would let us answer that question with data points like

"With custom DSTs, I reduced the size of my code base by 70%", or

"My examples now have a nicer syntax that's closer to what linear algebra users are used to. See the diff and the Python / MATLAB equivalent".

The experimental implementation doesn't need to bother with questions like "what's the best way to expose this feature (synatx, traits, etc.)", which has been one of the main topics of the RFC discussion, as long as it exposes the functionality.

the #[used] attribute.

Prevents LLVM from optimizing away a non-exported static that has no references to it, within a compilation unit (object file).

Seems like it would be useful in the bare metal context where a fine control over symbols and the linker is desired.

However, paired with #[link_section], #[used]can be used to achieve "life before main" (using the .init_array ELF standard) which seems like a bad idea in std programs because "before main" also runs before the Rust runtime is initialized. But "life before main" may not be such a bad idea in bare metal contexts where there's no Rust (std) runtime.

Again, experimentation can help answer these questions.

cc @aturon @nikomatsakis

[lifthrasiir]

Sounds interesting (of course, in views of casual observers), but one question:

If the feature didn't recieve the interest of the community, the associated feature branch will be deleted and the associated rust-lang/rfcs issue will be closed.

Wouldn't this break all links to the tree from issues when Github runs periodic git gc? I've heard that Github won't run gc on commits linked from PR, but I'm less sure about ordinary issues. We may possibly (ab)use the rust-lang-deprecated org to store such historical artifacts.

[japaric]

I wasn't sure about what repercussions deleting the branch would have so I added an unresolved question about that:

Should we keep the branches of failed experiments instead of deleting them?

It seems fine to me to keep those feature branches around given that they are properly "namespaced", i.e. experiment-$feature_name.

[steveklabnik]

I think this is a neat idea. Now that we have (well, will have soon) the Unstable Book, I was thinking we could categorize these as where they are in the process. This would be similar, but is more involved than just an annotation in the docs.

[nikomatsakis]

Hmm, interesting. As @japaric knows, but others may not, I had been contemplating proposing a “feature” process, but mine was different in some particulars. It’s probably worth highlighting those, they seem to bias towards different end-goals.

In particular, whereas this process keeps the experiment out of the main tree, I had intended to allow the feature to land in the main tree. This would allow people to more readily write code using it (but confined to nightly of course). I did not have the idea of a “time limit”.

Things that I like about @japaric’s version:

• It makes a very clear distinction between experimental features and those on the road to stabilization. Providing a better means to differentiate those things was a bit of “future work” in my plan.
• Since nothing lands on master, it will clearly not cause “unintended consequences” where the code causes regression for stable features.

Things that I’m wary of:

• An up-front “time limit” seems a bit surprising
  • Probably very hard to come up with a realistic estimate, but I guess we can extend it.
  • If the time limits are short, rebasing is good, but perhaps unlikely to gain much experience using the feature.
  • If the limit limit is long, rebasing is quite possibly painful.
    • But the branch could perhaps merge regularly from master to mitigate this, I suppose.
• Distinct branches means you can’t readily “combine” experiments.
• We have existing things that I view as “experimental” but which would not clearly fit this process. Naked functions are an example, interrupt ABIs are another.

I think by and large I am happy with @japaric’s proposal. It seems like a more conservative variant and a reasonable thing to experiment with (no pun intended). I’d be curious to hear from @alexcrichton or @brson regarding how much work it would be to produce the new binaries.

[japaric]

An up-front "time limit" seems a bit surprising

My original idea was to land the experimental features in tree (in the master branch). The "deadline" made more sense in that case as it would serve as a deterrent to experimental features becoming de facto unstable / stable features. But with one feature branche per experimental feature (what's proposed in the OP), the risk of that happening seems rather low.

In any case, instead of a hard deadline we could still declare an experimentation period that would instead serve as a remainder for "OK, let's check how the experiment went so far. Has the questions that we set to answer been resolved? Can we make conclusions from the user experience so far?".

Distinct branches means you can't readily "combine" experiments.

We could land some experimental features in tree provided that (a) they pose low maintenance burden and/or won't block the development of RFC-accepted features and (b) they still are clearly distinguished from unstable features (I mentioned #![feature(experimental($feature_name))] in the OP; that could be one way)

We have existing things that I view as "experimental" but which would not clearly fit this process.

I had this vague idea of "challenging" existing unstable RFC-less features (e.g. asm!), turning them into experimental features and using the experimentation period to answer the questions "Does this feature pull its weight? Can it ever become stable? What changes would be required to make it stable?". In those cases, the deadline would serve as an "ultimatum" for the feature where one possible outcome could be removing the feature. But the details are fuzzy and ,described like that, it may even sound aggressive (?).

[nikomatsakis]

We talked about this in the core team meeting. I had the job of summarizing what we said, but I must not have been getting enough sleep lately, because I’m having a sort of hard time remembering the details. I think I anticipated a lot of the comments in my post.

The short version was that there was a lot of skepticism about the idea of using topic branches. While it does an admirable job of shifting the pain of maintenance on to the experimenter, it also lessens a lot of the benefits of such an experiment, and raises the question of just what this “experiment” is buying exactly, versus making your own fork (I guess: official endorsement? infrastructure support for building binaries / rustup integration?)

I still continue to feel that there is some kind of gap in our process here, but it’s not obvious to me what precisely the best way to fill it. It may be that all we need is better organization of existing feature gates (i.e., more visibility into whether a feature gate represents an “internal hack”, “experiment”, “unstable feature”, “nearly done feature”, etc). I’d appreciate it if @aturon, @wycats, @brson or @alexcrichton would weigh in with their thoughts.

[Ericson2314]

I’d love love love something like this! I was concerned about combining features too, but if the team is fine landing in master that’s greatamd solves the problem. As I see it, @japaric is basically throwing you all a bone with the topic branches. If the relevant teams are fine without that, the result strictly more power to the community :).

I’d be curious whether 1) breaking up std and 2) std-aware cargo used in rustbuild are good for this. The former does involving changing unstable (as oppose to experimental) interfaces like liballoc, while the latter involves the rust and cargo repos on tamdem.

[canndrew]

I think forcing contributors to maintain features out-of-tree is asking a bit much. As a hobbyist rust contributor, having to rebase and rewrite changes over and over again while waiting for the main devs to review them and make decisions can get pretty tiring. (And that’s not a complaint. I realise that the devs have very limited bandwidth and much more important things to work on than anything I’ve submitted).

But if we’re going to make changes to how features are developed I’d like to see more of the legwork currently done by the devs done by community members instead. Then we can free-up the devs to spend more of their time reviewing RFCs, merging PRs and making decisions about the overall direction of the language.

[japaric]

@nikomatsakis

what this "experiment" is buying exactly, versus making your own fork

(I guess: official endorsement? infrastructure support for building binaries / rustup integration?)

All these actually. Take for example the avr-rust fork.

• To try this fork, you have to bootstrap the compiler yourself. That's the first barrier: 1h+ build time. Then most people won't know how to use this bootstrapped rustc with Cargo because the rustup toolchain link trick is not visibly documented AFAIK.

• Your bootstrap may fail because master may be broken. Master can break because there's no CI running on each PR. Although avr-rust has inherited rust-lang/rust CI, it's not really usable because it runs into Travis time limit before it tests everything. (IDK if forks have access to sscache?)

• The project lacks visibility. People routinely ask if Rust supports AVR, then they get pointed to the fork, then they get turned off because it's not officially supported.

• The project has pretty much only two developers. Again, probably due to the lack of visibility.

Having binary releases available would make testing this AVR feature straightforward and hassle free. And more visibility, either by listing the AVR experiment in rust-lang/rust's README or by listing it in 'This week in Rust', would bring more testers and developers on board.

avr-rust is also an excellent candidate for a feature branch because landing it in master would require bumping LLVM to 4.0 and that would break emscripten. This last part is probably no longer true because of recent development but a feature branch would have let us have an AVR experiment long time ago.

It may be that all we need is better organization of existing feature gates (i.e., more visibility into whether a feature gate represents an "internal hack", "experiment", "unstable feature", "nearly done feature", etc)

I think we also need a categorization like this rather than a catch-all "unstable feature" category.

[nikomatsakis]

@japaric

avr-rust is also an excellent candidate for a feature branch because landing it in master would require bumping LLVM to 4.0 and that would break emscripten. This last part is probably no longer true because of recent development but a feature branch would have let us have an AVR experiment long time ago.

Thanks, this is an interesting thought experiment. It's good to work from concrete examples.

I think a lot of the concerns with supporting forks boiled down to the idea that doing a good job maintaining a fork is a lot of work on the part of the people attempting to keep the fork going (e.g., regular rebasing or merging from rust-lang/master and so forth), and it's not clear that it's a good strategy to encourage. (After all, we will need to merge eventually, if we like the idea, and it may avoid us finding about interactions between experiments for longer.)

[alexcrichton]

I agree that the situation here isn’t great today, but on the other hand I’m quite wary of topic branches and rustup integration and such. Overall my gut feeling is that a proposal like this is just going to yet again increase the maintenance burden on rust-lang/rust. The list of things we’d be adding to our plate is:

• Management of branches on the rust-lang/rust repository for topics.
• We’re responsible for actively cleaning out old branches
• We’re responsible for the review queue still. PRs to topic branches would take away from the main repo
• We’re now responsible for distribution and maintenance of all these artifacts. We’re the ones that will get bug reports for distribution bugs and otherwise random system bugs
• We’re on the hook for adding rustup support, mantaining that, fixing bugs, etc

It seems the benefit of this change would be:

• An “official” location for various branches
• Decreased maintenance burden on these branches (as it’s shifted to us)
• No need for downstream users to compile a compiler.

I personally feel that the benefits of this do not outweigh the downsides. I would not be personally thrilled to take on this extra maintenance burden when we’ve already put so much effort into making the compiler easy for everyone to compile. If there’s remaining hurdles about how to actually use the compiler you just generated we should smooth those as well!

One thing I’m not sure how to solve though is the location for “official” branches. I agree that there’s one and only one AVR fork basically, and it’d be great to funnel everyone to one location for those looking for it. Today though if I google “avr rust” it has some pretty good results, so maybe it’s not immediate that we need to solve such a problem?

[jessstrap]

After reading the above posts I have an alternate, slimmer, strawman proposal. (Please, beat the stuffing out of it.)

1: Better feature flags, with CI enforcement.

#![feature(experimental($feature_name))] Cannot block other development. Rustbuild has a flag to cfg out all these features at compiler compile time. There is an entry in the build matrix that exercises this flag and gates pull requests. Usually enabled on nightly. If some non experimental development breaks experimental features, the non experimental pull request can just flip the flag on the config for rustbuild. Once the experimenter has done their fixing they flip the flag back on in their pull request.

#![feature(stable_track($feature_name))] Has an accepted RFC. Has a tracking issue. Can have nightly gating tests. Cannot have stable gating tests.

#![feature(internal_hack($feature_name))], #![feature(rustc_implementation_detail($feature_name))], #![feature(deprecated($feature_name))], ect… Just signalling about the purpose of this feature flag.

2: Recognized ‘friendly forks’.

Somewhere on rust-lang.org have links to a list of friendly forks. avr-rust would probably be on this list. A fork of rust targeting a GCC backend would go on this list. A fork of rust providing tier 1 level support for Plan-9 or Redox, or BSD would go on this list. The various rust for the GPU or (former) rust to asm.js forks would go on this list. Getting on the list and pruning the list would need some sort of process. I think RFCs would be to heavyweight. Perhaps a pull request to rust-lang/rust-www?

3: Make the Rust organizations infrastructure really, really, easy to clone.

This has largely been done already via the transition to travic-ci/appveyor and the creation of rust-central-station, but there are a couple of rough edges. Ideally it would be as simple as '1: Fork rust-lang/rust and clone rust-central-station. 2: Configure your clone (rust-lang/rust fork name, github username for the bots, irc username for the bots, irc channel and login information, S3 bucket identifier, api keys for github, travis, appveyor and aws, signing certificate, TLS key). 3: Run your clone somewhere.' The largest bit of additional support would be to make rustup able to pull from infrastructure clones based on a new optional parameter.

I also had some other ideas that are… less well formed.

• Sponsorship of infrastructure costs for friendly forks.
• A bors mode that automatically copies and r+'s PRs that land in other repositories. (To help friendly forks keep up to date with rust-lang/rust)
• A way for friendly forks to provide cargobomb instances and builds to the project.
• A way for friendly forks to provide advisory results on bors try runs on rust-lang/rust.

[nikomatsakis]

@jessstrap

After reading the above posts I have an alternate, slimmer, strawman proposal. (Please, beat the stuffing out of it.)

Ha, I am happy to see this. I was just coming to this thread to attempt to write-up an alternate proposal that looked a lot like yours, except that I wasn't sure what syntax to use for "experimental" features.

@jessstrap

#![feature(experimental($feature_name))]

Ah, I like this syntax! I have been trying to figure out how I think it should look. I think I would prefer a name other than "experimental". The reason is that we recently chose to call "feature-gated" things as experiments in rustdoc, and I think that those really correspond to "stable-track" things.

Looking at Ye Olde Thesaurus, I see that there are some other nice names for "experimental". I like the name "exploration" or "exploratory":

#![feature(exploratory(foo))]

I really wanted to work "eat-your-laundry" into this, but I can't think of a good way to do that.

@jessstrap

Cannot block other development. Rustbuild has a flag to cfg out all these features at compiler compile time. There is an entry in the build matrix that exercises this flag and gates pull requests.Usually enabled on nightly. If some non experimental development breaks experimental features, the non experimental pull request can just flip the flag on the config for rustbuild. Once the experimenter has done their fixing they flip the flag back on in their pull request.

I'm not sure I followed precisely what you had in mind here, but at the end of the day I suspect it's probably not going to work to try and "isolate" the code for exploratory features, and I think ultimately that's ok. We will have to be conservative in accepting things that have a big impact -- the deeper the changes that are required to the compiler, the higher the bar for the experiment, and that feels "as it should be". Something like naked fns that is pretty orthogonal we can accept with greater ease than custom DST.

One thing I would want to add to your proposal is that we modify our RFC acceptance procedure to include a new option, which is accepting for exploration (@rfcbot fcp exploration). The resulting feature-gate would require the 'exploration' tag. Moving from exploration to 'stable-track' requires a new RFC period -- this might just mean amending the existing RFC with the results from the exploration. You can introduce an RFC with the aim that it be accepted for exploration, which you might do because it has some big gaps, or you can introduce an RFC and we accept it for exploration, because of gaps.

@jessstrap

Recognized 'friendly forks'.

If this is just a matter of us advertising stuff, that seems fine, though I think forking is rarely the right answer, at least for the long term. I think I've come around to the idea that ultimately it's best to have stuff on master so that we get alerted.

@jessstrap

Make the Rust organizations infrastructure really, really, easy to clone.

Certainly a laudable goal; I suspect a lot of the challenge here is financial. That is, storage costs etc are non-trivial.

[jessstrap]

@nikomatsakis

I'm not sure I followed precisely what you had in mind here

To clarify this chunk of my strawman: Goal: Allow the CI to do the hard work of making sure exploratory features aren't impacting 'mainline' features. In tree there is a rustbuild config value for TEST_EXPLORATORY_FEATURES. In the inputs to rustbuild from the travis.yml or appveyor.yml there is a flag for --ALLOW_EXPLORATORY_FEATURES. In the rustc source there is a cargo feature exploratory. To the extent feasible (judged and enforced by reviewers) all code and tests for exploratory features are behind #[cfg(exploratory)] attributes. If rustbuild sees --ALLOW_EXPLORATORY_FEATURES and TEST_EXPLORATORY_FEATURES=TRUE it enables the exploratory cargo feature in that build, thus testing the exploratory features. No entries in the build matrix for stable or beta have --ALLOW_EXPLORATORY_FEATURES. In the build matrix for nightly there is at least one build with and one build without --ALLOW_EXPLORATORY_FEATURES.

If I have a pull request for stable feature S that breaks exploratory feature X I add a commit to my PR that sets TEST_EXPLORATORY_FEATURES=FALSE, it rides the bors go round, ignoring exploratory features, and things are good. Later I (or someone working on X) makes a PR either fixing X to work on top of my changes or removing the conflicting parts of X. The second pr includes a commit that sets TEST_EXPLORATORY_FEATURES=TRUE. It rides the bors go round, testing exploratory features and things are still good.

I hope that helps.

Off topic: A clarified strawman? Now I'm imagining a strawman made out of drop cloths and transparent plastic straws. Does he still scare crows if they can see through him?