Thoughts on RFC/stabilization reform in 2019

internals policy
1

I’ve been thinking a lot about RFC and stabilization reform. This has been partly inspired by my own experience and partly by insights from reading Rust2019 posts.

(EDIT: reformatted to make it less of a wall of text)

(EDIT: summary comment)

This post is intended as more brain-fodder. I had previously posted this pre-RFC which was an (admittedly extreme) proposal for attempting to fix some problems I and others have observed with current processes.

This post takes a more open-ended approach by listing some problems and potential solutions I came up with, smaller and more orthogonal steps. Also, I have intentionally avoided the question of where discussions should be held and what their format should be. I think that’s worthy of its own thread.

More automation

In order to scale processes, more needs to be automated.

rfcbot Table of Comments.

See (the pre-RFC). The idea was that rfcbot can be used to help maintain a structured, always-up-to-date summary of discussion. This can help reduce re-posting of the same idea, while also allowing people with new ideas to jump into discssions without having to read long threads.

Automate the unstable book.

The unstable features book was intended to be a way to keep track of what features are in the works. However, many of its pages are just obsolete, and in my experience, one just needs to go read the tracking issue anyway. I propose doing away with the unstable book and instead having rfcbot or some other bot keep track of every features status with a link to the tracking issue. This would be presented in the form of some webpage/dashboard with a big table. More below.

Automate unresolved questions and the creation of tracking issues, etc.

When an RFC is merged, some human (often @Centril :slight_smile: ) creates a tracking issue and adds a checklist to the OP with unresolved questions and other TODOs before stabilization. I propose that rfcbot should be used to do this instead. During the RFC discussion and FCP, team members can add an unresolved question (e.g. @rfcbot question Should the foo have a quux bar?). When the FCP merge is complete, rfcbot automatically merges the RFC, creates a tracking issue, and adds unresolved questions to the OP. Throughout the rest of the process, rfcbot can be informed via other commands when work items are completed.

Unified tracking from proposal to stabilization for features

If rfcbot is used for more process stuff, it has more semantic info about the process, which means it can expose all of this information in some sort of web dashboard. In my mind, I envision a big table including all RFCs under discussion, all RFCs in various stages of FCP, all RFCs that have been canceled or postponed, all unstable features in the compiler, libstd, and other tools, all stabilization FCP proposals, etc. Moreover, this dashboard would be searchable. It would be everyone’s one-stop shopping to see what is in-progress and how far it has gotten.

I already mentioned this, but I think it is important to expose closed and postponed discussions. This makes it easier to research for new proposals and documents paths that we have chosen not to take.

Experimentation without any guarantee of stabilization

I observe that there is a tension between wanting to play with a feature and try it out on nightly Rust but not being sure if we should ever stabilize it. The labeled-breal-value feature is an interesting example. Two language team members simply do not feel that Rust should have this feature at all, but other team members would like to see it stabilized. Currently, in my experience, the acceptance of an RFC seems to be more or less a guarantee that some form of the feature will be eventually stabilized. The eRFC process is intended as a way to experiment with changes and require an actual RFC before stabilizing. I would like to see this formalized and process-ified more:

I propose that we add a research and development (RnD) mode to rustc (e.g. -Z RnD). It is only available on nightly, and it enables RnD features. An entire RFC is required to make an eRFC feature into an unstable one. It must then go through the normal stabilization process. Notably, I think we should have a pretty low bar for adding RnD features: the feature must not regress stable or nightly compilers and must have support from a simple majority of the appropriate team.

Finish what is started… but don’t stress/oversubscribe team members

In the above-linked pre-RFC, I suggested bottlenecking the RFC process on a rust team member. The intent was to make sure we always have bandwidth to keep making progress on accepted proposals. However, I realized that this could also cause stress for team members.

While reading Niko’s Rust2019 post, it occured to me that we can use Working Groups to solve this problem: two or three persons from the appropriate WG would need to volunteer to shepherd an RFC to stabilization for it to be merged. This ensures the necessary bandwidth without placing a burden on any one person’s shoulders.

More holistic and objective evaluation.

While I realize that no such thing exists in most cases, I suggest that we can do better here.

We need some community-agreed set of design values by which we can holistically evaluate proposals and priorities. By necessity, I think this means we will need to de-prioritize or rule out some possibilities, as disappointing as that might be. I think we should be focusing on doing some things well, rather than everything OK.

These values, goals, etc. should be explicitly set out in a document somewhere and used as the ruler by which we measure all proposals.

We should continually re-evaluate the goals/values and our progress towards them to make sure they are still relevant.

One obvious candidate is for this to be a subsection in the annual roadmap.

Reducing noise and keeping discussions manageable.

I don’t really have much to add here beyond what I said in the aforementioned pre-RFC, but I think this is a real problem, I wanted it to be in this list.

12 Likes
2

:+1:

I will say that this is pretty easy already especially if you're actually taking the time to write a new RFC; anyone who does so should at the very least search through old proposals. One way to improve this is to label all the closed RFCs as well as open and closed issues properly. I've only labeled the merged ones but this took a lot of time and I'd appreciate the help with the rest.

3

One other thing to discuss is how to enact changes: we could do one big set of reforms, but I expect that would be contentious and take a long time. The other option would be to make many incremental improvements.

4

After reading some of the Rust 2019 blog posts, such as Niko’s and withoutboats’s, I had some ideas on how the RFC process could be improved and one of the rust team members suggested that I should post it on internals.rust-lang.org, and this seemed like the right place to share my ideas. (I hope that’s okay with @mark-i-m?)

One thing that was brought up by multiple people in the previous thread is the need for regular summaries. What follows are my thoughts about what this might look like and how it might be achieved. I am not very involved in the RFC process myself, so please feel free to just disregard all of this entirely – I’m just posting it here in the hope that it might lead to some useful ideas and eventually changes.

Summaries

The motivation for this is as withoutboats said – some RFCs get hundreds of comments on them, and no-one wants to read through all of them, but to make matters worse this results in comments that are redundant or duplicating of earlier comments (since they didn’t read all of the comments) and now there are even more comments that nobody wants to read through and the problem is even worse.

Some people have proposed summaries as a solution to this, and here is how I think that could work. (This is similar, but different, to the “Table of Comments” idea.)

What is a summary

Each RFC should be summarised regularly. Those summaries should be complete, clear, concise, and non-redundant.

  • Complete (or sufficient): The summary should include every point mentioned in the preceding comments. Thus by reading (only) the summary you can be sure that you are up-to-date on all points covered in the RFC comments.
  • Clear: The summary should be at least as clear as the original comments, but ideally more clear. An unclear comment is not an excuse for an unclear summary.
  • Concise: The summary should express all the points in the most concise way possible, collapsing related points together as necessary.
  • Non-redundant (or necessary): If there have been multiple comments expressing the same point they should be summarised only once. (This is true across multiple summaries as well - if a point has been mentioned in an earlier summary then don’t repeat it unless there is new information.)

Summaries can include links to the relevant comments that are detail-heavy.

The purpose of this is to optimise for reading. I realise that this might be a lot of work for the summariser, but the net saving for everyone else should far outweigh this. Consequently time should be saved overall. (You probably only summarise one or two RFCs, and benefit from summaries of all the others.)

Implementation

Collapsing comments

Every time the summariser posts a summary, the rfcbot marks all non-summary comments above the summary as “resolved”.

Top comment

The summariser edits the top post themselves. (Or, every time the summariser posts a summary, the rfcbot edits the top post to include the latest summary.)

Updates

In the case that the summariser edits the top comment to include a full summary we might also like there to be update posts that just include the new points so that it is easy for people to “keep up” with an RFC by just reading the updates. In any case, the summary remains the canonical source of information.

I’m sure there are other techniques and automations that would work for this. I think the main issue is the procedure/protocol itself, and automation can follow once that is decided.

Frequency of summaries

The frequency of summaries will likely vary depending on the rate of comments on the RFC. It would be useful to define how frequent they should be, e.g. each RFC should be summarised at least once per a week (if there was at least one comment), or each RFC should be summarised at least once per 30 comments, or each RFC should be summarised every day in which it has more than 10 comments, or some combination thereof.

The goal is that anyone reading the RFC can know when there will next be a summary posted and make an informed decision as to whether they would just like to wait until then or delve into the new comments.

Who is the summariser

In the simplest model, the RFC author must summarise all the posts.

N.B. It need not be a Rust team member. Whilst there should be rust team members involved in every RFC, anyone can summarise it. The team members that are involved in the discussion will likely be aware of all the points in the discussion, but the actual task of summarising need not fall on them.

A proponent, opponent model

One concern I had with the above model is that there could be a conflict of interests if the RFC author is summarising the entire discussion. To address this I had an idea of a “proponent”, “opponent” model.

The proponent summarises all points in favour of the RFC, and the opponent summarises all points against the RFC. This also includes relevant counterpoints, which might develop over the course of the discussion. They should try to use a similar structure to their posts so that points and counterpoints can be compared.

The proponent would be the RFC author.

The opponent would be selected randomly. If they felt that they had a conflict of interests, or they could not commit to the task, then they can “defer” and someone else would be selected randomly. (If everyone defers then presumably the RFC has no-one against it and can trivially be accepted! [Or everyone is really really far too busy, which is another problem altogether!])

The opponent need not be against the RFC, they just need to summarise in good faith (so playing devil’s advocate is fine and good).

Ideally, by reading all of the summary of points in favour, and summary of points against, one should be able to make an informed conclusion or comment about the RFC.

Rules about comments

The rules for commenting on RFCs should be strict and enforced. For example, if someone agrees with another persons post then a thumbs up reaction on the comment is fine, but a new comment is not. I’m sure there are many other useful rules.

3 Likes
5

I’ve been thinking that even before submitting a RFC to solve a problem, the problem itself should be defined.

That is, I think it would be valuable to start with an RFP (Request for Proposals) which describes a problem. The RFP would be discussed, the problem clarified, and then a decision would be made on what to do about it.

For example, I’ve seen several discussions on anonymous enums recently, where authors and participants have all investigated a significant amount of time and energy, notably bikeshedding syntax, even before the lack of anonymous enums was actually recognized as a problem. If we now end up deciding that actually, anonymous enums seem like a bad idea, then all that energy and good-will was wasted.

For another example, more extreme, consider the 2D Graphics proposal to the C++ Standard Library. It was reviewed multiple times over 3 or 4 years, each time receiving a lukewarm response and the committee asking for improvements/clarifications, for finally being rejected because 2D Graphics are not a good fit for the C++ Standard Library. Needless to say, its authors were terribly disappointed.

A process where first the problem need to be acknowledged would limit investment from all parties. Also, this is an excellent opportunity to simply postpone work on a problem, even before it started.

So the process would be first open a RFP, get it accepted, and then multiple RFCs can be opened referencing this RFC and proposing potential solutions of various scope. Also, even if the RFCs are rejected, a later RFC for this RFP will have all previous attempts at its fingertips immediately, rather than wondering whether their bibliography is exhaustive.

Examples of decisions on a RFP:

  • No: The problem described is not perceived as a problem. For example, the lack of M:N threading in std is deliberate.
  • Postponed: The problem described is acknowledged, but exploration of the solution space should be postponed for one reason or another. There may not be enough bandwidth, it may interact with another unstable or recently stabilized feature on which the team would prefer to gain more experience first, …
  • Low-Bandwidth: The problem described is acknowledged, the team/WG does not have time to think on it right now, but is open to starting discussions, albeit at reduced bandwidth. Do not expect mentoring or low-latency shepherding.
  • High-Bandwidth: The problem described is acknowledged, and there is commitment from the team/WG to see this through. Proposals are welcome, mentorship/shepherding is expected.
4 Likes
6

While I really want this to be a good idea, I’ve personally avoided proposing separation of problem from solution because in practice it’s often impossible to decide “this problem isn’t serious enough to warrant a language change” without knowing what the language change to solve it would actually look like. For example, I was on the fence about Ok-wrapping and related issues until https://github.com/rust-lang/rfcs/pull/2107#issuecomment-323515036. Having a concrete RFC to study was what it took to finally convince me it’s just not worth the cost.

But there’s probably something salvageable there about deliberately not engaging in surface debates when deeper potentially blocking issues remain unsolved. In the anon enums case, I remain unconvinced they’re worth their cost (and I’m fairly sure that’s not an uncommon opinion), so we should really not be debating the details of their syntax yet. The real debate probably ought to be “do nothing” vs enum impl Trait vs “anonymous enums” or something like that, but iirc that’s not the debate being had on those RFCs right now.

3 Likes
7

Collapsing comments and having a proponent/opponent system are really interesting ideas! I especially like the key point you pulled out: optimize for reading.

Perhaps one modification of the ToC idea would be to add some sort of rfcbot summary command that appends the body of the post to the OP and collapses all other comments. This would allow people to get the github email notification too.

Regarding frequency of summaries, I still think it should be after a certain number of comments, especially after all the initial typo comments subside.

8

I tend to agree that separating the design from the problem might make it hard to evaluate. On the other hand, there is something to be said for finer-grained roadmap-like discussions where we discuss future directions and explicitly acknowledge certain problems we would like to work on.

Currently, I think that is the main role of eRFCs. Going forward, perhaps the right balance would be to use the annual roadmap to layout a broad vision, and use eRFCs to flesh out parts of it by choosing specific problems to focus on?

9

I don't know how to do "complete" and "concise" in the same thing.

I do like the the proposal the the For and Against summaries are made by different people. My experience is that trying to summarize both side just gets people mad at me.

2 Likes
10

I'd also be afraid you'd run into cases where someone identifies a problem that is completely legitimate, but either very difficult or impossible to solve. This can quickly devolve into a completely nonconstructive discussion arguing whether a solution exists, rather than debating the merits of a given solution.

1 Like
11

I’ve started a prototype implementation of some sort of @rfcbot summarize command here: https://github.com/anp/rfcbot-rs/compare/master...mark-i-m:summaries?expand=1

Basically one uses it as follows:

@rfcbot summarize

This is my summary.

Blah blah blah
- a
- bunch
- of
- points

This would cause rfcbot to update the OP by appending the summary (i.e. everything from “This is my summary.” to “- points”.

Does anyone know how to use the Github API to collapse comments? Also, any other feedback is appreciated.

12

There is no API for that AFAIK.

1 Like
13

Perhaps "concise" is too strong. I think that anything can be made more verbose, either by adding redundancy, or just by explaining the same things but more laboriously, and the complete summary should be neither of those things. So I suppose what I'm looking for is the "minimal complete summary".

Yes - I hope this would avoid a problem where people who were against the RFC felt like their points were not being understood or not being heard because they were all going through the RFC author.

1 Like
14

I wonder whether it would be clearer to have rfcbot always post the first reply to any pull request (à la rust-highfive) and then update that post, rather than editing the original. It's a fairly minor difference, but it seems like there'd be less room for error that way.

5 Likes
15

That was my original proposal in the other thread, but it seemed simpler to do this. Is there a particular problem or error you had in mind?

16

The kind of complaints I've gotten on summaries have included disagreements on whether two points are the same, and thus the second should have been mentioned.

I like aping the Agile Manifesto as a way to phrase things such that they lead to actual guidance rather than lists of adjectives: "While we value both AA and BB, in situations where they're in conflict we will prefer AA over BB". That's way harder to get agreement on, but it's also way more useful.

(Or, said otherwise, if there are no arguments for the opposite it's unnecessary to state: for example, nobody will say that summaries should be "incomplete and loquacious".)

2 Likes
17

While we value concise and complete summaries, in situations where they conflict, I think we should completeness. Otherwise, people have to read arbitrary parts of the thread anyway despite the summaries.

1 Like
18

I’ve been thinking about a possible change to the rfcbot fcp process:

Rather than being opened with a specific direction in mind (merge or close), I think proposing FCP should allow each team member to say “merge” or “close”, and then the team needs consensus to move forward in that direction. If all but 1-2 members of the team say “close” and the remainder don’t respond, then it closes (as today); if all but 1-2 members of the team say “merge” and the remainder don’t respond, then it merges (as today), but if all but 1-2 members of the team say one direction and 1-2 say the other direction then the process continues until there’s a consensus, treating the divergence as indicating objections.

That would avoid the “inevitable momentum” problem where it feels like there’s no way to change direction once a direction is set.

8 Likes
19

I think this is what I mean by "minimal complete summary".

It has occurred to me, however, that the criteria could be relaxed slightly: If there is a comment that contains many details and sub-points, it could be acceptable in the summary to include just a sentence (or a paragraph) about it and then link to the relevant comment which contains all the details. (Rather than the summary just copying out all of the details.)

2 Likes
20

As a real example, how’s this summary I just did for the await bikeshedding? https://github.com/rust-lang/rust/issues/57640#issuecomment-455836523

(I sure hope it doesn’t get totally lost)

This is an interesting example since it’s such a bikesheddable topic (any syntax at all) and most of the “arguments” boil down to "I like how this feels: [example]`. That’s why the summary is really just a big matrix of examples.

2 Likes