Refining RFCs part 2: RFC staging

[aturon]

I’ve recently written a blog post enumerating several ideas for improving Rust’s RFC process. The post contains three separate (but interdependent) proposal sketches, which I’m eager for feedback and discussion on. I’m carving out each proposal as a standalone thread here on internals.

At the heart of Rust’s open development is the RFC process. Every major change to the language, compiler, core libraries, tooling, and policy go through an RFC writeup and consensus-building process. The process served us incredibly well in clarifying our technical direction on the road to 1.0, and has continued to be highly active since then, with on average about 2 RFCs merged every week.

But it’s not all roses. There’s been a growing sense among both Rust leadership and the broader community that the RFC process needs some further refinement as we continue to grow the community. I want to lay out my view of the problems and sketch some possible solutions, based on extensive discussion and brainstorming with many others on the team.

Each idea operates at a different scale (from big-picture to low-level mechanics), but they are intended to fit together into a whole; each one supports the others. Ultimately, these should result in a single RFC, but in the meantime I’ll start a discuss thread for each proposal.

There is a clear common theme to all of the problems I want to raise: communication. We need to find ways to better scale up lines of communication around the RFC process, and for Rust core development in general. There is also a cross-cutting concern: a need to increase our focus on mentoring and the path to team membership. @wycats has a great saying about measuring the health of the team structure:

Being a very active contributor who is not yet on a subteam should feel very close to actually being on that subteam.

Shooting for such a state of affairs has many benefits, not least of which is increasing the scalability of our community.

This thread is for the “RFC staging” part of the proposal.

---

Proposal: RFC staging

The problem

RFCs are hard to keep up with in part because reading a full design – and all the commentary around it – can be a lot of work, and there tend to be a large number of active RFCs in flight at any time. RFC discussions are often hard to follow, due to the overwhelming number of comments, sometimes stretching over multiple forums. Naturally, this problem is exacerbated by "controversial" RFCs, which is where we most need broad input and careful discussion. It can also be hard to track RFCs that are in some sense “competing” (offering alternative proposals for a common problem), or to correlate discussion between the discuss forum and github.

It’s also problematic to start off with a full proposal. What we really want is to get the community on the same page first about the importance of the problem being solved, and then to proceed to the design phase, perhaps considering multiple competing designs.

Finally, RFCs are sometimes closed as “postponed”, but ideally that should not simply terminate the discussion; instead, the discussion should simply continue elsewhere, or somehow be marked as being at a different stage.

The proposal

Idea: introduce stages into the RFC process, including one for reaching consensus on motivation prior to considering a design.

Idea: move the focus away from an RFC PR as the primary venue for RFC discussion.

Put differently, the idea is to orient the RFC process around problems first, and solutions second.

The rough phases I have in mind are:

1. Problem consensus
2. RFC drafting
3. RFC PR(s)
4. FCP
5. RFC merged

Concretely, what this would look like is having some venue for tracking the problems we might want to solve, perhaps a revamped version of the RFC issue tracker. Whatever this venue is, it would track the progression through all of the phases. Let’s call this venue the “Problem Tracker”.

• Phase 1: Problem consensus. The initial discussion is essentially about reaching consensus on the motivation section of an RFC, which should include examples and make a compelling case that solving the problem is important enough to warrant expending energy and potential complexity. The subteam would sign off on that motivation, at which point there is some level of commitment to solve the problem. That puts the focus where it should be – solving problems – and should make it much easier for subteam members to engage early on in the RFC lifecycle.

• Phase 2: RFC drafting. This phase can proceed in parallel with the previous one. During this phase, people sketch designs and work toward one or more full RFC drafts. Brainstorming and discussion on specific drafts would happen within dedicated “pre-RFC” discuss posts, which are linked from the Problem Tracker. In particular, newly-opened RFC PRs today often get an avalanche of comments and early revisions, making it very hard to join the discussion even a week later. Pushing early feedback to our forum instead will make the eventual RFC PR discussion more focused and easier to participate in.

• Phase 3: RFC PR(s). At some point, a shepherd (see below) can determine that an RFC draft is of sufficiently high quality and steady state that a PR should be opened, at which point discussion proceeds as it does today. Multiple RFC PRs might be open for the same basic problem – and indeed, this is a good way to take the “Alternatives” section more seriously. All open PRs would be linked from the Problem Tracker.

• Phases 4 and 5 work just as today.

One interesting aspect of this phasing: it’s possible to approve a Motivation section, then get all the way to RFC PR, only to close out the PR for one reason or another. In such cases, it should be possible to go back to the Problem Tracker and build a new, alternative RFC draft with the same Motivation section.

Note that, in this regime, you don’t ever open an RFC PR out of hand – it must go through the earlier phases, including the pre-RFC discuss post. While this may feel like more process, I think that globally it will make the whole thing more efficient, by weeding out poorly motivated RFCs earlier, by focusing attention on the problem, by producing higher quality RFC PRs, and (as we’ll see) by decentralizing the process a bit more. In addition, it makes it easier to cope with the problem of “Does this need an RFC?”

As part of this proposal, I think we should “reboot” the notion of a shepherd. The idea would be to create a broader network of people around a subteam who are empowered to help move the RFC process along in various ways, but aren’t necessarily responsible for the final decision. So, for example, we would have a larger set of “lang shepherds” who help lang RFCs progress. The powers and responsibilities of shepherds would include:

• “Calling to question” – that is, proposing that the subteam move to make a decision on problem consensus or moving to FCP.

• Working with the community to help brainstorm, draft, and revise pre-RFCs.

• Moving to from pre-RFC to RFC PR phase.

• Acting as the “scribe” for the RFC process, by keeping the Problem Tracker up to date. In particular, the subteams currently attempt to provide "summary" comments for contentious RFCs, to help people track the discussion. This proposal would give those comments more formal status, as something that would go directly on the Problem Tracker, and that any shepherd could provide at any point.

All subteam members can act as shepherds as well.

In general, I envision the Problem Tracker as the go-to place to see where things stand for a given problem/set of proposals, including summarization of discussion and pros/cons for the proposals. The shepherds would play a special role in establishing that official record.

I think these changes make the RFC process both more accessible and more scalable. More accessible because it’s easier to get involved and get quick feedback in lightweight ways (before writing up an entire design). More scalable because of increased parallelism, and because the big decision points happen at either an easier stage (establishing motivation) or with many fewer proposals in flight (the RFC PR stage).

Details to be worked out:

• What happens to current RFC PRs? Are they grandfathered in, or moved into this new process?
• Where does the “problem tracker” live?
• What are good guidelines around an initial “motivation”?
• How and where can we keep an “official record” of the progression of a problem, including links to (and summaries of) pre-RFC and RFC PR threads?

[AndrewBrinker]

I actually think some non-GitHub system for tracking and searching RFCs, past and present, accepted and in-progress, would be a good idea. The accepted RFCs would be added to the rfcs repo as they are now, with the tracker providing information on where an RFC is in the process, and links to the relevant page for that phase. So, in the problem consensus and RFC drafting phases, it would be a link to the discussion page for the proto-RFC. In the RFC PR page, it would be a link to the PR. If a PR fails, that history of the failed PR would be kept in the tracker, so people can see a full history of the issue. In terms of management of this system, it would probably make the most sense for the shepherd of a particular RFC to update that RFCs history tracker. This system wouldn’t keep any of the materials directly, but would be a way to understand the history of an RFC’s processing, and to see where it is in the process.

[Aatch]

I definitely agree with this. I’ve gotten quite frustrated with poorly-specified RFCs popping up. Worse is when the idea is popular, you end up with tons of discussion about the idea, but less about the actual RFC itself.

[ubsan]

This sounds absolutely amazing. I don’t have many comments, except that I want this, wholesale.

[withoutboats]

I think breaking the RFC process into two stages - identifying problems and proposing solutions - is a really great idea. I don’t think it makes sense at this stage for these two projects to be combined into a single process.

However, I think it may make more sense to recognize that a ‘new feature’ as described in an RFC may interact with multiple open problems. I don’t think we should think of this as a new step in the process for ‘an RFC,’ but rather as two systems - one, a forum for discussing and identifying problems that Rust has, another, a forum for proposing features that mitigate or resolve those problems.

An RFC then could have a more complex relationship with the open problems than a 1-to-1 ‘solves’ relationship. A single RFC could simultaneously solve open problem A, mitigates some of the issues described in open problem B without totally resolving them, and exacerbate open problem C. This also could make it easier to identify some of the drawbacks of an RFC.

Looked at it this way, the list of problems may be of widely varying scope. They could range from things which are probably resolved by a single RFC (like a gap in the standard library) to things that will probably be chipped away at by many different RFCs before they’re finally considered resolved - things like “string handling in Rust is very confusing to new users,” or “orphan rules can make certain kinds of generic code really frustrating.”

There should also be space for discussing open problems in an open ended way separate from drafting an RFC which proposes a specific feature. The idea that we are ready for opening an RFC PR should emerge from that conversation, though given this proposed many-to-many relationship between problems and RFCs, I’m not sure how the assignment of shepherding responsibilities would work.

Lastly, I’d note that the issues on the RFC repo currently serves some of this function, very informally.

[mgattozzi]

I mentioned this over on /r/rust but I think coming up with a system to aggregate all of the data (discussion threads, proposals, what stage the rfc is at) into a site that’s easily accessible and readable would hugely benefit this portion of the discussion. Being able to see which bucket an RFC is in, or to see which motivation it links too, or places that have been added where the discussion is happening, possibly hosting the video meetings that have been had as well by the sub teams could really help out. If it was an open project as well that the Rust community could get involved in I think it would help bring us all closer together in some way. Not everyone is going to be great at writing compilers but they could be good with web dev, so in some way they could help influence the RFC process, albeit in a more meta way. I think in terms of initiatives in the future this could be one worth looking into.

[djc]

I think the regular posting of summaries is a really important part of this. I’m informed in this mostly by my experience with Python’s PEP process. The mailing list threads can get pretty unwieldy, but as long as there is a roll-up after a few days of intense discussion of consensus points and open issues, that makes the process much more accessible to contributors with less time on their hands.

The focus on motivation early in the process seems exactly right.

[jntrnr]

+1 to rollups. I’ve simply been unable to follow some of the, specifically lang-related, RFCs because of the time commitment to read all the comments.

[nikomatsakis]

Some thoughts:

I think @withoutboats is correct that a given RFC may address or partially address several problems, so it makes sense to separate those two things out. But I still think there is a need for a space in between problem and final solution, which is what we are trying to introduce (to some extent, you could view issues on the RFC repo as “problems” and PRs as “solutions”). This space is sort of “draft solution”.

(I admit I am still not entirely sure what it will mean to write a motivation without any solution in mind. At least when I work on RFCs, I find that the two blend together a bit – the motivation is a good place to introduce the problems and try to describe the solution.)

On a very procedural note, one thing that bothers me today is that we no good single link to give for a particular RFC. I think it’d be awesome if we changed the format of RFC PRs so that they had no discussion. Instead, they were a collection of links:

• to the issues about the problems being solved;
• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
  • perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

[nikomatsakis]

In particular, newly-opened RFC PRs today often get an avalanche of comments and early revisions, making it very hard to join the discussion even a week later. Pushing early feedback to our forum instead will make the eventual RFC PR discussion more focused and easier to participate in.

I am a fan of this, definitely – I think an important role for shepherds (or maybe others too) ought to be summarizing. That is, in my ideal world, we’d start a thread with some amount of focus, discuss things, and then produce a link to the thread along with a link to an (in-thread) summary of what led up to it. It doesn’t have to address every post: but it should be enough that people don’t have to read all the prior posts to feel caught up.

Part of the reason to do the summary in-thread is to allow people to respond (in-thread) to clarify or add missing bits to the summary.

[anp]

• to the issues about the problems being solved;

• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
• perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

To chime in from an automation perspective – making sure this follows a standardized format would be very useful for providing some visibility into the statuses of various RFCs.

[nikomatsakis]

Yes. This reminds me that @brson and I have kicked around the idea of making a dashboard page for this summary instead.

[chriskrycho]

• to the issues about the problems being solved;
• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
  • perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

While in general I sympathize very much with all of this, I will say: some kind of central user interface as suggested above needs to exist, and the design of such an interface needs to be very carefully considered, or it will be even harder to understand RFCs than it is today. The downsides to GitHub-driven RFCs are fairly well-known in general and are well-summarized by @aturon at the top of the thread; but the upside is that there is one canonical source of truth for that RFC.

That said, if we have API-type access to all the relevant places where the discussion happens (Discourse threads, relevant GitHub issues indicating problems, etc.) in such a problem tracker, along with (perhaps?) a CMS-style interface for the shepherds to use for summarizing status, that could be a real win.

[glaebhoerl]

Just wanted to mention a couple more current “pain points” which are sort of complementary to the “it’s hard to get caught up with RFC discussions due to their length” problem:

• When discussion of an idea or proposal keeps moving to new threads (discourse, issue tracker, RFC postponements followed by new RFCs), thoughts I’d expressed in one of the earlier iterations are often still just as relevant, but now essentially lost as most people don’t go back to read earlier discussions (if they even know about them). Having to re-express the same thoughts all over again every time a new thread is started is exhausting, and starts to make it feel like a contest of stamina rather than a discussion about ideas. (The most egregious example of this was/is the “impl Trait” feature.)

• When a discussion thread gets long enough, it often happens that someone doesn’t have time to read the whole thread, but still wants to propose an idea or ask a question that seems “obvious” to them. Which just happens to be the same idea that’s been brought up in that thread, in the same fashion, 50 times beforehand. (Examples: “why can’t we just use fn foo<T>() -> T for abstract output parameters”; “why can’t we just optimize Box::new instead of adding a placement operator”.) Which then exacerbates the original issue by making the thread even longer. (Under the existing regime, all solutions to this problem are bad: either you erect an unreasonable barrier to entry for discussing RFCs by forcing everyone to read miles of discussion before commenting themselves, or you burden everyone else with the consequences when people don’t and then keep independently bringing the same ideas up over and over.)

All of these problems (including the originally-mentioned ones) seem to motivate the same kind of solution: having some kind of persistent document/page/site where the ideas, arguments, and counterarguments are mapped out, according to their logical or topological relationships, “once and for all”; and this would be grown and fleshed out as discussion progresses. (Which sounds broadly similar to the “Problem Tracker” described in the blog post.) But doing this “right”, or even “good enough” seems nontrivial: Who gets the right to edit these pages? How are the ideas even laid out – as a linear text document (with its attendant drawbacks), or as some kind of nonlinear tree or graph structure (laid out how)? You can’t even reliably silo off particular “big-ticket items” or “problem areas” from each other, the problems and/or their solutions are very often interrelated. (For example.) (Maybe the best solution to this last problem would be to employ a sophisticated device, known as the “hyperlink”; like a wiki, if not actually a wiki.)

[nrc]

Some thoughts on staging:

• separate motivation RFCs seems like a really excellent idea!
• also very much like the idea of expanding the pool of shepherds
• I’m a bit worried by the role of shepherds as scribes - shepheds are already overworked and not properly fulfilling their duties. I’d also like to encourage authors to be more responsible for RFCs and this seems a bit like babysitting. In the future I’d like writing and ‘maintaining’ an RFC to be more work, but much better supported, rather than less work, per se.
• others have mentioned collaborative authorship of RFCs, this sounds good to me and we have the tools (GitHub) and culture (OSS) to make this work.
• I’m not keen about moving more work to Discourse, I like it for conversation, but not for serious editing, etc. (Not that GitHub is perfect by a long shot, but still).
• The summary/state of the art conversation seems like a good candidate for being a living document/wiki on the dashboard.

[ubsan]

I would love for authors to be more included in RFC processes, and one issue I’ve seen as an RFC author is that it often feels pretty hopeless; there’s absolutely zero communication between the teams and you. I didn’t even know that there was such a thing as a “shepherd” until last week, despite writing… four RFCs over the past year.

[nrc]

there’s absolutely zero communication between the teams and you

I think this is really important to fix. ‘Better communication’ is something that I think would improve a lot of the friction around some of the process-ey stuff in the community. And you are right that communication between the sub-teams and the wider community (and in particular RFC authors) has sometimes been poor.

I’d like to see suggestions in this round of process updates for ways (formal and informal) to improve communication. I hope the increased pool of shepherds and increased level of interaction required to maintain the rolling updates will help. More ways would be better too.

[tomaka]

In my opinion, the biggest problems here is that github’s (and discourse’s) interface is not meant to be used for long conversations.

When commenting on an RFC, most people don’t read the entire conversation and bring up the same points that were already raised earlier in the conversation.

But how can you blame them when it’s so freaking long? For example I got interested in an RFC this morning, but the comments are 36 pages of text long! Either I read everything and it take me the whole morning, or I don’t read it and I’ll be “that guy” who brings up a point that was already discussed earlier. The end result is that I won’t participate in that discussion at all, even though I think I have an important point. I’ll wait until the RFC is approved, then if necessary raise my point in the tracking issue, which is usually less flooded.

The same problem arises if you open your own RFC. As you are supposed to defend your RFC, you are also logically supposed to read all the comments. But instead of reading few constructive comments, you get flooded by walls of text. I personally have two RFCs in mind that I’d like to propose, but that I won’t because of this point alone.

Compare this to reddit’s interface. Sometimes some RFCs have been linked on reddit and discussed on reddit, and it makes it pretty clear that their interface is much more readable for long discussions. Obviously I’m not saying that we should move the discussions to reddit, but I think it shows a clear problem.

[glaebhoerl]

The same problem arises if you open your own RFC. As you are supposed to defend your RFC, you are also logically supposed to read all the comments. But instead of reading few constructive comments, you get flooded by walls of text. I personally have two RFCs in mind that I’d like to propose, but that I won’t because of this point alone.

This is a good point too… and this somewhat characterizes my own behavior as well. (It hadn’t even occurred to me that it could be otherwise…)

[rkjnsn]

The problem I find with Reddit is that in can be hard to find new posts during the discussion, since one has to figure out under which subthread it appears. If one is interested in following the entire conversation, Discourse’s linear format works better. Newsgroups seemed to have the best of both worlds (easy to identify new messages in any subthread, but the conversation tree was clear and it was easy to ignore parts of the conversation that went off the rails). I’m still surprised none of the current web-based options seem on par.