Refining RFCs part 2: RFC staging

[AndrewBrinker]

I actually think some non-GitHub system for tracking and searching RFCs, past and present, accepted and in-progress, would be a good idea. The accepted RFCs would be added to the rfcs repo as they are now, with the tracker providing information on where an RFC is in the process, and links to the relevant page for that phase. So, in the problem consensus and RFC drafting phases, it would be a link to the discussion page for the proto-RFC. In the RFC PR page, it would be a link to the PR. If a PR fails, that history of the failed PR would be kept in the tracker, so people can see a full history of the issue. In terms of management of this system, it would probably make the most sense for the shepherd of a particular RFC to update that RFCs history tracker. This system wouldn’t keep any of the materials directly, but would be a way to understand the history of an RFC’s processing, and to see where it is in the process.

[Aatch]

I definitely agree with this. I’ve gotten quite frustrated with poorly-specified RFCs popping up. Worse is when the idea is popular, you end up with tons of discussion about the idea, but less about the actual RFC itself.

[ubsan]

This sounds absolutely amazing. I don’t have many comments, except that I want this, wholesale.

[withoutboats]

I think breaking the RFC process into two stages - identifying problems and proposing solutions - is a really great idea. I don’t think it makes sense at this stage for these two projects to be combined into a single process.

However, I think it may make more sense to recognize that a ‘new feature’ as described in an RFC may interact with multiple open problems. I don’t think we should think of this as a new step in the process for ‘an RFC,’ but rather as two systems - one, a forum for discussing and identifying problems that Rust has, another, a forum for proposing features that mitigate or resolve those problems.

An RFC then could have a more complex relationship with the open problems than a 1-to-1 ‘solves’ relationship. A single RFC could simultaneously solve open problem A, mitigates some of the issues described in open problem B without totally resolving them, and exacerbate open problem C. This also could make it easier to identify some of the drawbacks of an RFC.

Looked at it this way, the list of problems may be of widely varying scope. They could range from things which are probably resolved by a single RFC (like a gap in the standard library) to things that will probably be chipped away at by many different RFCs before they’re finally considered resolved - things like “string handling in Rust is very confusing to new users,” or “orphan rules can make certain kinds of generic code really frustrating.”

There should also be space for discussing open problems in an open ended way separate from drafting an RFC which proposes a specific feature. The idea that we are ready for opening an RFC PR should emerge from that conversation, though given this proposed many-to-many relationship between problems and RFCs, I’m not sure how the assignment of shepherding responsibilities would work.

Lastly, I’d note that the issues on the RFC repo currently serves some of this function, very informally.

[mgattozzi]

I mentioned this over on /r/rust but I think coming up with a system to aggregate all of the data (discussion threads, proposals, what stage the rfc is at) into a site that’s easily accessible and readable would hugely benefit this portion of the discussion. Being able to see which bucket an RFC is in, or to see which motivation it links too, or places that have been added where the discussion is happening, possibly hosting the video meetings that have been had as well by the sub teams could really help out. If it was an open project as well that the Rust community could get involved in I think it would help bring us all closer together in some way. Not everyone is going to be great at writing compilers but they could be good with web dev, so in some way they could help influence the RFC process, albeit in a more meta way. I think in terms of initiatives in the future this could be one worth looking into.

[djc]

I think the regular posting of summaries is a really important part of this. I’m informed in this mostly by my experience with Python’s PEP process. The mailing list threads can get pretty unwieldy, but as long as there is a roll-up after a few days of intense discussion of consensus points and open issues, that makes the process much more accessible to contributors with less time on their hands.

The focus on motivation early in the process seems exactly right.

[jntrnr]

+1 to rollups. I’ve simply been unable to follow some of the, specifically lang-related, RFCs because of the time commitment to read all the comments.

[nikomatsakis]

Some thoughts:

I think @withoutboats is correct that a given RFC may address or partially address several problems, so it makes sense to separate those two things out. But I still think there is a need for a space in between problem and final solution, which is what we are trying to introduce (to some extent, you could view issues on the RFC repo as “problems” and PRs as “solutions”). This space is sort of “draft solution”.

(I admit I am still not entirely sure what it will mean to write a motivation without any solution in mind. At least when I work on RFCs, I find that the two blend together a bit – the motivation is a good place to introduce the problems and try to describe the solution.)

On a very procedural note, one thing that bothers me today is that we no good single link to give for a particular RFC. I think it’d be awesome if we changed the format of RFC PRs so that they had no discussion. Instead, they were a collection of links:

• to the issues about the problems being solved;
• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
  • perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

[nikomatsakis]

In particular, newly-opened RFC PRs today often get an avalanche of comments and early revisions, making it very hard to join the discussion even a week later. Pushing early feedback to our forum instead will make the eventual RFC PR discussion more focused and easier to participate in.

I am a fan of this, definitely – I think an important role for shepherds (or maybe others too) ought to be summarizing. That is, in my ideal world, we’d start a thread with some amount of focus, discuss things, and then produce a link to the thread along with a link to an (in-thread) summary of what led up to it. It doesn’t have to address every post: but it should be enough that people don’t have to read all the prior posts to feel caught up.

Part of the reason to do the summary in-thread is to allow people to respond (in-thread) to clarify or add missing bits to the summary.

[anp]

• to the issues about the problems being solved;

• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
• perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

To chime in from an automation perspective – making sure this follows a standardized format would be very useful for providing some visibility into the statuses of various RFCs.

[nikomatsakis]

Yes. This reminds me that @brson and I have kicked around the idea of making a dashboard page for this summary instead.

[chriskrycho]

• to the issues about the problems being solved;
• to the issue about the motivation for this RFC;
• to a discuss thread (or multiple such threads) for discussing the RFC itself;
  • perhaps multiple threads, with summaries given where relevant;
• if the RFC is accepted, a link to the final text in the repo itself;
• if the RFC is not accepted, a link to the final rational

While in general I sympathize very much with all of this, I will say: some kind of central user interface as suggested above needs to exist, and the design of such an interface needs to be very carefully considered, or it will be even harder to understand RFCs than it is today. The downsides to GitHub-driven RFCs are fairly well-known in general and are well-summarized by @aturon at the top of the thread; but the upside is that there is one canonical source of truth for that RFC.

That said, if we have API-type access to all the relevant places where the discussion happens (Discourse threads, relevant GitHub issues indicating problems, etc.) in such a problem tracker, along with (perhaps?) a CMS-style interface for the shepherds to use for summarizing status, that could be a real win.

[glaebhoerl]

Just wanted to mention a couple more current “pain points” which are sort of complementary to the “it’s hard to get caught up with RFC discussions due to their length” problem:

• When discussion of an idea or proposal keeps moving to new threads (discourse, issue tracker, RFC postponements followed by new RFCs), thoughts I’d expressed in one of the earlier iterations are often still just as relevant, but now essentially lost as most people don’t go back to read earlier discussions (if they even know about them). Having to re-express the same thoughts all over again every time a new thread is started is exhausting, and starts to make it feel like a contest of stamina rather than a discussion about ideas. (The most egregious example of this was/is the “impl Trait” feature.)

• When a discussion thread gets long enough, it often happens that someone doesn’t have time to read the whole thread, but still wants to propose an idea or ask a question that seems “obvious” to them. Which just happens to be the same idea that’s been brought up in that thread, in the same fashion, 50 times beforehand. (Examples: “why can’t we just use fn foo<T>() -> T for abstract output parameters”; “why can’t we just optimize Box::new instead of adding a placement operator”.) Which then exacerbates the original issue by making the thread even longer. (Under the existing regime, all solutions to this problem are bad: either you erect an unreasonable barrier to entry for discussing RFCs by forcing everyone to read miles of discussion before commenting themselves, or you burden everyone else with the consequences when people don’t and then keep independently bringing the same ideas up over and over.)

All of these problems (including the originally-mentioned ones) seem to motivate the same kind of solution: having some kind of persistent document/page/site where the ideas, arguments, and counterarguments are mapped out, according to their logical or topological relationships, “once and for all”; and this would be grown and fleshed out as discussion progresses. (Which sounds broadly similar to the “Problem Tracker” described in the blog post.) But doing this “right”, or even “good enough” seems nontrivial: Who gets the right to edit these pages? How are the ideas even laid out – as a linear text document (with its attendant drawbacks), or as some kind of nonlinear tree or graph structure (laid out how)? You can’t even reliably silo off particular “big-ticket items” or “problem areas” from each other, the problems and/or their solutions are very often interrelated. (For example.) (Maybe the best solution to this last problem would be to employ a sophisticated device, known as the “hyperlink”; like a wiki, if not actually a wiki.)

[nrc]

Some thoughts on staging:

• separate motivation RFCs seems like a really excellent idea!
• also very much like the idea of expanding the pool of shepherds
• I’m a bit worried by the role of shepherds as scribes - shepheds are already overworked and not properly fulfilling their duties. I’d also like to encourage authors to be more responsible for RFCs and this seems a bit like babysitting. In the future I’d like writing and ‘maintaining’ an RFC to be more work, but much better supported, rather than less work, per se.
• others have mentioned collaborative authorship of RFCs, this sounds good to me and we have the tools (GitHub) and culture (OSS) to make this work.
• I’m not keen about moving more work to Discourse, I like it for conversation, but not for serious editing, etc. (Not that GitHub is perfect by a long shot, but still).
• The summary/state of the art conversation seems like a good candidate for being a living document/wiki on the dashboard.

[ubsan]

I would love for authors to be more included in RFC processes, and one issue I’ve seen as an RFC author is that it often feels pretty hopeless; there’s absolutely zero communication between the teams and you. I didn’t even know that there was such a thing as a “shepherd” until last week, despite writing… four RFCs over the past year.

[nrc]

there’s absolutely zero communication between the teams and you

I think this is really important to fix. ‘Better communication’ is something that I think would improve a lot of the friction around some of the process-ey stuff in the community. And you are right that communication between the sub-teams and the wider community (and in particular RFC authors) has sometimes been poor.

I’d like to see suggestions in this round of process updates for ways (formal and informal) to improve communication. I hope the increased pool of shepherds and increased level of interaction required to maintain the rolling updates will help. More ways would be better too.

[tomaka]

In my opinion, the biggest problems here is that github’s (and discourse’s) interface is not meant to be used for long conversations.

When commenting on an RFC, most people don’t read the entire conversation and bring up the same points that were already raised earlier in the conversation.

But how can you blame them when it’s so freaking long? For example I got interested in an RFC this morning, but the comments are 36 pages of text long! Either I read everything and it take me the whole morning, or I don’t read it and I’ll be “that guy” who brings up a point that was already discussed earlier. The end result is that I won’t participate in that discussion at all, even though I think I have an important point. I’ll wait until the RFC is approved, then if necessary raise my point in the tracking issue, which is usually less flooded.

The same problem arises if you open your own RFC. As you are supposed to defend your RFC, you are also logically supposed to read all the comments. But instead of reading few constructive comments, you get flooded by walls of text. I personally have two RFCs in mind that I’d like to propose, but that I won’t because of this point alone.

Compare this to reddit’s interface. Sometimes some RFCs have been linked on reddit and discussed on reddit, and it makes it pretty clear that their interface is much more readable for long discussions. Obviously I’m not saying that we should move the discussions to reddit, but I think it shows a clear problem.

[glaebhoerl]

The same problem arises if you open your own RFC. As you are supposed to defend your RFC, you are also logically supposed to read all the comments. But instead of reading few constructive comments, you get flooded by walls of text. I personally have two RFCs in mind that I’d like to propose, but that I won’t because of this point alone.

This is a good point too… and this somewhat characterizes my own behavior as well. (It hadn’t even occurred to me that it could be otherwise…)

[rkjnsn]

The problem I find with Reddit is that in can be hard to find new posts during the discussion, since one has to figure out under which subthread it appears. If one is interested in following the entire conversation, Discourse’s linear format works better. Newsgroups seemed to have the best of both worlds (easy to identify new messages in any subthread, but the conversation tree was clear and it was easy to ignore parts of the conversation that went off the rails). I’m still surprised none of the current web-based options seem on par.

[steveklabnik]

The problem I find with Reddit is that in can be hard to find new posts during the discussion,

The real problem is that they provide this, but only for people with Gold

That said, I would pretty vehemently oppose Reddit being a part of any official Rust process; its format isn’t the problem though.