I wrote a blog post describing a possible “staged RFC process”, which aims to improve upon our existing process. I wanted to open this thread for discussing the ideas there:
http://smallcultfollowing.com/babysteps/blog/2018/06/20/proposal-for-a-staged-rfc-process/
Though perhaps I should create a repository instead =)
Immediate reactions:
Explicit stages/priorities sounds like an excellent idea. We absolutely should do this. Nine stages seems like a bit much for the typical RFC, but I’m getting the impression you’re mostly thinking about how to improve RFCs/discussions around immensely huge and complex topics like modules, fields in traits, const generics, HKTs, dependent types, etc, and in those cases your list of stages seems reasonable.
I’m very skeptical of a separate repository per-RFC, because we’ve experimented with that a few times already and it hasn’t exactly been a resounding success, so I’m kind of surprised this is such a massive part of the proposal. Maybe there’s some communication failure or conflicting perceptions going on here, so I’m just going to be brutally honest with my perception of the experiments so far:
There’s probably a good way to make use of separate repos for some RFCs, though I don’t know what it is. Maybe this sort of completely overhaul is the way to make it actually work, and it’s weird today only because most people are only looking at the main repo.
Based on what you observed, that is what it sounds like. I think if, as Niko proposed, the main RFC Repo becomes an Index to all RFC-specific Repos then, this concern should be mitigated.
I wonder whether the stages would be for specific RFCs or for each repo. Many RFCs are refinements of topics covered by existing RFCs. In a repo-per-RFC world, I imagine the refinements would be added to the repo of the RFC they refine. In that manner, the "OP RFC" may be at the "Evaluating" or "Stabilized" stage, and the refinement effectively begins at the "Draft" stage, with spitballing and proposals taking place organically during implementation.
One possibility, especially for complex or contentious RFCs, may be that repos may include "marketing materials" for their proposals. On the one hand, it would be amazing for, say, the "async" repository to contain wthoutboats' posts on pinning. On the other, this could be exploited to make a proposal appear to have more support than it does. I think the best means to prevent that is by delegating responsibility for it to the champion or shepherd.
From @nikomatsakis' post:
I have hopes that we will find ways to use explicit repositories to drive discussions towards consensus faster.
I'm concerned about any process that tries to increase the speed at which we do things. In a deliberative environment, speed is often the enemy. A sense of urgency is easily used--often with the best of intentions--to short-circuit conversation. Rather than drive towards consensus faster, I'd hope that a revised RFC process would drive towards consensus more effectively. That is, by the end of the process, it should be clear that each option and its consequences are well-understood, that the acceptance criteria draw critical distinctions between the options, and, especially for heavily contested decisions, criteria are established determining when a proposal has failed to meet its objectives.
One thing I’d like to see noted in this proposal—essentially all of which I think is very good and makes a lot of sense for where Rust is in its adoption cycle and the increasing engagement that comes with it!—is the “offramps.”
There have been a number of ways people have gotten at this lately, all pointing at roughly the same thing: there’s an inherent tendency toward forward motion through the process as practiced today, and also as described and indeed pictured here!
An RFC may in fact end up being rejected at each stage along the way before final stabilization, though with the concerns driving such a move having to be increasingly serious at each stage in the proposal process. (Compare the staged TC39 process, which has precisely that behavior as well.) But it’s implicit, and simply stating it outright (as TC39 does) would be helpful.
Yes, I actually had intended to push some text about this, and I realized after posting the blog post that I never did add that. I do think it's a good question to iterate on. My expectation is that there can be off ramps potentially at any point, if things are not working out, but I think it's good to work out where we think the most likely off ramps are to be, and precisely what we would do with such RFCs.
My expectation is that there are a few possible outcomes, depending:
Well, they are not all equal. The "team decision points" in particular are not expected to last a long time. If they are, then the team clearly has not reached any form of consensus, and hence we should go back to the prior stage and not try to advance.
Clearly not all RFCs are equal. But I do think we should use this same process for all RFCs. My expectation is that smaller RFCs will just move quickly through the initial phases, since there won't be a need for extended "spitballing" -- in an extreme case, perhaps we'll just jump directly to the "Draft" stage in one step (which is essentially the same as the process today, in that case, except that discussion is taking place on a distinct repository instead of a pull request).
I too was initially skeptical, but I don't think your examples are prove very much (in fact, I don't even think they are necessarily problems -- though the delegation example is interesting!).
For example, the "fields in traits" repository is mostly dead -- but then, the PR was dead too! I think you are correct to say it is languishing until somebody has time to work on it, and that's ok. In fact, it's anticipated. This is the "prioritization feedback" I was talking about -- it's fine for a design to hang out in the Spitballing phase until some team champion has time to focus on it.
(I do think we want to be careful to avoid artificial scarcity -- it can be easy to forget that throughput problems can be resolved by growing the team as well -- but this is no different from today either.)
I think that the NLL design would have benefited a lot from having a repository that we used from start to finish. As it was, the repository was helpful even at the late stage in which it was created. That is not an RFC where I expect a wide set of contributors as it is deeply technical.
The delegation case is more interesting. I can't speak to the "crowdwritten" problem, I've not looked in detail at the RFC. It can be a challenge to develop things in an open way but keep a central vision. I don't know what's the solution there.
In contrast, though, the existing experience of using a single PR has produced pain in many, many instances. Anything remotely complex becomes a huge bear to manage. GitHub starts hiding key comments. Summaries kind of help but only so much.
A single PR feels "ok" if the RFC is short and the discussion simple -- but in that case, I would expect that the repository would just be very simple, and that doesn't seem like a problem?
I guess to summarize, I don't think we've really tried using a "repo per RFC" yet -- usually, we've created a repo either when the process had stalled or was effectively done (delegation, again, being an outlier).
I was thinking though that it would make sense to make a few real efforts at using the process before we decide whether to commit.
Ah, I wanted to bring up another example of where I think this new process could be really helpful. Often, you have a simple idea that anybody could describe -- but to make the RFC really actionable would require a detailed description of the mechanics of the idea that the original author may not be able to do.
A good example is the if let guard RFC (propose if let guard by strake · Pull Request #2294 · rust-lang/rfcs · GitHub) by @strake. If you read that thread, you'll see that I went back and forth about how much detail to require before we accept. I think the intentions of the RFC are clear enough, but there are some troublesome problems around the semantics that I would personally still like to see spelled out. But at the same time it doesn't seem reasonable to necessarily expect @strake to go learn about MIR just for us to decide if the RFC makes sense at a high-level.
Using a staged process, with a multi-author RFC repo, would allow us to move forward while deferring a detailed desugaring for later. Then @pnkfelix or myself or whomever could come along and either write the detailed desugaring ourselves or mentor @strake to do so (if they have interest).
(I suspect this would be part of the Implementing phase, though you could imagine it happening in the Designing phase as well.)
I’ve been mulling over a similar idea ever since the scalability issues came up during the pre-Impl period flurry of RFCs. However, I’ve never felt like I would have enough time to really champion it, so I never wrote up my proposal. It’s a fairly natural extension of the RFC process, so it’s not surprising that this looks pretty similar.
I too am skeptical about separate repos per RFC. While there might be a couple of cases in which that would be necessary, many RFCs are relatively small, and a repo per RFC could be a lot of repos; there are 71 open pull requests on the RFCs repo right now, and while that might be a little smaller due to the filter between the “spitballing” stage and “designing/prototyping” stage, it could still be a lot of repos to manage, even with extra tool support.
I think it would make more sense for it to stay as a single repo, in which pull requests for the initial proposal are the way to go from the “spitballing” to “design/prototyping” stage, and then tickets and PRs can just be tagged based on the RFC they apply to. For particularly big/complicated RFCs, it might make sense to split them out into an entirely separate repo, but I think the majority are small enough for that to be overkill.
Within a single repo, it would be possible to have a directory per RFC instead of just a single file, if you want to have more than one document.
@chriskrycho brings up a good point about offramps. It would be good to discuss what the options are when something doesn’t move forward, or when it needs to move back to a previous stage. A few cases that should be covered:
What would be good to spell out is whether things stay in the same state, go back to a prior state, or go into some alternative “postponed” or other states in these cases.
Another part I have a question on is the “find a champion from a team.” There have been several attempts in the past to have shepherds or champions of RFCs from one of the relevant teams, but I don’t know if that’s really happened. Instead, I see people propose RFCs when they feel like they have an idea that’s ready, and it will be accepted by the team or not.
How does one go about finding a champion? What happens if one would rather just let their proposal speak for itself?
(edit: apologies if a couple of these are already answered; starting writing this up and then hadn’t finished by the time Niko had posted a few replies)
One of the goals with this proposal is to create sort of a "funnel" of involvement by the teams. The initial stage represents zero team involvement; anyone can start a repo. But to move onto the next stage, someone from the team has to sign off on it: that's your champion. But it only requires one team member. Later stages require the entire team.
So it's not so much "speak for itself" as it is making sure that, as things progress, they fit into the team's goals, while also not bogging every single step of the way down in requiring all team members to sign off immediately.
As far as finding a champion goes, I imagine this is a good use of the team email aliases; once you think you're ready to move from spitballing to designing, you email the team and ask for a champion. If nobody on the team wants to sign off, that's a good sign that this either doesn't fit into the roadmap, needs to wait until someone's time is freed up (which isn't inherently a bad thing; it's part of prioritization), or something else. Ideally this is actual feedback given to the RFC author.
That said, that's just what I'm imagining; you're right that this is an important aspect of the whole thing, and so we should have a good story sorted out here.
You may have looked at this already, but in case you haven’t and for the sake of others reading this who haven’t—
The “prior art” here in the TC39 world actually works really well. Some of them are indeed small, and it’s a slightly too-high level of overhead, but it shines for more complex language design issues—not least because you can open discussion issues for different parts of the design rather than it all having to occur in a single thread. A prime example of that is the discussion level on some of the pipe operator and partial application proposals there. There is a lot happening in those proposals, and they’d be nearly impossible to manage as a single thread; as a repo it’s much more comprehensible.
It also makes it much easier to tune your level of engagement with the process in my experience (all as an interested observer, to date).
I do agree we’ll need some tooling improvements to support the change, but I strongly recommend to anyone who hasn’t done it yet to spend some time looking at the various TC39 proposal repositories.
It may be more clear to present the "team decision points" as gates or transitions rather than phases. In terms of state machines, they act more like inputs or events than states. This also lends itself naturally to describing the "off ramps" mentioned by @chriskrycho and @lambda
This seems like a natural fit with the role of the champion, as I envision it. The champion already serves as a gatekeeper. One of their responsibilities should be to evaluate the acceptance criteria of the RFC, which includes cohesion. As the conduit between the RFC author(s) and the related Rust team, they would also be responsible for ensuring feedback related to the central vision finds its way back to the RFC.
This could be built in to the web interface proposed on the blog:
The repositories will live in the rust-rfcs organization. There will be a convenient webpage for creating them; it will create a repo that has an appropriate template and which is owned by the appropriate Rust team, with the creator also having full permissions.
I imagine there would also be a way to provide feedback via the main rfc repo PR via rfcbot.
I agree that this feels overly focused on the giant RFCs which receive hundreds or thousands of comments of discussion. Most RFCs don't fall into that category, but they do stick out in memory. RFC per repo sounds like a reasonable idea, but it sounds like it's mostly just an attempt to solve a need for threaded discussion to me. From the comments here it sounds like it hasn't worked out as well as we'd like on RFCs that have tried it in the past.
I don't like the forced spitballing phase, personally. I'd like to see it made more explicit how smaller RFCs would skip through that period. For the majority of the RFCs I've written, I've not felt a need to write a pre-RFC (2 of them would have benefitted from that phase for sure). This feels like it should be left up to the author, rather than requiring team approval to move past it.
I am really torn on the entire idea around champions in general. I very much understand the issue of prioritization and making sure there is bandwidth available. One thing that is not clear to me from this proposal is whether the champions (who seem like they're essentially just what shepherds are/were supposed to be today) are required to be on the teams implementing the feature or not. To put it another way, is the problem we're trying to solve related to bandwidth of reviewing RFCs or bandwidth of implementing?
My biggest concern with requiring a champion is that it unfairly benefits those who have close relations with folks on the teams in question. Arguably those folks are most likely to write RFCs that could be accepted, but this feels like an arbitrary barrier that we should not be introducing.
You touched several times on the need to grow the team, but I don't think it's been given enough attention. Honestly I think that is the main problem we should be focusing on, rather than the RFC process itself. The biggest issue I see is that there's no clear path to start taking on these sorts of roles beyond getting involved in the implementation.
As you know, I personally tried to get more involved recently -- And while it was clear I didn't have the bandwidth to fully understand how the trait system is implemented and meaningfully contribute there, I'm certainly still qualified to help review RFCs, discuss them with the lang team, etc (and likely have bandwidth in that regard). But there's no clear path for me to do so. I suspect there are many out there like me in that regard.
One thing which worries me in “one repo” approach is that it becomes harder to write an RFC review and read them in one place. Yes, blogpost leaves place for summary FCP comments from the team, but I don’t think that it’s enough. For example when I read RFCs I often just briefly read motivation and summary and after that skim through comments in search for opinions and highlights of RFC’s weak/strong points. This is why here I’ve proposed to limit top-level RFC comments by “review” format.
We could add “reviews” issue, but how should reviews be discussed then? We either make this issue review-only (then we can’t see replies to it) or thread again becomes filled with noisy discussions. Or every review can go to the separate issue marked by tag, but then it becomes harder to read them in sequence.
The RFC process, especially the early parts, is inherently chatty. GitHub excels at reviewing and pushing code, but that doesn't mean a discussion about code that might one day materialise ought to live there from day 1. I've suggested using Discourse to a larger extent in the Rust RFC process before. Since it received some quiet approval and no pushback I hope you don't mind if I bring it up again.
I really like the flow of the new process, but I'm not a fan of the separate repository per-RFC proposal. By committing hard to GitHub so early in the discussion process you're setting yourself up for a lot of busywork like keeping track of all these disparate repos via some mix of manual curation and GitHub API hacks. Good points have been raised about how this workflow is primarily designed for "epics" rather than minor RFCs that are adequately covered in a single topic of discussion.
Here's how I would structure the early parts of the RFC process in a way that scales from small to big RFCs with minimal friction:
#pre-rfc tag to casually discuss any idea you might have. Minimal formalities.Create an RFC in the #rfc category. (Any related pre-RFC should be closed with a link to this RFC).
Every RFC topic must have a descriptor tag attached to it, like #semicolons-as-items. Discourse allows you to require this.
Branching discussions about a complex RFC can easily be created by starting a linked topic.
These new topics will automatically have the appropriate tag attached to them.
For extremely complex/controversial RFCs you might consider creating a whole new sub-category for discussion, e.g. #rfc:rust-epochs.
All of this categorisation is completely fluid and won't create any breaking links when changed.
Discourse will cover this part of the discussion process:
The way I see it, keeping everything in one place is a pipe dream. Before an RFC comes into existence there will always be informal discussions happening that can live just about anywhere. Better to use different platforms best suited for the different stages of discussion.
I see the stages of an RFC more like:
#rfc category)During the early discussion-heavy phase you want to facilitate discussion as best you can, and Discourse does discussion much better than GitHub.
And the open source nature of Discourse allows you to better formalise the RFC process, complete with UI conveniences, e.g.
A lot of these are feature we'd be happy to sponsor.
This is the key point. The mechanisms on Discourse are far superior to what is available on GitHUB.
These are features GitHub sorely lacks. It's also been pointed out that github thread collapsing often hides valuable feedback.
As a thread gets long, it typically starts to fragment. This is where splitting and merging would be useful, but that requires active management of the thread. As far as I've seen, this is not typical of the internals forum. Wouldn't this increase the burden of the moderation team? Are there any tools provided by Discourse that would allow the "Pre-RFC" author to prune the threads?
One benefit of the one-repo-per-RFC approach is it groups all conversations related to a feature. The try-related discussions (try fn, throw, pass, and various bikesheds), for example, would all live under one roof. What features does discourse have to facilitate this that are available to any member? It doesn't appear, for example, that #hastags work unless a category is set up, and I doubt we'd like to add a category for each "umbrella" RFC.
As a proposal to change the process so much, it is lacking Prior Art other than TC39 (I get that this is a draft 
).
Most other languages do not use an entire new GitHub repository for a new proposal. TC39 is a rare exception. Some projects like Elixir, TypeScript and Kotlin don’t even have a formal process.
Swift uses Discourse as the RFC discussion medium, similar to the suggestion above.
apple/swift-evolution repository with a link to the Pitch.The D process is also interesting in how it can forces a summary to long discussions by timeboxing:
dlang/DIPs repository.(Note that unlike Swift or D, Rust has an additional stabilization phase which we need to account for.)
Ref: https://isocpp.org/std/submit-a-proposal
| Stage | Discussion medium |
|---|---|
| Float | Newsgroup at std-proposals
|
| Draft | Newsgroup at std-proposals
|
| Submit | Email to WG21 LWG |
| … | Face-to-face meeting |
Changes to language will be exclusively relying on face-to-face meeting.
Ref: https://github.com/dotnet/csharplang
| Stage | Discussion medium |
|---|---|
| Discussion | GitHub issue on dotnet/csharplang labeled Discussion
|
| Proposal | GitHub issue on dotnet/csharplang labeled Proposal champion
|
The remaining stages are driven primarily within the C# Language Design Meetings (LDM).
Ref: https://github.com/dlang/DIPs/blob/master/PROCEDURE.md
| Stage | Discussion medium |
|---|---|
| Point of Contact | - |
| Development Stage (optional) | - |
| Draft Review | GitHub PR to dlang/DIPs
|
| Community Review | Forum https://forum.dlang.org/group/general |
| Final Review | Forum |
| Formal Assessment | Private GitHub gist (final edits) |
Ref: https://github.com/tc39/ecma262/blob/master/CONTRIBUTING.md#new-feature-proposals
TC39 will hold a face-to-face meeting every 2 months to promote or demote a proposal’s stage.
| Stage | Discussion medium |
|---|---|
| 0 (Strawman) | Mailing list, IRC |
| 1 (Proposal) | New GitHub repository (can be hosted anywhere) |
| 2 (Draft) | (the repository above) |
| 3 (Candidate) | (the repository above) |
| 4 (Finished) | (the repository above) + GitHub PR to tc39/ecma262
|
Design doc are submited as Gerrit codereview mail to https://go.googlesource.com/proposal. A final discussion at end of design doc stage will decide whether to accept or reject the proposal.
| Stage | Discussion medium |
|---|---|
| Proposal | GitHub issue on golang/go
|
| Design doc | (same issue as above) |
Ref: https://github.com/ghc-proposals/ghc-proposals
| Stage | Discussion medium |
|---|---|
| Community discussion | GitHub PR to ghc-proposals/ghc-proposals
|
| Committee process | Mailing list ghc-steering-committee
|
Ref: https://jcp.org/en/resources/guide
No thanks.
Ref: https://www.python.org/dev/peps/pep-0001/
The core developers (“BDFL” and “PEP czars”) will approve, reject or defer a PEP at the end of a review.
| Stage | Discussion medium |
|---|---|
| Presentation | Mailing list python-ideas
|
| Draft | GitHub PR or direct commit to python/peps (acquire PEP number) |
| Discussion | (anywhere; authors are encouraged to set up SIG mailing list or wiki pages for large PEPs) |
| Review | Usually mailing list python-dev
|
An FCP vote will be raised directly in the GitHub PR/issue to let team members give consensus whether to advance to next stage.
| Stage | Discussion medium |
|---|---|
| 0 (Pre-RFC) (optional) | Discourse https://internals.rust-lang.org/ |
| 1 (RFC) | GitHub PR to rust-lang/rfcs
|
| 2 (Tracking) | GitHub issue on rust-lang/rust
|
| 3 (Stabilized) | (the issue above) |
The team champion will open GitHub PR on rust-rfcs/rfcs to update the index; FCP will be raised there.
| Stage | Discussion medium |
|---|---|
0 ( Spitballing) |
New GitHub repository under rust-rfcs/
|
| 1 (✎ Designing) | (the repository above?) |
2 ( Implementing) |
(the repository above?) + GitHub PR to rust-lang/rust
|
3 ( Evaluating) |
(the repository above?) |
4 ( Stabilized) |
(the repository above?) |
Ref: https://github.com/apple/swift-evolution/blob/master/process.md
| Stage | Discussion medium |
|---|---|
| Pitch | Discourse https://forums.swift.org/c/evolution/pitches |
| Proposal | GitHub PR to apple/swift-evolution (acquire SE number) |
| Review | Discourse https://forums.swift.org/c/evolution/proposal-reviews |
I think making the stages more explicit is probably a very good idea.
I still have qualms about this use of GitHub. I feel that even the old-school pipermail email archives that for example python-dev has largely provide a better interface to reading these kinds of technical discussions, due to:
GitHub issues (and even Discourse) really don’t do so well in these departments, and I’m skeptical that splitting up into many issues (in separate repos) will be the right improvement.