Refining RFCs part 3: Async decisions

[aturon]

I’ve recently written a blog post enumerating several ideas for improving Rust’s RFC process. The post contains three separate (but interdependent) proposal sketches, which I’m eager for feedback and discussion on. I’m carving out each proposal as a standalone thread here on internals.

At the heart of Rust’s open development is the RFC process. Every major change to the language, compiler, core libraries, tooling, and policy go through an RFC writeup and consensus-building process. The process served us incredibly well in clarifying our technical direction on the road to 1.0, and has continued to be highly active since then, with on average about 2 RFCs merged every week.

But it’s not all roses. There’s been a growing sense among both Rust leadership and the broader community that the RFC process needs some further refinement as we continue to grow the community. I want to lay out my view of the problems and sketch some possible solutions, based on extensive discussion and brainstorming with many others on the team.

Each idea operates at a different scale (from big-picture to low-level mechanics), but they are intended to fit together into a whole; each one supports the others. Ultimately, these should result in a single RFC, but in the meantime I’ll start a discuss thread for each proposal.

There is a clear common theme to all of the problems I want to raise: communication. We need to find ways to better scale up lines of communication around the RFC process, and for Rust core development in general. There is also a cross-cutting concern: a need to increase our focus on mentoring and the path to team membership. @wycats has a great saying about measuring the health of the team structure:

Being a very active contributor who is not yet on a subteam should feel very close to actually being on that subteam.

Shooting for such a state of affairs has many benefits, not least of which is increasing the scalability of our community.

This thread is for the “Async decisions” part of the proposal.

---

Proposal: Async decisions

The problem

There is room for improvement around the way that the subteams themselves work. Today, subteams reach decisions on RFCs and other issues in (bi)weekly meetings. There are at least two problems with doing so. First, since the meetings have a limited duration, we often run out of time without finishing the active business, introducing delays; similarly, because of the high amount of RFC activity, the subteams often operate in “reactive” mode, more than actively leading.

Another issue is that meetings provide, in some sense, the “wrong defaults” for making decisions. We have to be careful to ensure that all the rationale for a decision is present in the online discussion thread, and that any new rationale that came up during a meeting means that the decision is delayed, to give the full community a chance to respond. The point is that, while we work hard to provide this transparency, it requires that extra work. At the same time, there is often good discussion in meetings wherein the subteam members build up a set of shared values – thereby missing the opportunity to argue for those values to the wider community. Finding a way to move decision-making to a more public, asynchronous system seems ideal, though meetings do have the benefit of providing a steady cadence to ensure that business is getting done.

The proposal

Idea: move away from video meetings for decision-making, instead reaching decisions entirely in the associated comment threads.

By moving the decision-making process fully online, we make it transparent by default. That is not to say that subteam members – or anyone else – will never have private conversation, of course. Just that this particular bit of business is better conducted online.

The key to making this work is automation. Right now, the meetings provide a convenient “forcing function” to ensure that decisions are being reached in a somewhat timely fashion. To ensure that we still make steady progress, we need a dashboard for every subteam member, showing them precisely what outstanding items they need to weigh in on – and that list needs to be kept manageably short.

We’ll need a dashboard tool that can pick up on special text from subteam members for:

• Calling an RFC/issue into FCP
  • “process: fcp”
• Approving/disapproving FCP
  • “process: fcp r+”
  • “process: fcp r-”
• Extending FCP
  • “process: fcp extend” (for one more week by default; possibly give parameter?)
• Approving stabilization/RFC merging
  • “process: r+” (ideally followed up by some commentary)
• Weakly objecting
  • Just leave a comment, followed by a “process: r+” once you are satisfied that the objection is addressed or that it’s OK not to address it.
• Strongly objecting (i.e. blocking acceptance)
  • “process: r-” (followed up with objection)
• Abstaining (possibly?)
  • “process: ack”

The dashboard tool would track the current status of RFCs/issues facing a decision, and would track the various timelines involved, e.g. that RFC FCP lasts for one week.

We can and should continue to hold video subteam meetings (they’re high bandwidth!), but for more forward-looking purposes: discussing specific early-stage RFCs, brainstorming, and prioritization. We can explore recording these meetings, and potentially opening them up to additional stakeholders who are not part of the subteam.

Details to be worked out:

• A plausible story for automation that retains the consensus process and is likely to keep things moving.
• Can the automation itself be responsible for moving to FCP/merging? Or at least provide a pushbutton way for doing so?

[AndrewBrinker]

This actually dovetails nicely with what I described in my comment on part 2. It may be that instead of RFC shepherds manually updating the RFC tracker, it could integrate with the tool described here, and be automatically updated.

Is there a system for writing bots for Discourse? Integrating with this forum as well as with GitHub would make the system work a lot more smoothly in the early stages, I imagine.

[ubsan]

This also seems really great. There’s a huge point of contention with me at least, because it feels like there’s discussion that happens outside the RFC process, and it’s not obvious why something is accepted or not, and this would make the process both more transparent, and more efficient. (and it’s not like the current process is very good at keeping RFCs from falling through the cracks anyways).

[brson]

I’m all for this.

Making the final call literally a vote (approving stabilization via ‘r+’) kinda makes me nervous. While it does reflect how Rust decision making effectively works today, it’s a big thing to encode into the process. Not something we can likely roll back.

I don’t see it in your description, but I imagine that the T-$team tags indicate to the bot which teams to expect responses from.

There will be times where team members don’t respond. What happens then? Does the process continue without their vote?

Offhand I can’t recall the difference between “approving/disapproving FCP” and “approving stabilization/RFC merging”. Isn’t FCP the last thing that happens before something is ‘done’. So e.g. doesn’t ‘fcp r+’ mean that an RFC is accepted? If so, what does plain ‘r+’ mean on an RFC PR? Oh, maybe “approving FCP” means the team has to vote publicly just to enter FCP?

https://github.com/dikaiosune/rust-dashboard/issues/61 is about some of the automation necessary.

[nrc]

Making the final call literally a vote (approving stabilization via ‘r+’) kinda makes me nervous

Me too! Could we retain the consensus requirement and allow people to change votes? Maybe with a possible override by the team leader if there is a majority but not consensus (in practice we seem to always reach consensus amicably).

[djc]

Sounds great. I’m confused about the meetings anyway, as it seems most of the teams have stopped adding their meeting minutes to the meeting-minutes repo? If the community/core wants to enable more people to contribute, doing more async is among the most important things.

[rkjnsn]

My (quite possibly incorrect) reading was that the idea was to require consensus: one could r+, abstain, or r-. An r- would block acceptance until the person thought their concerns were addressed enough to change to abstain or r+. Only when everyone on the relevant team had r+'d or abstained would the RFC be merged.

[nikomatsakis]

Making the final call literally a vote (approving stabilization via ‘r+’) kinda makes me nervous. While it does reflect how Rust decision making effectively works today, it’s a big thing to encode into the process. Not something we can likely roll back.

My assumption has been that this is not a vote, but rather just a way to survey whether consensus is achieved. The r+ or r- was a way to summarize the severity of your comments: is this a concern that must be addressed before you could accept the RFC? That’s r-. Is it a concern that could be addressed but you’d be willing to let it go forward? That’s r+.

Perhaps we want something a bit less…confrontational sounding than r-. Something that is more like “raising a concern” than “denying the RFC”. I recall that in bugzilla some people never give r- on reviews, they just change from r? to f+ (feedback granted), for similar reasons.

One simple thought is just to have r+ and ack – if you respond fcp ack, it sort of implies that you have a blocking concern. If you want to abstain, you write fcp r+ (abstain) or something like that.

There will be times where team members don’t respond. What happens then? Does the process continue without their vote?

I think that after some amount of time we just move on, if there is consensus from a suitable quorum. People who persistently don’t respond should perhaps be given the opportunity tove to “alumni” status (which basically means that they will not be pinged or expected to respond, but they can rejoin as an active member when they are able, if they choose).

[nikomatsakis]

Something else I would like from such a bot (or maybe another), is a way to request feedback from a specific team or from a specific person. Something like f? @aturon, which would then put this RFC onto aturon’s dashboard until he leaves another comment (which would be assumed to address my question).

I think this is important – I can imagine people for example wanting to ask whether a specific proposed solution addresses a concern that was raised by someone else, and I’d like a way to indicate “we are blocked from proceeding until you respond!”

[nikomatsakis]

Calling an RFC/issue into FCP

Ah, this reminds me. I think that part of going into FCP ought to be providing a good summary of the thread – of the sort we typically provide after a contentious process. The idea should be that once FCP begins, you have no need to read the rest of the thread – it’s all summarized for you (and of course if there are disputes with the summary, they can be covered below). This seems like it would be helpful to everyone involved. =)

[sfackler]

Perhaps we want something a bit less…confrontational sounding than r-. Something that is more like “raising a concern” than “denying the RFC”. I recall that in bugzilla some people never give r- on reviews, they just change from r? to f+ (feedback granted), for similar reasons.

Gerrit has a slightly larger scale that I think can be helpful for situations like these -

• +2 - Merge
• +1 - LGTM, but deferring to someone else to make the call to merge
• 0 - Nothing
• -1 - I have concerns, but not a hard blocker
• -2 - This may not merge

In my experience at least in a code review setting, the full range of ratings tends to be used. People will -1 a changeset after reviewing it if they want tweaks. Things tend to not merge with -1’s, but that’s because the -1’s turn into +1’s or +2’s after the requested changes are made. -2’s are more rare, and can be used for “this has fundamental problems” but is more frequently a scheduling tool to, for example, block a change from merging until a release is cut or something like that.

It’s not an exact equivalence in this kind of setting but I think it might be helpful in the same way.

[nikomatsakis]

Internally to the lang team we have a spreadsheet that we use for tracking the current state of people’s thinking that is very similar:

• +1 – yes!
• +0.5 – ok, if other think so
• 0 – abstain
• -0.5 – slight objections
• -1 – hard stop.

[nikomatsakis]

But while the range of options makes sense, I’m still wondering if we can find a “friendlier” phrasing. Maybe it doesn’t matter, it’s all about the text you write alongside, but I just think that people can get a “slap in the face” reaction if the first thing they see is “-1” instead of “tweaks requested”.

[sfackler]

Yep, that makes sense.

[glaebhoerl]

Maybe it doesn’t matter, it’s all about the text you write alongside, but I just think that people can get a “slap in the face” reaction if the first thing they see is “-1” instead of “tweaks requested”.

(With apologies for going off-topic, but this reminds me) on a similar note, is it possible to disable the thumbs-down reaction on the github repo? Thumbs-up and so on are useful (like here) just to get a sense of how many other people “in the background” agree with a comment and to avoid “+1 to this” comment spam, but dogpiling people with thumbs-downs for suggesting an unpopular idea feels unnecessarily hostile and unproductive. If there are disagreements, they should be elaborated and discussed. (And instead of -1ing the original comment, people can +1 the comment that best reflects why they disagree.)

[burntsushi]

Of all the proposals, I think this is the one I feel most strongly about. In particular, I very much identify with the notion that our subteam meeting has been quite reactive, and that we’ve adopted a kind of shared set of values. The dashboard idea feels worthy of trying, especially if it makes it easy to leave targeted feedback. Some other thoughts:

1. I don’t think I care too much about the specific vote/rating system we have, so long as there is a way to express “this has key issues that need to be addressed” in one way or another.
2. I often don’t have strong opinions on the matter, or the expertise to properly evaluate something, so I think being able to abstain is important.
3. I like @nrc’s suggestion of consensus with the team leader making a judgment calls.
4. Despite our meetings being reactive, I’ve really enjoyed the weekly video meetings as a way to interact with other team members. I worry that without the forcing function of using the meetings to make decisions, we may lose them altogether.

[nrc]

Some more thoughts on async decision making:

• The team dashboard seems like a really good tool for following what a team is doing. Currently there are a number of ways this is done (RFC repo, TWiR, sub-team reports, team meeting minutes, the proposed roadmaps) and consequently it is very different to track from the outside (at least I think so - e.g., I find it pretty impossible to keep track of what the libs team is doing) I.e., I think the dashboard could be useful for outsiders to track the teams’ work as well as for insiders to track their own work.
• random thought - could we use IRC meetings? Or combination video conferencing/IRC? I believe the community team have experimented with this and it is also used by the CSS working groups. It gives a way to broaden attendance at meetings and improve transparency whilst still keeping some of the benefits of sync meetings.
• I think it is good to keep in mind what sync meetings are good for:
  • getting triage work done (particularly the compiler team seems to benefit from this)
  • establishing a shared vision and building consensus
  • getting unstuck on contentious points
  • they’re fun and good for morale (especially for those of us who don’t see much of other contributors)

[eternaleye]

[I]s it possible to disable the thumbs-down reaction on the github repo? Thumbs-up and so on are useful (like here) just to get a sense of how many other people “in the background” agree with a comment and to avoid “+1 to this” comment spam, but dogpiling people with thumbs-downs for suggesting an unpopular idea feels unnecessarily hostile and unproductive.

The main objection I have to this is that it’s not a continuum between popular and unpopular - it’s two separate measures:

• How many people favor it strongly enough to mark it as such
• How many people object strongly enough to mark it as such

Variance in the former and variance in the latter can be very unconnected - there may be:

• Many people who go in each direction (Anyone remember when the tilde died?)
• A few people who go one direction, many in the other
• etc.

Some of these cases would become inexpressible - or at very least, easily misinterpreted. Consider if a few people favor an idea, but many dislike it. Having a quick summary of that sentiment balance on the first post of the thread is immensely useful. Without that, you have a “quick summary” of five thumbs-up… but someone reading the thread will find a lot of “me too” negative responses.

That seems both misleading (to the person trying to get a quick overview) and also even more negative to the person making the proposal (as now each negative response must escalate to a full comment).

[aturon]

I’ve been digesting feedback and having some offline discussions over the last week, and I’m going to respond to the comments in these threads over the next day-two.

I want to start with the mechanics of async decision-making. I think the original strawman I proposed was pretty flawed:

Making the final call literally a vote (approving stabilization via ‘r+’) kinda makes me nervous.

There will be times where team members don’t respond. What happens then? Does the process continue without their vote?

Could we retain the consensus requirement and allow people to change votes?

One simple thought is just to have r+ and ack – if you respond fcp ack, it sort of implies that you have a blocking concern. If you want to abstain, you write fcp r+ (abstain) or something like that.

… etc. Interestingly, there’s a push both toward making things a bit less formal/more fluid on the one hand, and toward highly fine-grained voting schemes on the other.

I agree strongly with the concern about “voting” not being in tune with our consensus process. After talking extensively with @wycats, I want to propose a revised scheme.

Dispositions

First, there’s a notion of a disposition toward an RFC:

• merge
• close: reboot from Motivation, saying that we still want to solve the problem but this RFC isn’t the desired route.
• postpone: essentially, put the original Motivation on hold, due to an issue that arose during the RFC process. Should be rare.

Process overview

• … pre-RFC stages …
• RFC PR is posted
• Discussion reaches a steady state (not necessarily consensus) and the RFC is up to date
• A shepherd/subteam member submits a Motion to FCP, along with a disposition for the FCP
  • This is the time for formal objections from the subteam
  • No fixed timeline; complete once all members have read the RFC and don’t object (see mechanics below)
• FCP period
  • One week long
  • Discussion focused on any major new arguments around the disposition (not minor details)
  • Normal result: disposition becomes final
  • Before subteam member makes the decision final, they must check that all new arguments have been adequately addressed.

Mechanics

• Motion to FCP is fcp: (merge|close|postpone)
• The tool then automatically posts a comment with a checklist of subteam members ([ ] aturon), with a check meaning “I have reviewed”
• Subteam members can post a comment with fcp: r- to lodge a formal (blocking) objection.
  • Usually this is cleared only on a new commit to the RFC.
• Motion to FCP completes when all subteam members have reviewed and all blocking objections have been addressed.
  • Members can file vacations etc to not be included.

My thoughts

I think this revised proposal addresses several concerns.

• Lightweight mechanics.
• Supports consensus and encourages momentum toward a decision by avoiding any form of voting, and instead focusing on consent and blocking objections
• Allows any shepherd to move things along through a motion to FCP.
• Gives a clear signal to subteam members who might not have been following that particular RFC closely that now is the time to review for any blocking objections. Because this is done prior to FCP, the FCP period itself is expected to be relatively smooth sailing. (Unlike today, where often FCP is often where subteam members leave the most comments.)

What do you think?

[aturon]

I think the dashboard could be useful for outsiders to track the teams’ work as well as for insiders to track their own work.

Definitely! I imagine this replacing today’s “subteam reports”.