Refining RFCs part 3: Async decisions

Yep, that makes sense.

(With apologies for going off-topic, but this reminds me) on a similar note, is it possible to disable the thumbs-down reaction on the github repo? Thumbs-up and so on are useful (like here) just to get a sense of how many other people "in the background" agree with a comment and to avoid "+1 to this" comment spam, but dogpiling people with thumbs-downs for suggesting an unpopular idea feels unnecessarily hostile and unproductive. If there are disagreements, they should be elaborated and discussed. (And instead of -1ing the original comment, people can +1 the comment that best reflects why they disagree.)

4 Likes

Of all the proposals, I think this is the one I feel most strongly about. In particular, I very much identify with the notion that our subteam meeting has been quite reactive, and that we’ve adopted a kind of shared set of values. The dashboard idea feels worthy of trying, especially if it makes it easy to leave targeted feedback. Some other thoughts:

  1. I don’t think I care too much about the specific vote/rating system we have, so long as there is a way to express “this has key issues that need to be addressed” in one way or another.
  2. I often don’t have strong opinions on the matter, or the expertise to properly evaluate something, so I think being able to abstain is important.
  3. I like @nrc’s suggestion of consensus with the team leader making a judgment calls.
  4. Despite our meetings being reactive, I’ve really enjoyed the weekly video meetings as a way to interact with other team members. I worry that without the forcing function of using the meetings to make decisions, we may lose them altogether.

Some more thoughts on async decision making:

  • The team dashboard seems like a really good tool for following what a team is doing. Currently there are a number of ways this is done (RFC repo, TWiR, sub-team reports, team meeting minutes, the proposed roadmaps) and consequently it is very different to track from the outside (at least I think so - e.g., I find it pretty impossible to keep track of what the libs team is doing) I.e., I think the dashboard could be useful for outsiders to track the teams’ work as well as for insiders to track their own work.
  • random thought - could we use IRC meetings? Or combination video conferencing/IRC? I believe the community team have experimented with this and it is also used by the CSS working groups. It gives a way to broaden attendance at meetings and improve transparency whilst still keeping some of the benefits of sync meetings.
  • I think it is good to keep in mind what sync meetings are good for:
    • getting triage work done (particularly the compiler team seems to benefit from this)
    • establishing a shared vision and building consensus
    • getting unstuck on contentious points
    • they’re fun and good for morale (especially for those of us who don’t see much of other contributors)

The main objection I have to this is that it's not a continuum between popular and unpopular - it's two separate measures:

  • How many people favor it strongly enough to mark it as such
  • How many people object strongly enough to mark it as such

Variance in the former and variance in the latter can be very unconnected - there may be:

  • Many people who go in each direction (Anyone remember when the tilde died?)
  • A few people who go one direction, many in the other
  • etc.

Some of these cases would become inexpressible - or at very least, easily misinterpreted. Consider if a few people favor an idea, but many dislike it. Having a quick summary of that sentiment balance on the first post of the thread is immensely useful. Without that, you have a "quick summary" of five thumbs-up... but someone reading the thread will find a lot of "me too" negative responses.

That seems both misleading (to the person trying to get a quick overview) and also even more negative to the person making the proposal (as now each negative response must escalate to a full comment).

I've been digesting feedback and having some offline discussions over the last week, and I'm going to respond to the comments in these threads over the next day-two.

I want to start with the mechanics of async decision-making. I think the original strawman I proposed was pretty flawed:

... etc. Interestingly, there's a push both toward making things a bit less formal/more fluid on the one hand, and toward highly fine-grained voting schemes on the other.

I agree strongly with the concern about "voting" not being in tune with our consensus process. After talking extensively with @wycats, I want to propose a revised scheme.

Dispositions

First, there's a notion of a disposition toward an RFC:

  • merge
  • close: reboot from Motivation, saying that we still want to solve the problem but this RFC isn't the desired route.
  • postpone: essentially, put the original Motivation on hold, due to an issue that arose during the RFC process. Should be rare.

Process overview

  • ... pre-RFC stages ...
  • RFC PR is posted
  • Discussion reaches a steady state (not necessarily consensus) and the RFC is up to date
  • A shepherd/subteam member submits a Motion to FCP, along with a disposition for the FCP
    • This is the time for formal objections from the subteam
    • No fixed timeline; complete once all members have read the RFC and don't object (see mechanics below)
  • FCP period
    • One week long
    • Discussion focused on any major new arguments around the disposition (not minor details)
    • Normal result: disposition becomes final
    • Before subteam member makes the decision final, they must check that all new arguments have been adequately addressed.

Mechanics

  • Motion to FCP is fcp: (merge|close|postpone)
  • The tool then automatically posts a comment with a checklist of subteam members ([ ] aturon), with a check meaning "I have reviewed"
  • Subteam members can post a comment with fcp: r- to lodge a formal (blocking) objection.
    • Usually this is cleared only on a new commit to the RFC.
  • Motion to FCP completes when all subteam members have reviewed and all blocking objections have been addressed.
    • Members can file vacations etc to not be included.

My thoughts

I think this revised proposal addresses several concerns.

  • Lightweight mechanics.
  • Supports consensus and encourages momentum toward a decision by avoiding any form of voting, and instead focusing on consent and blocking objections
  • Allows any shepherd to move things along through a motion to FCP.
  • Gives a clear signal to subteam members who might not have been following that particular RFC closely that now is the time to review for any blocking objections. Because this is done prior to FCP, the FCP period itself is expected to be relatively smooth sailing. (Unlike today, where often FCP is often where subteam members leave the most comments.)

What do you think?

2 Likes

Definitely! I imagine this replacing today's "subteam reports".

I absolutely think we should continue to have subteam meetings, likely on the same cadence as we currently do. My big goals here are (1) to focus those meetings more on proactive planning/brainstorming and (2) to make the decision-making process maximally transparent and efficient.

I agree with Nick's assessment on benefits of meetings, and I can definitely see using the meetings to talk through particular thorny "motion to FCP" situations. But I think the default should be for basic RFC business to take place on github etc.

Edit: oh, and re: IRC meetings -- I find them to be quite difficult and inefficient relative to video meetings. I would prefer to record/broadcast our meetings, and also to advertise a schedule of topics for meetings. In particular, if we're more focused on proactive discussion, I can imagine allocating a meeting to brainstorm one particular topic (e.g. an overhaul of rand), and we could open up that call to interested stakeholders as well.

1 Like

I like that the process you’ve proposed is relatively lightweight, and I think that we’ll want a form of “status page” for an RFC which shows which of the states in your “Process overview” an RFC is in. Especially for once a “motion to FCP” is made, I’d want a way to track any blocking objections and make sure they, for example, have a link to the comment resolving them or something like that.

I also like the fact that the subteam all manually signs off on a disposition before actually entering FCP, although it also seems like it’d be very hard for an RFC to reverse direction during the FCP once the entire subteam has signed off on the disposition of the RFC. For example, for each of the states:

  • merge - the only way to prevent this would be a major objection brought up for the first time during the FCP, which does make sense to me when I think about it, but good to acknowledge!
  • close - having any new comment convince the entire subteam to go towards merge instead during the FCP seems… quite unlikely!
  • postpone - similar to close, there’d have to be a comment (not brought up before theoretically) which reverses the thoughts of the entire team.

In that sense I wonder if FCP largely only makes sense for the “merge” disposition? I guess it also doesn’t hurt for the others, so perhaps could work for all!

I really dislike the idea of meetings in general, unless they are open to the public. It may be that we can make the meetings better, but I still have an awful taste in my mouth from the current meetings. As someone who is interested in the design of the rust language, the meetings as they stand feel like a way to shut out people; simply allowing people to watch isn’t enough. It’s still shutting people out.

I agree it's unlikely that any comment will convince the entire subteam to go towards merge, but a comment could certainly convince people (even one member is enough, in principle...) that it makes sense to keep discussing. In any case, I've certainly seen it happen that everybody considers some problem to be a blocking problem, but that a key comment changes peoples minds, and hence leads to a broad swing in opinion. So TL;DR I think FCP makes sense all the time.

I'm not sure if we need to build this into the bot per se. I have been debating about what the best term is for a "blocking objection" -- I wanted something that doesn't seem too "strident", especially since I think frequently "blocking objections" are more like "points of clarification" -- that is, something that you would like to discuss, or get more information about, but which may be just fine.

One thought I have is that for the motion to proceed, everyone should agree. This can take one of two forms:

  • you check the box and leave no comment;
  • you check the box and leave a comment saying you agree with the motion.

In other words, there is no special language to lodge a "blocking objection" -- you can just raise a concern. The special language is to say that you agree with the motion.

Anyway, I'm just noodling about the details. I think the overall proposal is very good and would love to adopt it yesterday!

One other related thing: I think we should also have the ability to write f? @somebody-or-other and to have the bot place a “Feedback request” on that person’s personal dashboard. This feedback request would be cleared whenever somebody-or-other leaves a comment. I think this will be really important for helping to drive async discussion to consensus, since you can write things like:

“This seems like a good compromise to me! f? @nikomatsakis, do you agree?”

and now I have a good place that will bug me about responding to you in a timely fashion.

1 Like

I think the value in doing something more formal is twofold:

  • It draws a more clear distinction between commentary on details versus, well, blocking objections. I think this clarity is useful for everyone, to know how/when things can proceed.

  • We can also use it as a way of tracking some of the major issues. For example, I could imagine next to each person's checkbox a link to each "blocking objection" they've raised, and some means of marking them resolved.

That said, certainly r- is harsh, and we could go for something more like fcp: concern or some such.

@dikaiosune I’m wondering what you think of this most recent proposal – is this something you think could be implemented easily, together with the per-person and per-team dashboards?

So yeah I’ve come around on this point about whether one has to be explicit about blocking vs explicit about accepting. It does seem to me that we should default to consensus, since this will help avoid unnecessary delay and seems to mirror more closely what happens “in real life” when you have a meeting.

I definitely feel though that writing “r-” or “-1” in order to block consensus is not ideal. I’d rather we write something more neutral, just because I find personally that if I see a response that starts with “r-” with “-1” I will immediately be put in a defensive posture, no matter what words come after it. So I’d rather see some other indicator.

Personally I kind of like the idea of using a funny emoji, such as :thinking:. Seems to set the right playful tone for me. But of course I’m open to anything.

(It may also be worth having some way to distinguish blockers where you feel strongly the design should be changed versus points that you would like to have clarified because the motion proceeds, but where you are unsure if the design is ok or not – but maybe not, since there is no big difference in practice.)

I do like of giving names to objections, because I find it helpful to have names as a shorthand in later discussion, but I’m not sure what’s a sufficiently lightweight way to do that. It could just be something that we try to do informally.

I think it's quite feasible in the broad strokes. Some questions:

The tool then automatically posts a comment with a checklist of subteam members ([ ] aturon), with a check meaning "I have reviewed"

Is this then edited directly by team members to induce state changes? I'm not familiar with GitHub's setup for these checklists, but in the past I've only seen repo owners & issue openers able to edit them. Also, I think it'll be slightly easier to implement if state changes are triggered by new comments, like @rfcbot reviewed or something similar. Perhaps I'm misunderstanding the exact mechanic you're picturing here.

Subteam members can post a comment with fcp: r- to lodge a formal (blocking) objection.

Regardless of the exact invocation (re: @nikomatsakis's comment), are RFC PRs guaranteed to only be tagged with exactly one T-team label before FCP? If not, we'd need to establish some mechanic for letting the bot know which members need to sign off.

Also, should something like this plan to register specific objections? Like:

@rfcbot not-enough-bikeshed
I don't think we've sufficiently fleshed out the various tradeoffs of red vs. yellow paint here.

Which would then save the not-enough-bikeshed objection along with the remaining text of the comment as a more detailed explanation? It'd be simple enough to then do @rfcbot not-enough-bikeshed to remove the blocking objection. If doing so, would this act in lieu of any sort of r+/r- framework? Or would it be in addition to the overall consensus review?

Motion to FCP completes when all subteam members have reviewed and all blocking objections have been addressed.

How should a bot like the proposed one handle FCP's closing? A comment on the PR, pinging subteam members to merge?

Members can file vacations etc to not be included.

A specific workflow would need to be set up here. I'm quite glad that so far I haven't needed to setup logins or user-facing authentication for the dashboard, so if this can come from GitHub or somewhere else, it'd be ideal from my perspective.

I think we should also have the ability to write f? @somebody-or-other and to have the bot place a "Feedback request" on that person's personal dashboard.

This is definitely do-able, assuming it's limited to subteam members. If opened up to the community, it presents a UI complication for individual dashboards. Hopefully GitHub ping emails are enough to get non-subteam folks' attention?

I think that was the idea. I'm not sure quite how our ownership is setup, but it seems pretty plausible that all team members are repo owners?

This is definitely important to know :slight_smile: Do edits (or checking of a checklist item, in particular) not trigger notifications? @rfcbot reviewed would probably also be ok, though it adds to the amount of work required.

I think the answer is no -- I would expect that the set of people implicated is the union of all teams that are tagged. For example, if we have T-lang and T-libs, then a list is created consisting of people belonging either to T-lang or T-libs.

I think this is a bit unclear, but it is one of the ideas that was being floated.

Note that once everyone has reviewed (e.g., left a checkmark, comment, whatever), that means that the final comment period begins, not ends. I sort of imagine that at this point it's out of the bots hands -- though it'd probably be nice to get a message once everyone has left a comment or a checkmark or whatever.

Seems like we should try to draw up a specific plan of action here :slight_smile: I suspect there are a few slightly different variations of what the bot should do. I'll try to write a proposal in a separate comment.

I'd strongly prefer if it worked for everyone. What is the UI complication exactly?

Oh, something I wanted to note (not related to the bot per se): I think that when someone makes a “move to [accept|close|postpone]”, they should always include a detailed summary of the prior conversation. It should be expected that people only have to read this summary to get a complete feeling for what has gone before (at least everything relevant to the current draft). People closely following the discussion can then add comments if they feel anything has been left out (but would be encouraged only to do so when it’s important, so as to keep the size of the thread under control).

@dikaiosune After chatting more with @nikomatsakis, here’s an updated proposal:

rfcbot

Commands

  • @rfcbot: fcp [merge|close|postpone]

    • “Motion to fcp”
    • Puts the RFC on the dashboard of everyone in any tagged T-team, until they’ve “reviewed” the motion
    • Should be accompanied by detailed summary of discussion and rationale for disposition
  • @rfcbot: fcp cancel

    • Cancel the motion to FCP, clearing it off dashboards
  • @rfcbot: concern foo

    • Where “foo” is an arbitrary name for a blocking concern that needs to be addressed before moving to FCP
    • The name spans from after “concern” to the end of the line
  • @rfcbox: resolved foo

    • Marks concern “foo” as being addressed
  • @rfcbot: reviewed

    • Marks the RFC as having been reviewed by the subteam member
    • Everyone on the tagged subteams should leave such a comment; can come before or after any concerns
  • @rfcbot: f? @user

    • Requests feedback from @user
    • Cleared when @user leaves a comment on thread

Bot actions

Optional functionality, but would be very nice:

  • Once everyone has reviewed and there are no outstanding objections, post a comment saying that “Consensus reached to enter FCP for [merge|close|postpone]” with link back to the motion to FCP.

  • Once a week has passed since entering FCP, if no new concerns have been registered, post a comment saying “Consensus reached to [merge|close|postpone]”

    • If concerns were raised, as soon as they are resolved (and it’s more than a week since FCP), post the comment
  • Comment @user when it has been a week since a motion to FCP and @user has not yet reviewed

Dashboards

Ideally, there’s just one dashboard you go to, which is personalized for you.

At the top:

  • Shows each RFC that still needs a reviewed comment from you (subteam members only)
  • Shows any outstanding f? requests

Below:

  • Shows all RFCs in motion to FCP, or actually in FCP, together with the status of all subteam members (reviewed, concerns w/) w/ links to originating comments
  • Organized by subteam
1 Like