Refining RFCs part 3: Async decisions

[aturon]

@burntsushi

Despite our meetings being reactive, I've really enjoyed the weekly video meetings as a way to interact with other team members. I worry that without the forcing function of using the meetings to make decisions, we may lose them altogether.

@nrc

random thought - could we use IRC meetings? Or combination video conferencing/IRC?

I think it is good to keep in mind what sync meetings are good for:

• getting triage work done (particularly the compiler team seems to benefit from this)
• establishing a shared vision and building consensus
• getting unstuck on contentious points
• they're fun and good for morale (especially for those of us who don't see much of other contributors)

I absolutely think we should continue to have subteam meetings, likely on the same cadence as we currently do. My big goals here are (1) to focus those meetings more on proactive planning/brainstorming and (2) to make the decision-making process maximally transparent and efficient.

I agree with Nick's assessment on benefits of meetings, and I can definitely see using the meetings to talk through particular thorny "motion to FCP" situations. But I think the default should be for basic RFC business to take place on github etc.

Edit: oh, and re: IRC meetings -- I find them to be quite difficult and inefficient relative to video meetings. I would prefer to record/broadcast our meetings, and also to advertise a schedule of topics for meetings. In particular, if we're more focused on proactive discussion, I can imagine allocating a meeting to brainstorm one particular topic (e.g. an overhaul of rand), and we could open up that call to interested stakeholders as well.

[alexcrichton]

I like that the process you’ve proposed is relatively lightweight, and I think that we’ll want a form of “status page” for an RFC which shows which of the states in your “Process overview” an RFC is in. Especially for once a “motion to FCP” is made, I’d want a way to track any blocking objections and make sure they, for example, have a link to the comment resolving them or something like that.

I also like the fact that the subteam all manually signs off on a disposition before actually entering FCP, although it also seems like it’d be very hard for an RFC to reverse direction during the FCP once the entire subteam has signed off on the disposition of the RFC. For example, for each of the states:

• merge - the only way to prevent this would be a major objection brought up for the first time during the FCP, which does make sense to me when I think about it, but good to acknowledge!
• close - having any new comment convince the entire subteam to go towards merge instead during the FCP seems… quite unlikely!
• postpone - similar to close, there’d have to be a comment (not brought up before theoretically) which reverses the thoughts of the entire team.

In that sense I wonder if FCP largely only makes sense for the “merge” disposition? I guess it also doesn’t hurt for the others, so perhaps could work for all!

[ubsan]

I really dislike the idea of meetings in general, unless they are open to the public. It may be that we can make the meetings better, but I still have an awful taste in my mouth from the current meetings. As someone who is interested in the design of the rust language, the meetings as they stand feel like a way to shut out people; simply allowing people to watch isn’t enough. It’s still shutting people out.

[nikomatsakis]

@alexcrichton

having any new comment convince the entire subteam to go towards merge instead during the FCP seems... quite unlikely!

I agree it's unlikely that any comment will convince the entire subteam to go towards merge, but a comment could certainly convince people (even one member is enough, in principle...) that it makes sense to keep discussing. In any case, I've certainly seen it happen that everybody considers some problem to be a blocking problem, but that a key comment changes peoples minds, and hence leads to a broad swing in opinion. So TL;DR I think FCP makes sense all the time.

[nikomatsakis]

@aturon

Subteam members can post a comment with fcp: r- to lodge a formal (blocking) objection.

I'm not sure if we need to build this into the bot per se. I have been debating about what the best term is for a "blocking objection" -- I wanted something that doesn't seem too "strident", especially since I think frequently "blocking objections" are more like "points of clarification" -- that is, something that you would like to discuss, or get more information about, but which may be just fine.

One thought I have is that for the motion to proceed, everyone should agree. This can take one of two forms:

• you check the box and leave no comment;
• you check the box and leave a comment saying you agree with the motion.

In other words, there is no special language to lodge a "blocking objection" -- you can just raise a concern. The special language is to say that you agree with the motion.

Anyway, I'm just noodling about the details. I think the overall proposal is very good and would love to adopt it yesterday!

[nikomatsakis]

One other related thing: I think we should also have the ability to write f? @somebody-or-other and to have the bot place a “Feedback request” on that person’s personal dashboard. This feedback request would be cleared whenever somebody-or-other leaves a comment. I think this will be really important for helping to drive async discussion to consensus, since you can write things like:

“This seems like a good compromise to me! f? @nikomatsakis, do you agree?”

and now I have a good place that will bug me about responding to you in a timely fashion.

[aturon]

@nikomatsakis

In other words, there is no special language to lodge a "blocking objection" -- you can just raise a concern

I think the value in doing something more formal is twofold:

• It draws a more clear distinction between commentary on details versus, well, blocking objections. I think this clarity is useful for everyone, to know how/when things can proceed.

• We can also use it as a way of tracking some of the major issues. For example, I could imagine next to each person's checkbox a link to each "blocking objection" they've raised, and some means of marking them resolved.

That said, certainly r- is harsh, and we could go for something more like fcp: concern or some such.

[aturon]

@dikaiosune I’m wondering what you think of this most recent proposal – is this something you think could be implemented easily, together with the per-person and per-team dashboards?

[nikomatsakis]

So yeah I’ve come around on this point about whether one has to be explicit about blocking vs explicit about accepting. It does seem to me that we should default to consensus, since this will help avoid unnecessary delay and seems to mirror more closely what happens “in real life” when you have a meeting.

I definitely feel though that writing “r-” or “-1” in order to block consensus is not ideal. I’d rather we write something more neutral, just because I find personally that if I see a response that starts with “r-” with “-1” I will immediately be put in a defensive posture, no matter what words come after it. So I’d rather see some other indicator.

Personally I kind of like the idea of using a funny emoji, such as . Seems to set the right playful tone for me. But of course I’m open to anything.

(It may also be worth having some way to distinguish blockers where you feel strongly the design should be changed versus points that you would like to have clarified because the motion proceeds, but where you are unsure if the design is ok or not – but maybe not, since there is no big difference in practice.)

I do like of giving names to objections, because I find it helpful to have names as a shorthand in later discussion, but I’m not sure what’s a sufficiently lightweight way to do that. It could just be something that we try to do informally.

[anp]

I think it's quite feasible in the broad strokes. Some questions:

The tool then automatically posts a comment with a checklist of subteam members ([ ] aturon), with a check meaning "I have reviewed"

Is this then edited directly by team members to induce state changes? I'm not familiar with GitHub's setup for these checklists, but in the past I've only seen repo owners & issue openers able to edit them. Also, I think it'll be slightly easier to implement if state changes are triggered by new comments, like @rfcbot reviewed or something similar. Perhaps I'm misunderstanding the exact mechanic you're picturing here.

Subteam members can post a comment with fcp: r- to lodge a formal (blocking) objection.

Regardless of the exact invocation (re: @nikomatsakis's comment), are RFC PRs guaranteed to only be tagged with exactly one T-team label before FCP? If not, we'd need to establish some mechanic for letting the bot know which members need to sign off.

Also, should something like this plan to register specific objections? Like:

@rfcbot not-enough-bikeshed
I don't think we've sufficiently fleshed out the various tradeoffs of red vs. yellow paint here.

Which would then save the not-enough-bikeshed objection along with the remaining text of the comment as a more detailed explanation? It'd be simple enough to then do @rfcbot not-enough-bikeshed to remove the blocking objection. If doing so, would this act in lieu of any sort of r+/r- framework? Or would it be in addition to the overall consensus review?

Motion to FCP completes when all subteam members have reviewed and all blocking objections have been addressed.

How should a bot like the proposed one handle FCP's closing? A comment on the PR, pinging subteam members to merge?

Members can file vacations etc to not be included.

A specific workflow would need to be set up here. I'm quite glad that so far I haven't needed to setup logins or user-facing authentication for the dashboard, so if this can come from GitHub or somewhere else, it'd be ideal from my perspective.

I think we should also have the ability to write f? @somebody-or-other and to have the bot place a "Feedback request" on that person's personal dashboard.

This is definitely do-able, assuming it's limited to subteam members. If opened up to the community, it presents a UI complication for individual dashboards. Hopefully GitHub ping emails are enough to get non-subteam folks' attention?

[nikomatsakis]

@dikaiosune

Is this then edited directly by team members to induce state changes? I'm not familiar with GitHub's setup for these checklists, but in the past I've only seen repo owners & issue openers able to edit them.

I think that was the idea. I'm not sure quite how our ownership is setup, but it seems pretty plausible that all team members are repo owners?

@dikaiosune

I think it'll be slightly easier to implement if state changes are triggered by new comments, like @rfcbot reviewed or something similar. Perhaps I'm misunderstanding the exact mechanic you're picturing here.

This is definitely important to know Do edits (or checking of a checklist item, in particular) not trigger notifications? @rfcbot reviewed would probably also be ok, though it adds to the amount of work required.

@dikaiosune

Regardless of the exact invocation (re: @nikomatsakis's comment), are RFC PRs guaranteed to only be tagged with exactly one T-team label before FCP? If not, we'd need to establish some mechanic for letting the bot know which members need to sign off.

I think the answer is no -- I would expect that the set of people implicated is the union of all teams that are tagged. For example, if we have T-lang and T-libs, then a list is created consisting of people belonging either to T-lang or T-libs.

@dikaiosune

should something like this plan to register specific objections?

I think this is a bit unclear, but it is one of the ideas that was being floated.

@dikaiosune

How should a bot like the proposed one handle FCP's closing?

Note that once everyone has reviewed (e.g., left a checkmark, comment, whatever), that means that the final comment period begins, not ends. I sort of imagine that at this point it's out of the bots hands -- though it'd probably be nice to get a message once everyone has left a comment or a checkmark or whatever.

Seems like we should try to draw up a specific plan of action here I suspect there are a few slightly different variations of what the bot should do. I'll try to write a proposal in a separate comment.

@dikaiosune

This is definitely do-able, assuming it's limited to subteam members. If opened up to the community, it presents a UI complication for individual dashboards.

I'd strongly prefer if it worked for everyone. What is the UI complication exactly?

[nikomatsakis]

Oh, something I wanted to note (not related to the bot per se): I think that when someone makes a “move to [accept|close|postpone]”, they should always include a detailed summary of the prior conversation. It should be expected that people only have to read this summary to get a complete feeling for what has gone before (at least everything relevant to the current draft). People closely following the discussion can then add comments if they feel anything has been left out (but would be encouraged only to do so when it’s important, so as to keep the size of the thread under control).

[aturon]

@dikaiosune After chatting more with @nikomatsakis, here’s an updated proposal:

rfcbot

Commands

• @rfcbot: fcp [merge|close|postpone]

  • “Motion to fcp”
  • Puts the RFC on the dashboard of everyone in any tagged T-team, until they’ve “reviewed” the motion
  • Should be accompanied by detailed summary of discussion and rationale for disposition

• @rfcbot: fcp cancel

  • Cancel the motion to FCP, clearing it off dashboards

• @rfcbot: concern foo

  • Where “foo” is an arbitrary name for a blocking concern that needs to be addressed before moving to FCP
  • The name spans from after “concern” to the end of the line

• @rfcbox: resolved foo

  • Marks concern “foo” as being addressed

• @rfcbot: reviewed

  • Marks the RFC as having been reviewed by the subteam member
  • Everyone on the tagged subteams should leave such a comment; can come before or after any concerns

• @rfcbot: f? @user

  • Requests feedback from @user
  • Cleared when @user leaves a comment on thread

Bot actions

Optional functionality, but would be very nice:

• Once everyone has reviewed and there are no outstanding objections, post a comment saying that “Consensus reached to enter FCP for [merge|close|postpone]” with link back to the motion to FCP.

• Once a week has passed since entering FCP, if no new concerns have been registered, post a comment saying “Consensus reached to [merge|close|postpone]”

  • If concerns were raised, as soon as they are resolved (and it’s more than a week since FCP), post the comment

• Comment @user when it has been a week since a motion to FCP and @user has not yet reviewed

Dashboards

Ideally, there’s just one dashboard you go to, which is personalized for you.

At the top:

• Shows each RFC that still needs a reviewed comment from you (subteam members only)
• Shows any outstanding f? requests

Below:

• Shows all RFCs in motion to FCP, or actually in FCP, together with the status of all subteam members (reviewed, concerns w/) w/ links to originating comments
• Organized by subteam

[nikomatsakis]

Looks pretty good to me, modulo a few nits:

• I think it’d be nicer to write f? @so-and-so, like we write r? @so-and-so. Seems lighterweight than @rfcbot.
• I just want to make sure it’s clear that I think the RFC bot would be useful if one can use it also on rust issues. In particular, there are many times when we must reach a decision about a particular PR, tracking issue, or other thing, and I think the same mechanic would apply.

[aturon]

One more thing that would be great to show on the dashboard: “stale” RFCs (e.g., that haven’t gotten a comment in, say, more than 3 weeks). Not sure what the precise criteria should be, but we can pick something simple and iterate.

[anp]

@aturon @nikomatsakis this looks pretty doable.

What's the interaction between concerns and reviews? If a concern is raised and/or resolved, should all tagged subteam members need to re-review, on the assumption that solutions to a concern may raise new concerns for that member?

Should outstanding f? requests block FCP from starting or ending?

@nikomatsakis

I'd strongly prefer if it worked for everyone. What is the UI complication exactly?

Thinking about it more, I think there are a few ways to do it. I'm imaging that there needs to be some page which lists all of the individual and team pages, and I was thinking it might be challenging to efficiently generate a list of users with outstanding f? requests, and to present them nicely. But I'm thinking now that there should be few enough of these requests that it won't be problematic.

EDIT: And re: feasability, I'm going to be at a low level of productivity for the next 2ish weeks, just as a heads up. I'm assuming that any consensus reached on this process will go through an RFC?

[nikomatsakis]

@dikaiosune

I'm assuming that any consensus reached on this process will go through an RFC?

I don't think so. This is basically a question of how the subteams organize themselves, which is meant to evolve organically and independently.

[nikomatsakis]

@dikaiosune

I'm imaging that there needs to be some page which lists all of the individual and team pages

This seems like an ok thing to have, but I'm not sure I'd say it's necessary. =) I just imagined that you would go to rusty-dash.com/user/foo and you would get the page for foo. But it might be nice to have a master list of outstanding f? requests and other such things.

[retep998]

@aturon

Once a week has passed since entering FCP, if no new concerns have been registered, post a comment saying "Consensus reached to [merge|close|postpone]"

We should make sure it is at least a week since the "this week in Rust" update which features the FCP, to ensure that people not closely following the RFC have an opportunity to comment.

[nrc]

I think part of the goal of these changes is that the dashboard becomes the place people go to be notified, so I believe FCP should be at least a week on the dashboard, rather than at least a week on TWiR (might need some adjustment period while people learn that the dashboard exists).