pre-RFC: RFC "Deliberation period"

internals policy
1

Summary

When an RFC is posted, we start with a “deliberation period” or “quiet reading time” about the length of FCP (10 days) during which no public comments are posted on the RFC itself. Everyone gets a bit time to read the RFC and send their personal thoughts to either the RFC author or someone in a moderator/shepherd role. At the end of the deliberation period, the RFC author/moderator/shepherd posts all the comments they received along with who made them, and a summary of the themes present in the comments. Then discussion opens as usual.

Motivation

The Rust community is growing, and the RFC process is starting to feel some growing pains. It’s becoming more and more difficult for people to get involved in the deliberation part of the RFC process without feeling overwhelmed with content. We’re missing out on feedback from folks who look at a recently posted RFC, see that it already has 50, 100, 200 comments, and close the tab (I have done this). The people who do comment have often not read all of the other comments, which leads to repetitive discussions that still expect a response but that all the people who are following the discussion have to experience over and over. A too-large number of comments on RFC threads are discussions that spin out of control and off topic. This may even be discouraging folks from submitting RFCs.

I’d like us to work on ways that we can get more feedback from more people without everyone getting overwhelmed. My proposal here is but one possible idea; this is at very early stages in my thought process and I’d like to see what others think.

Guide-level explanation

In order to explain the concepts of my idea, I would like to present it in the context of a thought experiment where we are all physically in the same room. This will hopefully keep us (myself included) focused on the concepts rather than the concrete implementation in GitHub or a custom web app, and which dropdowns/radio buttons/textareas go where, etc. I think it’s worthwhile to decide if this would be a good direction to try going in before we invest effort in thinking about how we’d implement it.

Ok, so we’re all in a room-- Rust team members, Rust contributors, Rust users, any interested party. The RFC author brings in a copy of the RFC for each person and hands them out. Then we start the deliberation period. The room is quiet so that we can each concentrate on reading the RFC for ourselves. Each person has some number of Post-it® Notes on which they can write any questions or comments they have while reading. In the front of the room where everyone can see, there are these prompts for people to think about:

  • What is your general opinion of this RFC and why? (in favor, agree on the problem but not the proposed solution, disagree with the problem premise, not sure, etc)
  • What use cases aren’t addressed in the text of this RFC?
  • What parts of this RFC are confusing and why?
  • More prompts along these lines, might vary per-RFC depending on what kinds of feedback the RFC author is most interested in

We can’t prevent people from leaving the quiet room and discussing the RFC amongst themselves in another venue, but people who walk into this room will be presented with the text of the RFC, blank sticky notes, and quiet. (If my metaphor is falling apart: we clearly can’t stop people from talking about an RFC on other forums like internals, users, r/rust, IRC, Twitter, email, etc. However, if we turn off comments on the RFC during this time, anyone who visits the RFC during the deliberation period will only feel like they need to read the RFC in order to participate, not the RFC + the comments made so far).

When each person is done reading the RFC and writing up their comments, they put their name on their notes and hand the notes to the summarizer. The role of the summarizer could be played by the RFC author, an RFC shepherd, or another volunteer. The summarizer reads all the comments and groups them into topic areas. If the summarizer has clarification questions about anyone’s comments, they can have a quiet, private discussion until they both are satisfied the comments are clear. The summarizer does have to experience the potential repetition of the comments, unfortunately.

At the end of the deliberation period, all notes are posted on a wall for anyone to read to continue to ensure transparency. They’re organized into groups of similar comments that the summarizer has created. The summarizer also posts a short description of each group of comments along with how many comments are in the group and who made the comments in the group. Counting the number of similar responses, while repetitive, will actually give us a sense of how many people hold which views, that we don’t get today since some people DON’T want to repeat what has already been said. Anyone can choose to read all the comments or just the summaries of the groups, thus eliminating the repetition of reading many similar comments unless you so choose to do so.

At this point, normal RFC discussion resumes as it proceeds today. If anyone feels like their comments weren’t represented accurately, they can provide more clarification. People can ask each other questions about their comments. The RFC author can respond via comments or via RFC modifications.

Reference-level explanation

Deliberately omitted at this stage; I would expect to flesh this out into actual implementation details if people like the ideas.

Drawbacks

  • There will still be a lot of repetition for the summarizer to read through, potentially more if people can’t see each others’ comments. This proposal doesn’t eliminate the repetition completely, but it does eliminate the experience of reading the same comments and discussions over and over (except for the summarizer) unless you decide to opt in to reading the full comments.
  • Summarizing is a lot of work. People sometimes provide summaries today, but as comments that also get lost in the sea of comments. We’re also not consistent with how often or whether there are people to summarize on every RFC.
  • Still might not encourage the kind of feedback we’re missing out on today
  • This may actually, and will definitely feel like it, slow down the progress of an RFC. This may not entirely be a bad thing!

Rationale and alternatives

I think the proposal here will take away some of the overwhelming aspects of participating in the RFC process that exist today without unduly hampering the benefits we get from the RFC comment process, which include:

  • Getting community buy-in before implementing an idea
  • Discovering missing use cases
  • Discovering new solutions that are better than the proposed solutions

Alternative: better summarizing of discussions/acknowledgement of points/answering of questions on a regular basis in the RFC PR description

Alternative suggested by natboehm:

  • Have multiple deliberation periods: one round of comment submission, then one comment summary, then another round of comment submission, then another comment summary.
    • Could make it easier to be inspired by others’ comments, have a discussion but slowly
    • If these are 10 days each, could make more opportunities for people to have time to get involved

Optional addition or orthogonal thing:

  • Today, RFC authors solicit feedback from people who represent important stakeholders and who might not participate organically. This process isn’t advertised very much, however. This could happen during the deliberation period and be explicitly called out in the summary. Example: “[RFC author] asked wycats, as a production Rust user, how this proposal would affect their workflow. wycats had this to say: […]”

I would like to hear about more alternatives!

Unresolved questions

  • Should the RFC author be allowed to revise the RFC text in response to questions/comments during the deliberation period, or should all modifications be held until the end of the initial period?
  • Should the summarizer be the RFC author? Or deliberately not the RFC author?
  • Is 10 days enough time for people who do Rust work as their job and people who do Rust work in their spare time to read and submit their comments?
  • When we open up the RFC for comments, will the discourse be worse than it is today?
7 Likes
2

I'd say the deliberation period would be a great time to use the received feedback to make clarifying changes to the RFC. From the authors perspective, it would turn it into a "clarification time".

For substantial changes I would find it easier to follow along if those were treated as a new version of the RFC. An author might even want to trigger a new deliberation/clarification period when something important has changed. To me it would mostly help by reducing the need to reread RFCs a lot.

2 Likes
3

It does indeed feel like that comments are an inefficient way to discuss complex problems like those that RFCs tend to the serve.

I think these variations could be considered:

  1. Perform the summarization incrementally: update the summary every time a comment comes in, rather than doing it all at once. This avoids the stark distinction before “pre-summary” and “post-summary” and thus allows to send comments at any time; however, it might take more time compared to doing it all at once.
  2. Perform the summarization asynchronously: comments are posted as normal, and when the summarizer has time, they fold their content into the summary and “archive” the comment (the “archiving” means the comment is hidden by default - not sure how to do that in Discourse exactly). This allows the comment information to be available immediately, even if it’s in a less easily understandable form; however, it might make reading more burdensome.
  3. Crowdsource the summarization: let people just edit a wiki summary instead of commenting. This does rely on the good faith of random anonymous people, but Wikipedia seems to work fine. If necessary, one could require the wiki changes to pass moderator review instead of being immediately applied, and other mitigating mechanisms could be conceived. It may be necessary to implement a mechanism that highlights changes to the wiki summary since the last time the user has viewed it.
  4. Rather than focusing the summary to be a “grouping” of comments with little or no editing (which seems to be what is proposed here, although it’s not very clear), make it more of a treatise on the problem the RFC is trying to solve and its solution (starting with the RFC text itself) informed by comments, which would imply a much heavier editing of the comments
1 Like
4

:-1:

IME, the original text of an RFC is often only a starting point for a discussion to hammer out the ideal design. Preventing interaction between comments, as this proposal would do, would make RFCs more of a “take it or leave it” scenario, or at least put all the burden onto the author to come up with a better proposal, rather than letting multiple other participants collaborate. Although there are downsides to excessive “design by committee” and bikeshedding, this goes too far in the opposite direction by discouraging all collaboration.

I for one would be highly disinclined to participate under these conditions.

From a personal perspective, I’d also worry about wasting time writing a critique that (unbeknownst to me) someone had already made more thoroughly; yet if I didn’t write it, I’d run the risk of letting the deliberation period end without anyone addressing the issue, potentially causing it to be ignored.

9 Likes
5

To continue in my low-tech metaphor, I'm imagining a group of post-it notes covered by a sheet of paper. On the sheet of paper is a summary of the sentiment expressed by the post-it notes, which may or may not include direct quotations of some notes at the discretion of the summarizer. Anyone can choose to just read the summary and move on, or lift the paper and read all the notes. Does that clear things up?

6

Moderators should probably lock this topic until August 13 :wink:

14 Likes
7

This just introduces a new phase prior to the current open comment period, would you stop participating in the open comment period because there was a quiet period before it opened?

2 Likes
8

I considered that, but this is a pre-RFC :wink: If this goes into an actual RFC, I will propose trying it out as part of the RFC process on this one :slight_smile:

9

I agree with the sentiment that the current process can be a little overwhelming and ineffective for some (but not all) RFCs.

My main question about this proposal is: where do people send their comments and how are they tracked? Reading, compiling and summarizing the comments is a lot of work. I’m sure most RFC authors wouldn’t care for a barrage of comments in their personal email inbox (just an example). For this process to work, it almost sounds like we need some kind of RFC management software.

10

I definitely have concerns about the implementation of this idea as well, but for now:

11

I guess in practice I'd ignore the quiet period and wait for the open period to start before even reading the RFC (since if I read it immediately, I'd probably forget what I was going to say about it by then, and/or lose interest). This would reduce my ability to influence the course of the discussion, but I suppose that's on me...

12

I don't see how this proposed change solves the problem it tries to remedy.

The problem

The people who do comment have often not read all of the other comments, which leads to repetition.

The proposed solution

At the end of the deliberation period, the RFC author/moderator/shepherd posts all the comments they received...

The first drawback

There will still be a lot of repetition for someone to read through, potentially more if people can’t see each others’ comments

13

Ah yes, I should clarify. In the comments currently, discussions among people often happen over and over on the same topics but involving different people. Not only does the author have to read these, but everyone following the RFC has to read these. And someone has to respond because there's an expectation that you should feel heard.

With the proposal, at least only one person (the summarizer) would have to read this repetition. And we'd actually get potentially more information this way-- right now, there are likely people who DO read the comments before posting that then don't comment at all because their point has already been expressed-- so therefore we don't know how many people have a particular use case, for example, or how many people are generally in favor of an RFC.

So you are correct in that this proposal does not eliminate repetition, but it changes how we experience it.

14

A subtlety in the metaphor is that comments are expected to fit on 3 inch square (7.62cm) pieces of paper -- so not take a lot of time :slight_smile:

15

This is definitely an interesting idea!

I’m concerned that this will just exacerbate the burden on our RFC authors and/or shepherds, that latter of which are generally pretty sketchy (I say as someone who has been shepherded and been a shepherd for several RFCs).

Like, I can imagine an RFC never leaving deliberation because no one has the time/energy to do this work.

3 Likes
16

I like the general idea, but starting with a quiet period is hardly optimal, for reasons that have already been said.

What I’d suggest is, start with comments open like today, an “opening salvo” if you will. When one of predetermined conditions occur (e.g. few days elapsed; 100 comments landed; people started trying to murder each other with words; etc.), comments are locked and deliberation period is started. The author and moderators work together to digest and summarize comments, and make preliminary adjustments to the RFC. When the summary and adjustments are done, only then will the “hidden comments” be enabled for a period of time, with everyone getting the chance to comment on the state of affairs without engaging with each other for a time. After this hidden comment period, summarize again and start another discussion salvo.

1 Like
17

There are some general concerns I have with the feature, and I also see some areas of improvement. Let me start with the areas of improvement:

  • I think there are clearly RFCs where this would be more red tape and less useful. Some RFCs simply don’t have as much stuff to comment on, due to their nature, or because there is simply no controversy on the topic. I think if its going to be implemented, then it should be only for a minority of RFCs that are expected to get lots of comments. Its really the top percentile we are talking about here.
  • Holding back questions about how to interpret the text (either because someone didn’t understand it, or because the text can be interpreted in various ways) is not really good IMO. I have experienced this multiple times, and sometimes my opinion on the RFC depends on my question so I can’t give a statement before knowing more.

More general criticism:

  • I’ve had a bad experience with summaries so far. I’ve seen fundamental criticism of the RFC being totally dropped by the summary, or only being mentioned very shortly while the summary talked in great lengths about possible changes to the RFC.
  • Even if we assume we could make summaries fair, I think its not good for ideas for improvement to be made available to the community only in a filtered form. Sometimes an idea is not good by itself, but someone reads it and takes it as inspiration to improve it so much that it becomes worthwhile for consideration. This might lead to worse concepts chosen.
  • If people feel critical about an RFC, but don’t see any :-1: comments, they might be afraid of voicing criticism of their own, especially if an RFC is authored by “official” people. This gives an unfair advantage to those RFCs and adds to the advantages that people with these positions hold.
  • This seems to me like a “tell me your gut feeling” game, and I think we should rather strive towards a discussion of informed opinions.

Suggestion

As suggestion for a better system, we could keep the main comment system open, but add a mechanism where people who feel overwhelmed by reading through the discussion of an RFC can give feedback (or ask questions!). Such comments would then be read by people familiar with the discussion and if they add to the discussion without duplicating something, they would be incorporated into the discussion.

1 Like
18

As someone who does a similar thing with large groups, I think it is a good idea! I have found that it bubbles up the main ideas and allows everyone to have a voice

1 Like
19

I think that the linear format of the discussion forum is a major problem: all sub topics get mixed together and it can become overwhelming to read the whole lot of it.

A tree based structure could more easily accommodate this. Each concern could be the head of a new branch. We could adopt a culture of keeping each sub topic limited to one concern and an author would open one sub topic (branch) for each of his/her concerns, or comment in an already existing sub topic.

13 Likes
20

I see the problem. I’m not convinced about the solution.

If somebody would read, sum up and collapse 100s of comments into few points, that would be ideal. Currently that’s sort-of done by restarting the RFC process with a new/updated proposal. Perhaps the RFC author should have power to hide comments that they think are addressed in the (updated) RFC?

Threading does help. I’ve survived the worst of W3C mailing lists by using an e-mail client that can hide all current and future replies to a sub-thread. Contentious topics create mega-threads, which are then easy to ignore (but tree-like threading is important for this. simplified top-level + 1 sub-level threading is not precise enough).

1 Like