pre-RFC: RFC "Deliberation period"

What's troubling to me is the way it seems normal to equate "a few days" with "100 comments." 100 comments is a lot of comments! Its impossible to keep up with an RFC that gets 140 comments in 2 days, as the extern crate RFC had. We desperately need this to process to become less noisy.

That doesn't mean less people participating (we want the opposite), but what we really need is less on-thread back-and-forth. It'd be much better if we could enable people to workshop specific problems outside of the thread and bring solutions back.

6 Likes

I really like the idea. I wonder if it’s necessary to do it for every RFC, since it does add some overhead. Typically RFC are not getting so much traffic because they are about things that are more complex and less polarizing. The “Revisiting Rust’s modules” RFC is unfortunate because it touches something rather simple, but affecting everyone, with people having very wide range of opinions about it.

Careful there, I'm not equating those, nor am I considering it normal. The point is not what is "normal" (what does the word even mean?), but what happens in extreme cases.

As @repax said, the linear format doesn't help anything. Vast majority of comments are not directly reacting to the RFC, but to other people's comments. On contentious issues, you have two choices -- either try to talk through all the differing opinions, and possibly make the comment thread 100-strong in a matter of hours, or ignore other people's opinions and give up on the RFC process altogether. But taking the discussion off-thread is faced with obstacles. As it is, any attempt to do so will likely be ignored, and the motivation for being the one special snowflake who does try is nonexistent when all you want to do is "write just one comment".

Github comment section is the metaphorical hammer that doesn't fit the purpose. We need better tools.

I know @carols10cents is trying not to get bogged down in implementation details, but: these kinds of discussions (RFC and otherwise) would be so much easier on a platform where there were more tools for self-organization. Threading helps some, but I don't think it's enough.

As an example, I've been in IRC discussions that could have become overwhelming but didn't, because people took advantage of the ease of creating new channels ("join #some-new-sub-channel") and started a bunch of smaller more-manageable conversations.

Or maybe a better example is what usenet used to be like if you had access to a local server; at CMU in the early 90's, any student could create as many newsgroups as they wanted in the alt.* hierarchy that only showed up locally, so there were groups like alt.somedorm.announce that spawned, as needed, alt.somedorm.springbash and then alt.somedorm.springbash.drinkscommittee.

TL;DR: If there's a way for people to create subgroups/channels, they will figure out by themselves how to organize and summarize complex discussions.

2 Likes

Yes, I think this is a valuable idea, and to keep implementation details from distracting from it, people in the room could choose to organize themselves into smaller groups to discuss particular aspects of an RFC. I will think about this idea more :slight_smile:

There's also this case though:

  • Person A, who is not on any of the teams, posts an RFC
  • Reader B feels generally positive about it
  • Team member C posts a generally negative comment about it
  • Reader B thinks "oh I must be missing something, if Team member C thinks this isn't a good idea, this must not be a good idea"

We're all influenced by the opinions of others, and I think one advantage of this quiet time could be that everyone gets a chance to make up their own mind first before being influenced by others.

To address the scenario you mention, and staying in metaphor, we could initially distribute the text of the RFC without revealing who wrote it.

Note that I explicitly said that all original text of all comments would be made available for anyone to read if they so choose, and that if someone feels they were summarized unfairly, they can bring that up once regular discussion begins.

I will think about this, it's certainly interesting, but it does feel kind of two-tiered. People who have more time will be able to participate in the "real" discussion, and the sidechannel wouldn't get as much attention, imo. I think I'm more interested in figuring out a way to get at least one round where everyone feels like they could be heard equally.

This is kind of what I'm trying to get at! There were a few reasons already mentioned, but your comment dosen't tell me which comments you agree with most, which you might have discovered on your own, if you thought it was an okay idea until you read others' comments, etc. I'd love to hear it in your own words!

This is interesting, but I think it still biases the framing of the discussion towards those who are loudest and with the most time/energy to spend in the initial discussion.

1 Like

The clarification period could also probably have a multi-sided effect. If I have issues with an RFC, a simple back and forth with the author about a small paragraph of a conceived downside/problem has a chance to bring us both on the same page and see where each one is coming from.

Concensus is a lot easier when everyone involved knows about what and why they disagree with each other in my experience.

The initial round of discussion usually quickly addresses obvious weaknesses of the proposal, getting the uncontroversial adjustments out of the way. I'm concerned that starting with deliberation period would result in most people simply stating the superficial flaws and moving on to other things.

The discussion is still biased towards those with most energy, because those will be the only ones taking the time to actually explore the topic to any useful degree. They still get to frame the discussion, and the controversial aspects will still produce the same back-and-forth as they do now. The initial deliberating would be nearly useless in that regard, because it was done without any knowledge of other people's perspectives.

That's the crux of the issue in my opinion. It doesn't take much time to understand the RFC itself -- it's other people's viewpoints that take time to digest. Everyone has a certain degree of resistance towards understanding and internalizing other people's opinions, and I fear that starting out just thinking about my own preexisting notions just cements that resistance, making discussion all the harder when we are finally exposed to the reality of disagreement.

2 Likes

That has not been my experience following RFCs at all. Many initial observations and adjustments are relitigated in the comments again and again.

I disagree that it would be "nearly useless"; the people with lots of energy won't be actively discouraging those with low energy to participate.

Does this part of my proposal not address this concern? If not, why not?

Also, if authors are allowed to make modifications to RFCs during the deliberation period, that can serve to clarify any confusion that any commenter brings up, to the benefit of everyone.

shrug

That is just stating the obvious. If your own experience supported my concerns, you wouldn't write down that scenario in the first place. It would be more helpful to know whether other people agree or disagree with my take on things (they probably don't, though; my brain works in mysterious ways).

Yes, I believe so. I was just pointing out that specific advantage from my angle, and that it's a two way street. If I understand the RFC authors position and motivation better in relation to my comments, it's much easier to find common ground and then work out the rest.

They come up again and again, but are they wrong? RFCs will always have bugs, and not being able to see any comments would just leave me wondering "so am I just misunderstanding something here or is the RFC not written clearly or did the author actually miss this important point"?

I agree there is a problem, but I do not think complete silence is the right solution. To stay in your analogy, what I think I would prefer is for people to write post-it notes as they read, and for someone to collect and group them in real-time so I don't have to wait a week to find out whether what bugs me in this RFC is me misunderstanding something or not. Of course this starts to slowly transform into something much more like the current situation, if people actually start chatting by writing post-it notes...

Having an entire forum / issue tracker (i.e., multiple threads -- one per issue) per RFC would go a long way. In fact, that's what Niko did for the NLL RFC. Also, faster updates of the RFC text itself (referring to the relevant discussion) could prevent the same questions from coming up again and again -- people reading the RFC later will have an easier time to find out that this issue or alternative approach has already been brought up, discussed, and maybe resolved.

2 Likes

I don’t think any form of non-public comment would be an improvement for the RFC process, or help with the problem of overly long comment threads. Personally, a deliberation period would encourage me to be silent until it’s over, or to only submit relatively simple comments that don’t take a lot of effort to write.

It sounds like part of the premise of this proposal is that many people take a long time to form their opinion on an RFC and are then discouraged by the length of the discussion generated while they were forming that opinion. While probably true to some extent, there’s clearly also a ton of people that form an opinion very quickly or simply post without attempting to read the entire discussion (to be clear, I am not implying either of these are a bad thing) or actually are willing to read the entire comment thread. I think a deliberation period would discourage some of the people who do post today and encourage some of the people who don’t. We simply don’t know if that would be a net win for participation. Even if it is a win for net participation, I would expect a drop in “high quality participation” (actual new ideas or arguments rather than repeats or "I agree"s) precisely because we can’t see what’s been said so far, and a smaller audience (at least to me) incentivizes less effort per post.

We are talking about a small minority of RFCs though. The vast majority of them are just fine with a single comment thread, because the volume of commentary is low enough that everyone who cares can easily read or at least skim all of it. Plus, in my experience, the minority where the comments get out of control (in number, not in content) often have some unique issues that don’t apply even to other hyper-chatty RFCs. For instance, the generators eRFC is the only time I’ve ever seen a large number of RFC comments with very non-concrete/unactionably vague suggestions, or ideas that simply don’t make any sense in a language like Rust. It’s also the only one that I would describe as a history lesson, since it turned out that explaining Rust’s past experiments with other threading models was key to establishing why stackless coroutines was the proposed solution. I think it’s safe to assume these particular issues won’t come up for the NLL RFC, any module system RFC, etc.

But despite all of that, I do agree there’s a problem here. Although I am a crazy person who reads literally every comment on every RFC, it’s ludicrous to expect everyone that’s capable of meaningfully contributing to an RFC discussion to invest that much time before writing their one comment. In my opinion, any solution that theoretically could address this problem would have to fall into one of these categories:

  1. Reducing the need to post any kind of feedback at all, by making the RFC “better” at the outset. We already do a TON here. Honestly, I don’t think we can do much better at this category. “Bad RFCs” just aren’t the problem.

  2. Reducing the need to post comments by encouraging other forms of feedback. For most RFCs there’s not much we can do here, but for highly subjective ones like the module system threads, a lot of people (myself included) tend to post “I like X but not Y” comments that don’t add any new ideas future commenters need to read before writing their own comments. But when there’s an obvious and easy-to-use form of directly aggregetable feedback separate from the full-blown comment thread, a lot of people would probably use that and thus generate fewer comments, or make the comments they do generate more focused on the new, interesting idea they have than a like/dislike list. I’ve heard this is the main reason Github added “reactions”. Polls and surveys also go under here.

  3. Reducing the number of comments newcomers need to read before posting their own.

  • 3a) Periodically replacing comment threads with summary comments, so all the relevant information remains available but easier to digest. Typically, this is done by closing the current thread and opening a new one, and the first post of the new thread attempts to be a summary of all past discussion. I believe that’s already the best you can do in a lot of cases. In theory we could do more summarizing and thread rebooting, but as pointed out above, that relies on the shepherd or author doing more work, and it relies on all past/future commenters agreeing the summary is accurate. For highly technical RFCs (NLL? const generics?), this is what I would focus on. For highly subjective RFCs like those involving the module system, this seems completely unrealistic, so I’d instead focus on #2 and #3b.

  • 3b) Breaking up the comments/information into subthreads, so newcomers can completely skip the information that’s not relevant to them. Yes, half of the other posts in this thread already suggested this. They’re all right. On the rare, exceptionally chatty RFCs for which a standard RFC thread is not enough, this is by far the best thing we could possibly do to improve the process.

My conclusion: We should give the module system discussion its own github repo. There should be separate issues for every significant problem with the current system and proposed change, so everyone can easily throw +1s and -1s at each individual idea, and comment only if they have something new to add. People can easily make new issues for side discussions or new ideas worth voting on. The repo itself would contain all the strawmen proposals so far and a summary of all ideas and feedback we’ve managed to gather. Issues where some sort of consensus has been achieved (not necessarily on the solution, but at least what the leading candidates and their pros/cons are) then get closed by pull requests which update that summary. Newcomers would only have to read the summary, “vote” on any open issue they have an opinion on, and read the comments only on issues they have a strong interest in. That seems more manageable than catching up on even one of the module threads on these forums today.

5 Likes

The modules system RFC definitely has a thread about the history behind why paths aren't relative. I've also thinking that we could explicitly add a prompt for "what other ideas have been tried in the past and how did they work out?" in the "rationale and alternatives" section of the RFC template.

That seems like something that would be a great short-term thing to try. Going forward, do we let all RFCs try the current process and get a lot of comments before moving them to a repo? If yes, I'm just going to wait to see if an RFC will be moved into a more manageable format or not before I even bother reading it. If we try to decide before an RFC is posted whether it should start as a PR or a repo, how do we tell which RFCs will be contentious enough to apply this process? The eRFC text still says "This RFC is intended to be relatively lightweight and bikeshed free", so it's not always possible to predict which RFCs will garner lots of comments and which won't. If all RFCs require a repo, is that a reasonable amount of extra process/overhead to ask of RFC authors and commenters?

Not always possible, but I think the vast majority of the chatty RFCs have been ones we expected to be chatty. I was not at all surprised to see a ton of chatter on that eRFC, despite that line in the text. Though that eRFC is also an odd case where a separate repo might not have made sense because so many of the issues people raised were "already settled" in past discussions...though maybe it still would've been useful for forcing us to write summaries of that history. I dunno.

Agreed. iirc, it turned out that the kind of "relative" path that didn't work in the past is not quite the same as what most people in the thread suggesting relative paths had in mind, but that's exactly why such a prompt would be valuable. I'd definitely want this to get mentioned in any real module system revamp RFC, since I assume it'll change paths somehow (or explain in great detail why it chose not to change paths).

What about using Discourse, and encouraging the use of the "reply as linked topic" feature to split off subdiscussions?

(That feature is now buried under the Share button, btw; I had to look it up.)

1 Like

So, first of all, I have a lot of sympathy for this proposal; the concept resonates with me. And there are certain contexts where I feel like it does makes sense; in particular, we’ve talked before about raising “problems without solutions”, thoroughly discussing and analyzing the problem, then working on solutions. That would fit in the spirit of “hold off on proposing solutions”, which tends to produce better results, and avoids having people latch onto an early proposed solution and just doing local search within the nearby design space.

However, for RFCs themselves, as an RFC author, I have some significant concerns with this as written:

  • Most importantly, I feel like this will result in the RFC author getting numerous duplicate mentions of the same issues. If the RFC author missed something obvious, and many people can easily see that, I’d rather have one person comment and say “hey, this looks great, but you missed X, could you fix that”, rather than 50 people emailing the RFC author because none of them coordinate with the other 49.
  • This also causes those people to duplicate effort. Better for one person to find a bug or typo or similar and report it, rather than 50 people doing so and each taking the effort to write it up separately.
  • Related to that, I think it puts a lot of burden on the RFC author to collate comments from day 1. I’ve helped mentor new RFC authors, and I’d like to avoid putting more work on them right from the start. Later on, absolutely, they should help summarize and collate information. But not every RFC will end up with 200 comments and a thread that needs detailed summaries, and an RFC thread with 20 comments seems preferable to putting 20 comments worth of work on the RFC author to re-post.

In short, I feel like this approach has scaling issues, which will get worse as the community gets larger, rather than getting better.

I do think we have a problem with people not reading existing comments before posting their own, and with people providing Yet Another Take on an issue without seeing to what extent their views align with an existing take. I think we can solve that problem with a little more curation, though. In particular:

  • Let’s have a bot that always posts the first comment on an opened RFC. Provide a correct “Rendered” link (linking to tip-of-branch, not initial commit hash), thank the proposer, provide some info on the RFC process, and remind people of the following things:
  • Remember to read the thread before commenting. Before you add another take on the situation, make sure you’re presenting new information, rather than duplicating an existing comment. Try, hard, to see if someone has already made the same point, and if so, use reactions to indicate support for that point. Post a comment if you feel you have novel information to add, not to restate existing information. And whenever possible, concur with one or more existing comments and just state the specific novel points you want to add.
  • As an RFC author, please incorporate common comments and issues into the “Drawbacks” and “Alternatives” sections as appropriate. Even if you don’t agree with them, they’re worth documenting.

I do also feel that for the most controversial RFCs, having a separate repository for the RFC that can itself have issues and pull requests filed would be a really good idea. I wish that idea had occurred to me when working on the union RFC; I think that would have helped. (This is something that git and github themselves ought to have better support for.)

Regarding the notion that we won’t always know what the most controversial RFCs are in advance: that was once the case, but I think we should now be able to identify those at the pre-RFC stage.

7 Likes

Does Discourse have a way to give each RFC its own subforum? I wouldn't want an RFC with 10 different concerns to spam i.rl.o the way the module RFC is doing now.

I think the problem we have here is basically a UI problem - a single Github PR does not work for an RFC for which there are multiple alternatives and multiple concerns (because e.g. some syntax bikeshedding can easily drown out everything else, and if an RFC is changed, it's not obvious whether an old comment is addressed by it), and I think that with a Github repository it's too hard to track which issues are active ( . Therefore, it needs a UI solution.

5 Likes