Triagebot Notifications

dtolnay and I have been working on a potential new system for tracking notifications for Rust team members. The primary reasoning behind this is that we've wanted to surface team member queues to folks doing triage and the community at large, as a way to help build both empathy for the length of the queue and make it easier to tell how "soon" a PR might be reviewed.

Currently I believe it's just myself and dtolnay using the tooling, but it is open for any Rust team member (specifically, listed on rust-lang/team) to try. We do not currently have plans to expand beyond that, though there is nothing intrinsically limiting it.

So, how does one look at the notifications? Presuming you're a Rust team member, you can visit https://triagebot.infra.rust-lang.org/notifications?user=$GITHUB_USER_NAME. For example, see dtolnay's page.

The UI is quite dry, and the HTTP interface is entirely read-only today. Acknowledging and adding meta information to notifications is done through Zulip today. See the wiki for details on supported commands.

Currently only direct or indirect team pings (i.e., @Mark-Simulacrum or @rust-lang/release) will automatically add a notification, across the rust-lang organization. It is intended that in the future we will support other modes of notifications, such as "subscribing" to particular threads and so forth.

It's very early days so far, but dtolnay has told me that this has been very helpful, and I have personally experienced the same. We're announcing this to a wider audience primarily to give team members something to try out. We'd love to hear feedback of any kind! I'm sure that this is not the right tool for everyone, but even if it improves the lives of a small subset of team members that would already be great!

Note that all Rust team members currently have a dedicated page, but this does not mean that they are using the tool. There are plans to surface this in some way.

15 Likes

This is awesome! *goes off to comment on things*

Oh, also, forgot to mention. Feel free to leave feedback here, but also https://github.com/rust-lang/triagebot/issues is fine to file issues if you have feature ideas/requests/bugs.

cc @dtolnay, also

I am very excited about this system, since the number of emails I get from GitHub is out of control to the point that email is no longer a reasonable way for me to track and manage actionable notifications. I expect as Rust's momentum grows, other team members have already experienced the same or will soon.

As I see it there are four major improvements that Mark's triagebot work brings to three audiences:

  • Team members see where their attention is needed without juggling zillions of emails. This is the most apparent benefit, but not the most important one from my point of view.

  • Triage team has a way to see what is on each team member's plate and make intelligent load-balancing decisions. Previously, if a PR sat for a while without review, the state of the art was for triage to make a shot in the dark to assign it to a random other member of the reviewing team, which is not necessarily productive.

  • Community members who send PRs or make issues now have something to track while waiting on feedback.

    One of the nightmares for me as a maintainer is what @yaahc describes in this meetup talk: I remember distinctly one time leaving a comment on an issue very shyly being like "hey would it be okay if I take this issue?" and waiting two days for a response and overthinking it and freaking out, and eventually I just deleted the comment and tried to find somewhere else because I was too stressed about it.

    Imagine if the comment that @rust-highfive leaves on new PRs included a link to the reviewer's triagebot page. This gives a place for the PR author to track as the reviewer makes progress toward the specific PR, and a clear way to internalize that there is a lot going on across the project – the lack of immediate attention isn't because the PR is stupid, or the author is a minority, or whatever other overthought reason. This makes for a friendlier environment than waiting in an unresponsive void.

  • Finally, most excitingly, the new system opens a way for community members to participate actively in the triage process. Not called out in Mark's post but documented in the wiki link, we've opened up write permissions on any team member's notification queue to anyone with a zulip id registered with triagebot.

    So for example anyone can manipulate my queue in all the same ways I can, by prefixing their triagebot commands with "as dtolnay".

    I strongly encourage people to take advantage of this to add or reorder my items! With the volume of things I am involved in, things slip by me and I can use the help keeping track of where my attention is most needed.

6 Likes

Sweet! Dumb question: how do we actually interact with the Zulip bot? I didn't see a username or room mentioned on the wiki page.

@sfackler, I've added a link to the wiki — it's https://rust-lang.zulipchat.com/#narrow/pm-with/261224-triage-rust-lang-bot.

Looks like there are quite a few false-positives, e.g. https://github.com/rust-lang/rust/pull/68997#issuecomment-586694101 is in the list for me, but it's just a PR that I r+'d.
I'll try to review the list closer this evening.

There are also notifications that I ignored because I don't care about them, I guess I'll have to don't care explicitly now :slight_smile:

Looks pretty useful if false positives are pruned somehow.

1 Like

The tool does indeed expect everyone to acknowledge everything explicitly, including in that leaving a comment is not an acknowledgement.

I think that there's definitely room for improvement there: for example, it's long been a goal to make triagebot take over from high-five for r? management, and if we did that then it would likely be not too hard to consider such a comment resolved with an approval. Currently the tool lacks any heuristics like that, which certainly simplifies the design, but also does add to the potential human workload. I imagine that some heuristics are probably good ones to add, though I think finding exactly where to stop might be nontrivial.

As to notifications you don't care about, could you elaborate some more? Is that in the context of a team-wide cc perhaps? One thing we don't surface in the UI is whether a comment originates from such a ping or from a direct invocation of a username, but we obviously have this information. Would it be helpful to have that highlighted?

2 Likes

When I've thought about suggesting this tool in the past, I was going to suggest an explicit notation like f? @nikomatsakis as a way to explicitly request feedback from someone (vs just @-mentioning people randomly).

Something else I would really like:

I tend to want to order my notifications by "how frequently did I interact with this issue" as measured in number of comments. In other words, if I posted, the next comment is very likely in response to what I wrote, and I would tend to want to see it -- this is much more true than if there have been 222 comments in the interim.

Not necessarily team-wide, just something I either cannot usefully answer, or something that I need to investigate before answering, but won't investigate because it's not important to me.
Probably not automatable.

I think I agree that it's better to say something meaning "I won't answer" explicitly instead of just keeping silence, and that includes team-wide pings too.
Perhaps a triagebot list can help with that.

This is fantastic!. Going to try it out as soon as I have my zulip-id PR added on team :blush:

So, I went through the list of https://triagebot.infra.rust-lang.org/notifications?user=petrochenkov and it was mostly double work.

The largest part of the list are links to PRs from my review board (https://github.com/pulls/assigned) in various review stages, even closed ones (https://github.com/pulls?q=is%3Apr+assignee%3Apetrochenkov+archived%3Afalse+is%3Aclosed), sometimes multiple links to different comments in the same PR.
I already track those through the review board, from which they disappear automatically on PR merge/close, I wouldn't want to manually cleanup them from the triage board as well.

From the remaining items, some I've seen before and ignored as "meh, not my business", but some were new. The number of those remaining items wasn't large, so it should be ok to acknowledge them manually.