More precise regression tracking

[brson]

The way we track regressions today with ‘regression-from-$channel-to-$channel’ tags leaves a lot to be desired. In particular, it is impossible to automatically derive any useful historical information about regressions per release since the tags do not provide enough information. I would very much like to be able to issue reports quantifying whether we are doing better or worse at preventing and fixing regressions.

Here is the type of information that would be useful:

• How many regressions were discovered in version X?
• How many regressions were fixed in version X prior to release?
• How many regressions were not fixed in version X prior to release?
• How many regressions in version X were closed as ‘expected breakage’?

It would be nice to change our process sooner than later so that someday we can do that analysis without going back and retriaging every reported regression ever.

I haven’t thought much about the process I’d prefer but here’s one possibility:

• Regressions are just tagged a hot-pink “regression” (or maybe I-regression)
• Every release has a “X.Y.Z regressions” milestone, and every regression is dumped into a milestone
  • We leave these milestones open for tracking during the nightly/beta period, but close them once the release is out so they don’t pollute the milestone page
• There is also a “Stable regressions” milestone that stays open forever
  • As regression milestones are closed those issues are additionally added to the stable regressions milestone
  • This could also be a “stable-regression” tag
• There is an “expected-breakage” tag
  • This seems the hardest thing to track. I wouldn’t expect devs to remember to tag closed issues like this, so probably somebody would have to go through each regression milestone before closing it and make sure everything was tagged appropriately

With this system we could easily find the regressions for each channel, and do queries that answer the above questions. Again, I have not put much thought into the specifics. Just a strawman proposal.

Opinions?

[alexcrichton]

I like the idea of using milestones and tagging with just a plain “regression” tag, saves us from having to migrate between tags as well. We could presumably just have a blanket “X.Y.Z” milestone for each release, right? (regressions and perhaps other features would go in here too).

The only tweak I’d suggest from what you were thinking is to not have a stable regressions milestone but instead just have that be ambient about when the issue itself was reported. Any “regression”-tagged issue that’s still open for more than 12 weeks should be a stable regression, right? I’d also be fine with just a “stable-regression” tag in addition to “regression”. That way closing a milestone is just adding tags on issues (stable-regression or expected-breakage + close)

[brson]

We could presumably just have a blanket “X.Y.Z” milestone for each release, right? (regressions and perhaps other features would go in here too).

Yes, and we do have those milestones today. I had some reason for wanting to split them up, but on second thought don’t think that is necessary, so yeah we could just put all regressions into the release milestone.

Any “regression”-tagged issue that’s still open for more than 12 weeks should be a stable regression, right?

The math isn’t quite that simple since regressions can be opened at any point during the 12-week prerelease period. It would be more accurate to say “any regression that was open later than its associated milestone was closed” was a stable regression, or actually, that won’t work since milestones don’t necessarily get closed on schedule; more like “any regression that was open later than the release date of its associated milestone”. Though there is a fuzzy window between the branch and the release, that’s still probably a fine approximation.

[brson]

Oh actually, just doing that math to figure out stable regressions doesn’t satisfy one of the reasons I wanted a bucket for them. When doing triage we need to be able to query nightly, beta, and stable regressions, and there needs to be some way to separate out those three categories. I’m not sure there’s a way to formulate a github query that says ‘is open and not in either of these milestones’.

[alexcrichton]

Ah ok that makes sense! Maybe just “stable-regression” tag along with an “expected-breakage” tag? That way triage would just open up a milestone and filter by “regression”, right?

[brson]

Unless somebody objects I intend to make these changes before the next release.

[nikomatsakis]

I have to admit I find the current tags quite perplexing. The idea of I-regression feels pretty good. I think basically I feel like we treat regressions as P-high all the time, and the thing I usually want to know is roughly “do I have to backport this?!” – how would I derive that information in this proposal? By checking out which milestone it’s in (and comparing that to the rust release numbers I see on rusty-dash or whatever?)

[brson]

That’s a good question, hm. You could easily see which release the fix needs to be in by looking at the milestone but not whether it actually is in that release.

Today you can mostly tell by the tags whether a patch needs to be backported, as long as some human has kept the tags up to date and promoted the ‘nightly’ tags to ‘beta’ etc, though there is a whole lot of room for error and we do sometimes fail to catch needed backports.

The only way I can see to do it foolproofly is:

• For every issue in the milestone
• Look it git whether the commits for the associated pr is in the branch
• If not, backport

[FraGag]

Perhaps the branch name could be part of the milestone’s name? For example: 1.20 (nightly), 1.19 (beta), 1.18 (stable). Then the older versions have no branch name at all (e.g. 1.17). When a release moves to a different branch, the milestones are renamed to update the branch name; when the stable version is replaced, the previous stable version loses its branch name.

[brson]

Perhaps the branch name could be part of the milestone’s name? For example: 1.20 (nightly), 1.19 (beta), 1.18 (stable). Then the older versions have no branch name at all (e.g. 1.17). When a release moves to a different branch, the milestones are renamed to update the branch name; when the stable version is replaced, the previous stable version loses its branch name.

Even with that though, you can’t know what channel the version was on at the time any given issue was closed, which is what determines whether a backport is needed. For example if I put my issue on “1.20 nightly”, fix the issue, then rename to “1.20 beta”, I don’t need a backport; but if I put my issue on “1.20 nightly”, rename it to “1.20 beta”, fix the issue, then I do need a backport.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.