Role of Crater in beta release criteria

[hsivonen]

Rust 1.23.0-beta.1 broke encoding_rs and, therefore, Firefox. Thankfully, the problem was discovered and fixed before 1.23.0 graduating to the stable channel, but I’m still left to wonder:

Why wasn’t this caught by Crater and what could be improved in the future?

I thought graduation from nightly to beta involved a Crater run. Is that not the case? Or is the Crater run in debug mode only and, therefore, a release-mode-only bug was not caught? If so, could a release-mode Crater run be added to the process of a Rust train moving from nightly to beta?

[petrochenkov]

Looks like the last “regular” crater run was performed before Rust 1.19 1.20, then @brson left and no regular crater runs were performed since then. (Unless infra team runs crater without reporting results.)

cc @rust-lang/infra

[steveklabnik]

(Unless infra team runs crater without reporting results.)

Yes, as far as I know they’ve been doing that; or at least, I vaguely remember seeing people talk about it in the infra channel.

[aidanhs]

Thanks for starting this thread though! I’ll raise your points at the next infra meeting.

My thoughts and background:

When requested on PRs, we will do a crater run for that PR and put the results in the PR. Beta runs tend to be requested (and reported) more informally in the #rust-infra channel on IRC, and I can see that we did not do a run for any 1.23 betas (though we did do them for 1.21, 1.22).

A ‘regular’ schedule isn’t a bad idea, but consistently doing it for betas would be a better place to start.

All testing is done without --release. Doing release runs is also an excellent idea for nightly->beta, but I cringe at the thought of how long it would take to complete - serde + all deps (for example) takes 4x as long to compile in release mode. Two weeks is a long time to get a result - performance improvements are possibly a prerequisite here.

[nikomatsakis]

A ‘regular’ schedule isn’t a bad idea, but consistently doing it for betas would be a better place to start.

to this, I think we should definitely be doing regular beta runs.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.