Rust 1.23.0-beta.1 brokeencoding_rs and, therefore, Firefox. Thankfully, the problem was discovered and fixed before 1.23.0 graduating to the stable channel, but I’m still left to wonder:
Why wasn’t this caught by Crater and what could be improved in the future?
I thought graduation from nightly to beta involved a Crater run. Is that not the case? Or is the Crater run in debug mode only and, therefore, a release-mode-only bug was not caught? If so, could a release-mode Crater run be added to the process of a Rust train moving from nightly to beta?
Looks like the last “regular” crater run was performed before Rust 1.19 1.20, then @brsonleft and no regular crater runs were performed since then. (Unless infra team runs crater without reporting results.)
Thanks for starting this thread though! I’ll raise your points at the next infra meeting.
My thoughts and background:
When requested on PRs, we will do a crater run for that PR and put the results in the PR. Beta runs tend to be requested (and reported) more informally in the #rust-infra channel on IRC, and I can see that we did not do a run for any 1.23 betas (though we did do them for 1.21, 1.22).
A ‘regular’ schedule isn’t a bad idea, but consistently doing it for betas would be a better place to start.
All testing is done without --release. Doing release runs is also an excellent idea for nightly->beta, but I cringe at the thought of how long it would take to complete - serde + all deps (for example) takes 4x as long to compile in release mode. Two weeks is a long time to get a result - performance improvements are possibly a prerequisite here.
to this, I think we should definitely be doing regular beta runs.