Thanks for starting this thread though! I’ll raise your points at the next infra meeting.
My thoughts and background:
When requested on PRs, we will do a crater run for that PR and put the results in the PR. Beta runs tend to be requested (and reported) more informally in the #rust-infra channel on IRC, and I can see that we did not do a run for any 1.23 betas (though we did do them for 1.21, 1.22).
A ‘regular’ schedule isn’t a bad idea, but consistently doing it for betas would be a better place to start.
All testing is done without
--release. Doing release runs is also an excellent idea for nightly->beta, but I cringe at the thought of how long it would take to complete - serde + all deps (for example) takes 4x as long to compile in release mode. Two weeks is a long time to get a result - performance improvements are possibly a prerequisite here.