So, oftentimes we get regression reports that are troublesome to reproduce – for example, they might be on a platform we don’t have ready access to. As it happens, we now have “PR-by-PR” artifacts available that really help us to narrow down precisely where the problem was introduced. It would be so cool if we had a tool
cargo bisect-rustc that, given some inputs, find the exact PR that triggered caused the build to fail.
In my ideal world, the tool would require zero input from the user, though they would be able to optionally supply some stuff to make it go faster.
First, it would verify that the build fails with the current nightly that is in use.
It would work by doing backwards jumps over nightlies to find a “working” date. So, it would start with the previous nightly, then try 2 nightlies ago, then 4, then 8, etc. until the build starts to pass again.
Then it would narrow down to a specific nightly.
Naturally, the user should also be able to supply bounds explicitly, and also to specify the failing command expected result (e.g., is build not supposed to pass? or maybe it’s a test/example that fails?).
All of that can basically always be done. But then we can go further and use the PR-by-PR builds. We can find the SHA1 info from the nightlies where the breakage occurs and use rustup-toolchain-install-master to install their builds, and then do bisection.
This would basically just be an easier-to-use, zero configuration version of the various tools that already exist – e.g., @est31 I think has some bisection script, @kennytm wrote the rustup-toolchain-install-master, etc.
Naturally such a tool isn’t just for users – it would help us too. Some of us – cough me cough – always get hives thinking about running
git bisect, for example, since it requires me to think hard, take a bunch of manual steps, etc.
What do you all think?