Crater is the tool that Rust teams use to check for regressions in the compiler, in order to detect bugs and guide implementation of new features or soundness fixes. If you’ve never seen some results from Crater you can take a look at this report for this pull request as an example.
It started life as ‘Crater’, then a rewrite called ‘Cargobomb’ arrived, and now we’ve decided to bring the ‘Crater’ name back into use! In future we’d like to have all whole-ecosystem regression testing live under the ‘Crater’ umbrella. The rename is still ongoing and you might see references in report urls, but the user-facing rename should be mostly complete - comment on that tracking issue if you spot any we’ve missed!
While you’re thinking about Crater, I’d like to take the opportunity to note that Crater is part of the impl period in the Infrastructure team section - there’s the usual gitter channel and tagged issues with help available, and I’ve recently merged some changes to make it much simpler to get started locally with Crater, so there’s no time like the present to get involved!
I just want to mention how invaluable Crater is to the Rust team, for discovering regressions, testing out theories about the ecosystem, and checking for potential breakage with permitted changes.
With all of that said, the tool could be so much better! Working on it is a great chance both to make an impact, and to use Rust in a particularly interesting setting – especially as we work toward managing distributed builds, etc. Please join in!
At the moment it’s limited to testing Rust toolchains against all of crates.io. https://github.com/brson/cargo-crusader is a tool along the lines of what you want – not sure of the current status though.
Is there any possibility to use the new crater in old-style mode (build only, without running tests)?
It’s 90% as useful as full testing, but doesn’t require waiting for many days and even weeks for each iteration.
Full testing is probably necessary only for regular per-release runs and especially critical PRs with possible silent runtime changes.
My impression is that the value of tests tends to be more that they are actual uses of libraries and so may fail to compile, rather than flagging silent runtime changes. So what may be more useful is the --no-run flag to cargo test (this may be what taskcluster-crater did,).
That said: in rough order of priority for speeding up crater runs, my current list is: 1) less human interaction, 2) more parallelism, 3) reduced coverage runs. With 1 and 2 I think we can probably get wait time down to about 2 days from the initial request (currently it’s ~4 days from when the run is started).
some parts are to work around bugs (e.g. cutting down logs which are too long)
some parts just need some scripting added (e.g. gathering info from github to inform which rustc commits to download)
each step needs manual interaction - once some prerequisite bugs are fixed, this can be resolved by just chaining the steps together
slightly related to being manual - it can be easy to forget about stages of runs and when they complete
From the perspective of people waiting for runs:
it’s slow (it takes a while to be started because its manual, it isn’t as parallel as it could be etc)
there’s little insight into progress of crater runs
As I mentioned in the post above yours, my first priority is to resolve my pain points However, if you’re interested in crater I’d love any contributions - there are tagged issues for the impl period you can take a look at, or you can visit the #rust-infra channel on IRC to ask questions. There’s a bunch of low hanging fruit available!
However, if you’re interested in crater I’d love any contributions - there are tagged issues for the impl period you can take a look at, or you can visit the