Crater plans


#1

Crater is this tool I made to test versions of Rust against crates in the wild, and we’ve been using to good effect to make decisions about breaking changes. People seem glad to have the tool, which makes me glad.

Still, since got it running before 1.0 I haven’t been doing much work on it besides fixing bugs. This is partly because I want to do a complete rewrite and haven’t built up the enthusiasm to do it until now.

In the rewrite I’m going to change the design quite a bit to reflect what we’re actually using it for and what it works well for - evaluating changes between two specific toolchains at a specific moment in time. Originally, I had grander plans of a tool for visually exploring the history of Rust regressions, but in order to get it done, I intend to limit the scope to something very simple. So here’s what I’m thinking the next revision of crater is going to be like.

First, it’s going to be written in Rust. The current version is JavaScript and I don’t like writing JavaScript. This is going to require an AMQP client library, and bindings to TaskCluster. The TaskCluster bindings shouldn’t be too hard. Hopefully the AMQP library already exists.

I’m torn about whether crater should be a local tool or hosted. Locally, it would require a database (probably sqlite so that the credentials can just be embedded in the tool), credentials for Pulse (the Moco AMQP server), and credentials for taskcluster. Hosted, the data will be archived for future analysis and most credentials can be managed by the server. Right now I plan to give it a client server architecture and host it.

In the current model, all results are shared, so if one test run tests num-0.1.0 with nightly then another does the same test, the results will be overwritten. This makes me uncomfortable because it’s discarding data. So in the new revision all data will be isolated by ‘job’, so my data is independent from niko’s data.

The initial UI is going to be a CLI, since that’s easy to build and work with.

The new usage model will be simple, compared to the existing:

  • Create a job, which is described by a ‘before’ toolchain and an ‘after’ toolchain.
  • If either of the toolchains is custom, they will be built automatically.
  • The status of the job can be checked periodically, in which case the tool will report whether the toolchain is available, whether the crate builds have started, and the status of the crate builds.
  • A report can be generated and automatically gisted at any time.

To start with authentication will just be with a shared password.

Potential command examples:

# Test some releases
$ crater new comparison nightly-2015-05-01 beta-2015-05-05

# Test some custom builds
$ crater new comparison https://github.com/brson/rust#22f3d3516824b151b4b7923ddc60cc8f34ab7ace https://github.com/brson/rust#299b34dde1ab67ad81fce2320a9cd73313da4e05

# Check the status of a job
$ crater status 1

# Print a report
$ crater report 1

# Gist a report
$ crater gist-report 1

Does this seem sufficient for now? My main goal is to let other people start doing crates.io regression testing without my help as quickly as possible.


#2

Would be nice if this could test crates on Windows as well.


#3

This all sounds great to me! My only worry would be making sure the hosted server stays online, but I imagine that won’t be too hard for us to do.


#4

This sounds great to me, thanks for this indispensable tool @brson!