Update on the CI investigation

Hi everyone!

The infrastructure team started investigating new CI providers for the rust-lang/rust repository a few months ago, and we wanted to post an update on the work we’ve been doing.

We’re grateful to all the providers that posted on the initial thread and we investigated each of them. We also asked a few members of the community for feedback, learning from their hands-on experiences. The most promising option is Azure Pipelines and we’re going to evaluate it live on the repository for the next few months before reaching a decision.

To make things clear this is an evaluation and staying on Travis CI is still on the table, especially because they made good progress in the past few months on some of the issues we had. We’re also not planning to migrate our Windows workload (currently running on AppVeyor) for the time being, focusing most of the evaluation efforts on Linux (including cross-compiled tier 2 and 3 platforms) and macOS.

This process won’t affect other repositories in the rust-lang or rust-lang-nursery organizations: the infrastructure team is mainly responsible for rustc’s CI and other projects have always been free to use whatever CI platform they prefer. If the evaluation is successful and we end up migrating rustc, other rust-lang repos that want to follow will be welcome to do so, but they will also be able to continue using Travis CI as they’re doing today.

We sketched out this plan for the evaluation:

  1. Configure a few builders (spanning all the CI host platforms) and run them for around 3 weeks without gating PRs, fixing issues on the configuration as they come up.
  2. Gate PRs on the builders for a week to ensure we don’t get spurious failures. This step might be delayed if the week happens to be close to a release since we can’t afford CI weirdnesses those days.
  3. Hold an infrastructure team meeting and decide whether to continue with the evaluation, given the experiences we got after trialing the service.

If the team will agree to continue the evaluation we’re going to repeat the same “three weeks plus one gated week” process with all the builders enabled, and then decide how to continue. We’re still going to run all the builders on Travis CI and AppVeyor for the whole evaluation period, treating them as the source of truth.

We already have the bulk of the new configuration ready to go, but we’d be glad if anyone from the community (especially with Azure Pipelines experience!) wants to help us. You can find us in the #infra channel on Discord.

Pietro.
— Rust infrastructure team

44 Likes

Exciting! Curious to hear what factors led the infrastructure team to conclude that Azure Pipelines is the most promising option.

1 Like

If it helps at all, I just started a series of posts over setting up Azure Pipelines, originally targeting Python binary packages. I’m also interested in the factors that made it promising; we (Scikit-HEP) decided it was best for us mostly because it provided an easy method to support all three OS’s at the same time, which seems to be not one of the factors considered here.

The first PR adding the Azure Pipelines configuration is now open!

There are multiple providers that tick all our boxes feature-wise, but as we said one of the most important factors for us is reliability and customer support, and we think Azure is better on that front.

5 Likes

The initial PR adding the Azure configuration landed a few hours ago! There are still a few issues with it to fix though, and I opened an issue for each of them with the azure-evaluation label. If you want to help fixing them please claim an issue (typing @rustbot claim in a comment on it) and start working on it!

I also want to thank a lot everyone involved in getting the initial PR landed. It’s deeply appreciated!

6 Likes

Have you considered Mozilla’s Taskcluster?

Yes, Taskcluster was one of the options we evaluated. The main issue with Taskcluster is that it’s more of a toolbox to setup your own CI the exact way you want and that comes at the cost of a way bigger maintenance overhead, both in maintaining the builders and the configuration itself. @SimonSapin posted a summary if you haven’t read it already.

4 Likes

As an update on our CI evaluation process - we are transitioning rust-lang/rust’s CI from Travis CI to Microsoft’s Azure Pipelines. This is the result of a generous sponsorship offer from the Azure Pipelines team. This pipeline has been in the testing process for several weeks and we expect to use it to prepare the 1.37 release. We are still in the process of fine-tuning the results, but we expect to make a more complete (and more public) announcement in due time.

We’re still working with Microsoft to get 4 core machines up and running: in the meantime builds will be a bit slower than they used to be, but that should be resolved soon. There is also a work in progress PR to add Azure Pipelines support to the error messages bot.

Other repositories from the rust-lang and rust-lang-nursery that are currently using Travis will be shifted over to their free tier in the next few days. That should generally be enough for most of the organization. For those cases where it is not, members of the infra team will be reaching out to the repository owners to make arrangements.

If you have any questions or concerns about this change, don’t hesitate to reach out to us in our Discord channel.

19 Likes

Just to have asked the question: What happens when this Microsoft sponsorship ends?

Mozilla pays for it, I guess. Just like they used to with Travis CI.