The infrastructure team started investigating new CI providers for the rust-lang/rust repository a few months ago, and we wanted to post an update on the work we’ve been doing.
We’re grateful to all the providers that posted on the initial thread and we investigated each of them. We also asked a few members of the community for feedback, learning from their hands-on experiences. The most promising option is Azure Pipelines and we’re going to evaluate it live on the repository for the next few months before reaching a decision.
To make things clear this is an evaluation and staying on Travis CI is still on the table, especially because they made good progress in the past few months on some of the issues we had. We’re also not planning to migrate our Windows workload (currently running on AppVeyor) for the time being, focusing most of the evaluation efforts on Linux (including cross-compiled tier 2 and 3 platforms) and macOS.
This process won’t affect other repositories in the rust-lang or rust-lang-nursery organizations: the infrastructure team is mainly responsible for rustc’s CI and other projects have always been free to use whatever CI platform they prefer. If the evaluation is successful and we end up migrating rustc, other rust-lang repos that want to follow will be welcome to do so, but they will also be able to continue using Travis CI as they’re doing today.
We sketched out this plan for the evaluation:
- Configure a few builders (spanning all the CI host platforms) and run them for around 3 weeks without gating PRs, fixing issues on the configuration as they come up.
- Gate PRs on the builders for a week to ensure we don’t get spurious failures. This step might be delayed if the week happens to be close to a release since we can’t afford CI weirdnesses those days.
- Hold an infrastructure team meeting and decide whether to continue with the evaluation, given the experiences we got after trialing the service.
If the team will agree to continue the evaluation we’re going to repeat the same “three weeks plus one gated week” process with all the builders enabled, and then decide how to continue. We’re still going to run all the builders on Travis CI and AppVeyor for the whole evaluation period, treating them as the source of truth.
We already have the bulk of the new configuration ready to go, but we’d be glad if anyone from the community (especially with Azure Pipelines experience!) wants to help us. You can find us in the #infra channel on Discord.
— Rust infrastructure team