Rust-lang/rust's CI provider investigations

[aidanhs]

The previous thread on this topic got a ton of interest from a number of parties (which is great) and so I wanted to take the time to outline how we intend to go about our investigations.

First, let’s set the scene.

The rust-lang/rust repository CI is a constant concern for the Infra team - we’ve reached a point where you hit the inevitable edge cases of interacting with many different components of infrastructure (networks, system toolchains, interacting and conflicting build systems, caching) for extended periods of time. In the process of working on these different components we evaluate different aspects of our CI process, and so it makes sense to evaluate overall CI solutions as well.

To be clear, I and other individuals on the team are huge fans of Travis as a CI service (I personally fully intend to continue using it myself), and remaining with Travis is one of the top options on the table. We also fully expect problems with whatever we end doing - I and other members of the team have worked on production infrastructure and it’s just a fact of life.

Notes:

• We are carefully scoping this to the rust-lang/rust repository. Other repositories are free (as always) to use whatever they like.
• Given the need for investigation and evaluation, any hypothetical CI migration would not happen before May (likely longer!).

To roughly outline the first steps of our investigation process:

1. We have a big list of suggestions from the previous thread that we’re going to whittle down based on the original criteria laid out.
2. We will review the integrations we have with our current CI and investigate what we would need to reach parity if we were to want to move, and review the possible providers with these integrations in mind.
3. We will contact providers to get more information about their services where we have questions, particularly to find out about projects with similar characteristics to rust-lang/rust’s.
4. We will experiment with these new services ourselves to get a feel for how they might compare against what we have today.

We’ve sketched out additional steps after this, but these four will keep us plenty busy for now and it gets a bit more fluid after that.

We’ll endeavour to keep this thread updated as we move through the list!

[thoroughburro]

The new owners of Travis CI, Idera, just laid off a substantial portion of their workforce, including a lot of seniors with institutional knowledge. There is a thread here with ongoing reports. It sounds like it was handled extremely poorly, which doesn’t bode well.

Can we assume that Travis CI will fix the many problems we’ve been experiencing, given this development? Even if they do, do we want to work with a company that would treat its employees this way?

[aidanhs]

I’ve been observing what’s going on and as of right now I don’t see any of the current plans or investigations needing to change - it doesn’t seem to necessitate an accelerated response that would have us doing anything before May, which gives us time to evaluate as we go.

[skade]

I’ve been in touch with a couple of Travis people (not from the Rust side, just because I’m friends with some) and while this has been handed down in a classic US fashion, it’s handled as “extremely poorly” as many other of such situations. One of the impactful things here is that this is the first time Travis lays off a lot of people. The thread you link (and also the Twitter thread) don’t give much insight who and why people were let go. Travis has an extremely good standing as an employer in the Berlin tech scene and that history should not be completely forgotten.

On the level of a project, it is also hard to make decisions on these axis. As much as I support workers rights and prefery giving money to good organisations, it’s also extremely hard to figure out stuff based on tweets and a reddit thread. There’s still a ton of very good people working there. I’d start thinking about that issue with that background not before giving it a month space.

I would prefer not to have that storm sway our path too much.

[pietroalbini]

An update with the next steps of our investigation was just posted!