We’re building Hercules CI, which is a CI for the Nix ecosystem. We’re just opening up preview access and we would love to help with open source projects like Rust.
Hercules CI builds a Nix expression in your repository, so you have to provide one. The community’s carnix tool helps with that.
Going into details about your requirements:
Hard requirements
- The service must be operated by a company we can contact directly for support. Building and maintaining a CI system in a reliable way for a project as big as ours takes a lot of time, especially to test on macOS, and most of the Rust infrastructure team is not paid to work on infrastructure. If the service requires us to use our own servers, the maintenance work we have to do should be minimal.
The CI (scheduling and UI) is hosted. The build agents would be in your hands, easy to manage and deploy because they are disposable and we provide deployment configuration files.
- Support should be direct, prompt (as appropriate), and helpful.
Since we’re just starting out and don’t have a lot of users, we’ll mostly focus helping everyone stabilize their CI. But we’d ask for a transition period to make sure everything works smoothly.
- The service must provide both Linux and macOS machines. Windows support could be nice, but switching away from AppVeyor is not a priority for us.
Nix supports many platforms including Linux and MacOS, but no native Windows support. Hercules will support what Nix supports.
- The service must allow us to increase timeouts and the available resources in the VMs.
- Anecdotally we need at least 4-core machines.
- 14 Windows + 5 Mac + 38 Linux = 57 current builders per PR
Since agents are self-hosted, you can choose whatever configuration you want.
- The service must be able to build and execute Docker containers (or a comparable system for enabling a level of reproducible builds).
We’d recommend using Nix to achieve reproducibility and avoiding containers, but you can build Docker containers with Nix even without using Docker.
- The service should be somewhat established, in the sense that we won’t have to go looking for a new solution in a year.
Nix is 15 years old technology, but the CI will be fresh. It’s written in Haskell and Elm, so we have high confidence.
Nice to have
- Evidence of usage on a project of a similar size to Rust (number of parallel builders, build length)
Sadly, not yet.
- The ability to hook custom hardware up alongside hosted builders. This would allow us to easily add CI for targets that might want to become Tier 1 in the future.
That’s the default. We might add hosted builds later if there’s a need.
- The ability to log into the builders remotely to investigate spurious failures.
They are under your control. Moreover, Nix allows you to build everything locally in the same sandbox so there should be little to no difference between what runs on CI and on developer builds.
- The ability to easily run and debug builds locally (this already mostly possible for the docker-based Linux builds, but support for other platforms would be nice as well).
As said above 
- The ability to share the same plan between multiple orgs (
rust-lang and rust-lang-nursery )
Pricing is per seat, but we’re leaning towards sharing the plan between orgs.
- The ability to prioritize builds from a repo (like rustc) over builds from the other ones.
This is not yet built but planned (not in short term).
- Built-in caching support: rustc doesn’t use it, but if we migrate other repositories away from Travis CI we’re going to need it.
That is already doable with Cachix and you can share all binaries with the world.
- Pay-for-what-you-use rather than a subscription model. We’re somewhat permanently under-utilizing capacity on Travis by design as we don’t want to get clogged, but it means we’re paying a bit more than we otherwise would.
You could configure the agent machines to trigger autoscaling and thus minimize your build costs.
