Building our own CI
This is a quick and dirty analysis/brainstorm of what it might take to build our own CI service just for the rust-lang/rust repo. I propose that all the other repos can stay with Travis CI for now.
This is just one point in the design space intended to start a discussion.
I propose that we can leverage AWS Spot Instances (maybe GCP or Azure has something similar?) and incremental compilation to achieve very high cost-efficiency by allowing builds to be interruptable.
I don’t know anything about the budget or personal bandwidth the infra team has, so I just layout my rough estimations here, and I will let them consider the numbers and respond as they feel comfortable.
Design Sketch
A central build scheduler would keep track of all scheduled and running builds. This could be bors or the GitHub issue tracker if we wanted. The point is that there is some central location that knows the state and progress of each build.
Each build runs in a container (e.g. via docker), as we currently do with craterbot, for security reasons.
On AWS, spot instances get a 2-minute warning when they are about to be interrupted. We could use this 2-minute warning to cache build artifacts to AWS S3 and notify the build scheduler that the build should be restarted later (and the names of the artifacts in S3 to be used later). We can clean up old artifacts in S3 lazily or eagerly when the build terminates.
There would also be integration with GitHub, bors, and rust-highfive to allow the workflow to remain roughly the same. Also, we would likely want some sort of dashboard to expose the progress and build logs for each build.
Normal CI builds (non-bors)
IIUC, these builds are less of a bottleneck and can be more variable time-wise (within reason). I propose that we choose the cheapest AWS spot instances that give reasonable CI time. Alternately, we could overcommit VMs (as Travis does)… whatever makes financial sense and achieves reasonable performance.
bors builds
These are traditionally the bottleneck. Currently, our process suffers from
- Long build times
- Variable build performance
- Timeouts, time travel, CI service flakes
The hope is that by controlling the CI infra ourselves, we can eliminate the second two, which are a significant source of frustration. We may also be able to improve build times by adjusting the instance types used for builds.
Cost/benefit
Financial
Currently, we pay Travis CI “a lot”. Looking at their public pricing info, it looks like the lowest tier is $130/month and their highest tier is $500/month. The $500/month plan allows 10 concurrent builds.
For comparison, I will use AWS prices from Jan 18 at about 11am CT. The numbers here are all pretty approximate, so take it with a cup of salt.
My laptop has 8 cores and 8GB RAM, and a full clean build (minus LLVM) takes ~20 minutes. IIRC, the full test suite take ~1 hour, and an LLVM build takes ~30 minutes, but I haven’t run these in a while. I will take this as performance baseline.
An AWS c5.2xlarge instance has 8 3GHz cores and 16GB RAM, so I expect it would get similar performance, or maybe a bit better.
The spot instance price for a c5.2xlarge (currently) is ~$0.10 per hour or ~$75/month/instance if left running continuously. According to the AWS page, these instances get interrupted about 10-15% of the time. I think with incr compl, we can easily tolerate this without too much frustration. One open question: how often are the spot instances available; would we spend more time waiting for builds to run?
Using on-demand instances would be too pricy, I think: $0.34/hour => $250/month/instance (running continuously). This might be problematic if we want to do 40 builds for each bors r+, as we currently do. Some potential alternatives would be to use smaller instances, which might slow down build times: t3.large (2 cores, 8GB RAM) = $0.08/hour; t3.xlarge (4 cores, 16GB RAM) = $0.16/hour.
Supposing that we wanted to run spot instances for everything, ~42 c5.2xlarge build instances continuously running would cost around $2100/month, which is not cheap. If we limit everything to 10 concurrent builds, this comes down to about $700/month, which might be competitive with Travis while (hopefully) avoiding some of the problems we’ve experienced.
Human
Obviously, the human cost here is that we would need to maintain and run the CI service. Hopefully, it would be simpler software because it doesn’t have to be as general-purpose as Travis CI, but it’s still more software than we have to maintain now.
Overall, I would estimate that the initial implementation would take a couple of man-months total, and hopefully maintainance/ops workload would be pretty low after that. Perhaps others with more experience could give better estimates?
Finally, since this would be a big experiment, there is the “unknown factor” – the problems we don’t know about yet that come up along the way.
On the other hand, the human cost of dealing with spurious CI failures would hopefully be relieved, which would be good.
Technical
The main technical cost, I think, would be the need to implement and maintain a CI service. Also, integration with existing services, such as GitHub, bors, and rust-highfive might be less straightforward than they are now. However, because the service doesn’t need to be general-purpose, I think it will not be too complex of an undertaking.
In exchange, we get all of the following technical improvements:
- Easier debugging (hopefully), since we own the code.
- More consistent performance => hopefully, fewer timeouts and flakes.
- More control over performance:
- Can control VM overcommittment and scheduling.
- Can control instance types.
- Can use different instance types for CI and bors.
- More control over caching.
- With spot instances, can take advantage of incr compl to do interruptable builds
Related discussions
There have been some previous discussion of our CI processes. A quick search brings up the following:
