We’re in the process of making a number of changes to Rust’s integration and release infrastructure, with the goals of making it simpler for the community to participate in Rust’s infrastructure maintenance, making the infrastructure more reliable, expanding the release build process so that it can accommodate more Rust standard tooling, such as the RLS and clippy, and accomodating distribution of the new book and other Rust docs.
Here I’m going to explain what’s happening and why.
Some of the factors at play in this transition include:
- Our buildbot-based CI / release infrastructure cannot be maintained by community members, is generally bottlenecked on Alex and myself.
- Our buildbot configuration has reliability issues, particularly around managing dynamic EC2 instances.
- Our nightly builds sometimes fail for reasons not caught during CI and are down for multiple days.
- Packaging Rust for distribution is overly complex, involving many systems and source repositories.
- The beta and stable branches do not run the test suite today. With the volume of beta backports each release receives this is a freightening situation.
- As certain core Rust tools mature we want to deliver them as part of the Rust distribution, and this is difficult to do within the current infrastructure / build system design. Distributing additional tools with Rust is particularly crucial for those intimately tied to compiler internals, like the RLS and clippy.
At the end of this process we will expect that
- Anybody can modify the majority of Rust’s CI / release infrastructure by submitting a PR to rust-lang/rust.
- Rust’s CI builds and testing will be running on Travis and AppVeyor, not private buildbot instances.
- Rust will not be running buildbot at all.
- There are binary builds available for every PR that is merged, for use in testing and bisecting.
- Every day that a PR lands there will be a nightly.
- Beta and stable branches receive the same amount of automated testing as the master branch.
- Rust release builds are produced on Travis and AppVeyor.
- The rust-lang/rust build itself will optionally compile, package, and install an expanded Rust distribution that includes cargo, and will be extensible to other core tools such as RLS and clippy in the future, laying the groundwork to install the RLS with Rust via rustup.
Note: in this document I make a distinction between “rustc” and the "platform", when discussing the build, where the platform refers to rustc plus the critical tools beyond rustc that are tied to rustc internals or that otherwise need to be distributed with Rust. I realize the word “platform” is tainted because of the previous library discussion, but the use of the term here shouldn’t be conflated with the distribution of additional libraries with Rust.
Moving CI to Travis and AppVeyor
Already underway and near completion, we are converting our CI builds - the ones that run on every PR and must pass before a PR is merged - to run on Travis and AppVeyor. This will make it easier for people to help fix the automation, since the configuration is stored directly in the source tree, and since others have easy access to their own instances. It should also make the builders more reliable since these systems are widely used and they create consistent, clean environments for every build.
These builds will still be coordinated by bors and the workflow for contributors and reviewers will remain the same.
We have dedicated instances from both Travis and AppVeyor.
Publishing builds from CI
Today the release build process tends to break in ways that are not detected during CI and nightlies are often not produced.
To address this, we are going to modify the CI build so that it
produces full release artifacts on every merge. Every time bors builds
Rust it will also run
make dist and upload the binaries to S3 under
a directory named after the commit sha. So we know that every time
bors merges a PR we have a full set of release binaries for all
supported platforms. And by having binaries of every merged PR
available it will make it easier to bisect regressions.
We can later use this same infrastructure for producing the actual releases.
Integrating Rust tools into the build process
Today the build process for producing full Rust releases is quite convoluted, it can’t be reasonably be reproduced outside our infrastructure, and it is difficult to extend to include additional tools that we want to distribute with Rust, like the RLS.
To address this we are going to integrate all the scripts and build steps involved in producing a full Rust release directly into the rust-lang/rust source tree. This includes functionality that currently lives in rust-packaging and rust-buildbot, as well as the complete cargo build. By doing this anybody should be able to easily produce the same set of release artifacts we do, it will be easy for Linux distros to keep their cargo build properly paired with rustc, and it will be easy to expand our on release builds to include other tools.
Even though this will expand the set of build artifacts produced by the rust-lang/rust build, we will make it optional, leaving the rustc developer experience as-is.
When building / testing a complete release one will pass the
--enable-platform flag, that is “enable the full Rust platform”
(though we may need to pick a different name than “platform” since
that word is tainted). When this is enabled several things happen:
- at the end of the final stage build, cargo is built
- during final stage testing, cargo is tested
- during ‘make dist’, cargo is packaged
- during ‘make dist’, combined ‘rust’ installers are produced that include cargo
- during ‘make dist’, other package formats like msi are produced
- ‘make install’ installs rust and cargo
This logic extends to future additions to the Rust distribution, like the RLS and clippy.
Adding the RLS and clippy to the Rust distribution
Once the aforementioned work on the build system is complete, that is, once we have an extended “platform” build that can be optionally enabled to build the full Rust distribution, we can begin adding other tools to Rust. This is most important for the RLS and clippy. Both tools are, or will be, widely used, an important part of the Rust developer experience, and both are intimately tied to the compiler’s internals. Because of this the most reasonable way for them to be distributed is as part of Rust - otherwise it is too difficult to pair the correct build of the tools with the correct build of rustc. An alternative would be to augment cargo in some way so that it can understand installation of toolchain-specific crates, but considering that this is exactly what rustup does, the most obvious path forward today is to distribute it through rustup or otherwise as part of the Rust distribution.
As mentioned before with the addition of cargo to the platform build, once e.g. rls is part of the build, it will be optionally built, tested, and packaged with every build of Rust; and thus installable via rustup.
For the most part it will be a straightforward addition. The major question is how to maintain the RLS within the tree. We could e.g. create an RLS submodule, but I think this is likely to be quite difficult to maintain since changes to rustc will break the RLS, and RLS tests will need to pass for rustc changes to land. For that reason, I believe the most practical solution is simply to pull the RLS in tree and deprecate the RLS repo. Again, because the extended platform build is optional, this should have little affect on day to day development, but Rust developers will be on the hook to keep the RLS building. Since delivering an RLS is a key part of our product strategy, this seems reasonable to me.
On the other hand, merging projects will also merge their issue trackers, and possibly slow down their development.
Future changes to doc build and distribution
This plan naturally extends to the treatment of Rust’s docs. Documentation issues on the horizen include the distribution of Rust’s new book, the future of the nomicon and the reference, distributing Cargo’s docs.
Like with tooling, docs will become part of the extended build,
continuing to produce the single
rust-docs package. Like with
tooling, the question about where the documentation is maintained -
either in-tree or out-of-tree - I think will be mostly driven by the
stability of its subject matter. My preference is for things that can
be maintained out of tree, to be maintained out of tree.
So we might end up with the new book, the nomicon, and the reference, as submodules, all discussing stable matters, and a separate in-tree document for unstable matters.
Then we can think about expanding the standard Rust bookshelf further. The first candidate will be the cargo docs, which should be accessible the same way as Rust’s other documentation.
As part of this process we’ll take a build-time dependency on mdBook, only when the extended build is enabled. This should be fine once the old build system is removed.
Producing releases from CI builds
As natural fallout of the above changes, we can then eliminate almost all of the nightly, beta, and stable build machinery. After a successful PR merge we will have the complete set of binaries available on S3, nearly ready for release as-is.
By releasing these binaries, it does mean that there is another intermediate party involved with our release infrastructure compared to today: in addition to AWS / macstadium we are also depending on Travis and AppVeyor to maintain the integrity of our release build environment.
We’ll need to consider how to make this setup reasonably secure, and it’s clear that we need to start making Rust’s release security better, not worse. We’ll think through this more clearly before doing anything.
The final steps after the binaries are produced is to sign them, generate the manifest used by rustup, and publish them to their final location for distribution.
To accomplish this we would have a small machine that periodically does the work. Exactly when it will run will depend on the release channel, but it will do a few things:
- Download all the artifacts for the commit being released to local disk
- Generate the release channel manifest
- Sign all the artifacts
- Upload them to static.rust-lang.org’s dist folder and archives
- Run CDN invalidations
As an extension, we can also produce rustup-installable releases for every single bors-merge commit, not just once per day.
Beta / stable release changes
The beta and stable branches work just like the master branch, PR’s go through bors, run all tests. Whereas today commits are just merged into beta and stable.
Once change here is that, with releases keyed off of bors merges to beta and stable, there will be times when we need to submit a no-op pr just to trigger a build.
More efficient bootstrapping
As part of the switch away from buildbot, to fit within our time budget on Travis, we will modify the bootstrap to build fewer stages by default. We will only build the compiler and standard library twice, instead of three times, once with the bootstrap compiler, and then once more (I’m not going to talk about stages here because the way stage artifacts are intermingled makes it too confusing to communicate about, instead I will talk about the “first build”, “second build”, etc.). Today we always do three builds of std+rustc during bootstrap, tomorrow we will do two by default, and validate that the third continues to work.
In the Rust bootstrap process, the second and third builds are assumed to be ABI-identical (the first and the second builds are often not ABI-identical). In practice today there is no validation that this is true, and the purpose of doing the third build is to confirm that rustc can rebuild itself.
So by default, the rustc build will only do two builds, and we will have one integration machine testing the third build.
Travis/Appveyor build matrix
This next section is simply cataloging all the configurations used in the new system (the “build matrix” in Travis terms).
These are distinguished along three dimensions: whether they are building/testing rustc, whether they are building/testing the extended "platform" tools, or whether they are just providing other coverage.
Some of the division here is motivated by reducing total build time.
Here “host” means “platforms that run rustc”, and “target” means "platforms that run std".
There are 46 configurations, publishing bins for 44 platforms.
Cross host rustc/platform builders
These are the large set of cross-compiled host platforms that can be built from Linux. They do not run tests and therefore can afford to build both rustc and the extended platform. They publish rustc, the platform tools, and combined installers.
Native host rustc builders
These build host toolchains on their native build platforms, as well as additional targets. Hosts are tested. Targets may be tested. They publish only the target artifacts, not the host artifacts, which are published by the platform builders in the next section.
- x86_64-unknown-linux-gnu (not tested, not uploaded)
- asmjs-unknown-emscripten (test)
- wasm32-unknown-emscripten (test)
- i586-unknown-linux-gnu (test)
- i686-unknown-linux-musl (test)
- x86_64-unknown-linux-musl (test)
- x86_64-unknown-linux-gnu (not tested, not uploaded)
- arm-linux-androideabi (test)
- x86_64-unknown-linux-gnu (not tested, not uploaded)
- armv7-linux-androideabi (test)
Native host platform builders
These are build / testing host platform bins on the native OS. These don’t run the rustc test suite but do test the additional platform components. All create combined tarballs and platform-specific installers and publish their artifacts.
- x86_64-apple-darwin (+pkg)
- i686-apple-darwin (+pkg)
- i686-pc-windows-gnu (+msi)
- i686-pc-windows-msvc (+msi)
- x86_64-pc-windows-gnu (+msi)
- x86_64-pc-windows-msvc (+msi)
These are additional configurations for test coverage. They publish nothing.
- x86_64-unknown-linux-gnu (nopt tests)
- i686-unknown-linux-gnu (nopt tests)
- x86_64-unknown-linux-gnu (debuginfo in compiler, no test)
- x86_64-unknown-linux-gnu (cargotest, grammartest)
- x86_64-unknown-linux-gnu (LLVM 3.7, tests)
- x86_64-unknown-linux-gnu (makefiles, tests)
- x86_64-apple-darwin (makfiles, tests)
- x86_64-pc-windows-msvc (makefiles, tests)
- x86_64-pc-windows-msvc (cargotest)
- i686-pc-windows-gnu (makefilest)
- x86_64-unknown-linux-gnu (distcheck)
- x86_64-unknown-linux-gnu (stage2 bootstrap)
CI / release build transition roadmap, and how to help
There’s a lot here, some immediate, some more speculative. In the short term we need to finish moving to travis/appveyor and configure them to be able to produce complete rust releases. The steps to that are part of this tracking issue. The faster we get through this the faster we can have the rls just work!
Beyond that is less clear, but we’ll be reworking the release process,
and packaging the rls, the book, clippy, and other things. We can
begin adding those to the build in relatively short order, once we
--enable-platform flag, even if we don’t begin distributing
them immediately. Here are some of the tasks that need to be done:
- add rls submodule, add to rustbuild, build / test when --enable-platform
- teach rustbuild to create the rls installer
- add clippy submodule, add to rustbuild, build / test when --enable-platform
- teach rustbuild to create the clippy installer
- add new book submodule, removing / deprecating the old