Rust release infrastructure changes and RLS packaging


The plan of record is to move our release infrastructure from buildbot to Travis/AppVeyor plus a small sign+release bot. As part of that process we will be incorporating the RLS into the build, and releasing it via rustup. We will deliver the RLS via rustup and not cargo because rustup correctly pairs tightly-coupled compiler components, and has binary packaging. For the RLS beta and initial release the RLS will be an optional component. This document describes the steps to get there. Let me know what you think.

I want to make it very clear the impact these changes have on Rust development and users. The RLS will be an optional part of the Rust build, landing patches to rust-lang/rust will be gated on the RLS working, and users will install the RLS via rustup, not via cargo.

See the end of this for more details on a proposed RLS in-tree maintenance model.

The steps below have all been added to the above tracking issue.

Some constraints:

  • Immediate goal is to create the option to deliver RLS in Rust 1.17 beta, which branches March 16
  • I feel a need to have a forward-looking security story in place, and at least tacit signoff from Firefox releng, before we switch to new release infrastructure, and am skeptical about doing it for 1.17.
  • We should allow plenty of time on nightly/beta to evaluate the bins produced by the new process before committing to a stable release.

I think this means we need to be producing the RLS packages by March 16. I suggest we plan on releasing the RLS via the buildbot system, and not blocking the RLS on a conversion to the new release model.


  • Figure out rls submodule maintenance model
    • in-tree vs. out-of-tree and how
  • Add rls submodule/source to src/
  • Move rls into rust-lang-nursery (tools team)
  • Add a --enable-extended / --enable-platform flag
    • teach bootstrap to only build, test and dist the rls when this flag is set
    • think about the name of this flag because it’s important
  • Add --enable-extended to the Trivas/AppVeyor release builders
    • this will start uploading rls packages to rust-lang-ci
  • Modify rust-buildbot to pass --enable-extended to host dist builds
    • Starts building and uploading the RLS

At this point we’ve got rls sufficiently packaged that we can flip the switch. Next steps are to flip that switch.

  • Make and deploy patch to rust-buildbot to add rls to manifests as optional
  • Make a new nightly to test RLS deployment
  • Add rls to the combined installer via rust-packaging
    • tarballs, .pkgs, .msis
    • so that non-rustup users can access

We’d want to do all the above before the March 16 branch for 1.17 beta.

At this point we’re delivering the RLS via the buildbot release builders. Next steps are to follow through on the release builder conversion.

  • Add --enable-dist-msi and --enable-dist-pkg flags to configure
    • move those packaging steps from rust-packaging into the tree as part of make dist
    • remember that the rls is not included when --enable-extended is not enabled… hm
  • Again in-tree, create a new build target, dist src/publish-ci-build (or something) to do the final steps of arranging the bins, signing, building the manifest, and uploading to
    • This is in-tree because it gives us a path for others being able to produce their own rustup dist servers, and also because it’s just better to have all the code for packaging in the same git commit.
    • Being in-tree has the complication that we have to run ./configure even though we don’t need to build any Rust…
    • All the steps need to (eventually) be discrete so that people who want to build manifests without our CI infrastructure can do so, but for our purposes we can have one dist publish-ci-build
    • ‘publish-download’ step:
      • for any given channel, download the appropriate bins from rust-lang-ci s3 bucket, put them in the right file system layout
      • this might want to verify hashes and/or some intermediate signature
    • ‘publish-hash’ step:
    • ‘publish-signing’ step:
    • ‘publish-manifest-build’ step:
    • ‘publish-upload’ step:
  • Build a docker container that runs a the above as a service
    • publish to some staging area for testing before we turn it on in prod
    • batch job every night
    • cron job in a docker container?
  • Set up aws machine to run that service
    • let it run in parallel to existing build service
    • test with RUSTUP_DIST_SERVER
  • port cargo build to the rust extended build system
    • simultaneously turn off cargo’s existing publication
  • modify docker images to use correct old glibcs
    • keep using crosstool ng
  • disable llvm assertions for all DEPLOY travisveyor configs
  • switch over to the new release builds, turn off buildbot

RLS maintenance model

Right now it looks like we’re going to be including the RLS in the build as a submodule, not moving the RLS code directly in tree. This is the preference of @nrc and @jonathanturner because it lets the RLS maintain its own issue tracker and PR queue, which is better for their development velocity and team morale (I think this is similar reasoning for why we are moving toward having seperate repos for the docs).

Being a submodule that is tightly coupled to the compiler though makes maintenance difficult though - every time you break an internal API that the RLS uses, then somehow the RLS repo needs to be updated and the rust RLS submodule needs to be updated in one operation that appears ‘atomic’ from either’s commit history. The good news though is that @nrc says the RLS uses relatively stable APIs and he doesn’t expect frequent RLS breakge. If you think about how often the syntax crate breaks downstream, we might expect RLS to break less frequently.

So what will happen with PRs that break the RLS? Here’s a sketch of the best idea we have for the process.

  • The RLS repo is set up such that it has a branch for each Rust release, like rust-1.14.0. This provides backport targets when the RLS has bugs that need to be patched during beta, and is how cargo is currently set up. This won’t actually matter for rust patches against master, as in the following example, but will matter for patches against beta.
  • Contributor submits a patch to master and breaks RLS
  • Reviewer instructs them on how to set up their workspace to test and upgrade the RLS. Probably it’s complex enough that we will just need a doc to link to.
  • Contributor makes and submits RLS patch against RLS master, while updating their rust pr to point the submodule to that commit.
  • RLS merges their patch to a temporary branch so that the original rust pr can find it by submodule sha.
  • Rust PR gets approved and merged.
  • RLS merges their temporary branch into master

As long as that last step succeeds, this sequence ensures that all RLS commits checked into Rust continue to exist in the RLS history.

The mechanics of this sequence seem about as simple as one might achieve, but who is responsible for what is unclear to me. We both want the Rust developers to have some responsibility to keep RLS building (it is going to be part of the Rust product so they must be on-board with delivering it), but we really can’t make it burdensome on contributors.

One other step will be involved during the release process: every time a Rust branch is promoted fram master to beta, the RLS needs to get a new rust-X.Y.Z branch. Same process as cargo today.


If you think about how often the syntax crate breaks downstream, we might expect RLS to break less frequently

Hopefully an order of magnitude less frequently.


Oh here’s another challenge. Right now the set of proxy binaries installed by rustup is fixed, they are all provided by required packages and configured at rustup install time. The rls is a new, optional binary that rustup doesn’t know about.

So rustup is going to need to be modified to create the rls proxy in ~/.cargo/bin. I’ll probably just adjust the logic so that any time rustup installs a binary to the sysroot/bin folder it also ensures a proxy exists. There are some uninstall questions too. But no matter what happens users will need to do a rustup update to get a working rls. So that’s something that should be done soon to get the update out there.