RFC: Impending changes to the release process

1

At the workweek in August we discussed making changes to the release process1, primarily moving to a branching ‘release train’ model, similar to that used by Firefox and other projects. Conceptually, this is a fairly minor change from our current time-based releases, but because it introduces more active development branches, it requires more automation and more process.

There would always be three branches in development: ‘nightly’, ‘beta’, and ‘stable’. Nightly is exactly as today, and where development occurs; a separate ‘beta’ branch provides time for vetting a release before it gets wide use. Each release cycle beta gets promoted to stable (the release), and master gets promoted to beta.

The perceived benefits of this model are a few:

  • It provides a window for testing the next release before committing to it. Currently we release straight from the (very active) master branch, with almost no testing.

  • It provides a window in which library developers can test their code against the next release, and - importantly - report unintended breakage of stable features.

  • It provides a potential opportunity for us to test unstable features in the nightly release channel, while disabling them in the stable channel.

This proposal lays out the workflow for dealing with three release channels: nightly, beta, and stable; particularly focusing on the buildbot setup, as well as the human process required. At this time I am not proposing further changes to bors to account for release channels, meaning that integration for the beta and stable channels will, in the short-term, require manual intervention. The proposed setup is intended to be compatible with a borsified future. This proposal does not deal with the possibility of turning off unstable features in stable releases; that can be dealt with independently.

Automation and process flow changes

I’m going to use the term ‘distribution set’ to refer to the entire set of artifacts we create for releases/nightlies.

At the switch to a new release cycle we do the following:

  • Push the beta branch to stable, retiring the current stable branch
  • Push the master (aka ‘nightly’) branch to beta
  • Create a beta ‘distribution set’ (the new beta)
  • Create a stable ‘distribution set’ (the new release)
  • Do all the other release time activites for stable
  • Bump the version number on master

We keep the nightly branch named ‘master’ for familiarity.

Unlike the current process, nobody will need to manually append -pre to version numbers during the beta period. The default build will just always append -pre, unless configure switches (via automation) tell the build system that we’re doing a nightly, beta, or stable release build, in which case the build system automatically appends -nightly, -beta, or nothing to the version.

The master (nightly) branch

People continue to submit PR’s against master by default. The process on master continues just as it does today, with bors automatically doing integration. Nightlies continue to be produced on a 24-hour schedule as today.

The beta branch

The beta branch holds the code for the current beta, and like the master branch today it is not where the actual testing happens; there is a pre-commit integration step before landing onto beta.

A new set of ‘auto-beta’ builders are configured as the current 'auto’ builders are - every time a commit lands on ‘auto-beta’ it will build and test on all our configurations.(The current ‘auto’ builders are renamed ‘auto-master-*’ for consistency, and bors changed to push to the ‘auto-master’ branch).

A new set of ‘dist-beta’ builders mirrors the current 'dist2’ builders: a manual push to the dist-snap-beta branch will trigger a beta dist build, producing all our various artifacts. This is again a manual step to push out a beta build. The configuration for these will involve telling the build system to append -beta to versions, as today’s nightly builders append -nightly. (The current 'dist2’ builder is renamed ‘dist-stable’ for consistency).

See ‘Distribution set upload’ below for further changes to the upload process.

The beta commit process

During the release cycle beta binaries will be available for testing. We’ll encourage library developers to work off of this branch.

Beta testers will want a way to specify that a bug affects beta. GitHub offers no mechanism for this, since bug reporters can’t tag issues. The development policy will be updated to state that bug reporters should tag issues titles with the word '[BETA]if they have encountered the issue while testing the beta; triagers will then apply theC-beta` tag (‘beta channel’).

Any PR’s fixing bugs in beta should also go into master. We tell people to continue filing PR’s against master, but indicate that this is a beta fix (again GitHub provides no mechanism for this since users can’t tag, so they should write [BETA] in the title). The bug is reviewed and approved as normal. At the end of the day, somebody reviews all landed beta PR’s, rolls them up and pushes them to the ’auto-beta’ branch, where they will be tested.

In the morning, if the previous night’s beta push succeeded, then somebody will push ‘auto-beta’ to both ‘beta’ and to ‘dist-beta’ for distribution.

The stable branch

Automation and process for the stable branch is essentially identical to the beta branch.

The only difference is that every stable release must bump the version number, unlike nightlies or betas. Bumping the version number will remain a manual process.

It’s not clear yet what commitments we want to make yet to maintaining short-term or long-term stable point releases, and that is not the subject of this proposal.

Version and artifact naming

Today’s nightlies are versioned with the -nightly suffix, e.g. 0.12.0-pre-nightly. Under this proposal the -pre is dropped and nightlies are called e.g. 0.12.0-nightly. Similarly, betas are called e.g. 0.12.0-beta.

Nightly and beta distribution artifacts, like today, will not contain the version number, but only the ‘nightly’ name, e.g. rust-nightly-i686-apple-darwin.tar.gz. This reinforces that nightlies and betas are transient stepping stones to stable releases. Once a new beta is released the old is gone forever. It also reduces the burden of modifying the version number for each dist release, which for nightlies happens every day, and betas happens more frequently than stable.

Stable releases have no suffix, and the dist artifacts contain the complete version number, for archival purposes.

Builds from source will automatically attach -pre to the version number to reinforce that they are not official releases. Likewise for the artefacts produced by dist-builds from source.

Distribution set upload

Today our bots upload the dist artifacts from each builder as they are finished. This causes problems when not all dist builders succeed, and our artifacts become ‘out-of-sync’, with some being one one revision and others on another.

To solve this problem for the beta and stable channels, which require manual intervention to make a release, the beta and stable dist builders will upload not to the dist/ directory of our s3 bucket but to dist-staging-beta/ and dist-staging-stable/. A manual file move is required to complete the process.

Branch and builder summary

Ok, that’s a lot to digest. Here’s the final set up of builder families and branches.

Branches:

  • master - ‘nightly’
  • beta
  • stable
  • auto-master - bors mastert integration branch
  • dist-snap-beta - for manually creating beta distribution sets
  • dist-snap-stable - for manually creating stable dist sets (same as dist-snap today)
  • try-master - the current try branch, without --disable-unstable
  • try-beta - a try branch with --disable-unstable
  • snap-stage3 - today’s snap branch

Builders:

  • auto-master-* - Tests bors’ commits the ‘auto-master’ branch
  • auto-beta-* - Tests commits to the ‘beta’ branch
  • auto-stable-* - Tests commits to the ‘stable’ branch
  • nightly-master-* - Nightly dist builds of ‘master’, as today
  • dist-beta-* - Manual dist builds of the ‘dist-snap-beta’ branch, in ‘beta’ configuration
  • dist-stable-* - Manual dist builds of the ‘dist-snap-stable’ branch, in ‘stable’ configuration
  • snap3-* - Snapshots as today

We will have over 100 builders for testing Rust. Note that this won’t much affect the number of VM’s we require as the three release channels will share VM’s.

2

I think a release candidate is a better name for a beta.

Generally, people would expect betas to be somewhat different from the stable releases that come after them. but the final RCs (the last ones that come before the stable releases) are expected to be identical to the stable releases.

And the candidate part means that, if serious bugs are found in the RC, then in the next release cycle, only a new RC will be released, but the new stable release (that should be the old RC) will be skipped. (Sans bugfixes, the new RC should be backward-compatible with the old RC.)

3

Also I think an “Aurora” channel can still have its use. The releases on this channel will be like the point releases that we have today, in that they are just snapshots of master with no additional testing. They will be released in sync with beta(rc)/stable, but can be skipped at will.

So, what is the point of Aurora? “Marketing”.

Every time a major language feature/standard library addition is deemed “mostly usable”, we do an Aurora release in the next cycle. So more people will be aware of the new features and be more willing to test them (not with the Aurora but with the following nighties).

Or maybe Aurora would be unnecessary after all.

And why do we disable the unstable/experimental parts in beta(rc)/stable releases? People should be able to opt-in, and if they do so, they are taking their own risks, and this is fine. (Library authors will be generally advised against this practice, of course.)

4

@CloudiDust thanks for the feedback!

Great observation that ‘release candidate’ is also a good name for the beta. I’m not clear on the distinction you are making between ‘aurora’ and ‘nightly’: ‘snapshot of master’ is what nightly is.

An important consideration is that the proposal to have three release channels at 6 week intervals already greatly increases the ongoing effort necessary to keep the Rust project moving. More channels increases that burden.

As to disabling unstable features I mentioned that this proposal and this thread is not the place to discuss that. Right now I am just working on the mechanics of release channels and do not want to get bogged down in what is an independent and more controversial topic.

5

I think aurora can be used for “marketing” purposes. They are not expected to be actually used, but are used as milestones “documenting” the changes to the experimental/unstable bits between release cycles, like the current point releases.

But on second thought, simply doing “what’s new in nighties” posts in sync with beta(rc)/stable releases is enough.

6

EDIT: Oh I think I misunderstood the proposal. A version bump happens after a stable is released. So the following discussion is not correct.

I think if we go with the name release candidate, then it is natural for us to exactly specify for which release the candidate is for. (Actually, I think people would expect betas to do so too.) So I propose the following change:

Stables do not bump version numbers, but Betas/RCs do. Each Beta/RC should either:

  1. Bump the patch version by 1;
  2. Bump the minor version by 1 and reset the patch version;
  3. Bump the major version by 1 and reset the minor and patch versions.

All nighties after a Beta/RC also get their versions bumped.

In the next circle, the current Beta/RC becomes the new Stable by dropping the -beta or -rc suffix, and a new Beta/RC with a bumped version number will be created.

If a stable release is skipped, the new Beta/RC would still bump the version number. So it is possible to have “version holes” between stable releases. Rust 1.1.3 may be followed by Rust 1.1.5, not Rust 1.1.4. This is like Java where the `Update" patch version number may not be successive.

Personally I believe this is a more natural scheme.

Or we can go with 1.1.4-beta1, which gets abandoned and followed by 1.1.4-beta2, which is promoted to be the stable 1.1.4, so there will be no holes in the stable version numbers. But the point is, let Betas/RCs bump version numbers, not Stables.

7

Still I think a rule should be added: Even if a stable release is skipped, we should still do a version bump (or a beta/rc suffix number bump). Otherwise we will end up with two different Betas/RCs with the same version string, and this would be a mistake.

8

@CloudiDust Can you restate why having two different betas with the same version would be a mistake? The main downside I see is that bug reports can’t point to a unique version, but they can still specify the commit ID since we include that.

9

Oh, I forgot the commit ID part. Then this will not be a problem.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.