At the workweek in August we discussed making changes to the release process1, primarily moving to a branching ‘release train’ model, similar to that used by Firefox and other projects. Conceptually, this is a fairly minor change from our current time-based releases, but because it introduces more active development branches, it requires more automation and more process.
There would always be three branches in development: ‘nightly’, ‘beta’, and ‘stable’. Nightly is exactly as today, and where development occurs; a separate ‘beta’ branch provides time for vetting a release before it gets wide use. Each release cycle beta gets promoted to stable (the release), and master gets promoted to beta.
The perceived benefits of this model are a few:
It provides a window for testing the next release before committing to it. Currently we release straight from the (very active) master branch, with almost no testing.
It provides a window in which library developers can test their code against the next release, and - importantly - report unintended breakage of stable features.
It provides a potential opportunity for us to test unstable features in the nightly release channel, while disabling them in the stable channel.
This proposal lays out the workflow for dealing with three release channels: nightly, beta, and stable; particularly focusing on the buildbot setup, as well as the human process required. At this time I am not proposing further changes to bors to account for release channels, meaning that integration for the beta and stable channels will, in the short-term, require manual intervention. The proposed setup is intended to be compatible with a borsified future. This proposal does not deal with the possibility of turning off unstable features in stable releases; that can be dealt with independently.
Automation and process flow changes
I’m going to use the term ‘distribution set’ to refer to the entire set of artifacts we create for releases/nightlies.
At the switch to a new release cycle we do the following:
- Push the
stable, retiring the current stable branch
- Push the
master(aka ‘nightly’) branch to
- Create a beta ‘distribution set’ (the new beta)
- Create a stable ‘distribution set’ (the new release)
- Do all the other release time activites for stable
- Bump the version number on master
We keep the nightly branch named ‘master’ for familiarity.
Unlike the current process, nobody will need to manually append
to version numbers during the beta period. The default build will just
-pre, unless configure switches (via automation)
tell the build system that we’re doing a nightly, beta, or stable
release build, in which case the build system automatically appends
-beta, or nothing to the version.
The master (nightly) branch
People continue to submit PR’s against master by default. The process on master continues just as it does today, with bors automatically doing integration. Nightlies continue to be produced on a 24-hour schedule as today.
The beta branch
beta branch holds the code for the current beta, and like the
master branch today it is not where the actual testing happens;
there is a pre-commit integration step before landing onto
A new set of ‘auto-beta’ builders are configured as the current 'auto’ builders are - every time a commit lands on ‘auto-beta’ it will build and test on all our configurations.(The current ‘auto’ builders are renamed ‘auto-master-*’ for consistency, and bors changed to push to the ‘auto-master’ branch).
A new set of ‘dist-beta’ builders mirrors the current 'dist2’
builders: a manual push to the
dist-snap-beta branch will trigger a
beta dist build, producing all our various artifacts. This is again a
manual step to push out a beta build. The configuration for these will
involve telling the build system to append
-beta to versions, as
today’s nightly builders append
-nightly. (The current 'dist2’
builder is renamed ‘dist-stable’ for consistency).
See ‘Distribution set upload’ below for further changes to the upload process.
The beta commit process
During the release cycle beta binaries will be available for testing. We’ll encourage library developers to work off of this branch.
Beta testers will want a way to specify that a bug affects
beta. GitHub offers no mechanism for this, since bug reporters can’t
tag issues. The development policy will be updated to state that bug
reporters should tag issues titles with the word '[BETA]
if they have encountered the issue while testing the beta; triagers will then apply theC-beta` tag (‘beta channel’).
Any PR’s fixing bugs in beta should also go into master. We tell
people to continue filing PR’s against master, but indicate that this
is a beta fix (again GitHub provides no mechanism for this since users
can’t tag, so they should write
[BETA] in the title). The bug is
reviewed and approved as normal. At the end of the day, somebody
reviews all landed beta PR’s, rolls them up and pushes them to the
’auto-beta’ branch, where they will be tested.
In the morning, if the previous night’s beta push succeeded, then somebody will push ‘auto-beta’ to both ‘beta’ and to ‘dist-beta’ for distribution.
The stable branch
Automation and process for the stable branch is essentially identical to the beta branch.
The only difference is that every stable release must bump the version number, unlike nightlies or betas. Bumping the version number will remain a manual process.
It’s not clear yet what commitments we want to make yet to maintaining short-term or long-term stable point releases, and that is not the subject of this proposal.
Version and artifact naming
Today’s nightlies are versioned with the
0.12.0-pre-nightly. Under this proposal the
-pre is dropped
and nightlies are called e.g.
0.12.0-nightly. Similarly, betas
are called e.g.
Nightly and beta distribution artifacts, like today, will not contain
the version number, but only the ‘nightly’ name,
rust-nightly-i686-apple-darwin.tar.gz. This reinforces that
nightlies and betas are transient stepping stones to stable releases.
Once a new beta is released the old is gone forever. It also reduces
the burden of modifying the version number for each dist release,
which for nightlies happens every day, and betas happens more frequently
Stable releases have no suffix, and the dist artifacts contain the complete version number, for archival purposes.
Builds from source will automatically attach
-pre to the version number
to reinforce that they are not official releases. Likewise for the artefacts
produced by dist-builds from source.
Distribution set upload
Today our bots upload the dist artifacts from each builder as they are finished. This causes problems when not all dist builders succeed, and our artifacts become ‘out-of-sync’, with some being one one revision and others on another.
To solve this problem for the beta and stable channels, which require
manual intervention to make a release, the beta and stable dist
builders will upload not to the
dist/ directory of our s3 bucket but
dist-staging-stable/. A manual file move
is required to complete the process.
Branch and builder summary
Ok, that’s a lot to digest. Here’s the final set up of builder families and branches.
- master - ‘nightly’
- auto-master - bors mastert integration branch
- dist-snap-beta - for manually creating beta distribution sets
- dist-snap-stable - for manually creating stable dist sets (same as dist-snap today)
- try-master - the current try branch, without --disable-unstable
- try-beta - a try branch with --disable-unstable
- snap-stage3 - today’s snap branch
- auto-master-* - Tests bors’ commits the ‘auto-master’ branch
- auto-beta-* - Tests commits to the ‘beta’ branch
- auto-stable-* - Tests commits to the ‘stable’ branch
- nightly-master-* - Nightly dist builds of ‘master’, as today
- dist-beta-* - Manual dist builds of the ‘dist-snap-beta’ branch, in ‘beta’ configuration
- dist-stable-* - Manual dist builds of the ‘dist-snap-stable’ branch, in ‘stable’ configuration
- snap3-* - Snapshots as today
We will have over 100 builders for testing Rust. Note that this won’t much affect the number of VM’s we require as the three release channels will share VM’s.