At the workweek in August we discussed making changes to the release
process1, primarily moving to a branching ‘release train’ model,
similar to that used by Firefox and other projects. Conceptually, this
is a fairly minor change from our current time-based releases, but
because it introduces more active development branches, it requires
more automation and more process.
There would always be three branches in development: ‘nightly’,
‘beta’, and ‘stable’. Nightly is exactly as today, and where
development occurs; a separate ‘beta’ branch provides time for vetting
a release before it gets wide use. Each release cycle beta gets
promoted to stable (the release), and master gets promoted to beta.
The perceived benefits of this model are a few:
-
It provides a window for testing the next release before committing
to it. Currently we release straight from the (very active) master
branch, with almost no testing.
-
It provides a window in which library developers can test their code
against the next release, and - importantly - report unintended
breakage of stable features.
-
It provides a potential opportunity for us to test unstable features
in the nightly release channel, while disabling them in the stable
channel.
This proposal lays out the workflow for dealing with three release
channels: nightly, beta, and stable; particularly focusing on the
buildbot setup, as well as the human process required. At this time I
am not proposing further changes to bors to account for release
channels, meaning that integration for the beta and stable channels
will, in the short-term, require manual intervention. The proposed
setup is intended to be compatible with a borsified future. This
proposal does not deal with the possibility of turning off unstable
features in stable releases; that can be dealt with independently.
Automation and process flow changes
I’m going to use the term ‘distribution set’ to refer to the entire
set of artifacts we create for releases/nightlies.
At the switch to a new release cycle we do the following:
- Push the
beta branch to stable, retiring the current stable branch
- Push the
master (aka ‘nightly’) branch to beta
- Create a beta ‘distribution set’ (the new beta)
- Create a stable ‘distribution set’ (the new release)
- Do all the other release time activites for stable
- Bump the version number on master
We keep the nightly branch named ‘master’ for familiarity.
Unlike the current process, nobody will need to manually append -pre
to version numbers during the beta period. The default build will just
always append -pre, unless configure switches (via automation)
tell the build system that we’re doing a nightly, beta, or stable
release build, in which case the build system automatically appends
-nightly, -beta, or nothing to the version.
The master (nightly) branch
People continue to submit PR’s against master by default. The process
on master continues just as it does today, with bors automatically
doing integration. Nightlies continue to be produced on a 24-hour
schedule as today.
The beta branch
The beta branch holds the code for the current beta, and like the
master branch today it is not where the actual testing happens;
there is a pre-commit integration step before landing onto beta.
A new set of ‘auto-beta’ builders are configured as the current 'auto’
builders are - every time a commit lands on ‘auto-beta’ it will build
and test on all our configurations.(The current ‘auto’ builders are
renamed ‘auto-master-*’ for consistency, and bors changed to push to
the ‘auto-master’ branch).
A new set of ‘dist-beta’ builders mirrors the current 'dist2’
builders: a manual push to the dist-snap-beta branch will trigger a
beta dist build, producing all our various artifacts. This is again a
manual step to push out a beta build. The configuration for these will
involve telling the build system to append -beta to versions, as
today’s nightly builders append -nightly. (The current 'dist2’
builder is renamed ‘dist-stable’ for consistency).
See ‘Distribution set upload’ below for further changes to the upload
process.
The beta commit process
During the release cycle beta binaries will be available for
testing. We’ll encourage library developers to work off of this
branch.
Beta testers will want a way to specify that a bug affects
beta. GitHub offers no mechanism for this, since bug reporters can’t
tag issues. The development policy will be updated to state that bug
reporters should tag issues titles with the word '[BETA]if they have encountered the issue while testing the beta; triagers will then apply theC-beta` tag (‘beta channel’).
Any PR’s fixing bugs in beta should also go into master. We tell
people to continue filing PR’s against master, but indicate that this
is a beta fix (again GitHub provides no mechanism for this since users
can’t tag, so they should write [BETA] in the title). The bug is
reviewed and approved as normal. At the end of the day, somebody
reviews all landed beta PR’s, rolls them up and pushes them to the
’auto-beta’ branch, where they will be tested.
In the morning, if the previous night’s beta push succeeded, then
somebody will push ‘auto-beta’ to both ‘beta’ and to ‘dist-beta’ for
distribution.
The stable branch
Automation and process for the stable branch is essentially identical
to the beta branch.
The only difference is that every stable release must bump the
version number, unlike nightlies or betas. Bumping the version
number will remain a manual process.
It’s not clear yet what commitments we want to make yet to maintaining
short-term or long-term stable point releases, and that is not the
subject of this proposal.
Version and artifact naming
Today’s nightlies are versioned with the -nightly suffix,
e.g. 0.12.0-pre-nightly. Under this proposal the -pre is dropped
and nightlies are called e.g. 0.12.0-nightly. Similarly, betas
are called e.g. 0.12.0-beta.
Nightly and beta distribution artifacts, like today, will not contain
the version number, but only the ‘nightly’ name,
e.g. rust-nightly-i686-apple-darwin.tar.gz. This reinforces that
nightlies and betas are transient stepping stones to stable releases.
Once a new beta is released the old is gone forever. It also reduces
the burden of modifying the version number for each dist release,
which for nightlies happens every day, and betas happens more frequently
than stable.
Stable releases have no suffix, and the dist artifacts contain the complete
version number, for archival purposes.
Builds from source will automatically attach -pre to the version number
to reinforce that they are not official releases. Likewise for the artefacts
produced by dist-builds from source.
Distribution set upload
Today our bots upload the dist artifacts from each builder as they are
finished. This causes problems when not all dist builders succeed,
and our artifacts become ‘out-of-sync’, with some being one one
revision and others on another.
To solve this problem for the beta and stable channels, which require
manual intervention to make a release, the beta and stable dist
builders will upload not to the dist/ directory of our s3 bucket but
to dist-staging-beta/ and dist-staging-stable/. A manual file move
is required to complete the process.
Branch and builder summary
Ok, that’s a lot to digest. Here’s the final set up of builder
families and branches.
Branches:
- master - ‘nightly’
- beta
- stable
- auto-master - bors mastert integration branch
- dist-snap-beta - for manually creating beta distribution sets
- dist-snap-stable - for manually creating stable dist sets (same as dist-snap today)
- try-master - the current try branch, without --disable-unstable
- try-beta - a try branch with --disable-unstable
- snap-stage3 - today’s snap branch
Builders:
- auto-master-* - Tests bors’ commits the ‘auto-master’ branch
- auto-beta-* - Tests commits to the ‘beta’ branch
- auto-stable-* - Tests commits to the ‘stable’ branch
- nightly-master-* - Nightly dist builds of ‘master’, as today
- dist-beta-* - Manual dist builds of the ‘dist-snap-beta’ branch, in ‘beta’ configuration
- dist-stable-* - Manual dist builds of the ‘dist-snap-stable’ branch, in ‘stable’ configuration
- snap3-* - Snapshots as today
We will have over 100 builders for testing Rust. Note that this won’t
much affect the number of VM’s we require as the three release
channels will share VM’s.
