Release channels, git branching, and the release process

[brson]

Now that the beta release channel is operational we have some procedural changes to work through in order to integrate betas into the process. This document will explain the branching strategy for Rust release channels, as well as the process reviewers and release managers will use to integrate fixes to beta and stable releases. It is mostly targeted at those who review patches to rust-lang/rust, as they will have new responsibilities to identify patches that should be backported to the current beta (and subsequent stable) release.

Branching

This section will talk specifically about how the git branching works with release channels. Other aspects of the workflow will be illustrated later.

I’m going to be using graphs like this to show the branching:

master: A - B - C
             \
beta:         D - C'

The branch name is on the left, and commits are capital letters. Primes (') are used to indicate backported (cherry-picked) commits. In this diagram release branched off of master at B by applying D, and C' was backported from C on master.

Rust has more than three commits on master so far, but we’ll pretend like all that history doesn’t exist for simplification purposes.

The summary is that Rust has three active branches: master, beta, and stable. Every release cycle, beta and stable are reset from master and beta, respectively, by a force-push. Fixes to beta and stable are applied by cherry picking.

The rest of this section will illustrate with examples.

We’ll start off by showing how master and beta relate, which reflects how development is working right now, prior to 1.0. Then we’ll add in stable development as well.

Here’s Rust development on the master branch:

master: A - B

At the beginning of each cycle the contents of the master branch are pushed to beta.

master: A - B
             \
beta:         .

I’ve used the dot here to indicate that B was pushed from master to beta without any changes. At this point master and beta are identical.

All regular work will keep going on the master branch:

master: A - B - C - D
             \
beta:         . 

This is the important part of this strategy. Everything lands on master first. master is the canonical source of truth from which all else flows. New -beta branches always end up forking off of master.

When a committer determines that a commit should be backported to the beta branch, they’ll open a second PR against the beta branch directly. We could also possibly automate this through @bors, though some commits may need massaging, of course. And generally speaking, after 1.0, backports should be relatively rare.

master: A - B - C - D - E
             \
beta:         . - E'

Now beta's history has diverged from master. And work continues…

master: A - B - C - D - E - F
             \
beta:         . - E'

And other commits are backported:

master: A - B - C - D - E - F - G
             \
beta:         . - E' - G'

I’m going to shorten the commits a tad to make it smaller:

master: A - B - C - ... - G
             \
beta:         . - E' - G'

Anyway, then, on May 15, a release happens and a new cycle begins.

master: A - B - C - ... - G - H
             \                 \
beta:         \                 .
               \
stable:         . - E' - G'

The branch that was beta is force-pushed to stable, and master force-pushed to beta. For the moment master and beta are the same again. beta's history still exists because it is named stable now. At the same time, we’ll sign and tag the new stable branch and call it e.g. 1.0.0, whatever the version number is. Now the commit of the stable release is recorded forever.

master: A - B - C - ... - G - H
             \                 \
beta:         \                 .
               \
stable:         . - E' - G'
                         |
1.0.0:                   .

At this point we’ve started the next development cycle. If this was the 1.0 release, then 1.0 has been released to stable, the future 1.1 is on beta, and the future 1.2 is on nightly (master).

Ideally, the stable release won’t need any further changes. But let’s say we discover a critical security bug, and fix it with I:

master: A - B - C - ... - G - H - I
             \                 \
beta:         \                 . - I'
               \
stable:         . - E' - G' - I'
                         |    |
1.0.0:                   .    |
                              |
1.0.1:                        .

I needs to be backported against every relevant branch. Here beta gets a cherry-pick of I' to fix the issue in the next scheduled stable release (1.1), stable gets the cherry-pick as well, which is immediately released and tagged 1.0.1. The old 1.0.0 tag is untouched because it represents a signed and completed release.

1.1 and beyond

Let’s do some more work, J:

master: A - B - C - ... - G - H - I - J
             \                 \
beta:         \                 . - I'
               \
stable:         . - E' - G' - I'

and backport a new commit to beta:

master: A - B - C - ... - G - H - I - J
             \                 \
beta:         \                 . - I' - J'
               \
stable:         . - E' - G' - I'

Now it’s time to move to the next development cycle and release 1.1. We do the same as before: force push master to beta, and beta to stable. stable is then released and tagged 1.1.0.

master: A - B - C - .... - G - H - I - J
             \                  \       \
beta:         \                  \       .
               \                  \
stable:         \                  . - I' - J'
                 \                          |
1.0.0:            . - E' - G'               |
                           |                |
1.0.1:                     . - I'           |
                                            |
1.1.0:                                      .

Again, what was on beta is now on stable, and beta is force-pushed to be identical to master.

Just for completeness, let’s add one more security patch, K, and apply it.

master: A - B - C - .... - G - H - I - J - K
             \                  \       \
beta:         \                  \       . - K'
               \                  \
stable:         \                  . - I' - J' - K'
                 \                          |    |
1.0.0:            . - E' - G'               |    |
                           |                |    |
1.0.1:                     . - I'           |    |
                                            |    |
1.1.0:                                      .    |
                                                 |
1.1.1:                                           .

Notably, none of these patches were backported to any non-current stable release - it’s only the current stable release that is maintained, at least to begin. In the future, we may add a ‘long-term stable’ channel, where we select one stable release to maintain for an extended support period. In that scenario, an lts channel should be a natural extension of nightly, beta, and stable, but for now we are not considering it.

How to process beta PR’s on GitHub

The previous section described how the git history works. This will describe the human process for applying patches to beta.

At some point there will be automation help for this process, but for now there is not. This section is written assuming that @bors does not help us with the beta branch.

The process begins as it does today - with users submitting PRs against master.

When a reviewer sees a patch that should be backported to beta, they apply the ‘beta-nominated’ tag, and take no further action. At this point the patch undergoes the long process of integrating to master, and may undergo a number of changes. The ‘beta-nominated’ tag stays applied the whole time to remind us to revisit it later.

Patches that should be backported to beta are rare, so reviewers should be selective. Potential beta candidates include fixes for security issues, fixes for regressions, major fixes to new features, and last-minute re-gating of new features due to beta feedback.

Periodically, perhaps once a week, a team member will go through all the closed PRs with the ‘beta-nominated’ tag, cherry-pick them to a local branch based off of beta, remove the ‘beta-nominated’ tag, run tests as they see fit, then push back to beta.

(TODO: Could also open PRs. Could also have ‘beta-nominated’ triage to make decisions about which PRs to actually backport)

Note that the actual testing regimine for ‘beta’ PRs is left open for now. It’s not clear how vigilant we need to be about testing cherry-picks to beta: they’ve already been vetted on master, and they can’t be released without running through the ‘dist’ builders, which provide decent coverage.

At the start of each cycle no PRs should have the 'beta-nominated’ tag.

For stable backports, there is no such nomination process - stable updates are a rare ‘drop-everything’ event and we’ll all know when there’s a patch that needs to be deployed over an existing stable release.

The release process, outlined

At the switch to a new release cycle we do the following:

• Commit the next beta’s release notes to master
• Push the beta branch to stable, retiring the current stable branch
• Push the master (aka ‘nightly’) branch to beta
• Publish a beta ‘distribution set’ (the new beta)
• Publish a stable ‘distribution set’ (the new release)
• Do all the other release time activites for stable, including tagging the commit
• Bump the version number on master
• Create a new value for CFG_FILENAME_EXTRA

The version number bump happens immediately after the development cycle starts, so at all times the nightly, beta, and stable channels reflect the version number they ultimately will become (nightly is $current_release + 0.2, beta is $current_release + 0.1).

Beta release concerns

Each beta within a development cycle needs to have a unique ’prerelease’ version, e.g. the ‘.2’ in ‘1.0.0-beta.2’, for easier identification. These prerelease versions must be updated manually under one scenarios: after each beta is published, a beta-only patch should bump the prerelease version. Because this number is only ever bumped on beta, on master the prerelease version is always ‘.1’.

[brson]

Thanks @steveklabnik for writing much of this.

[nagisa]

To kick the discussion off, I want to ask one thing related to git. Does this scheme mean we will have to git pull -f every 6 weeks when stable is forcibly replaced with beta HEAD and beta is forcibly replaced with master HEAD?

[brson]

@nagisa I think you will only need to force pull if you are on a local tracking branch of beta and stable. If you are developing on master, as most people will be, there will never be a conflict.

That said, I don’t use ‘git pull’ or tracking branches, so I can’t say for sure.

[Manishearth]

think you will only need to force pull if you are on a local tracking branch of beta and stable. If you are developing on master, as most people will be, there will never be a conflict.

This is correct.

Most projects follow this backporting pattern; I see no issue here. Not making PR-makers handle the backports is a good decision IMO -- this stuff can get confusing and it's an unnecessary burden for someone to learn.

[brson]

Added a note that CFG_FILENAME_EXTRA should change every release.

[gus]

Since I see this came up in the weekly meeting (https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2015-04-28.md#snapshots), I’d like to mention it directly here:

It would be very useful for the Debian packaging efforts if we updated the stage0 snapshot to something with the same “breakage generation” as the release we’re about to make, before each release. Specifically, I’d like to be able to build a rustc release with itself (forced to channel=dev so I can “opt in” to the unstable features required by rustc source).

Background: Debian packages need to build without network access, and must be built on the native architecture (no cross-compiled packages are released into the Debian archive).

With my change proposed above, each release looks like this:

• Bundle pre-built stage0 for one arch (eg: x86_64) with rustc source package.
• Use that to build a “bootstrap rustc” that is forced to --release-channel=dev and --host=$target (effectively I’ve just built my own stage0 snapshot for $target)
• On $target, rebuild rustc natively using that bootstrap rustc
• Release that last native build

Without proposed change:

• Bundle stage0 snapshot for every Debian architecture we want to support
• Build natively using the prebuilt stage0 for local architecture
• Release that

… and the issue with the last version is that “every stage0” is pretty big :-/

In the past I’ve asked if we can support building stage1 with the previous channel=stable release and I appreciate this is too restrictive to consider at this point in the Rust maturity curve. Just highlighting that this and the above are two different feature requests, and either one of them is better than none of them for us

[Manishearth]

You can build a rustc release with itself, just skip the stage0 build (or, well, do the stage0 build with the cfgs removed).

A ./configure option to build starting from stage1 given a packaged compiler would be nice.

[gus]

Huh, that is an excellent point. I’ll dig into the Makefiles further and find the right target(s).