Tweaking how betas are produced

tools and infrastructure
1

Good morning! I’ve been thinking recently that our beta process is falling a little behind and was wondering if we could help automate it a bit more to require less human intervention. I was motivated to think about this when a reported regression was later fixed on master. Later on this was fixed on beta but unfortunately we never released a new beta! In other words, we merged fixes to beta but didn’t have a chance to validate that the issues were actually fixed, but thankfully we got lucky with the stable release!

Currently we release a new beta whenever the version number changes. The version number is a mysterious constant in the build system that’s manually updated whenever we’d like to release a new beta. Otherwise if new changes are merged to beta they won’t actually get released! To me this seems ripe for automation!

Continuous betas

I’d proposed that we continuously and automatically release betas in the same manner as nightly releases. That is, whenever a backport to beta is r+'d we’d automatically release the beta without any human intervention required (aka you hit r+, wait 24 hours, rustup update, and there’s the new beta). This I think will help ensure that we never get into this situation again where a fix is merged for over a week but we never get around to testing whether the underlying issue is fixed!

The first question this brings up is “how do we number betas?” I originally was thinking we would just drop the prerelease version and work similarly to nightlies where the git rev is the version sorta. @kennytm had a great idea though in that a better way may be to use git rev-list --count, basically counting git commits. This would give us a human-readable number which is often useful.

To do this we’d calculate the prerelease version instead of encoding it into the build system, for example:

$ git rev-list --merges --count `git merge-base origin/beta origin/master`...origin/beta
2

So the current beta branch (2a65c6a54c8f5e7fd53d0aecef479d5d321b7f2f) would be listed as beta.2 (2 PRs have been merged).

Publishing betas

Once we update the beta version number to change automatically, we’ll also need to update how betas are published via rust-central-station. Unfortuantely the way we’ve structured everything we can only actually produce one beta a day. If we did more than that then the beta artifacts would actually overwrite each other and we’d lose data!

A relatively easy solution to this, however, would be to basically just adjust the cron job to release a beta once a day instead of once an hour (release if a change is detected, that is). That way we can guarantee we’ll never overwrite data but we’ll still get betas out promptly.

List of changes

I believe that automating the release of betas will require two PRs:

  • First we’ll need to delete CFG_PRERELEASE_VERSION as a constant and switch to calculating it whenever we need it. It would be calculated probably via the git command I listed above. This may end up being pretty tricky, though, because we by default try to do shallow clones on CI. We’ll need to figure out how to have the necessary git information on CI or otherwise be able to query for it. Maybe we can use GitHub’s HTTP API for this? Input would be welcome here!
  • Next we’ll need to adjust the cron job for rust-central-station, which should hopefully be as easy as just changing a line!

I’m curious to hear what others think of this! If you’d like to volunteer to help out implementing this also please just let me know!

12 Likes
2

Oh no, shallow clone ruined everything :sweat_smile:. We could use git fetch --deepen=N to increase the shallow clone depth, but we need to fetch both beta and master branch to calculate the merge base. Doing so much just to calculate a release number seems out of proportion.

Since we are going to release a beta at most once a day, perhaps we could just derive the number using the date modulo the 6-week cycle:

$ echo $(( ($(date +%s) / 86400 - 21) % 42 ))
6

Edit: using date of commit instead of date +%s, for sure.

3

Beta builds happen pretty rarely though, right? And a git fetch is slow but not like half-hour-slow I think? I’d agree yeah an alternative numbering scheme would be fine (that doesn’t need git data) but fetching just on betas probably wouldn’t be too hard to implement.

4

The API looks good enough to me:

curl -i https://api.github.com/repos/rust-lang/rust/compare/master...beta
1 Like
5

It should be fast, though there's a chance of spurious network error. OTOH, spurious network error isn't a serious issue since the error have to happen very early, so we at most waste 10 minutes.

6

@cuviper ah interesting! You’re thinking we could parse out the merge_base_commit there and just count commits from there?

That way we could probably set the clone depth to something like 100 and I doubt we’ll ever have >100 patches on beta (and if so we can just bump that number)

7

I don't think you even need that -- we could directly use the total_commits. Or if you really only want to count merge commits, then we could go through the given commits array and count those with multiple parents. The git checkout can still be completely shallow.

2 Likes
8

Oh nice, in that case this sort of counting sounds perfect!

9

I’ve sent a PR to rust-lang/rust and rust-central-station for these changes.

1 Like
10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.