Proposal: move to bors-ng

tools and infrastructure
1

bors-ng is a rewrite of bors/homu in Elixir. Interesting things:

  • uses the new GitHub integration APIs.
  • automatic rollup/bisect. No manual action needed to clear the queue.
  • can be used with any CI that posts commit status. This means we can potentially move to TaskCluster at some time.

Concerns: Will the algorithm work well with the flaky CI infrastructure?

1 Like
2

From what I can tell, the automatic rollup is decent, but in my opinion not all that good for us. We build artifacts on every PR merge which are then used for bisection and performance reporting; this makes rollups of PRs which have potential to change perf and introduce accidental breaking changes problematic for us. I donā€™t think thereā€™s really any major advantage beyond the automatic rollups to using bors-ng other than the possibly better handling of various states due to more active maintenance.

3

So are you having problems do to the manual rollups we currently do? Why would procedurally generated rollups be different?

4

When doing manual rollups we (or at least I) try to limit to only simple or less likely to be the bisection cause PRs. Itā€™s possibly not the best approach, but it does work. As far as I can tell, bors-ng by-default rolls up all PRs, not just those marked with rollup. This may be incorrect though.

5

Youā€™re correct.

If you want the artifact cache to work the same way as itā€™s been working, youā€™re going to want parallel building, which sounds like something Travis CI would love :money_mouth: (but Iā€™m going to guess that it would be within Rustā€™s budget, if the code were implemented).

6

BTW, I think Rust would also like the (admittedly very simple, so it shouldn't be that hard to add to homu) "try arguments." Basically, you leave a command like this:

bors try --platform=linux

And bors-ng writes this into the commit message:

Try #NNN: --platform=linux

Your test harness can then use the git cli to grab (and sed to parse?) the commit message and use it to determine which platform to test on. Rust already does this kind of stuff to limit the subset of the test suite that's run when PRs are opened.

Of course, the ideal setup would throw a second Integration into the mix, to make this work:

bors try --cargobomb

That would probably require a custom webhook, since cargobomb is not going to run within Travis.

7

Indeed parallel building is the best way to speed up testing and leave the artifacts intact. This Is best described in the documentation for zuul. But as alexcrichton pointed out ā€œRight now we unfortunately just donā€™t have capacityā€.

The discussion on adding procedurally generated rollups to the existing infrastructure is at homu/issues/102. Although sofar it is mostly me working conceptually. I would love to hear what people are concerned about and what strategies they would or would not find acceptable.

Also one day Iā€™d love to see this in rust, but those projects are still pretty far off.

8

I still find that automatic rollup would be much better, because:

  • Itā€™s hard to predict whether a PR is 100% safe or not. We have oversights.
  • PRs get rolled up all times. This means that we can stay at a relatively idle state, avoiding the rollup mistakes when the queue gets filled with too much PRs (pressure) that can potentially fail.

But well, @notriddle I found the lack of priority a blocker for migration. Letā€™s get it implemented soon. bors-ng is really well maintained :slight_smile:

9

The described batching algorithm does seem like it has a high potential to not work well for Rust, with our long turnaround times, and high intermittent rates. Iā€™d be more comfortable adopting it if it supported our existing workflow and let us gradually experiment with others.

If bors-ng is going to be maintained, and it can support our workflow, and somebody is willing to do the work of the transition, and preferably if it can remain under the venerable ā€˜borsā€™ account, then it might be worth pursuing. Weā€™ve long wanted to replace homu. Iā€™m glad somebody has finally done the work - I didnā€™t realize it was ongoing.

Hereā€™s what it looks like in action: https://bors-app.herokuapp.com/

Sadly thereā€™s no activity in the queue.

1 Like
10

Also re the batching algorithm, we often use bors in ways where we probably donā€™t want batching. For example, just last week I threw a PR at bors several times, fully expecting it to fail, just so I could see what the CI windows results would be.

11

It would be prudent to identify our current bors wishlist, and see how bors-ng will address them. Weā€™ve often discussed changes to homu we want, but donā€™t make because nobody wants to hack on homu.

3 Likes
12
  • Every repository gets itā€™s own queue.
  • Right now, the front dashboard page only shows repos that you have reviewer permissions for. Thatā€™s not going to change: on the public bors-ng instance, where anybody can add their repo, a global ā€œawaiting reviewā€ list would be too long.
  • Right now, the individual repo pages are only accessible to reviewers. That shouldnā€™t be hard to fix, by providing a sort of ā€œread onlyā€ mode when non-reviewers open them.
  • From an adminā€™s POV, I can see that thereā€™s webhooks hits every single day. I should probably set up a public metrics collector?
  • Right now, my main concern is fixing the hit-by-a-bus factor. I assume this would be a high priority for Rust, as a potential user.
13

Hereā€™s an example of what it looks like in RO mode. The PR to enable this is in review, now.

Edit: PR landed and deployed. Hereā€™s the queues for:

1 Like
14

In the context of finding enough people who know the language and are willing to improve the bors-ng codebase, is the usage of elixir expected to be a benefit, a detractor, or neutral?

15

Compared to Rust, I expect it to be neutral as far as finding developers. Compared to Python, however, it probably makes it harder to find developers.

16

I am working on understanding how homu works so I can hack on it. I am starting with batching, I am trying to design something very small and conservative to start with.

If we have a wishlist, then letā€™s get it together so persons can help.

As bors-ng seams better maintained, when it meets most of the requirements then Iā€™d advocate for the switch. Not that I have much say in the matter.

1 Like
17

It does prioritization now. Shout-out to @khodzha for implementing it, BTW!

1 Like
18

@alexcrichton I think itā€™s a good time to migrate to bors-ng. Are you interested in rolling it out on all nursery repositories, or even cargo?

19

Itā€™s already being adopted at crates.io. Be patient; thereā€™s still a few missing features before the bigger rust-lang repos can use it.

2 Likes
20

any progress ?