Homu queue woes, and suggestions on how to fix them

[davidtwco]

Are there any hard numbers on how much time is lost to spurious failures from Travis or Appveyor?

[benh]

The infrastructure team consists exclusively of full-time Mozilla employees?

[JeanMertz]

No, there’s not. That’s the whole point I’m making; only The Infrastructure Team 1 has the ability to deploy fixes and updates to Rust’s servers, and, for infosec reasons, they can’t just outsource it to volunteers.

Right. That’s even better, since you don’t get access to the Kubernetes main node, as it’s fully managed by Google Cloud.

You can’t just set up a Jenkins instance for them, since they aren’t going to trust strangers on the Internet with access to the Rust release pipeline.

I’m not suggesting I would (although I’m willing to do that, if push comes to shove). I’m suggesting I could help others do that, or just in general help flesh out the project scope and the roadmap on how to get there.

At some point, yes, someone would have to do something to get this running.

Well, alright, Jenkins. Not Kubernetes. Same point, though, that you’re adding more load to the infra team.

Yes, true. If learning how Jenkins works (up to a certain level), or finding people to add to the team of volunteers that know how Jenkins works is out of the question, then my whole suggestion of using Jenkins is moot.

[pietroalbini]

Not sure why crowdfunding needs to be reliable over time. if the crowdfunded budget runs out, CI gets slower. Insert coin to receive faster builds.

Thankfully we don’t have this problem

Is there a good place to read up on bors/homu strategies that people have considered? It’s fun to think about and I’m super tempted to try to come up with some microoptimizations, but I’m sure a lot of this has been thought through before…

If you’re interested you (and everyone else!) can find us in the #infra channel on Discord!

No, there’s not. That’s the whole point I’m making; only The Infrastructure Team has the ability to deploy fixes and updates to Rust’s servers, and, for infosec reasons, they can’t just outsource it to volunteers. You can’t just set up a Jenkins instance for them, since they aren’t going to trust strangers on the Internet with access to the Rust release pipeline.

By the way, even inside the infra team only few people have access to more critical machines (like the one doing releases).

Are there any hard numbers on how much time is lost to spurious failures from Travis or Appveyor?

I don’t think we have aggregated numbers, but @kennytm is doing awesome work tracking most of the spurious failures in a spreadsheet. The most common failure we have right now is travis-ci/travis-ci#9696, which is a network failure that happens every day between 6 AM UTC and 7:30 AM UTC since the start of June.

The infrastructure team consists exclusively of full-time Mozilla employees?

Nope, only few of the members are Mozilla employees as far as I know.

[notriddle]

The infrastructure team consists exclusively of full-time Mozilla employees?

No, but I’m pretty sure all of them work for a company or university that sponsors them (Pingcap, Integer32, Rails, and Mozilla are the ones I know of). The really important part, though, is that none of them are strictly anonymous; Mozilla knows their names and occupations, so if any of them pull something, Mozilla can ban them and guarantee that they just don’t rejoin under a new name.

Right. That’s even better, since you don’t get access to the Kubernetes main node, as it’s fully managed by Google Cloud.

GKE only relieves the Infra Team of having to maintain the underlying container orchestrator. But, because Google will not help with Jenkins problems, it’s still on the Infra Team to diagnose any downtime, to figure out whether it’s a Jenkins problem that they need to fix or a Kubernetes problem that they just wait for Google to fix. That is a significant regression compared to the status quo where the Infra Team only have to maintain homu/bors.

[JeanMertz]

GKE only relieves the Infra Team of having to maintain the underlying container orchestrator. But, because Google will not help with Jenkins problems, it’s still on the Infra Team to diagnose any downtime, to figure out whether it’s a Jenkins problem that they need to fix or a Kubernetes problem that they just wait for Google to fix.

Yes, that’s correct, if the current team structure/responsibilities stay the same.

At the same time, because the Jenkins setup is immutable and restarting it is a matter of stopping/starting the “pod/instance” that is running it, which is done via a CLI command, not by accessing the underlying infrastructure, you could widen access to that command if necessary, and reduce the burden on the infra team.

Again, there are downsides, I’m just stating that there can also be significant upsides, and the downsides can be mitigated, with some investment.

That is a significant regression compared to the status quo where the Infra Team only have to maintain homu/bors.

From what I understand from @pietroalbini on the chat (and also what started this thread), Travis isn’t working so great right now, so there’s more than just maintaining homu/bors. If a workable solution can be found that reduces the time needed from any volunteers, that’s obviously the best outcome, but it does sound like Travis will most likely not be a part of that solution.

Either way, I’ll let this discussion rest. I’m still offering a helping hand, also if some other solutions turn out to be a better fit, and I’ll reach out in chat to see where I can help

[graydon]

Zuul sounds like a perfect fit for the Rust project at this point.

Yes, Zuul and the Bors-NG algorithm are similarly optimistic-rollup oriented; though from the look of it (not completely clear from the docs) Zuul looks like it is going to burn more speculative work eagerly by doing every sub-prefix it can immediately and merging the maximal prefix from that set, whereas Bors-NG starts maximal and bisects-down to either a prefix it can commit or a failure it can kick out, which can take longer than the best-case (can burn log(prefix) worth of bisection time), but might make more sense given the limited width of your cluster (i.e. you don’t have enough money to do the Zuul approach).

[graydon]

(To make this more concrete: if you have 50 queued PRs and there’s a bad one at the 15th PR, assuming I’m reading the docs right, Zuul will run 50 jobs in parallel – one per possible prefix – of which the first 14 will pass and the next 36 will fail, and it’ll commit the 15-PR prefix. This is time-optimal but burns a lot of money: 15x speedup for 50x cost. Bors-NG will run one failing 50 PR job, then a failing 25 PR job, then a passing 12 PR job that gets merged. That’s longer than Zuul – took 3 cycles to merge 12 PRs – but it only cost you 3 jobs of CPU time to find that prefix, not 50. 4x speedup for 3x cost amplification. In general the balance depends on the bugginess of your PR queue, and the quality of guesswork going into the existing rollups.)

[alex]

For what it’s worth, Zuul was built for OpenStack, an open source cloud platform. As a result, there were a lot of cloud providers donating compute (HP, Rackspace, probably some others). Hopefully that explains the design decisions.

[graydon]

Hopefully that explains the design decisions

Sure, or like I could totally be misreading the docs! It’s just worth considering that there are a few possible ways to do the optimistic strategy and they will affect both the bill and net speedup in not-necessarily-linear ways.

[Mark_Simulacrum]

I’m fairly certain this graph is accurate (but, if it seems wildly wrong, then perhaps it is – for reference, the code used to generate this is here: https://github.com/Mark-Simulacrum/landing-time; it does not include the gnuplot script and will likely not run on anyone else’s computer).

I’ve downloaded all rust-lang/rust PRs since, well, the repository existed and all associated comments so if there’s other data that would be helpful here I can try and get it out. Currently I’m checking for approval via a simple does the comment contain bors r+ or bors r= which seems to work fine (only one merged PR not approved in the last 6000).

Based on the graph, it looks like most PRs are landing within ~1-2 days with essentially all landing within ~5 days of the last approval. Note that this means that if the PR bounces due to needing rebasing we’ll consider that a “new” approval – I’ve not implemented any fancy detection there. Each point on this graph is a single PR; the axes are otherwise labeled.

Let me know if this is helpful – I’m not actually sure that I have derived much meaning from it yet, so not sure what my reaction is.

[jethrogb]

Note that this means that if the PR bounces due to needing rebasing we’ll consider that a “new” approval – I’ve not implemented any fancy detection there.

I think this is a crucial piece of information that is missing.

[Mark_Simulacrum]

Okay – the new data is in. This time I’ve implemented “last approval” a little differently: it is now the last time the pull request was r+-ed after being in the “proposed” (i.e., just created) or “denied” (i.e., r--ed) state. Merge conflicts no longer affect the last approval. I’ve also changed the graph to have the 85th, 90th, 95th, and 98th percentiles. It becomes clear that there has been a spike to ~2 weeks of wait time in the last month or so which is likely what brought this on. Generally, as we’ve gotten closer to the edition merge times have gotten progressively worse. Note that the descent at the very end is because we’ve simply not merged many pull requests (and those that do merge, merge in rollups and as such rapidly).

[Mark_Simulacrum]

I’ve also put together CI duration over the past year (split into Travis and AppVeyor, plus an “overall”). This is computed only for PRs that get merged themselves (i.e., are not rolled up). The overall measurement is from the time of the “Testing” comment posted by bors to the “Merging” comment posted by bors; Travis and AppVeyor lines are drawn from data provided by their APIs.

The data does suggest that at least one reason as to why queue length has been worse over the past month is that the cycle time for one CI build has been higher – over 3 hours. I’ve not yet attempted to track down why CI builds are taking more than 3 hours; that should be impossible as I believe both Travis and AppVeyor cap us at 3 hours… presumably not all builders are starting simultaneously.

The attached graph shows the 95th percentile with each week of data grouped together.

[pietroalbini]

that should be impossible as I believe both Travis and AppVeyor cap us at 3 hours…

By the way, at the moment our AppVeyor timeout is 4 hours instead of 3.

[josh]

The fact that this chart is measured in “days” and not in “hours” suggests we still need significant improvement here.

[inejge]

The most common failure we have right now is travis-ci/travis-ci#9696 , which is a network failure that happens every day between 6 AM UTC and 7:30 AM UTC since the start of June.

As a FYI, I’ve just cleaned up and published a basic DNS-over-HTTPS NSS module which might be useful in the Travis DNS brownout scenario. I’m not using it on Travis, but the motivation was similar: have a resolver of last resort when the regular DNS service is flaky, but HTTPS connections go through.

[pietroalbini]

As a FYI, I’ve just cleaned up and published a basic DNS-over-HTTPS NSS module which might be useful in the Travis DNS brownout scenario. I’m not using it on Travis, but the motivation was similar: have a resolver of last resort when the regular DNS service is flaky, but HTTPS connections go through.

Thanks for linking this! Travis found the cause of the issue (something is disabling ip forwarding every day on their images breaking all the networking inside docker) and they’re going to fix it on their end, so we shouldn’t need that. Also it won’t really be useful since all the networking goes away :(

[KiChjang]

Something that hasn’t been mentioned yet on this thread is a way to track intermittent errors; Servo currently does this with the intermittent tracker.

The way it works is that it pulls data from Servo’s GitHub issues labelled “I-Intermittent” to determine whether the failed test in the PR matches the filename reported on that issue, e.g. this Servo issue. The tracker does not automatically create new intermittent issues – this still requires manual input by a contributor/maintainer to create a new GitHub issue and flag it appropriately.

With the current number of open issues under the rust repo, filing intermittent error issues would balloon the issue tracker to sizes that make it hard to sift through non-intermittent issues. If we are going forward with implementing this on rust-lang, I would instead suggest that we keep a top-level TOML file that lists all the intermittent tests, and requeue the PR when all failures on the PR are all matched on the list.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.