Three points:
-
Please do not consider disabling an integrate-what-you-test gateway on the basis of this observation. There are at least two likely-good options (the following points) to pursue at length before considering giving it up. I know nobody’s suggested it yet but IME it always comes up in conversations like this, and it should be resisted with all your power.
-
Profile the build and get aggressive about pruning costs in it. A casual sampling of the logs shows you’re spending at least 20 minutes per host rebuilding LLVM every time still (as has been the case since 2011 or so?) and 10 minutes on stage0 (?) and a variety of other things that give the impression of “low hanging fruit”. Similarly, as many of the long steps do seem to be rustc-building-rustc, maybe this is a good community signal to focus effort still further on making rustc faster. Pretty sure it’s still one of the top complaints about the compiler.
-
Consider adopting the Bors-NG algorithm, in which the integrator attempts a maximal integration each time, bisecting down to either a successful multi-PR prefix (a “rollup”) or an isolatable/skippable failure. This is very much an improvement in the state of the art and (IMO) everything in this family of tools ought to be using that algorithm now.