Blessing tests for rustc too hard for average PC

tools and infrastructure
1

I decided to make my first pull request to rustc. Everything was fine, I build rustc using LLVM build from CI and it worked good, I made PR and it was almost ready...

Until I needed to bless some tests because some MIR output changed.

Then, I learned that I need to build unoptimized LLVM to do this on my Ubuntu PC. It was very hard, I spent 2 days for this.

First, I just failed to build it few times because it just fails to find few headers. After few tries, I made a wild guess and switched compiler from gcc to clang with this changes in config.toml.

[target.x86_64-unknown-linux-gnu]
cc = "clang-11"
cxx = "clang++-11"

Then it started to compile but it used almost all place on my little SSD and compilation constantly killed by OOM.

Then I decided to build it on my gaming PC. It runs Windows so I needed to setup HyperV and run a VM. And still all build take more than 4 hours with 10 cores (I need to limit number of cores because some compilation processed was still killed by OOM with higher parallelizm) and it used near 53 Gb of RAM during linking. Also, build folder have 66G size.

Are there any thoughts to move blessing to some build agents? I think, we cut some possible contributors from work because they just haven't such good machines. I have 64 gb RAM on my PC but I don't know anyone other who has. It woulds be nice if we can call something like @bors bless-tests to generate commit with blessed tests.

Finally, I think that we should

  1. update docs and mention, that it is possible to use LLVM from CI (I noticed that only from reading config.toml), and mention that user should use clang as compiler (this I got only from my guess);
  2. add some way to bless tests using CI because it is too hard task for average computer.

P.S. And a PR which I made if anybody wants to see how confused I was during a process.

P.S.S. And all config.toml changes which I made to bless tests:

[llvm]
...
optimize = false
...
[build]
...
profiler = true
...
[target.x86_64-unknown-linux-gnu]
cc = "clang-11"
cxx = "clang++-11"
12 Likes
2
  • update docs and mention, that it is possible to use LLVM from CI (I noticed that only from reading config.toml), and mention that user should use clang as compiler (this I got only from my guess);

Are you interested in making a PR for this? It should a lot easier than a PR to rust-lang/rust, just update https://github.com/rust-lang/rustc-dev-guide/blob/master/src/building/how-to-build-and-run.md :slight_smile:

3

Maybe on weekend.

4

I uploaded PR #83755 in response to the frustrations described here. This should help in the future.

6 Likes
5

I'd love to see something like this. I think it would need to be a two-pass thing: something like @bors try bless to run a try build and generate the diff for blessing tests, and then either the PR submitter could apply the diff from bors and push, or the person merging could do something like @bors r+ bless accept to commit and merge the diff from the previous @bors try bless. I don't think we should support doing that in one pass, because that would prevent reviewing the blessed test output before merging.

13 Likes
6

Hi, not sure if this is relevant here, but LLVM is known to eat up a lot of ram when linking in debug mode.

The possible remedies are:

  • Use release mode
  • Restrict parallelism, LLVM has the -DLLVM_PARALLEL_LINK_JOBS=2 CMake flag for exactly this purpose
  • Use lld as the linker instead of bfd. lld has a much lower memory usage, so it can link everything without constraining parallelism. It’s also a lot faster (~30 s for bfd compared to ~4 s for lld).
7

Well, the problem occured exactly because we needed to use debug build of LLVM for this.

Thanks to @richkadel , it wouldn't be needed after Simplify coverage tests by richkadel · Pull Request #83755 · rust-lang/rust · GitHub

2 Likes
8

Ah, so the problem was related to coverage tests?

I am asking because I contribute to rustc regularly and I never had to do a debug build of LLVM. The worst I ever had to do for blessing tests is ./x.py test src/test/ui --bless, which takes a while since it's 10k tests... usually I add --test-args to only bless the tests that are likely affected. So I am curious what it is that made your PR different.

9

You can read discussion in my PR.

10
11

One way I solve this when the error in test is small is I just follow and applied the diff myself, usually it's just a few files that are different and not more than 30 lines. So applying those by hand is still possible, I just look at the error, copy and paste and fixed the test without bless (I blessed it myself).

But if you want to have access to faster machines, you can try to apply to https://cfarm.tetaneutral.net/, that is probably a lot faster than using your own laptop, a bless there took like half an hour including the build IIRC. On my old machine it took more than 5 hours probably so I cancelled it.

1 Like
12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.