Rust compiler with parallel build

[sundeep-kokkonda]

We wanted to make use of parallel front-end compiler mentioned in Faster compilation with the parallel front-end in nightly | Rust Blog while building the rust compiler. As I understood currently the parallel front-end compiler can be used only with Cargo to build rust applications. But I tried to extend the usage to build rust compiler itself by doing below -

Cloned rust from git clone GitHub - rust-lang/rust: Empowering everyone to build reliable and efficient software. Added the below config in config.toml as given in doc,

[build]
rustflags = ["-Z", "threads=8"]

and gave a build with x.py build command and that shows below error.

Building bootstrap
    Finished dev [unoptimized] target(s) in 0.57s
failed to parse TOML configuration 'config.toml': unknown field `rustflags`, expected one of `build`, `host`, `target`, `build-dir`, `cargo`, `rustc`, `rustfmt`, `docs`, `compiler-docs`, `library-docs-private-items`, `docs-minification`, `submodules`, `gdb`, `nodejs`, `npm`, `python`, `reuse`, `locked-deps`, `vendor`, `full-bootstrap`, `extended`, `tools`, `verbose`, `sanitizers`, `profiler`, `cargo-native-static`, `low-priority`, `configure-args`, `local-rebuild`, `print-step-timings`, `print-step-rusage`, `check-stage`, `doc-stage`, `build-stage`, `test-stage`, `install-stage`, `dist-stage`, `bench-stage`, `patch-binaries-for-nix`, `metrics`, `android-ndk` for key `build`
Build completed unsuccessfully in 0:00:00

Is there any way that we can make use of nightly compiler & Cargo to build the rust compiler?

[bjorn3]

The config.toml read by x.py is an entirely different format from the .cargo/config.toml read by cargo and setting .cargo/config.toml for x.py is not officially supported. You can still use the RUSTFLAGS env var though.

[sundeep-kokkonda]

So, which config.toml file I should use to pass these flags. I tried passing directly to build with below patch but it seems not working.

diff --git a/src/bootstrap/src/core/builder.rs b/src/bootstrap/src/core/builder.rs
index 753b41abaf4..f4cc4e93c37 100644
--- a/src/bootstrap/src/core/builder.rs
+++ b/src/bootstrap/src/core/builder.rs
@@ -1320,6 +1320,9 @@ pub fn cargo(

         let mut hostflags = HostFlags::default();

+        cargo.env("RUSTFLAGS", "-Z threads=8");
+        //cmd.arg("-Zthreads=8");
+
         // Codegen backends are not yet tracked by -Zbinary-dep-depinfo,
         // so we need to explicitly clear out if they've been updated.
         for backend in self.codegen_backends(compiler) {

[steffahn]

Just passing them manually in your terminal as you invoke the built should work, no patches required, I believe. See also here.

[petrochenkov]

Nope.

Building bootstrap
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `E:\msys64\home\we\rust\build\x86_64-pc-windows-gnu\stage0\bin\rustc.exe - --crate-name ___ --print=file-names -Zthreads=8 -Ctarget-cpu=native -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings --crate-type bin --crate-type rlib --crate-
type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit code: 1)
  --- stderr
  error: the option `Z` is only accepted on the nightly compiler

Setting both RUSTFLAGS_BOOTSTRAP and RUSTFLAGS_NON_BOOTSTRAP instead of just RUSTFLAGS seems to work though. Building bootstrap itself with -Zthreads=8 would also be nice.

[steffahn]

Makes sense. Unstable flags can’t be used with beta compiler which compiles the bootstrapping script. There’s a workaround for that, which is also used internally for the bootstrapping itself (which is why your builds of rustc don’t complain about the flag, and might work if you really want to build the bootstrapping script with -Zthreads=8: You could try setting the environment variable RUSTC_BOOTSTRAP=1.

[bjorn3]

What @steffahn said should work. Or if you don't mind the first compilation stage to happen without -Zthreads=8, you can set RUSTFLAGS_NOT_BOOTSTRAP instead of RUSTFLAGS when running x.py. This will skip it for the bootstrap compiler

[petrochenkov]

@steffahn

RUSTC_BOOTSTRAP=1

Just RUSTC_BOOTSTRAP=1 doesn't work.
Hmmm, maybe need to also add some -Z unstable-options as well, let me check.

[steffahn]

In a test project where I just combined RUSTC_BOOTSTRAP=1 together with a -Z threads=8 option, I’ve noticed that the error might be cached and the RUSTC_BOOTSTRAP=1 not recognized as a significant change to trigger actually retrying the build. Try to clean first, or otherwise force a re-build to actually be tried (e.g. touching a relevant source file, or manually deleting relevant artifacts).

Edit: Just noticing, in my test, it was actually sufficient to slightly alter RUSTFLAGS, e.g. by changing between -Zthreads=8 without space and -Z threads=8 with a space, to make it re-try properly.

[sundeep-kokkonda]

Where these flags were added, since the x.py config.toml is not accespting these flags, I passed them as below:

diff --git a/src/bootstrap/src/core/builder.rs b/src/bootstrap/src/core/builder.rs
index 753b41abaf4..b3e5f1be5a9 100644
--- a/src/bootstrap/src/core/builder.rs
+++ b/src/bootstrap/src/core/builder.rs
@@ -1319,6 +1319,13 @@ pub fn cargo(
         let out_dir = self.stage_out(compiler, mode);

         let mut hostflags = HostFlags::default();
+       cargo
+               .args(["-Z", "unstable-options", "-Z", "threads=8"])
+                .env("RUSTC_BOOTSTRAP", "1");
+

And, the compiler complains as -

error: unknown -Z flag specified: threads

The bootstrapping beta compiler will have these newly introduced flags in nightly builds? or am I passing the flags incorrectly?

(Also, I tried Setting both RUSTFLAGS_BOOTSTRAP and RUSTFLAGS_NON_BOOTSTRAP in same way)

[steffahn]

The error message looks like the message from cargo, not from rustc. Cargo cannot accept the -Z threads argument directly, it needs to go through to rustc and the easiest way is to set the RUSTFLAGS environment variable accordingly, so that cargo applies the relevant command line arguments to rustc.

Isn’t setting RUSTFLAGS working for you? Why are you not setting them yourself in the terminal instead of modifying the bootstrap code? Couldn’t you start out by following the suggestion other people have already tried before trying to bake it into the bootstrap/src/core/builder.rs?

Or does your environment for some reason not enable you to set an environment variable for your x.py code? You could even just modify x.py itself^[1] for a similarly direct approach, I suppose, but even still. Are you aware how one sets an environment variable for the execution of a command on your operating system?

---

1. adding something like this perhaps…

   os.environ["RUSTFLAGS"] = "-Z threads=8"
   os.environ["RUSTC_BOOTSTRAP"] = "1"
   

   ↩︎

[sundeep-kokkonda]

I tried setting the env var RUSTFLAGS="-Z threads=8" from terminal and also the build command as RUSTFLAGS="-Z threads=8" ./x.py build. In both cases I got error -

Building bootstrap
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/home/00_Work/rust/build/x86_64-unknown-linux-gnu/stage0/bin/rustc - --crate-name ___ --print=file-names -Z threads=8 -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
  --- stderr
  error: the option `Z` is only accepted on the nightly compiler

  help: consider switching to a nightly toolchain: `rustup default nightly`

  note: selecting a toolchain with `+toolchain` arguments require a rustup proxy; see <https://rust-lang.github.io/rustup/concepts/index.html>

  note: for more information about Rust's stability policy, see <https://doc.rust-lang.org/book/appendix-07-nightly-rust.html#unstable-features>

  error: 1 nightly option were parsed

failed to run: /home/00_Work/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/00_Work/rust/src/bootstrap/Cargo.toml
Build completed unsuccessfully in 0:00:00

(I make sure the default rust compiler as nightly compiler)

[steffahn]

As per the discussion above, have you also tried RUSTFLAGS="-Z threads=8" RUSTC_BOOTSTRAP=1 ./x.py build? (And if you try it now, try e.g. adjusting between "-Z threads=8" and "-Zthreads=8" to force a change noticed by cargo, so the error message isn’t cached).

[sundeep-kokkonda]

This is a general question w.r.t the Faster compilation feature I tested on rust build.

I build the rust compiler with these flags RUSTFLAGS="-Z threads=8" RUSTC_BOOTSTRAP=1 ./x.py build -v and I made a few iterations of the build and observed only a 2-8% of improvement in build time with these flags.

Also, I build the ripgrep test application given in Faster compilation with the parallel front-end in nightly | Rust Blog with the same flags and observed a 36% improvement in build time.

What will be the reason for very less improvement in build time on rust compiler build and is that expected with rust compiler build?

FYI... Below are the build times I noted w/ rust compilation.

Iteration1
Normal verbose build - Build completed successfully in 0:08:15
Z Thread flag verbose build - Build completed successfully in 0:07:48
Improvement - ~8%

Iteration2
Normal build - Build completed successfully in 0:10:11
Z Thread flag build - Build completed successfully in 0:09:44
Improvement - ~6%

Iteration3
Normal verbose build - Build completed successfully in 0:06:42
Z Thread flag verbose build - Build completed successfully in 0:06:29
Improvement - ~2%

Iteration4
Normal build - Build completed successfully in 0:06:33
Z Thread flag build - Build completed successfully in 0:06:12
Improvement - ~3.3%

Iteration5
Normal verbose build - Build completed successfully in 0:05:52
Z Thread flag verbose build - Build completed successfully in 0:05:30
Improvement - ~4%

There is no further improvement observed with thread count increase also e.g., -Z threads=64

[steffahn]

@sundeep-kokkonda

What will be the reason for very less improvement in build time on rust compiler build and is that expected with rust compiler build?

Probably because the Rust compiler is already split up into pieces in the form of many smaller crates, so for a large part of the compilation, those pieces can already be compiled in parallel and thus make effective use of your processor’s parallelism.

[the8472]

you should run compilation under time or perf stat to see your CPU core utilization. If the cores fully utilized for most of the compilation then adding more parallelism won't speed things up. If the less-than-perfectly-parallel bottlenecks are not in the frontend then the parallel frontend won't help either. And last but not least, the parallel frontend does not make things embarrassingly parallel. Ultimately the query system still is subject to the dependencies between queries which can form lengthy chains which can't be resolved in parallel. E.g. the core crate parallelizes rather poorly for whatever reason.