Incremental Compilation Beta

michaelwoerister · 2017-02-04T00:52:17.572Z

This is issue should be gone in the most recent nightly. So whenever Servo updates to that, you should see some gains.

GolDDranks · 2017-02-04T04:39:36.844Z

I’m running into a bug: tried to build the release version of Tokei with it, but got a bunch of linker errors: https://github.com/Aaronepower/tokei/issues/102

Without the incremental flag it builds successfully.

SimonSapin · 2017-02-04T11:05:16.888Z

Indeed!

My benchmark is: build Servo, run touch components/style/lib.rs (so nothing other than a timestamp actually changes), then build again. With -C codegen-units=4 on a fast machine, this second build takes 147 seconds. With -C codegen-units=4 and CARGO_INCREMENTAL=1 (for both builds) on the same machine, the second build takes 108 seconds.

So incremental compilation cuts 26% of the time. On one hand this is significant, on the other hand I kind of hoped for more since no source code has actually changed.

michaelwoerister · 2017-02-04T14:11:55.465Z

Thanks for the report! I opened an issue on Github: https://github.com/rust-lang/rust/issues/39534

michaelwoerister · 2017-02-04T14:21:32.786Z

SimonSapin:

So incremental compilation cuts 26% of the time. On one hand this is significant, on the other hand I kind of hoped for more since no source code has actually changed.

This sounds like you are no getting any re-use, just the speed-up from the increased number of codegen units. If you specify -Zincremental-info via RUSTFLAGS, can you look for lines like incremental: re-using 0 out of 1 modules?

aochagavia · 2017-02-04T17:15:46.944Z

I would be interested in the fuzzing tool, but don’t have enough time to do a full GSOC project. Is there anyone who would like to mentor?

SimonSapin · 2017-02-04T18:05:33.955Z

I’m skeptical this is just codegen units, since the non-incremental result was with -C codgen-units=4 on a 4 cores (8 threads) machine. With -C codegen-units=200 non-incremental, I get 161 seconds, slower than 147 seconds for 4 units.

Back to incremental builds, -Zincremental-info says everything is reused: incremental: re-using 1252 out of 1252 modules. Here is the output with RUSTFLAGS="-Zincremental-info -Ztime-passes" for the largest crate (script) https://gist.github.com/anonymous/996ddd7dae5da732c02e517f368a08bf Extract:

time: 3.367; rss: 830MB	expansion
time: 2.508; rss: 1086MB	compute_incremental_hashes_map
time: 16.472; rss: 1607MB	load_dep_graph
time: 7.488; rss: 2957MB	item-bodies checking
time: 7.160; rss: 4005MB	translation
time: 17.666; rss: 3945MB	serialize dep graph
time: 9.442; rss: 2305MB	linking
    Finished dev [unoptimized + debuginfo] target(s) in 79.13 secs

alexcrichton · 2017-02-04T19:52:39.855Z

@michaelwoerister I’ve tested this out on Cargo and unfortunately get an ICE when compiling tests (but normal builds work great!). Would it be useful to open an issue for this and post details, or are we still at early enough stages that such large projects end up being too large to reduce in a reasonable amount of time?

nikomatsakis · 2017-02-04T21:17:32.657Z

The enormous time spent in load_dep_graph and serialize_dep_graph is certainly a problem. We have a pending PR that will help, but I’d like to dig more into it.

UPDATE: it landed, actually.

SimonSapin · 2017-02-05T13:32:54.177Z

This helps. I built Rust master locally since that change is not in Nightly yet. Rebuilding the script crate with incremental compilation after touch components/script/lib.rs now takes 52 seconds instead of 79. Output like before: https://gist.github.com/anonymous/a6baf2e4794acfc7961a3a4bf51464e1 and extract (2+ seconds lines):

   Compiling script v0.0.1 (file:///home/simon/servo/components/script)
time: 2.681; rss: 797MB	expansion
time: 2.314; rss: 1102MB	compute_incremental_hashes_map
time: 2.014; rss: 1153MB	load_dep_graph
time: 7.397; rss: 1942MB	item-bodies checking
time: 7.088; rss: 3274MB	translation
time: 8.728; rss: 3140MB	serialize dep graph
time: 8.303; rss: 1662MB	linking
    Finished dev [unoptimized + debuginfo] target(s) in 52.24 secs

Rebuilding all of Servo with incremental compilation after touch components/style/lib.rs now takes 81 seconds instead of 108 seconds.

Edit: “instead” here means compared to rustc from a few days ago, not compared to non-incremental compilation.

Edit 2: my second benchmark is therefore: 81 seconds with incremental compilation vs 117 without, which is ~30% less time.

eulerdisk · 2017-02-05T21:24:55.242Z

That’s interesting. I wonder if, in general, recompiling after a very small change (for a project having a big dependency graph to [de]serialize) will still be relatively slow cause all the incremental stuff management cost. Is it an absolute limiting factor or is only a problem with the current implementation ?

michaelwoerister · 2017-02-06T08:31:01.042Z

@alexcrichton Yes, please open an issue. We do some testing of compiling and running unit tests but not a lot. We should change that!

michaelwoerister · 2017-02-06T08:38:39.396Z

eulerdisk:

Is it an absolute limiting factor or is only a problem with the current implementation ?

The current approach does these things non-incrementally, so effort will be O(n) where n the size of the dependency graph / code base. We don’t have any immediate plans to change this, but it’s still looking like it’s taking way too much time! We’ll have to profile to find out more and hopefully reduce the time.

SimonSapin · 2017-02-06T10:53:44.152Z

michaelwoerister:

so effort will be O(n) where n the size of the dependency graph / code base.

n is particularly large in my benchmarks above. The output of --pretty expanded for Servo’s script crate is 657k lines, 47 MB. -Zincremental-info counts 1252 modules. (A lot of this code is generated.)

michaelwoerister · 2017-02-07T07:25:34.071Z

SimonSapin:

n is particularly large in my benchmarks above. The output of --pretty expanded for Servo’s script crate is 657k lines, 47 MB

That’s an interesting data point. The lower bound for the effort an incremental computation solution has to spend is O(size of change). Maybe we can move more towards that.

SimonSapin · 2017-02-07T10:54:00.565Z

michaelwoerister:

The lower bound for the effort an incremental computation solution has to spend is O(size of change).

Of course something has to be O(size of everything) at least to find out what changed, but reducing the constant factor of that part would be nice, yes.

SimonSapin · 2017-02-07T15:00:41.798Z

It has come to my attention that official Nigthly builds enable LLVM assertions which significantly affect compiler performance, while the default configuration when building rustc from source does not.

This means that my previous benchmarks compare timings with more than one parameter varying (different compiler version with incremental-compilation-related performance fixes, and different configuration with and without LLVM assertions). These results are therefore worthless.

lilith · 2017-02-07T18:18:38.851Z

What is (roughly) the perf difference w/ LLVM assertions?

I’ve been struggling with some pathological compilation times for Imageflow, but so far most bugs can be traced back to use of HashMaps in Cargo. Could we perhaps purge HashMaps as sources for Cargo fingerprints? The randomized order somewhat interferes with the creation of a meaningful hash (when more than 1 element exists, as can happen). I have pushed (my first) straw-man PR to sort values in Cargo: https://github.com/rust-lang/cargo/pull/3659

I’d be wiling to undertake such a refactoring with some guidance (perhaps comment on the PR above, as part 2 of this question is off-topic).

SimonSapin · 2017-02-07T18:44:05.001Z

lilith:

What is (roughly) the perf difference w/ LLVM assertions?

I’ve measured this today. An official Nightly build (which apparently has LLVM assertions) takes ~20% more time to do the same compilation than the same rustc commit built locally (with the default config where these assertions are not enabled). (This is a single data point: one set of crates on one machine, timed once with each compiler.)

SimonSapin · 2017-02-19T00:59:03.132Z

New findings:

LLVM assertions are even more significant in release mode.
With a hot incremental compilation cache, compiling in release is faster than debug mode.

I’ve changed my config for Servo development to use release mode, incremental compilation, and no LLVM assertions. (And made that last one the default.) All of these add up. Together with buying a faster CPU, I’ve reduced the cycle time in my typical workflow by an order of magnitude (almost down to a single minute, from over a dozen minutes). Many thanks to everyone who’s worked on this!