Brainstorming improvements for new rustc devs

[brson]

Since so much of this is about how disorganized the makefiles are, it seems like the entire picture will change once we’ve moved fully to rustbuild.

As far as bootstrap times go, every time we talk about it I don’t see any way to cut out any part of it, particularly without dropping dylib plugins. But we still might be able to create a better defalut workflow for developers that short-circuits parts of the bootstrap in the right scenarios. For example, there’s a clear difference in the possible workflows of compiler devs and std devs. std devs can generally short-circuit everything but the stage2 stdlib.

So in that light one way we could re-design the build process is toward ‘roles’. In the ‘std-developer’ role after the full rebuild it only rebuilds std (we could additionally make it so stage2 build stop right at std/test and doesn’t build librustc_*). Doing partial rebuilds like this though affects which parts of the test suite can be expected to work correctly, and not having this relationship well-coordinated is one of our problems here.

Since plugins are the main reason that builds need to go all the way through stage2, we might be able to stop by default at the end of stage1+std (no librustc), but just turn off all the test suite that deals with plugins (printing some warning that you aren’t running the full thing). That means everybody would build the std + rustc once, then std one more time to get one compatible with the new rustc. I think that’s basically the minimum amount of building we can reasonably expect to get a working compiler.

One problem the build system traditionally has here is that it doesn’t differentiate between the set of libraries necessary for std, and the set of all libraries we provide, it just considers every crate a ‘target library’, so even if you just need to build std, you are still building all the way through librustc, which in the final stage is only used by plugins. rustbuild has a more fine-grained opinion about which libraries are grouped together and will let us more easily just stop building at std.

[brson]

we might be able to stop by default at the end of stage1+std (no librustc), but just turn off all the test suite that deals with plugins (printing some warning that you aren’t running the full thing). That means everybody would build the std + rustc once, then std one more time to get one compatible with the new rustc.

Just a few more thoughts on what the minimal set of crates you need to build to do any useful development. For the minimal development build one could cut out most of stage0 too. Imagine if you are just fixing a one-liner in rustc_driver than doesn’t impact the ABI. The only thing that actually needs to change to validate that it works is to rebuild rustc_driver (and the few other crates that depend on it) - the new rustc should still be able to use the exact same std it was built with. So you could imagine a really clever build system reusing all the stage0 artifacts from the snapshot up to rustc_driver, rebuilding just rustc_driver, promoting that combination of snapshot stage0 bins and the new rustc to stage1, then testing the stage1 compiler using the stage0 sysroot. It should all work, as long as you haven’t broken any ABIs that the test suite cares about. But automatically understanding the impact on the test suite is super hard.

As an aside, I’ve long disliked the way the bootstrap uses stageN host artifacts to build stageN target artifacts, then copies those target artifacts to stageN+1 (iow that the compiler for one stage builds its own libraries for that stage; that artifacts from two different stages are mixed together). I created this scheme, but I’d rather have it so the stageN compiler builds the stageN+1 libs and then copies those to become the stageN+1 host compiler (both schemes have complications though). The more time passes though the more impossible it seems that we could ever make such a major change.

Ugh, so as usual every time this comes up, I don’t have any concrete plan as how to fix it - the issues are tiny and interrelated and complex. But here are some scenarios with their minimal builds and impact on the test suite

• std-basic (stage0 host + stage0 sysroot)

  • rebuild stage0 std
  • do not rebuild rest of stage0 libraries
  • reuse snapshot rustc to test std
  • any test that exercises plugins will fail because librustc hasn’t been built

• rustc-basic (stage1 host + stage0 sysroot)

  • Minor change to rustc that affects no function signatures, metadata or other ABI matters:
  • reuse stage0 std
  • reuse stage0 libs up to the rustc lib with the code change
  • rebuild stage0 rustc
  • promote stage0 target to stage1 host
  • do not rebuild stage1 std, instead use stage0 sysroot
  • all test should pass because no ABI’s have changed. everything under test is directly from the snapshot except the minor rustc change

Well, I’m bored of this now. But you could imagine that everybody starts out in either the std-basic or rustc-basic “roles”, then when they run into weird test failures could fallback to the full build.

OK, another suggestion to speed things up: observe that in the stage0 build if the source of std has not changed then rebuilding it will result in the exact same crate that is in the snapshot. In stage0 the build system could short-circuit the build of every crate that hasn’t actually been touched by copying them from the snapshot.

Sorry for rambling.

[brson]

OK, some more rambling. For people that really care about this, I think it would be worthwhile to prototype a completely different build system that is focused on developer speed (hopefully reusing the rustbuild libs but maybe not), while sacrificing correctness in some scenarios. The existing build system works well for validating that the entire system works, but most devs don’t care about that at all until right before (or after) submitting a PR. Maybe a fresh start would shed some light on the problem. It would not be a bad thing at all if active developers use a less full-featured build system for day-to-day work if it lets them do some large percentage of what they need more quickly than the complete bootstrapping build.

I’d suggest that before doing this it would be good to get a complete list of all the tricks that active developers use to short-circuit the existing build in particular scenarios, as well as understanding more thoroughly the various types of maintainers we have (std, rustc, docs, etc).

[jethrogb]

Just a few more thoughts on what the minimal set of crates you need to build to do any useful development. For the minimal development build one could cut out most of stage0 too. Imagine if you are just fixing a one-liner in rustc_driver than doesn’t impact the ABI. The only thing that actually needs to change to validate that it works is to rebuild rustc_driver (and the few other crates that depend on it) - the new rustc should still be able to use the exact same std it was built with. So you could imagine a really clever build system reusing all the stage0 artifacts from the snapshot up to rustc_driver, rebuilding just rustc_driver, promoting that combination of snapshot stage0 bins and the new rustc to stage1, then testing the stage1 compiler using the stage0 sysroot. It should all work, as long as you haven’t broken any ABIs that the test suite cares about.

Right. This is basically what I was doing with the scripts linked above.

[arielb1]

rustc-basic (stage1 host + stage0 sysroot)

Minor change to rustc that affects no function signatures, metadata or other ABI matters: reuse stage0 std reuse stage0 libs up to the rustc lib with the code change rebuild stage0 rustc promote stage0 target to stage1 host

do not rebuild stage1 std, instead use stage0 sysroot all test should pass because no ABI’s have changed. everything under test is directly from the snapshot except the minor rustc change

I don’t think this makes sense - if anyone has made an ABI-breaking change in this release, stage0 and stage1 will produce incompatible libraries.

What I sometimes do is to:

1. pre-build stage1 sysroot for master (make rustc-stage1).

2. modify the compiler in a way that does not change the ABI

3. build rustc only (make x86_64-unknown-linux-gnu/stage1/bin/rustc)

4. manually perform the tests I need.

5. if not done, goto step 2

6. run the stage2 test-suite locally/on travis.

This is basically as optimal as it can be (when we get incremental compilation, it will be even faster, as a 1-liner change to librustc will not require any recompilations) unless you do evil things like manually building some .so-s and copying them over.

However, this strategy is hackish and has several problems:

1. if I do change the ABI, I can expect a mess.

2. if I accidentally run make at steps 3/4/5, it ruins the stage1 sysroot and I have to rebuild it.

3. if I git commit and then recompile rustc_metadata, the metadata guard will complain.

A less-hackish infrastructure will fix (2), and disabling the metadata guard for development builds will fix (3).

[arielb1]

I think the most annoying issue with the normal build process is that make/make check builds a full set of stage3 libraries, which should be identical to the stage2 libraries. OTOH I rarely ever do it locally – make rustc-stage2 builds the stage3 sysroot, which is fairly fast.

The second problem with that is that the --test tests are also a part of stage3. This means that if I have a syntax error in a test, I have to go back all the way to stage1. If I am actively developing something, I can compile it with --test using the stage0 compiler (and stage1 dependencies), but if a refactoring breaks something, this can be very annoying.

Unless there is some underlying issue, we can just copy the stage2 binaries to the stage3 binaries, and do the --test tests using the stage1 compiler.

[arielb1]

I think a possible fix for everybody would be:

• make stage3 libxyz.so a copy of stage2 libxyz.so
• have the --test tests be a part of stage2 (i.e. built by stage1)
• have a rustbuild lock stage1/rustbuild unlock stage1 commands.

rustbuild lock stage1 makes sure stage1 is up-to-date, and prevents any further calls to rustbuild from recompiling it. rustbuild unlock stage1 reverts.

This means that ABI changes while stage1 is locked will cause stage2 and stage3 to differ - so this is unsafe, and works best if we weaken the metadata guard.

However, now everything works perfectly - the std tests now only depend on stage2 libstd and its dependencies, and if you change a part of the compiler, it only builds the compiler libs (incrementally, when we get that) and promotes the old stage2 to stage3.

If you change the ABI, you just need to rustbuild unlock stage1; rustbuild lock stage1 to recompile stage1 (maybe combine it: rustbuild relock stage1).

I think we will still want a separate stage3 on bors, to assure us that our compiler can compile itself correctly.

[sanxiyn]

For me, better support for out-of-tree (i.e., my distros package) LLVM would be nice.

This works for me, with ./configure --llvm-root=/usr/lib/llvm-3.8. If it doesn’t work for you, please file a bug. This is supposed to work. I agree it is important to keep this working.

[sanxiyn]

My understanding is that lldb-350 is buggy and lldb-360 is fixed.

[jntrnr]

Right - though lldb-350 seems to be the currently available version as updating xcode via the App Store does not update it beyond that.

[nikomatsakis]

Well, I’m bored of this now. But you could imagine that everybody starts out in either the std-basic or rustc-basic “roles”, then when they run into weird test failures could fallback to the full build.

I definitely think that being able to manually control what gets rebuild would be useful. Another very common example for me is that I change some #[test] fn, which requires rebuilding a ludicrous amount of stuff. To some extent incr. comp. will help here, but due to the nature of bootstrapping less than you might think – so long as we are still committed to preserving correctness.

I’m not sure that “roles” is the way to think of it. I feel like just having flags you can pass to rustbuild telling it to ignore the fact that things are out of date, or specialized targets that start from a given point regardless of what is out of date, might be good enough. We just have to document them well, and make clear their shortcomings.

This has the advantage that you start out with the safe targets and kind of “opt in” to the more advanced ones.

Still, thinking about “roles” – or at least common use cases – is a great idea to come up with what those targets are.

[stebalien]

Ah. I see. Actually, it appears I’ve tried this before (based on some bugs I’ve filed).

Unfortunately, this isn’t accessible to new devs as it’s not really documented anywhere except ./configure --help and, on many distros (Arch at least) requires an additional --disable-codegen-tests flag due to some LLVM binaries that aren’t distributed by default (but the only way to figure this out is to google around and read old bug reports).

[sanxiyn]

Maybe we should automatically disable codegen tests if FileCheck is not found. At the moment, debuginfo-gdb (respectively, debuginfo-lldb) tests are disabled if gdb (respectively, lldb) is not found. We could handle codegen tests similarly.

[vadimcn]

What is rustbuild’s story around build parallelism?

Right now if I want to utilize all of my CPU cores, I need to use make -jN, which works ~okay, though occasionally things fail, presumably because dependencies are not specified 100% correctly. Re-running make usually fixes it.

Unfortunately this breaks down on compiletest, which does its own parallelism, so we end up with N^2 threads of execution instead of N.

Would be nice if our build system was integrated end-to end in this respect.

[alexcrichton]

The rustbuild build system currently relies purely on parallelism in the tools that it calls, it doesn’t actually invoke anything in parallel yet (but that may change). Each step of the build, though, always maxes out cores (e.g. cargo or libtest suites), so it ends up not mattering much.

[matklad]

I submit PR to rustc from time to time, and figuring out what is the correct make command to run interesting tests is always a mystery to me.

Could there perhaps be a script which analyses changes and suggests what is the most appropriate command to run? Like, if you edit parser.rs, then it should suggest syntax tests, if it’s standard library, then make check-stage1-std NO_REBUILD=1, etc.

And is there some way to avoid building LLVM? I also have to use @stebalien’s algorithm because of this

[cuviper]

And is there some way to avoid building LLVM?

Use --llvm-root=/usr from your distro, or even point to your own cached build – assuming the thing you’re testing is independent of exact LLVM versions.

[matklad]

Should this perhaps be documented in CONTRIBUTING.md?

[cuviper]

Maybe so. There’s already a small section about configure options. And I forgot that you might need --disable-codegen-tests to go with it, as @stebalien mentioned, if your LLVM doesn’t have FileCheck. (e.g. Fedora doesn’t package that.)

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.