Distributed ThinLTO support in Rustc

compiler
1

Hi.

Clang supports optimizing programs with ThinLTO for distributed builds. This option is not well documented in the official documentation but some pieces of information about that could be found in the following places:

Having support for Distributed Thin LTO is an important thing for warehouse-like Rust applications that want to use ThinLTO as an additional optimization step - that's why Google added support to distributed ThinLTO for Clang (for C and C++ apps).

As far as I understand, right now, Rustc does not support this option. Did anyone try to implement distributed ThinLTO for Rust applications? If the option is already supported, it's worth to document it. If the option is not supported, then we can add it to the Rustc's backlog. I think that this functionality is important enough for large-scale development in Rust. So before creating an issue for Rustc, I wanted to raise a discussion about that here.

I am not a first person who is asking about distributed builds in Rust :wink:

For implementing distributed ThinLTO case, it's not enough only implement it in the compiler - we also need to tweak correspondingly distributed build systems like sccache (I will create a corresponding issue later for them).

2

@durin42 you got distributed ThinLTO working, right? At least I remember your PR rustc_codegen_llvm: add support for writing summary bitcode by durin42 · Pull Request #125345 · rust-lang/rust · GitHub added something on the rustc side that was missing for supporting this.

3

Yes, we have it 99% working in bazel - sadly there's still some pending work in bazel to expose the right extension points out to the starlark rules, but we've definitely made it work.

I need to hassle some bazel experts on that front, I expect we'll roll it out widely this year.

1 Like
4

Great to know it, thanks for your work!

If you have (almost) working implementation right now for Rustc, could you please share something like step-by-step guide on how can I call Rustc compiler directly to perform distributed ThinLTO? At least smth like we have for Clang here. It can help to other people for their non-Bazel use cases.