Idea: Easier cross lang LTO?

compiler
1

The current cross lang LTO using linker-plugin-lto doesn't work well with cross compilation tool such as cargo-zigbuild

Why can't rustc package external libraries with their corresponding rust crate together, mixing their code and then:

  • if LTO not enabled, perform LTO across the rust crate and external lib, and generate one assembly archive for each crate
  • If LTO enabled, perform the LTO on all crates and external libs, and then call the system linker on one LTO-ed object

that would save a tons of trouble, is there any reason the linker-plugin-lto delays everything to the system linker for LTO, instead of simply merging the external lib with rust crate, and then let rust's own LTO process do the LTO?

2

That still requires the C libraries to be compiled by a compatible version of clang, but with cargo-zigbuild presumably the same clang version would be used as the linker plugin. In addition the LTO support in rustc uses the legacy method for performing LTO afaiu, which for example doesn't handle weak symbols correctly. This is not a problem for pure-rust LTO as #[linkage] is unstable, but it may be a problem with C libraries. And it would actually prevent cross-lang LTO for staticlibs as there rustc doesn't know about all libraries that will be linked together in the end.

3

IIRC llvm ir code generated by old version of LLVM generally can be accepted by newer LLVM (except <=3.0.0), so even if zig-cc uses an older LLVM it'd be ok, as so far zig-cc often lag behind rustc as of LLVM version.

Plus zig-cc linker doesn't like getting passed linker plugin.

Hmm so the biggest obstacle here is the lack of weak symbols support?

Yeah if it's per crate then it definitely can't figure it out, it would probably has to skip it and just not to perform LTO on any weak symbols.

Same for dynlib regardless of LTO requested.

But if user asks for LTO across crates to generate an executable, then surely it can be done?

4

(Edit: however, doing LTO before calling the system linker is a great idea, dodging a lot of linker integration complexity)

There's a proposal for this:

5

Thanks I know about it, but my issue is with the linking stage during cross compilation.

I use cargo-zigbuild for cross compilation, and it doesn't support linker plugin LTO.

The same problem would happen with other cross compilation tool, the final linker used might not be lld or support linker plugin.

What I wish is that the LTO process is done differently, so that the rustc LTO can also handle external libraries as well, instead of delaying that to system linker, that'd be far more reliable.

1 Like
6

Part of the difficulty is that when telling cargo/rustc extra linker arguments, the expectation is that those arguments will be used when linking together with the system libraries. Yes, the libraries themselves are specified as libraries to link to rather than as just linker flags, but the ability to set those custom flags still makes things more complicated.

1 Like
7

Yeah -Clinker-plugin-lto itself would be affected, if rustc LTO is extended to also include external libs, and probably tuning of LTO behavior/passes.

Maybe we could have a rustc/cargo flag for tuning rustc's LTO processes?

8

Would it make sense to write a proposal for it?

I honestly think cross-lang-lto is unnecessarily complex and a lot of burden is pushed to the end user trying to do it to pick the right LLVM version, or have workaround if using zig-cc or something

9

The first step is to expose Rust's clang/llvm through the rustup toolchain, so it's easier to do external LTO, which already has its proposal. Once that's available on nightly, it becomes much easier to discuss how cargo/rustc can streamline the process even further.

3 Likes