Add new build-script instruction cargo::no-more-influence-on-crate-compilation-after-this

motivation

Currently, all crates that bindgens, compiles or finds external dependencies have to wait for compilation/lookup of compiler artifacts to be done before compilation can continue.

For example, zstd-sys takes really a long time to compile zstd and all crates that depend on it have to wait until they are compiled.

Given that bindgen usually doesn't take that long (if user enabled zstd-sys/bindgen), the dependents of zstd-sys could start compilation earlier.

proposed solution

I propose that we introduce a new flag cargo::no-more-influence-on-crate-compilation-after-this (or under a different name since this is too long), it would tell cargo that the build script won't do anything that affects compilation of the rust code of this crate (generating code, issuing cfgs, etc), and so the rust crate can generate the rmeta and compile the code.

Once the rmeta is generated, dependents of this crate can start compiling without waiting for the whole compilation to be done.

If the build-script violates the protocol and actually tries to issue build-script instructions for cfgs then compilation would fail.

Generating code or changing already generated code that the rust crate includes!() would be considered the same as trying to modify the source code of the crate when rustc is compiling, rustc/cargo does not guarantee whether it would be read, and rustc could read the files when build script is in-place updating it.

Q&A

how would that benefit systems with less than 4 cores, e.g. GHA?

It definitely cannot benefit single-core system, however for system with 2-4 cores it could still benefit.

build-script using cc-rs

P.S. I am one of the maintainers of cc-rs

For crate using cc-rs to compile the code, cc-rs cannot fully utilise every core at all time.

Particularly, cc-rs runs is_flag_supported logic in sequential, and that logic actually tries to compile a dummy c file to check if the compiler supports the flag.

cc-rs also has other places where it has to run command to retrieve information (compiler family etc).

During parallel compilation (if feature cc/parallel is enabled), cc is not guaranteed to spawn a new job immediately when a job token is freed by another process (e.g. other build scripts, rustc, etc).

compilation using external systems

Compilation using external systems such as configure + make, or cmake (potentially through cmake-rs) typically involves a setup/env detection and makefile/ninja generation process that AFAIK is single thread and takes quite some time.

It might be enough for rustc to compile the crate itself and enable the dependents to be compiled in parallel.

And since ninja/make has an internal dependencies resolution process, it is not likely to fully utilise all cores at all time, unless the project doesn't have any internal dependencies.

2 Likes

That would also require zstd-sys to use disable the bundle modifier for the dependency on libzstd.a, right? Otherwise rustc will already attempt to read and bundle the libztd.a file before it actually got created.

Would it be possible to defer the archive step?

Previous thread with what looks like the same idea: Proposal: Add "cargo:rustc-compile-crate-without-waiting-for-build-rs" for build.rs

3 Likes

That would be -Zno-link/-Zlink-only (though that may currently still require the static libraries to already be present during the -Zno-link phase), which is unstable and currently broken for the case where you do need to invoke the linker.

1 Like

Thanks I almost forgot that I posted about this before

Reading the post again I don't think it's that easy to split bindgen and compilation, it often comes together in one build script , e.g. zstd-sys

Even if we eventually have this built in to cargo, the proposal would still help reduce the compilation time.

Ahh right, so this can't happen until the build-script/proc-macro can utilise pipelining.

I skimmed through the doc, seems like -Zno-link would generate a rlink file and object files?

Would it be possible to put external archive into it?

If you do -Zno-link for an rlib crate, then the -Zlink-only step would assemble the rlib. Only assembling the rlib needs bundled static libraries to be available. But I wouldn't be surprised if we currently already check for the presence of the static library during the -Zno-link step.

Good thing it's still unstable, so it can still be changed

Would it make sense to put up a proposal to see if there's interest in this?

Seems like this could be implemented if -Zno-link does not require external static lib, or modified to be, so if better to determine if rust wants it before it get stablised?

1 Like

No objections from me.

1 Like

For spelling, I'd suggest either cargo::no-new-rust-code or cargo::no-new-directives to match the wording of "No New Privileges" in OS interfaces.

Making thin-local LTO/linking wait until build script complete would satisfy this use case, right? You should be able to make all the CGUs then wait.

1 Like

Thanks, that's a much better name, I've changed it to cargo::no-new-rust-code-change since cfg doesn't exactly count as code, though I don't have a strong preference on the name

Put up a proposal:

Opened

and closes the rust lang one since it's more to do with cargo internals

1 Like